ClearBlade QA Metrics (Draft)
Make our QA measurable, define metrics so we can measure our QA progress over time. If we can measure QA, we can set objective goals for a new QA hire.
Known Issue: A high priority unfixed bug that has the potential to compromise confidentiality, integrity, availability of the platform and/or edge.
Labeling all significant bugs in MONSOON with Known Issues tags, and tracking the number of Known Issues (Label all backend bugs which seem to be a high priority)
JIRA search for known issues: https://clearblade.atlassian.net/issues/?filter=10844
Evaluating ROI for using something similar to Crashlytics for Golang and send crash reports back home, reducing the number of crashes over time
- Automatically sends crash reports on any platform failure to ClearBlade for all instances
Value: - Allows us to be proactive and thus, ensure we take care of all bugs - Also, it helps solve the log-rotation problem, since important stuff is recorded ahead of time
Code Coverage calculated and logged every CI build
Set Code Coverage Goal of 80% (Make note of tool used)
Perform Gap Analysis for Features: https://docs.google.com/spreadsheets/d/15iNjOBTjbL6YrH3OfBpqD7k8SFYNByvV41TJYbnhZDc/edit#gid=0
Value: - Keeps us aware of untested features - Allows us to prioritize their testing - Helps us ensure that all our features are tested
High steady load, until failure
Ex. Publish 1,000 messages per second until the platform fails, maybe from a memory leak over course of 48hours of running
Increase stress until failure
- Endurance Test Outputs for first 5 scenarios (already available)
Value: - Allows us to give numbers on how much stress can platform/edge sustain - Allows us to give Node-Specs to Customers - Helps us measure feature optimizations better - Helps us determine accurate numbers for parallelism
- Performed by QA Security Testing is done to check how the software or application or website is secure from internal and external threats. This testing includes how much software is secure from the malicious program, viruses and how secure and strong the authorization and authentication processes are.
Verifying Ex: This also includes tests to find out bugs like developer logging in instead of user, when using ClearBlade Library.
Resiiliency to DoS and DDoS. Ex: Rate Limit testing
Value: - Allows us to consider rate-limiting our end-points - Makes us more available - Continuous testing ensures we always adhere to the security rules when deploying new features
Devops Testing: Integration testing
Involves Smoke testing of all our components like HAProxy, CB Containers, Console, Exporters working together. Ex: Smoke testing that we get logs on the exporters, when we perform a blade-runner test on the platform. Current tests will be just smoke testing
Value: - Allows us to test any infrastructural changes we make
Customer Scenario Testing
- Make customer system scenarios easy to import
- Mock using blade-runners
- Stress and load testing these scenarios
- Number of customer scenarios we can mock
- How quickly can we mock a newly added customer
- Find the break-points before customers find them
- We can also demo these and also help customers test their new feature before they deploy on production
Allows us to recommend Node Specs for instances, based on customer’s use cases.
Tentatively planning on using this built-in tool
go test -cover tentative goal of 80% code coverage for the clearblade repo.