Measure Your Testing

MEASURE YOUR TESTING

I was always a numbers guy, who goes in for showing everything with quantities and referencing outcomes to objective measures. In private life, it is sometimes hard and sometimes very easy. In order to show my love and dedication to my wife, I always remind her with numbers. Sometimes these can be the dates of our anniversaries and sometimes they can be the number of days we haven’t seen each other.

In testing, these measurements are easier to extract and present. When you are doing any unmeasured or unquantifiable testing activity, you need to be very careful. Without any metrics and measures, software testing becomes a meaningless and unreal activity. Imagine, in any project, that you are not aware of the total effort you have expended and total number of defects you have found. Under these circumstances, how can you identify the weakest points and bottlenecks?

I always remember and respect the famous saying: “If you want to improve something, you need to measure it first.” And I truly believe this. If you want to show the bad as well as the good, you need to measure your testing and present quantifiable results. In general, logical and clever managers require numerical evidence; they are not satisfied with any nonnumeric information.

Let me list some of the favorite test metrics that I love to capture and display. I believe some of them should be very easy for you to get, and for the rest you may try your chances. They are:

• Total Test Effort and Test Efficiency (including schedule and cost variances)
• Test Execution Metrics (# of Pass, Fail, Block, Inconclusive test cases)
• Test Effectiveness (# of defects found in system test / total # of defects found in all phases)
• Total # of Defects, Priorities, Severities, Root Causes (system test, regression, UAT, and live)
• Defect Turnaround Time (time required for developers to fix defects)
• Defect Rejection Ratio (ratio of false/invalid defects)
• Defect Reopen Ratio (ratio of unfixed/failing “ready to test” defects)
• Defect Density (per development day or line of code) and Defect Detection Ratio (per day or per testing effort)
• Test Coverage, Requirement Coverage, Requirements Stability, and some others you may like more…

If you collect and analyze the above metrics accurately, you will be several steps ahead of others who don’t.