PERFORMANCE TESTING TOOLS NEED CALIBRATION!
Unfortunately, performance testing tools do not emulate the exact same behavior a real user would show when accessing any application. You need to calibrate them first to have accurate results.
-Roland Van Leusden
Frankly speaking, so far I haven’t seen an organization engage in performance testing properly. Maybe I am a little bit unlucky or a little bit pessimistic (or even a little bit exaggerating), but this is the case! In some organizations we were using inappropriate tools, in some we were lacking deliverables, processes, and reporting, and in others we were lacking performance requirements.
But most importantly, for all of the above we were unaware of the performance testing tool calibration phenomenon. Putting it another way, we were 100 percent relying on the tools’ behavior and output. Having realized this fact, I can easily say that processes, deliverables, reporting, requirements, and everything else you have successfully adapted in performance testing are worthless if you are neglecting tool calibration. I have told you; truth is bitter!
Do all these mean that the performance tests so far (at least the ones I have executed!) do not reflect real results (and can be discarded as trash)? No, I cannot say that; but I do have serious doubts. Hopefully, my ex-supervisors and managers won’t read this blog post!
At Romania Testing Community Conference, an interesting session was held by Roland Van Leusden, a colleague and a good old friend from the Netherlands, which explained this fact in perfect detail. I have great respect for Roland’s research and know-how about the subject, so I asked him to give me some details about it. Here is what he had to say:
“When we buy a ruler, we expect that it is correct within a certain tolerance. When we buy a more expensive ruler, it comes with a calibration certificate that states under which conditions the ruler will be exactly one meter long. With performance test tooling, there is no such calibration and certification; you just have to believe that when you run with fifty users it will behave and produce a load equal to fifty real users.
After investigation I found that this is not the case when you use the tools with their default settings. You need to analyze what the user is doing at client, network, and server level with your application; this will be your reference. The good thing is that most tools can be adjusted to create similar behavior to the real user; it requires, however, in-depth knowledge of your application at the various levels and tooling. To get accurate, production-like results, you need to calibrate the tooling and validate the output, to make the right decisions for the best user experience.”
IT professional with 15+ years of experience as IT Consultant, Software Engineer, Software Developer and Software Tester for many different organizations. Published a number of papers and books within the Software Engineering profession and contribute to the field by regularly attending conferences as a public speaker.
- Sourcing Approach for Mobile Testing
- Test Execution Techniques
- Your Users Will Be Your First Guideline
- Think Wider, Test Better!
- Mobile Testing is a Challenge, Meet It!
- Convincing Your Manager To Fund Testing
- An Overview of Test Design Techniques
- User Acceptance Testing Has Many Meanings
- Testers Never Assume; Testers Ask Questions
- Testing in a Cross-Cultural Team
- Be “A Little” More Flexible in Hiring Junior Testers
- Buy Tools if You Really Need Them
- Mobile Testing & The Importance of Test Metrics
- Measure Your Testing
- Improve Your Test Processes by Improving Your Test Data
- Negative Testing Can Be Really Positive
- 20 Practical Advice To Help You Better Test Your Mobile Apps
- Testers Can (!) Stop Bad Software
- Mobile Usability Testing is Not Rocket Science!