I've recently been asked to get involved in helping a client to produce management information (MI) that helps to quantify the value that performance testing is adding to their business.
This puts me in a quandary. I know that much of the testing that they do at the moment adds little or no value. I've seen examples of performance tests which have suffered from high error rates due to out of date test data, but have still been reported as "passed" purely based on the fact that transaction response times compare favourably with other tests. Very few tests actually monitor the performance of application servers and I'm not sure that enough thought goes into ensuring that tests accurately reflect production loads and customer behaviour.
Despite the list of problems that I've identified (and the above list is by no means exhaustive), even this testing has some value. Even poor testing will catch some "show stopper" problems, but how do you decide when the testing that you do isn't worth doing?
Due to the trend towards Agile application development, incorporating continuous builds with automated deployments and tests; we've heard plenty about "shifting-left" (testing earlier and more frequently in the software development lifecycle). In some cases, such as the examples above where test teams lack the necessary skills, testing toolsets and methodology or have insufficient time to perform high quality tests, it may be more appropriate to "shift-right".
For applications with complex data requirements and intricate application architectures, that aren't easily reproduced in a test environment; it can sometimes be preferable to perform basic functional tests, perhaps with some automation, rather than adding to a project's risk and timescale by demanding complex performance tests.
In this case it may be worth considering A-B testing. Deploying a new version of code to one server in a farm and monitoring application performance as well as looking out for errors on that server, may be preferable to engaging in a complex, time-consuming cycle of testing. I know that it may seem odd for an application tester to argue against testing, but I think that you need to be open minded as a tester and only test when you can add value and there isn’t an easier or more cost-effective option.
If you need help deciding whether testing is appropriate for you, get in touch with us at Trust IV.