How do you decide when to stop testing?

There is always more testing that  can be done on a feature but there are reducing returns on testing a feature over time. At some point you need to stop testing, and the question as to when to stop is sometimes a difficult call. Also, sometimes there is more than one feature to be tested. Choosing when to stop testing the first feature and move to testing the second feature can be a difficult choice.

If I test a feature for a while I find that after a while I am finding less issues, and after a while longer I stop finding issues. At the point when you stop finding issues it makes sense to stop testing. Sometimes you don’t have the time to keep testing until you stop finding more issues and it makes sense to stop testing when you are finding only minor issues. In both situations I find that I find more issues at the beginning of testing than at the end and it is the slowing rate at which I find issues that prompts me to consider stopping testing.

That I find more issues, and more serious issues, early in testing than I do later is a pattern I have identified. The pattern is not unlike the Pareto Principle. Pareto was a nineteenth century  Italian mathematician who observed that 80% and Italy’s land was owned by 20% of the population and found that this ratio applied in many other situations too. When I am testing it feels that 20% of the testing effort finds 80% of the issues.

One of the key principles of Joseph Juran, the American evangelist for quality, was to use the Pareto Principle. This meant identifying “the vital few and the trivial many” in order to focus on the key areas to improve quality. The Pareto Principle is the pattern that I observed in my testing where I found more issues earlier in my testing and less issues later in my testing. The “vital few” tests were those that I made earlier in my testing and the “trivial many” were the tests that I ran later and found less issues.

We need to know when to stop testing a feature and the Pareto Principle can help to decide when to stop. I stop testing when my testing ceases to find issues and I know that my testing has moved from the “vital few” tests to the “trivial many” tests.

Further Reading: https://www.juran.com/blog/a-guide-to-the-pareto-principle-80-20-rule-pareto-analysis/

Published by Mike Harris

Mike has been working in testing for 20 years and is currently the lone tester for Geckoboard. He has been a Test Lead and has also worked as a part of waterfall, lean and agile teams. He is also Programme Secretary of BCS SIGiST. Mike has a B.Sc.(HONS) from Middlesex University and is an Associate of the University of Hertfordshire. He has set up and led a Testing Community of Practice and been part of a successful agile transition. He is Co-Programme Chair of the British Computer Society’s Specialist Interest Group in Software Testing. He also contributed to the e-book Testing Stories and has had articles published by the Ministry of Testing. Follow Mike on Twitter: @TestAndAnalysis

Leave a comment

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Create your website with WordPress.com
Get started
%d bloggers like this: