How do you decide when to stop testing?

There is always more testing that  can be done on a feature but there are reducing returns on testing a feature over time. At some point you need to stop testing, and the question as to when to stop is sometimes a difficult call. Also, sometimes there is more than one feature to be tested. Choosing when to stop testing the first feature and move to testing the second feature can be a difficult choice.

If I test a feature for a while I find that after a while I am finding less issues, and after a while longer I stop finding issues. At the point when you stop finding issues it makes sense to stop testing. Sometimes you don’t have the time to keep testing until you stop finding more issues and it makes sense to stop testing when you are finding only minor issues. In both situations I find that I find more issues at the beginning of testing than at the end and it is the slowing rate at which I find issues that prompts me to consider stopping testing.

That I find more issues, and more serious issues, early in testing than I do later is a pattern I have identified. The pattern is not unlike the Pareto Principle. Pareto was a nineteenth century  Italian mathematician who observed that 80% and Italy’s land was owned by 20% of the population and found that this ratio applied in many other situations too. When I am testing it feels that 20% of the testing effort finds 80% of the issues.

One of the key principles of Joseph Juran, the American evangelist for quality, was to use the Pareto Principle. This meant identifying “the vital few and the trivial many” in order to focus on the key areas to improve quality. The Pareto Principle is the pattern that I observed in my testing where I found more issues earlier in my testing and less issues later in my testing. The “vital few” tests were those that I made earlier in my testing and the “trivial many” were the tests that I ran later and found less issues.

We need to know when to stop testing a feature and the Pareto Principle can help to decide when to stop. I stop testing when my testing ceases to find issues and I know that my testing has moved from the “vital few” tests to the “trivial many” tests.

Further Reading: https://www.juran.com/blog/a-guide-to-the-pareto-principle-80-20-rule-pareto-analysis/