How long should an automated test pack take to run? And what should we do about it?

How long should an automated test pack take to run?

Some people say that a certain number of minutes is too long, and others say that a different number of minutes is too long. How can we determine what is “too long”? And what can we do about it?

What is an acceptable length of time for a test pack to run depends on several factors. Your team’s cycle time will affect how long it is acceptable for a test pack to run. Cycle time can be defined as a development process “ that transforms identified customer needs into delivered customer value at a reliable, repeatable cadence”[1] If your team is practising continuous deployment or continuous delivery and has a cycle time of a few hours there will be pressure for automated test packs to run more quickly than if your team makes monthly or quarterly releases. Teams that practice continuous deployment or continuous delivery work in small batches so the test packs will need to be small batches too.

Another factor that affects how long it is acceptable for a test pack to run is whether the length of time the tests take to run is a constraint on development. In some situations, a test pack of thirty tests that runs in fifteen minutes may be an enabler and in other situations, it may be a constraint. An example of this would be; if the test pack that you are concerned about takes no longer to run than other test and deployment processes then it is unlikely to be a constraint, however, if it takes much longer than other test or deployment processes then it may be a constraint on how developers work.

What can the test engineer responsible for the test pack do about the length of time it takes to run the test pack? 

The engineer who is responsible for the test pack should not be defensive about the test pack, instead, they should lead the discussion about how long it takes the test packs to run. They can, for instance, go to developer meetings and create a dialogue with developers by talking about how they are developing the test pack. This dialogue should include how long it takes for the pack to run. If the test automation engineer is involved in a dialogue they will find out how the test packs are affecting developers. They can also tell developers what they are doing to make the tests run faster and discuss with developers ideas as to how to improve the running of the test packs. Testers and developers should then use this dialogue to start to cooperate on improving test packs, including their running times.

What is an acceptable time for a test pack to run depends on several factors. We need to understand our development system and cooperate with our development team to improve test packs. 

References

[1] Implementing Lean Software Development: From Concept to Cash by Mary and Tom Poppendiek (2006,p98)