A chart for measuring the quality of “spinning plates”

Quality metrics can be like measuring the wobble on spinning plates. 

The engineering teams are merging to the main branch, code is being deployed and you need metrics to show if there are issues with this process that require your attention. The teams working are rather like spinning plates. The teams are working and management needs to know if something is going wrong that they need to address. You can see that the plates are spinning and wobbling. You need to know if the plate is wobbling too much and will fall. If it is going to fall then you need to intervene.

Walter Shewhart created control charts to solve this problem at Bell Telephones. He saw variations in how the telephone production line worked and wanted a method to show when he needed to intervene. He plotted the metric he wanted to measure on the chart and then used the standard deviation to calculate the Upper Control Limit and Lower Control Limits. If the curve went above the upper limit or below the lower limit he needed to intervene. If the chart remained between the upper and lower limits he did not need to intervene.

Control charts are now used in many industries and can help us in software engineering.

John Willis recently wrote in his blog about how he used control charts. He used them to track his teams’ deployments to help him understand whether the team was improving.

I had a friend who was testing the work of two teams. One team was releasing new features and moved from making monthly releases to continuous delivery. The other team made monthly releases and needed developers to work extra hours to fix bugs when a release was made. In order to find out the difference between the two teams my friend decided to use control charts to analyse the work the two teams were doing. He created control charts that showed the number of unit tests added by each team each week. He found that the team practising continuous deployment created a fairly consistent number of unit tests each week and the curve on its chart was between the upper and lower limits. The chart for the team making monthly releases was quite different. There were weeks when the team making monthly releases added unit tests, there were also weeks when the team deleted unit tests and there were weeks the number of unit tests did not change. The curve on their chart often went below the lower limit. The control charts showed that a difference between the two teams was that the team practising continuous delivery were writing unit tests every week and the team making monthly releases frequently were not writing unit tests. The charts helped my friend to understand what was happening; the team that was making monthly releases were not writing unit tests. The understanding gained from the control charts enabled my friend to start a conversation with management about the team making monthly releases. 

Control charts are a useful tool that can help us understand quality. If the ‘wobble’ on a spinning plate is within the lower and upper limits you do not intervene. You only need to intervene if the ‘wobble’ goes outside the lower and upper limits. 

Thank you to Krizia Cureg and Nick Smith for the conversation that led to this blog post.

Further reading:

Books:

• Statistical Method from the Viewpoint of Quality Control by Walter A. Shewhart
• Out of the Crisis by W. Edwards Deming

Blog:

• Enumerated and Analytical Statistics (Part 1) by John Willis

Websites:

• Control Chart from the American Society for Quality
• How to create a control chart in Excel?

eLearning:

• DemingNext