Are you testing output and outcome?

There is a discussion in the DORA Community about “the problem with DORA metrics is that they focus on maximizing outputs rather than outcomes”.  Some people have argued that Dora metrics will reward teams with a high number of commits, which would be considered ‘output’ but not reward teams whose commits benefit customers, which would be considered ‘outcome’.

This discussion could also apply to testing. 

We need to test the output. This could be described as testing that we are ‘building the thing right’. Testing should include static analysis, unit testing and regression testing as these things all test that we have ‘built the thing right’. However, does testing sometimes concentrate too much on testing output?

We also need to test the outcome. We need to test that the results of the software change will create a positive outcome for the customers, that is that we are ‘building the right thing’. Testing should also include consideration of the outcome of the software change. 

To have confidence that the outcome of a software change will be successful we need to test the requirements. We need to test the requirements of the work so that any misunderstandings in the requirements are found before the new software is released.

Many bugs are caused by ‘breakdowns in information handling’ [1]. These errors occur when information about the required software change is communicated to the development team and within the development team. If we use only the teams’ understanding of the required change we may only be testing the output because we are only testing that ’the thing has been built right’ according to the specification. 

Another way to test that we are ‘building the right thing’ and testing for a positive outcome is to test from the customer’s point of view. I try to understand the customer’s point of view.  I attend user testing regularly so that I see how customers use the product. I also use the Theory of Jobs to be Done to help me understand customers’ use of our product.

Does testing sometimes concentrate on testing output?

References

[1] The Art of Software Testing by Glenford J. Myers (1979, p103)