Testing qualities not quality

To help me test I used to find it useful to think about what quality is for the application that I am testing.  “There has been a tendency to conceive of quality as indicating the goodness of an object.”[1] There are many aspects to quality and I have found that this idea of quality is ”too indefinite for practical purposes”[1].  I now think about the qualities of the application, not its quality.

Walter Shewhart worked to improve the quality of telephones and wrote that “every conceptual “something” is really a group of conceptions in more elementary form. The minimum number of conceptions required to define an object may be called the qualities thereof”. “In this sense a thing has qualities not a quality”[2]. Shewhart’s approach to quality is useful because it prompts us to ask what are the qualities of the software that we are testing. 

Shewhart quoted Jevons[1] who wrote that “two fragments of rock may differ entirely in outward form, yet they may have the same colour, hardness, and texture. Flowers which agree in colour may differ in odour. The mind learns to regard each object as an aggregate of qualities, and acquires the power of dwelling at will upon one or other of those qualities to the exclusion of the rest”.[3]

When we are testing we need to find the qualities that differentiate things that appear to be the same. Testing can be, as Jevons suggested, like observing two fragments of rock that have similarities. One fragment could be our understanding of how the product should be and the other fragment could be the product that we are testing. The two fragments of rock could also be two versions of the application that have the same backend but different UIs. We need to find the differences between the two and discovering the qualities of each helps to do that. Performance, testability or usability could all be qualities of the product that we are testing.

The practice of analysing the qualities of a product led to the use of quality attributes for testing. 

Defining the qualities of the application under test will also help with defining the quality criteria used when defining quality coverage, as in Melissa Fisher’s recent blog post: Test coverage — how about quality coverage?

Asking questions about qualities helps me break down the functionality into smaller batches for testing. I can test one quality at a time, for example, I can test accessibility.  This approach can also help me gain insight into one or more of the qualities and give me new paths for testing through the application.

References

[1] Economic control of quality of manufactured product by Walter Shewhart (1931, p55)

[2] Economic control of quality of manufactured product by Walter Shewhart (1931, p56)

[3] The principles of science : a treatise on logic and scientific method by William Stanley Jevons (1913, p24 )