It is better if we build quality into the product instead of trying to test quality in

“Cease dependence on inspection to achieve quality. Eliminate the need for inspection on a mass basis by building quality into the product in the first place.” [1] is one of W. Edwards Deming’s 14 Points for Management.  Inspection can be defined as testing after development has been completed. Some people have interpreted Deming’s point as an argument against testing. However, “Dr. Deming is often misunderstood on this point. The key word is dependence.”[2] “Is Deming saying that we should abolish inspection? No. of course not” [3].

“Dependence on mass inspection is fraught with danger and with high cost”. [4] An example of a development team being dependent on mass inspection would be a team that relies on manual regression testing before making a release. The manual regression testing will probably find bugs which probably need to be fixed. The bugs that need fixing are rework and fixing them will cost the company time and money. 

“Building quality into the product”[1] is cheaper for the company than inspection and is something that testers can play an important role in. It is cheaper because the earlier in the process an issue is found the cheaper it is to resolve, for example, it is cheaper to fix a fault in a specification than it is to fix a bug in production. Testers can help write and review specifications, and so help find issues early in the process. When we do so we are helping to build quality into the product.

“Inspection is too late. The quality, good or bad is already in the product”[5]. To build quality into the product we need to test as early as possible, for example, testing the designs of a feature before the feature is developed [6]. 

 “Deming said that quality isn’t so much about improving the product as it is about improving the process”[7]. Exploratory testing can help improve the process by identifying issues in the product. These issues can show faults in the development process. “There is a world of difference between, on the one hand dependence on inspection as an attempt to provide the customer with something he won’t complain about and, on the other, the use of inspection to provide guidance toward improvement of a stable process”[8].

The second part of the Deming Cycle, also known as the plan-do-study-act cycle, is to “carry out the change or test decided upon”[9]. Deming does not say who should carry out the testing. Testers’ knowledge of testing techniques will help the team conduct the tests the team needs to build quality into the product. 

An operational definition is a tool that can be used to build quality into a product or service. An operational definition requires there to be a specification for an item, for there to be criteria to show whether the item has met the specification and a decision on whether the item met the criteria.[10] Operational definitions are something that testing professionals can a team use to build quality into the product. Automated tests can help a team decide whether its product meets the criteria in operational definitions. Developing automated tests that give developers the confidence to release their software and find bugs before code is deployed to production is another way that testing professionals help to build quality into the product. 

Testing professionals have analytical skills and using systems thinking for analysis will help the team gain a better understanding of their situation. Systems thinking is a part of Deming’s System of Profound Knowledge.[11] Testers can help their teams build quality into the product by using systems thinking to understand the systems that our product is part of, for example, systems thinking can be used to understand that the company and its customers are one system.

“Inspection merely discovers a lack of quality” [12]. If, instead of only inspecting the product, testers help their team build quality into the product testers will be helping their team to improve and helping to improve the quality of the product. By helping to build quality into a product, testers will also develop a broad skill set that will serve their team, their company and themselves well. 

References

[1] Out of the Crisis by W. Edwards Deming (1986, p23)

[2] Four Days with Dr. Deming by William J. Laztko and David M. Saunders (1995, p48)

[3] The Deming Dimension by Henry R. Neave (1990, p297)

[4] The Deming Dimension by Henry R. Neave (1990, p298)

[5] Out of the Crisis by W. Edwards Deming (1986, p29)

[6] Why should we test in the design phase? How should we go about it? by Melisa Fisher

[7] Deming’s Journey to Profound Knowledge by John “Botchagalupe” Willis with Derek Lewis (2023, p193)

[8] The Deming Dimension by Henry R. Neave (1990, p298)

[9] Out of the Crisis by W. Edwards Deming (1986, p88)

[10] Out of the Crisis by W. Edwards Deming (1986, p277)

[11] The New Economics by W. Edwards Deming (1994, p95)

[12] Deming’s Journey to Profound Knowledge by John “Botchagalupe” Willis with Derek Lewis (2023, p193)

Additional resources:

If you are weighing up which tool to use for test automation have a look at Playwrights Trace Viewer, it may persuade you to use Playwright

When we are automating and maintaining tests we need good feedback on test runs. Playwright gives quick and useful feedback in its Trace Viewer which provides feedback on the run of tests. Trace Viewer’s feedback is one reason I like automating tests with Playwright. Through Trace Viewer you can:

  • View the UI of the application under test at each step and before and after each step:
  • See on the UI where the mouse has clicked:
  • See the UI of the application under test over time:
  • Read the content of the network and console tabs from dev tools as they were at any point in the test run:
  • See how long each step in the test takes:
  • See the source code of the test that is executed at each step:
  • If a test fails I can read an error message that includes the name of the file and line number where the test failed:
  • Get metadata about your test:
  • Find the built-in locator for an element in the UI, I could not find this feature in the documentation. To find the built-in-locator for an element:
    • click on the circles next to Locator
    • click on the UI element in the screenshot that you would like to know the built-in-locator for
    • then the built-in locator will be displayed in the Locator tab:

If a Playwright test fails, Trace Viewer can help you:

  • see a screenshot of where the test failed
  • investigate the UI of the application under test at the point it failed and use the browser’s dev tools to investigate the failure
  • see the lines of test code in the test where the failure occurred
  • because you can share the URL from CI of the Trace View 

Trace Viewer can help you develop new tests because

  •  run locally by adding an option to the command line
  • it can show you the built-in locator for an element 
  • if a test fails it shows you where in the test it failed

I have automated tests with several tools and never found a feature as helpful as Playwright Trace Viewer. I will also likely find more ways that Trace Viewer provides valuable feedback over time. If you are considering automating tests with Playwright I would recommend investigating Playwright’s Trace Viewer because it may be the factor which persuades you to use Playwright!

Sometimes you need to assert that Playwright tests are not running too fast!

Tests that are automated in Playwright run fast, which is great. However, sometimes they run so fast that assertions need to be used to stop the test being flaky.

Playwright uses Auto-waiting to help it run tests fast. Auto-waiting means that Playwright performs several checks before making an action, for example, it checks that an element is visible before clicking on it. I have found two situations in which steps in Playwright tests ran so quickly that they caused tests to be flaky. An assertion, that checks that an action has been completed, can be added to a test to prevent this from occurring. The assertion controls the flow of the test so that an action is completed before the next action is executed.

If you automate a test for a configuration with optional items, the button to save those choices is enabled while the user makes their optional configuration choices. After a configuration option has been checked the ‘save’ button will pass the checks that auto-waiting makes and Playwright can click the ‘save’ button before the chosen configuration option has been saved. The test flakes if the optional configuration option has not been saved. So that the test did not flake I added an assertion that checks that the optional element has been selected before the test clicks on save, and the test no longer flakes. An example of this would be:

const locator = page.getByLabel('Subscribe to newsletter');

await locator.click();

await expect(locator).toBeChecked();

await page.getByRole(‘button’{ name: /save/i });

I also found that when Playwright selects an option from a select element, the application may not have completed selecting the option before Playwright makes the next action. The next option, say a button, passes the test for auto-waiting and so Playwright clicks on it before the application completes selecting the option. If the correct option has not been selected the test flakes. To prevent the test flaking I added an assertion that checks that the correct option has been selected before the test makes the next action. An example of this would be: 

await page.getByLabel('Choose').selectOption('Test.xsls');

const locator = page.getByLabel('Spreadsheet');

await expect(locator).toHaveText('Test.xsls');

await page.getByRole(‘button’{ name: /continue/i });

Asserts are a good way to make a test wait for an action to complete because an assert waits for its condition to be true instead of waiting for a fixed time. 

Playwright uses auto-waiting to wait the minimum time before making an action. Sometimes it is necessary to use an assertion to make the test wait for an action to complete before it takes the next action.

Get insights from “The World of W. Edwards Deming” by Cecelia S. Kilian

Cecelia S. Kilian was W. Edwards Deming‘s long-term secretary. She created this book which contains her memories, and many curated documents from Deming. I read this book with the Profound Deming Book Club and it made me think about why I am interested in Deming. The details of Deming’s life are interesting, however I am interested in him because I want the same things he did. I want companies to be successful businesses by continually improving the quality of their products and services. 

The most interesting chapters in the book are Deming’s notes on what he taught Japanese management in 1950 at lectures which he was invited to give by the Japanese Union of Science and Engineering. The book includes notes on his lecture at Mt. Hakone. If I understand the history of the idea I understand the idea more deeply. I found it fascinating to read in the notes of lectures from nearly 75 years ago many ideas that are used today. These points are taken from the chapters about Deming’s 1950 lectures :

  • Building quality into a product: “You can not inspect quality into a product”. ”You must build in quality” [1]
  • The use of control charts to analyse processes, as is done in many industries today. [1]
  • User testing, which he describes as consumer research.” Consumer research is communication between the manufacturer and users and potential users” of the product or service”. [3]
  • The plan-do-study-act cycle. Today the plan-do-study-act cycle describes a software development team’s iterations[4] and “the Toyota Production System .. embodies the learning cycle of ” the plan-do-study-act cycle [5].
  • New product development with the plan-do-study-act cycle. To begin the development “of a product on a pilot scale and to build up its production on a sound economic basis, only as fast as market conditions indicate, re-designing the product from time to time in the light of consumer needs and reactions”. Today, we would probably describe that form of new product development as ‘lean startup’.[6]
  • Viewing product development as a system. He uses a flow diagram to describe production as a system, showing how  “improvement of quality envelopes the entire production line”[7]. The only key change I would make to the diagram to make it relevant to software development is to have data as input instead of raw materials.
  • A non-blame culture. We should not blame an individual for a fault because the fault is probably common to the system. Fixing these common causes of faults is an important way to improve quality.[8]
  • Quality attributes, which Deming called quality-characteristics. [9]

It is worth considering how engineering, including testing, would have been different without these lectures. Would the Toyota Production System have been the same? And if Toyota was not the same, would we have had lean and agile?

Another useful chapter in the book is an appreciation of Deming’s “friend, mentor and colleague” Walter A. Shewhart. [10]

The book also includes moving recollections of family life, childhood memories,  Deming’s sheet music, a description of being introduced to the Emperor of Japan, and accounts of his meetings with the Toyoda and Ishikawa families.

If you are interested in Deming as a person this book is for you because it enables you to get to know Deming as a human being. If you would like to learn from his notes on what he taught in Japan in 1950 you will find this book useful because it has some chapters which describe the lectures he gave in Japan.

References

[1] The World of W. Edwards Deming by Cecelia S. Kilian (1992, p70)

[2] The World of W. Edwards Deming by Cecelia S. Kilian (1992, p63)

[3] The World of W. Edwards Deming by Cecelia S. Kilian (1992, p67)

[4] Scrum: The Art of Doing Twice the Work in Half the Time by Jeff Sutherland (2015, p36)

[5] The Toyota Way by Jeffrey K. Liker (2004, p263)

[6] The World of W. Edwards Deming by Cecelia S. Kilian (1992, p69)

[7] The World of W. Edwards Deming by Cecelia S. Deming (1992, p24)

[8] The World of W. Edwards Deming by Cecelia S. Deming (1992, p80)

[9] The World of W. Edwards Deming by Cecelia S. Deming (1992, p62)

[10] The World of W. Edwards Deming by Cecelia S. Deming (1992, p87)

The Pesticide Paradox is a reason to do exploratory testing

Whichever technique is used to create tests they will contain assumptions about the nature of bugs. Each technique targets a different set of bugs. If development teams react to bug reports by fixing the bugs that have been reported and by finding and fixing similar bugs, then running the same tests is unlikely to find bugs.[1] 

This is “called the ‘pesticide paradox’ after the agricultural phenomenon, where bugs such as the boll weevil build up tolerance to pesticides, leaving you with the choice of ever-more powerful pesticides followed by ever-more powerful bugs or an altogether different approach”. [1]

The “Pesticide Paradox” is more relevant now than when Beizer wrote about it in 1995 because today unit testing is widely used and every bug fix should have unit tests to ensure that it does not reoccur. 

Automated tests are run over and over again so the ‘Pesticide Paradox’ does raise questions about the role of automated tests. There are lots of reasons to automate tests but in my experience, automated tests are unlikely to find many bugs. 

A reason to do exploratory testing is the ‘Pesticide Paradox’. We need to “learn, create and use new tests based on new techniques to catch”[1] bugs and “exploratory testing involves scouting around the areas”[2] that tests do not cover.

References

[1] Black-box testing by Boris Beizer (1995, p9)

[2] Explore It! By Elizabeth Hendrickson (2013, p5)

A review of ”The Checklist Manifesto How to Get Things Right” by Atul Gawande

As testers, we often use checklists, such as a Definition of Done, so I was drawn to this book when it was suggested at the Profound Deming Bookclub. 

Atul Gawande tells the story of how he was asked by the World Health Organisation(WHO) to develop a global programme to reduce avoidable deaths and harm from surgery and used a checklist to do so.  He used knowledge from other professions to create the checklist.

Ineptitude and ignorance are causes of failure. We can reduce failures caused by ineptitude by applying “the knowledge we have consistently and correctly”. [1]

As a result of a test flight for a Flying Fortress that resulted in an air crash, a checklist was created for pilots. The checklist enabled the plane to be flown safely.[2] A checklist reduced infections in Michigan’s hospitals [3]. The construction industry has checklists that specify communication tasks which make everyone work as part of a team that takes each other’s concerns into account. [4]

Checklists “are not comprehensive how-to guides”. “They are quick simple tools aimed to buttress the skills of expert professionals”[5]

The first checklist that Gawande created for the WHO was too long, and lessons were taken from the aviation industry to improve it. A new checklist for the WHO was tested in eight institutions around the world. The test showed that the checklist was successful. The team was concerned that this was due to the Hawthorne Effect, but they could prove that it was not. The checklist “aimed for a team conversation” [6] and it had improved communication. The improvement in communication was the key to the improved results.[7] The two-minute WHO checklist helped teams embrace a culture of teamwork and discipline[8]. The WHO checklist has saved and is saving, many lives. 

Some people may feel that checklists are “beneath us”. If that is how you feel you should remember that Captain Chesley  B. “Sully” Sullenberg used checklists to save lives when birds hit his plane over New York.[9]

The WHO checklist was created by drawing on the experience of professionals from more than one industry. Checklists can improve communication and help teamwork. This book can be used to help testers create checklists by learning from the experience of others, for instance by creating checklists that help to improve communication within their team.

References

[1] The Checklist Manifesto How to Get Things Right by Atul Gawande (2011, p10)

[2] The Checklist Manifesto How to Get Things Right by Atul Gawande (2011, p34)

[3] The Checklist Manifesto How to Get Things Right by Atul Gawande (2011, p44)

[4] The Checklist Manifesto How to Get Things Right by Atul Gawande (2011, p65)

[5] The Checklist Manifesto How to Get Things Right by Atul Gawande (2011, p128)

[6] The Checklist Manifesto How to Get Things Right by Atul Gawande (2011, p137)

[7] The Checklist Manifesto How to Get Things Right by Atul Gawande (2011, p156)

[8] The Checklist Manifesto How to Get Things Right by Atul Gawande (2011, p160)

[9] The Checklist Manifesto How to Get Things Right by Atul Gawande (2011, p173)

Drive out fear to improve psychological safety

We need psychologically safe workplaces.  Testers, and all other members of development teams, need to feel safe enough to be able to ask questions and express opinions about the project we are working on. 

The need for psychological safety is not a new issue. W. Edwards Deming saw it as an important issue. Drive out fear is one of Demings’ 14 Points for Management which he published in 1982[1]. He first spoke about his 14 Points for Management at a conference in Tokyo in 1978[2]. The early versions of his Points for Management, when he spoke in Japan, did not include Drive Out Fear because Japanese managers did not need to be told to Drive Out Fear. [3]

He wrote that “fear takes on many faces. A common denominator is loss from impaired performance and padded figures”[1]. No one can put in their best performance unless they feel secure[4]. Examples of fear include[5]:

  • I am afraid I will lose my job
  • I am afraid that I may not always have an answer when my boss asks something
  • I am afraid to admit a mistake
  • My boss believes in fear

Drive out fear is one of Deming’s Points for Management. If you are working in a climate of fear then it is not your fault, it is the responsibility of management. Instead of command and control management, “Deming directed managers to build trust throughout the organisation.” [6]

“Management in authority will struggle over every one of the” 14 Points for Management [7], including Drive Out Fear. Deming wrote that the Plan-Do-Study-Act Cycle will be helpful as a procedure to follow for improvement[8]. Management can use the Plan-Do-Study-Act cycle to drive out fear and improve psychological safety. They can plan an initiative to improve psychological safety, such as providing training on how to facilitate meetings. Then “do” the initiative, study its results and then act on what they learned from the study.

Innovation in the twenty-first century is being supported by Deming’s 14 Points for Management, including Drive Out Fear, for example, they are the roots of DevOps.[9]

Psychological safety is important for everyone at work. It has particular importance for testers because we need to be able to ask questions. Drive out fear is a good example of how Deming’s 14 Points for Management provides a framework to create a great place for testers, and everyone else, to work.

References

[1] Out of the Crisis by W. Edwards Deming (1982, p59)

[2] Quality or Else by Lloyd Dobyns and Clare Crawford-Mason(1991, p102)

[3] The Deming Dimension by Henry R. Neave (1990, p37)

[4] Out of the Crisis by W. Edwards Deming (1982, p58)

[5] Out of the Crisis by W. Edwards Deming (1982, p60)

[6] Out of the Crisis by W. Edwards Deming (1982, p88)

[7] Out of the Crisis by W. Edwards Deming (1982, p86)

[8] Deming’s Journey to Profound Knowledge by John “Botchagalupe” Willis with Derek Willis (2023, p200)

[9] Deming’s Journey to Profound Knowledge by John “Botchagalupe” Willis with Derek Willis (2023, p163)

Further Information:

A review of “Understanding Variation The Key to Managing Chaos” by Donald J. Wheeler

This book is both insightful and useful. It was recommended to me by members of the Deming Profound Book Club. Wheeler describes how to create control charts and analyse processes using control charts. This book can help you start to use control charts to analyse data from your test and development processes.

Walter Shewhart invented control charts when he was working for Bell Telephones. For the last one hundred years managers and teams have been using control charts. They are used in many industries, but only sometimes used in software development. Testers can use control charts to provide a useful way to analyse the results of performance tests. 

The use of goals and targets to analyse data is contrasted with the use of control charts. Goals and targets are described as “the Voice of the Customer” because it defines what you want. Control charts are described as the “Voice of the Process” because they define what you will get from a system. [1]

Control charts contain time series data, a central line as a visual reference, and upper and lower control limits. Wheeler describes how to calculate the control limits from the variation in the time series data. 

All variation is characterised by control charts as either predictable (within the control limits and so in statistical control) or unpredictable (outside the control limits and so out of statistical control). [3]

A major advantage of “the Control Chart Approach filters out noise by the construction of the control limits. Signals are indicated by points which fall outside the control limits or by obvious non-random patterns of variation around the central line” [4]

Wheeler uses control charts to tell stories about the economy and companies. Each story throws light on a different aspect of the valuable insights provided by using control charts. These stories include sections about how to analyse events that occur rarely and how a company’s performance is affected by an improvement plan.

I used this book to help me create control charts using Geckoboard Datasets AP, the product that I am testing. I have learned so much about the product I test and about how to use control charts by creating control charts. I would encourage you to use this book to start experimenting by creating control charts and so gain insights from your process.

I would also like to thank Rob Park for his help when I had questions about creating and analysing control charts.

References

[1] Understanding Variation The Key to Managing Chaos by Donald J. Wheeler (1993, p79)

[2] Understanding Variation The Key to Managing Chaos by Donald J. Wheeler (1993, p23)

[3] Understanding Variation The Key to Managing Chaos by Donald J. Wheeler (1993, p112)

[4] Understanding Variation The Key to Managing Chaos by Donald J. Wheeler (1993, p30)

Testing qualities not quality

To help me test I used to find it useful to think about what quality is for the application that I am testing.  “There has been a tendency to conceive of quality as indicating the goodness of an object.”[1] There are many aspects to quality and I have found that this idea of quality is ”too indefinite for practical purposes”[1].  I now think about the qualities of the application, not its quality.

Walter Shewhart worked to improve the quality of telephones and wrote that “every conceptual “something” is really a group of conceptions in more elementary form. The minimum number of conceptions required to define an object may be called the qualities thereof”. “In this sense a thing has qualities not a quality”[2]. Shewhart’s approach to quality is useful because it prompts us to ask what are the qualities of the software that we are testing. 

Shewhart quoted Jevons[1] who wrote that “two fragments of rock may differ entirely in outward form, yet they may have the same colour, hardness, and texture. Flowers which agree in colour may differ in odour. The mind learns to regard each object as an aggregate of qualities, and acquires the power of dwelling at will upon one or other of those qualities to the exclusion of the rest”.[3]

When we are testing we need to find the qualities that differentiate things that appear to be the same. Testing can be, as Jevons suggested, like observing two fragments of rock that have similarities. One fragment could be our understanding of how the product should be and the other fragment could be the product that we are testing. The two fragments of rock could also be two versions of the application that have the same backend but different UIs. We need to find the differences between the two and discovering the qualities of each helps to do that. Performance, testability or usability could all be qualities of the product that we are testing.

The practice of analysing the qualities of a product led to the use of quality attributes for testing. 

Defining the qualities of the application under test will also help with defining the quality criteria used when defining quality coverage, as in Melissa Fisher’s recent blog post: Test coverage — how about quality coverage?

Asking questions about qualities helps me break down the functionality into smaller batches for testing. I can test one quality at a time, for example, I can test accessibility.  This approach can also help me gain insight into one or more of the qualities and give me new paths for testing through the application.

References

[1] Economic control of quality of manufactured product by Walter Shewhart (1931, p55)

[2] Economic control of quality of manufactured product by Walter Shewhart (1931, p56)

[3] The principles of science : a treatise on logic and scientific method by William Stanley Jevons (1913, p24 )

Developing your listening skills is really useful

We all know that speaking up and getting your point over at a meeting is important. However, I am sure that we have all taken part in meetings when everyone is so keen to speak that we do not listen to one another. 

It is not only important to speak, it is also important to listen. If people speak but do not listen then there is no conversation. If people do not listen to each other we can’t learn from each other and they cannot cooperate to achieve a goal. 

If people are talking and are not listening some people will find it difficult to get to speak because everyone else is busy speaking. Listening to each other will help improve inclusion because it will help everyone to speak at meetings. 

Sometimes it is hard for us to listen because, instead of listening, we are waiting to speak. I use a technique to help me listen. I have learned to ask myself what is going to be the last thing that someone is going to say. This technique helps me to listen to every word someone is saying and not to interrupt the person speaking. 

Asking myself what will be the last thing that someone says on a subject during a discussion also helps me when I speak because I can take into account what others have said. 

Listening is an important skill for testers. We need to build a rapport with developers and listening to them will help. We also need to hear every point of view on a subject because we may hear something that will help us test. We need to listen to stakeholders to know what they expect and hope for. We also need the team to work cooperatively because teams that work cooperatively achieve more.

This podcast from Amy Gallo via Harvard Business Review is a great resource with useful tips for improving your listening skills: Practice Your Active Listening Skills.

TIL: more about using zero in testing

Three testers walk into a bar…and we all know that they order, one beer, zero beer and minus one beer! We also all know that they need to order zero beers because the behaviour of zero can be different from the behaviour of other numbers.  We all test with zeros for this reason. Recently I have learned something more about the behaviour of zero that is helping me to test..

I am doing Typescript tutorials to help me automate tests and recently learned more about behaviour with zeros which helped me to understand the cause of a bug. I have recently learned about conditional narrowing, which can be used to check the type of a value in TypeScript. If the && operator or the || operator are used for truthiness narrowing then zero is coerced to false. The && and || operators treat zero in the same way as are NaN, “”(the empty string), 0n (the bigint version of zero), null and undefined. This means that truthiness narrowing does not treat zero as a number, it treats zero as false

A friend of mine used this knowledge and found a bug in which the user could enter zero but was not able to save the zero that they had entered. The application could not save zero because its logic said that zero was false so was not a number.

Zero is a curious thing and I am using what I have just learned about to help me test. 

Additional Learning Resources:

Test using Quality Characteristics\Factors\Attributes that you create.

It can be useful to define aspects of a product that describe the quality of the application that you are testing. I have found examples of doing this in seven different decades. Sometimes these aspects have been put into sets, lists or groups and have sometimes been called quality characteristics, sometimes quality factors and at other times they have been called quality attributes. These sets, lists and groups have also been used in different ways to improve quality.

Walter Shewhart wrote in 1931 about the “conception of the Quality of a Thing as a set of characteristics[1]”, and in 1939 wrote that the “aim is to produce a sequence of objects having a specified quality characteristic within some specified limits”[2]. The quality characteristics of the objects would be controlled by the use of control charts. 

In 1950 in Japan in his lecture at Mt Hakone, W. Edwards Deming spoke about “communication between buyer and the seller of the quality-characteristics of a product is provided by the same statistical techniques that are used for improved testing, production and inspection”[3]. The statistical technique he was suggesting is that of the control chart. The quality characteristics he spoke about would have been for physical products. 

The Western Electric Statistical Quality Control Handbook was first published in 1956 and describes the Statistical Quality Control process developed by Shewhart that used control charts. The Handbook says that  “the word “Quality” means much more than the goodness or badness of a product. It refers to the qualities or characteristics of the thing or process being studied”[4].

A report for the USAF in 1977 listed quality factors under headings in groups[5], for example under the heading testability it lists simplicity, modularity, instrumentation and self-descriptiveness. The report also recommends which factors should be measured[6], and how to measure them.[6]

Attributes to specify quality were defined in 1988 as “a quality concept or resource concept which describes a system quantitatively” [7]. These attributes are to be “specified and controlled throughout the project”[7]. The attributes could be expressed as a hierarchy, for example:

  • System performance
    • Weekly work-load capacity
    • System availability
    • Repose time to questions[8]

In 1998 the list of quality factors from the 1977 report is used as a list of quality attributes. “These attributes include both subjective and objective measures and are only a subset of the total attributes of software programs”, and “the applicable elements should be selected by the quality professional for application to each ..program”[7].

A list of eighteen quality attributes was listed in 2002 under Testing Techniques, with the advice “to determine whether a feature is defective, ask yourself how you would prove that it lacks or violates one of these attributes”[10]

Seven product dimensions, which included a quality attribute dimension, were taken from business analysis to help describe quality in 2015. It is “helpful to consider all seven dimensions during release, feature and iteration planning”[11]

Quality professionals have been using quality characteristics/factors/attributes of a product to describe the quality of products and services for a long time. I have not given an example of their use from the 2020s, it is up to us to find how best to use quality characteristics/factors/attributes to help us with the testing challenges we face today!

Thank you to Dennis Sergent for his help with this blog post. 

References

[1] Economic Control of Quality of Manufactured Product by Walter Shewhart (1931, p55)

[2] Statistical Method from the Viewpoint of Quality Control by Walter Shewhart with a foreword by W. Edwards Deming (1986 – originally published in 1939, p7)

[3] The World of W. Edwards Deming by Cecelia S. Kilian (1992, p62)

[4] Western Electric Statistical Process Control Handbook (1982, p2)

[5] FACTORS IN SOFTWARE QUALITY Concept and Definitions of Software Quality By Jim A. McCall, Paul K. Richards CD Cene, F. Walters (1977, p44)

[5] FACTORS IN SOFTWARE QUALITY Concept and Definitions of Software Quality By Jim A. McCall, Paul K. Richards CD Cene, F. Walters (1977, p38)

[6] FACTORS IN SOFTWARE QUALITY Concept and Definitions of Software Quality By Jim A. McCall, Paul K. Richards CD Cene, F. Walters (1977, p62)

[7] Principles of Software Engineering Management by Tom Gilb (1988, p134)

[8] Principles of Software Engineering Management by Tom Gilb (1988, p136)

[9] Computer Applications to Quality Systems by Frederic I. Orkin and Danliel Olivier in Jurans Quality Handbook 5th Edition edited by J.M. Juran and A. Blanton Godfrey (1998, p10.7)

[10] Lessons Learned in Software Testing by Cem Kaner, James Bach, Bret Pettichord (2002, p60)

[11] More Agile Testing by Janet Gregory and Lisa Crispin (2015, p130)

How long should an automated test pack take to run? And what should we do about it?

How long should an automated test pack take to run?

Some people say that a certain number of minutes is too long, and others say that a different number of minutes is too long. How can we determine what is “too long”? And what can we do about it?

What is an acceptable length of time for a test pack to run depends on several factors. Your team’s cycle time will affect how long it is acceptable for a test pack to run. Cycle time can be defined as a development process “ that transforms identified customer needs into delivered customer value at a reliable, repeatable cadence”[1] If your team is practising continuous deployment or continuous delivery and has a cycle time of a few hours there will be pressure for automated test packs to run more quickly than if your team makes monthly or quarterly releases. Teams that practice continuous deployment or continuous delivery work in small batches so the test packs will need to be small batches too.

Another factor that affects how long it is acceptable for a test pack to run is whether the length of time the tests take to run is a constraint on development. In some situations, a test pack of thirty tests that runs in fifteen minutes may be an enabler and in other situations, it may be a constraint. An example of this would be; if the test pack that you are concerned about takes no longer to run than other test and deployment processes then it is unlikely to be a constraint, however, if it takes much longer than other test or deployment processes then it may be a constraint on how developers work.

What can the test engineer responsible for the test pack do about the length of time it takes to run the test pack? 

The engineer who is responsible for the test pack should not be defensive about the test pack, instead, they should lead the discussion about how long it takes the test packs to run. They can, for instance, go to developer meetings and create a dialogue with developers by talking about how they are developing the test pack. This dialogue should include how long it takes for the pack to run. If the test automation engineer is involved in a dialogue they will find out how the test packs are affecting developers. They can also tell developers what they are doing to make the tests run faster and discuss with developers ideas as to how to improve the running of the test packs. Testers and developers should then use this dialogue to start to cooperate on improving test packs, including their running times.

What is an acceptable time for a test pack to run depends on several factors. We need to understand our development system and cooperate with our development team to improve test packs. 

References

[1] Implementing Lean Software Development: From Concept to Cash by Mary and Tom Poppendiek (2006,p98)

A theory of management for improvement of quality vs a quality improvement plan, which helps us more?

What can we learn from comparing Deming’s 14 Points for Management and Crosby’s 14-Point Quality Improvement Plan? Which will help us more to improve quality?

W. Edwards Deming first presented his 14 Points at a conference in 1978 in Tokyo[1] and published his 14 Points for Management in 1982[2]. Philip B. Crosby published his 14-point Quality Improvement Plan in 1979[3]. I have not found anything to suggest that their both having 14 points/ steps is anything but a coincidence.

Deming’s 14 Points for Management [2]Crosby’s 14-Step Quality Improvement Plan [3]
1. To create constancy of purpose towards improvement of product and service, with the aim to become competitive and stay in business and to provide jobs
2 .Adopt the new philosophy. We are in a new economic age. Western management must awaken to the challenge, must learn their responsibilities, and take on leadership for change. 
3. Cease dependence on inspection to achieve quality.
4. End the practice of awarding business on the basis of price tag. Instead, minimise total cost. Move toward a single supplier for any one item., on a long-term relationship of loyalty and trust.
5.  Improve constantly and forever the system of production and service, to improve quality and productivity, and thus constantly decrease costs.
6. Institute training on the job.
7. Institute leadership. The aim of supervision should be to help people … do a better job. 
8. Drive our fear, so that everyone may work effectively for the company.
9. Break down barriers between departments. People in research, design, sales and production must work as a team.
10. Eliminate slogans, exhortations and targets for the workforce..such exhortations only create adversarial relationships.
11. Eliminate management by numbers..substitute leadership.
12. Remove barriers that prevent people from taking pride in their work
13. Institute a vigorous programme of education and self-improvement.
14. Put everyone in the company to work to accomplish the transformation. The transformation is everybody’s work.		1. Management Commitment –  to make it clear where management stands on quality
2. The Quality Improvement Team –  to run the quality improvement programme
3. Quality Measurement – to provide a display of current and potential nonconformance problems in a manner that permits objective evaluation and corrective action
4. The Cost of Quality – to define the ingredients of the cost of quality and explain its use as a management tool
5. Quality Awareness – to provide a method of raising the personal concern felt by all personnel in the company toward the conformance of the product or service and the quality reputation of the company
6. Corrective Action  – to provide a systematic method of resolving forever the problems that are identified through previous action steps
7. Zero Defects Planning – to examine the previous activities that must be conducted in preparation for formally launching the Zero Defects programme.
8. Supervisor Traning – to define the type of training that supervisors need to actively carry out their part of the quality improvement programme
9. Zero Defects Day – to create an event that will let all employees realise through a personal experience that there has been a change.
10. Goal Setting – to turn pledges and commitments into action by encouraging individuals to establish improvement goals for themselves and their groups.
11. Error Cause Removal – to give the individual employee a method of communicating to management the situations that make it difficult for the employee to meet the pledge to improve.
12. Recognition – to appreciate those who participate
13. Quality Councils – to bring together professional quality people for planned communication on a regular basis.
14. Do it over again – to emphasise that the quality improvement program never ends

What do Deming’s points and Crosby’s steps have in common? 

  • both aim to help businesses improve the quality of their product or service
  • both were part of the Quality Movement
  • both aim to prevent errors
  • both view quality as management’s responsibility

What are the key differences between Deming’s points and Crosby’s steps?

  •  Deming described his 14 Points as a “theory of management for improvement of quality, productivity, and competitive position”[4]. In contrast, Crosby’s steps are a quality improvement program comprising 14 steps.
  • Deming’s 14 points are a coherent philosophy, each point of the 14 points is implicit in all the other points. An example would be training (point 6) will be ineffective unless barriers that prevent people from taking pride in their work are removed (point 12). Crosby’s plan is a series of steps rather than points that are implicit in each other.
  • The role of management is different:
    •  Deming’s points include instituting leadership so that the role of a manager ”is a coach and counsel, not a judge”[5]. In contrast, Crosby’s plan has a more traditional hierarchical view of management’s role in which it is management who runs the Quality Improvement Programme. 
    • The manager’s role in Deming’s Points includes driving out fear, this means that managers take responsibility for psychological safety. Staff working in a company using Crosby’s plan would probably need to use the Error Cause Removal step to raise issues relating to psychological safety.
  • Deming’s points include developing people’s potential through education and training for all staff, whereas Crosby’s plan only requires training for supervisors.
  • In Deming’s points the whole company, including its suppliers, is seen as a system in which everyone should cooperate, and the manager’s role is to help everyone work effectively. Crosby’s plan puts responsibility on individual members of staff, for example, on Zero Defects Day individual staff are asked to sign pledges to improve quality.
  • Deming’s Points include cooperating across company departments and between the company and its suppliers to improve quality. Crosby’s steps do not include cooperation.
  • Crosby‘s plan sets goals and looks at costs, whereas Deming’s points ask for constant improvement.
  • “Building quality in” instead of “inspecting quality in” is covered by Deming’s point 3. Crosby has a step on Corrective Action to resolve problems.
  • Crosby aims for Zero Defects, whereas Deming aims for continual improvement.
  • Deming’s 14 Points for Management are the roots of DevOps[6] and so are part of innovation in the twenty-first century. I am not aware of any twenty-first century innovation based on Crosby’s 14 Steps.

We should note that Deming also used his System of Profound Knowledge, and the plan-do-study-act cycle to help businesses. Crosby also used a Quality Management Maturity Grid.

Deming’s ideas are powerful “on cooperation and human potential”[6]. We need cooperation and the realisation of human potential to improve quality and I wouldn’t want to work anywhere that was very different to Deming’s 14 Points for Management.

References

[1] Quality or Else by Lloyd Dobyns and Clare Crawford-Mason (1991, p102)

[2] Out of the Crisis by W. Edwards Deming (1982, p23)

[3] Quality is Free by Philip P. Crosby (1979, p137)

[4] Out of the Crisis by W. Edwards Deming (1982, p19)

[5] The New Economics by W. Edwards Deming (1994, p126)

[6] Deming’s Journey to Profound Knowledge by John “Botchagalupe” Willis with Derek Willis (2023, p163)

Additional Resources

  • Deming NEXT – eLearning from the Deming Institute with a free two-week trial
  • Deming’s Journey to Profound Knowledge by John “Botchagalupe” Willis with Derek Willis  –  Chapter 19 gives contemporary true-to-life examples for each of Deming’s 14 Points. 

Gaining a knowledge of React is helping me automate tests

React is an open-source User Interface (UI) JavaScript library created by Facebook. I am doing some React tutorials to help me better understand the UI and what I am learning is helping me to automate tests.

The React tutorials I am doing are here: https://react.dev/learn. The tutorials include exercises that can be done in a code sandbox, so you do not need your own development environment to do the exercises. I have gained a better understanding of the course content by doing the exercises. I have done some exercises in my dev environment and some in the code sandbox.

I have learned that React apps are made out of components. I am now learning about React components and understand that if there is a bug in a component the bug may be present wherever the component is used. I have also learned about CSS classes, and how they can be used to create a similar appearance throughout the app. 

I now better understand the developers when they talk about “props” and “lifting state up”.  This helps me contribute to discussions at planning meetings.

The React tutorials have given me a better understanding of the UI that I am automating tests for. When I have had a problem automating an action I have been able to look in the repo and work out how to interact with the React component that I am trying to automate. On one occasion the DOM element was nested and I found that by reading the code I could see which element handled an onClick event, and so could automate the click that I wanted.

Doing the React tutorials is making me more of a ‘T’ shaped tester. I have deep testing skills but I also now have a growing understanding of React, and this knowledge of the UI is helping me automate tests.

Design a site like this with WordPress.com
Get started