Gain insights by using control charts to analyse your performance test results

On Friday 16 May 1924 Walter Shewhart gave his manager at Bell Telephone Laboratories a memo.  The memo “suggested a way of using statistics to improve quality in telephones.[1]” Shewhart’s memo proposed using Statistical Process Control, including Control Charts for visualisation, to improve quality. Shewhart sparked “a revolution in quality control”[2] that can help us analyse the results of our performance tests. 

Control charts visualise time series data along with the data’s average and upper and lower natural process limits. “Faults from fleeting events[3]” are described as special causes and appear outside a control chart’s upper and lower natural process limits. Faults of the system are described as common causes and appear on a control chart within the upper and lower natural process limits. The natural process limits are calculated from the variations in the time series data[4]. 

The above control chart shows UK inflation as it spiked post covid. The control chart shows that UK inflation had a fault outside the natural process limits and so had a special cause. Seeing whether the data from performance tests falls inside our outside natural process limits enables a tester to understand if the faults have a special cause ( a ‘fleeting event’) or a common cause (a fault of the system). This is very useful when analysing the results of performance tests because a fault due to a fleeting event should be treated differently from a fault with the system.

Eight patterns, known as ‘Nelson Rules’, can be used to analyse data variation within the chart’s natural process limits. 

A control chart lets managers track variation and compare datasets. After the introduction of control charts managers could find and fix the causes of defects, rather than blame the staff for faults.

Control charts can also be called Process Behaviour Charts.

We now have one hundred years of learning from using control charts, which are now widely used. If you have a Fitbit you will be using control charts because the personal ranges for your health data are based on control charts. Here are links to guides to using them in different processes including DevOps automated governance, analysis of unit testing, load testing, customer satisfaction, health care, nuclear power, car manufacturing, education quality management system, managing stock levels, and marketing Campaigns. 

Thank you to John Willis, Dennis Sergent, Rob Park and the Deming Profound Book Club members for helping me to gain a deeper understanding of control charts. 

Here are some resources that will help you if you would like to use control charts to gain insights from your performance test results:

Introductory guides to control charts:

A useful guide if you want to create a control chart:

A video introduction to control charts:

An API that can be used to create control charts

A One Day course at Coventry University:

References

[1] Quality or Else by Lloyd Dobyns and Clare Crawford-Mason (1991, p52)

[2] Manufacturing the Future by Stephen B. Adams and Orville R. Butler (1999, p216)

[3] Out of the Crisis by W. Edwards Deming (1982, p314)

[4] Understanding Variation: The Key to Managing Chaos by Donald J. Wheeler (1993, p35)

How long will that test automation take?

Sometimes testers are asked how long it will take to automate a batch of tests. Planning how long your test automation should be simple, however, the plan will have missed some requirements.

There are three types of requirements[1]:

  1. Known requirements – these are the requirements that have been clearly defined in planning meetings. 
  2. Overlooked requirements – we are all human, we need to allow for missing things so we need to recognise that we will have missed some requirements. The initial analysis could have overlooked functionality that requires an automated test.
  3. Emergent requirements – these are requirements “that surface through the act of building the product.” [1]. Examples of emergent requirements in test automation could include:
    • A feature of the testing tool does not work as expected, and the test automation engineer needs to find a new solution to a test automation problem.
    • An element in the UI is difficult to interact with, for example, it is difficult to identify the element with a CSS selector, XPath or the test tools built-in locator.
    • The testing tool does not support something that the test requires and you need to extend the capability of the testing tool using native code such as TypeScript.
    • When you start to automate a new test you discover that a function needs to be refactored and as a result of refactoring the function, other tests also need to be refactored..

“We will never understand all the requirements of a story ahead of time”[2], this applies to test automation stories too.

Instead of creating a plan for test automation, it is better to create a test automation backlog. To make a backlog think about your vision for test automation and “start breaking down the things you need to execute that vision” [3]. “Don’t over plan, just estimate”[3], and create a list of the tests you want to automate. “The picture will change” [3]. “The reason for doing this type of planning is to create transparency within the organisation”[3].

References

[1] Agile Requirements Gathering: Three Types of Requirements by Mike Cohn (2020)

[2] Agile Testing by Janet Gregory and Lisa Crispin (2009, p133)

[3] Scrum: The Art of Doing Twice the Work in Half the Time by Jeff Sutherland (2015, p200) 

Do Chromatic tests complement Playwright tests?

I am using Playwright to automate end-to-end tests, and have started to complement my Playwright tests with Chromatic tests. 

The Playwright tests are really useful, but each test needs to run through several steps to create a scenario I want to test. Creating a Playwright test for each input variation for each part of the UI would be expensive. I would like a test automation tool to check a wide range of inputs for each element of the UI that does not require automating a large number of steps.

I have been introduced to Chromatic by a developer. Chromatic is a test tool that is made by the team behind Storybook. If your team uses Storybook you can use Chromatic to test for visual regressions in UI components. Chromatic tests can be described as “unit tests for visual states”. 

Chromatic can be used to test each component in the UI. With Chromatic you can, for instance, create tests that check how a component handles the values that can be entered into an input field. Chromatic tests are like unit tests and do not require steps to set them up, also each test runs quickly so you can create Chromatic tests for all the inputs you would like without worrying that the test pack will take a long time to run. A Playwright test of each of these inputs would have the disadvantage of requiring automating several steps to set the test up before testing the inputs.

Chromatic tests can be written in javascript or typescript. An example test is below:

Chromatic tests can be written in more than one way:

Chromatic is a test automation tool that can complement Playwright, this is because I can write end-to-end tests in Playwright that test flows through the application and write “visual unit tests” in Chromatic that test a wide variety of inputs into individual UI components.

Cooperation helps to improve testing, helps testers and helps the company

Testers can help increase cooperation across the company, and cooperation will help us and the company. 

Testers give feedback to developers when we test. We should also get feedback from other departments and customers on our work and the product we are testing. The feedback we get from cooperating with other departments brings the perspectives of customers and other departments into our testing. These additional perspectives help our testing, help us learn about the company and help break down barriers within the company. 

When testers get feedback on the product from other departments, we also foster cooperation between them. Cooperation helps the company. If departments cooperate rather than compete against each other they will accomplish more. The company can be described as a system. If you improve one part of the system separately you can also destroy another part of the system.[1]  Cooperation helps the whole system improve.

Cooperation also helps develop testers’ potential. Learning from other departments in the company will also help a tester if they would like to progress in their career because it will help them gain a broader perspective of how the company functions.

We can cooperate with developers and learn from them for example, I was introduced to GitHub Copilot by a developer. We can also learn from Developers how they perceive risks and then use what we have learned when we test.

“Everyone in engineering design, purchase of materials, testing materials and testing performance of a customer has a customer…Why not get acquainted with the customer?”[2] I have learned a lot about customers from talking to Customer Support\Success, and have taken that learning into my testing and the development team’s work. Talking to Salespeople is also useful. I once created a stakeholder map from conversations with the Sales Team and shared the information it contained about our customers across the company. 

“Teamwork is sorely needed throughout the company. Teamwork requires one to compensate with their strength someone else’s weakness, for everyone to sharpen each other’s wit with questions”[2]. We can help improve teamwork across the company through discussions with people in different roles than our own. These discussions will help us learn from others in the company and we can take what we learn into our testing. We may, for instance, learn that a particular feature is often discussed between customers and customer support, this learning can be taken into our testing.

Cooperation is part of W. Edwards Deming’s theory of management. Point 9 of his 14 Points for Management is: We should “break down barriers between departments. People in research, design, sales and production must work in a team, to foresee problems of production and in use that may be encountered with the product or service.”[3] 

Testers can improve cooperation across the company, which is good for testing, testers and the company.

References

[1]  2003 Ackoff Seminar Part 1 of 4 from the Deming Cooperative (2019)

[2] Out of the Crisis by W. Edwards Deming (1986, p62)

[3] Out of the Crisis by W. Edwards Deming (1986, p24)

Think slowly when you are testing And Think slowly when you are automating

I am interested in how thinking slowly can help me test and I have been reading Gene Kim and Steven Spear’s new book  Wiring the Winning Organization: Liberating Our Collective Greatness Through Slowification, Simplification, and Amplification with the Deming Profound Book Club. Spear and Kim acknowledge how Daniel Kahneman, who recently passed away, influenced them when they show thinking slowly helps you improve. They also describe a tool that helps slow thinking. 

We can think fast and we can think slowly. We think fast when we are under pressure. We use established habits, routines and heuristics to do this. Fast thinking has its limitations, for example, our biases may guide our decision-making. Fast thinking also does not allow time for us to learn. We need to think slowly so we can improve. [1]

We can apply this to testing. If we use ‘fast’ thinking when testing we may test in the way we always have. If we use ‘slow’ thinking when we test we can view testing as a learning process and learn more about the product we are testing. 

If we use ‘fast’ thinking when we automate tests we may follow a known pattern for automating the tests. If we use ‘slow’ thinking when we automate tests we can learn a new way to automate a test or learn more about the application we are testing.

When we test and when we automate there are benefits in “Slowing ourselves down so that we can be more deliberative and self-reflective”[2] 

Sometimes I have had to repeat a path of exploratory testing because I tested too quickly and missed something I meant to test. When I repeated the testing I worked through the path slowly tested more slowly and found more issues with the product. I thought slowly when I repeated the testing and found more issues because I was learning more and thinking through the consequences of the tests I was executing.

I have sometimes merged an automated test and then regretted it. This happened because I was thinking quickly and ‘completed’ automating the test using existing patterns. After I had merged and thought slowly about the test, I realised there was a better way to automate the test, and I refactored the test.

Spear and Kim recommend ‘slowification’ which they define as “shifting problem-solving from performance (operation, execution) back to practice (preparation) and planning”[2]. They say that “Dr. W. Edwards Demings learning cycle of Plan-Do-Study-Act (PDSA) is a tool to encourage slowification”[3]. Testers can use the Plan-Do-Study-Act cycle to slow down our testing and automating so that we need to learn more while testing and automating. We can:

  • Plan our testing or automation
  • Do our testing or automation
  • Study our testing or automation
  • Act on what we learn from the Study
  • And then repeat the cycle
The Deming Cycle

If we use Plan-Do-Study-Act we will slow our thinking by studying what we are doing and acting on what we learn. We can use the Deming Cycle as a tool to help us slow our thinking because the cycle includes making time to study.

We can achieve more as testers if we slow our thinking and use the Deming Cycle to do so.

References

[1] Wiring the Winning Organization: Liberating Our Collective Greatness Through Slowification, Simplification, and Amplification by Gene Kim and Steven Spear (2024, Chapter 4)

[2] Wiring the Winning Organization: Liberating Our Collective Greatness Through Slowification, Simplification, and Amplification by Gene Kim and Steven Spear (2024, Chapter 3)

[3] Wiring the Winning Organization: Liberating Our Collective Greatness Through Slowification, Simplification, and Amplification by Gene Kim and Steven Spear (2024, Chapter 5)

It is better if we build quality into the product instead of trying to test quality in

“Cease dependence on inspection to achieve quality. Eliminate the need for inspection on a mass basis by building quality into the product in the first place.” [1] is one of W. Edwards Deming’s 14 Points for Management.  Inspection can be defined as testing after development has been completed. Some people have interpreted Deming’s point as an argument against testing. However, “Dr. Deming is often misunderstood on this point. The key word is dependence.”[2] “Is Deming saying that we should abolish inspection? No. of course not” [3].

“Dependence on mass inspection is fraught with danger and with high cost”. [4] An example of a development team being dependent on mass inspection would be a team that relies on manual regression testing before making a release. The manual regression testing will probably find bugs which probably need to be fixed. The bugs that need fixing are rework and fixing them will cost the company time and money. 

“Building quality into the product”[1] is cheaper for the company than inspection and is something that testers can play an important role in. It is cheaper because the earlier in the process an issue is found the cheaper it is to resolve, for example, it is cheaper to fix a fault in a specification than it is to fix a bug in production. Testers can help write and review specifications, and so help find issues early in the process. When we do so we are helping to build quality into the product.

“Inspection is too late. The quality, good or bad is already in the product”[5]. To build quality into the product we need to test as early as possible, for example, testing the designs of a feature before the feature is developed [6]. 

 “Deming said that quality isn’t so much about improving the product as it is about improving the process”[7]. Exploratory testing can help improve the process by identifying issues in the product. These issues can show faults in the development process. “There is a world of difference between, on the one hand dependence on inspection as an attempt to provide the customer with something he won’t complain about and, on the other, the use of inspection to provide guidance toward improvement of a stable process”[8].

The second part of the Deming Cycle, also known as the plan-do-study-act cycle, is to “carry out the change or test decided upon”[9]. Deming does not say who should carry out the testing. Testers’ knowledge of testing techniques will help the team conduct the tests the team needs to build quality into the product. 

An operational definition is a tool that can be used to build quality into a product or service. An operational definition requires there to be a specification for an item, for there to be criteria to show whether the item has met the specification and a decision on whether the item met the criteria.[10] Operational definitions are something that testing professionals can a team use to build quality into the product. Automated tests can help a team decide whether its product meets the criteria in operational definitions. Developing automated tests that give developers the confidence to release their software and find bugs before code is deployed to production is another way that testing professionals help to build quality into the product. 

Testing professionals have analytical skills and using systems thinking for analysis will help the team gain a better understanding of their situation. Systems thinking is a part of Deming’s System of Profound Knowledge.[11] Testers can help their teams build quality into the product by using systems thinking to understand the systems that our product is part of, for example, systems thinking can be used to understand that the company and its customers are one system.

“Inspection merely discovers a lack of quality” [12]. If, instead of only inspecting the product, testers help their team build quality into the product testers will be helping their team to improve and helping to improve the quality of the product. By helping to build quality into a product, testers will also develop a broad skill set that will serve their team, their company and themselves well. 

References

[1] Out of the Crisis by W. Edwards Deming (1986, p23)

[2] Four Days with Dr. Deming by William J. Laztko and David M. Saunders (1995, p48)

[3] The Deming Dimension by Henry R. Neave (1990, p297)

[4] The Deming Dimension by Henry R. Neave (1990, p298)

[5] Out of the Crisis by W. Edwards Deming (1986, p29)

[6] Why should we test in the design phase? How should we go about it? by Melisa Fisher

[7] Deming’s Journey to Profound Knowledge by John “Botchagalupe” Willis with Derek Lewis (2023, p193)

[8] The Deming Dimension by Henry R. Neave (1990, p298)

[9] Out of the Crisis by W. Edwards Deming (1986, p88)

[10] Out of the Crisis by W. Edwards Deming (1986, p277)

[11] The New Economics by W. Edwards Deming (1994, p95)

[12] Deming’s Journey to Profound Knowledge by John “Botchagalupe” Willis with Derek Lewis (2023, p193)

Additional resources:

If you are weighing up which tool to use for test automation have a look at Playwrights Trace Viewer, it may persuade you to use Playwright

When we are automating and maintaining tests we need good feedback on test runs. Playwright gives quick and useful feedback in its Trace Viewer which provides feedback on the run of tests. Trace Viewer’s feedback is one reason I like automating tests with Playwright. Through Trace Viewer you can:

  • View the UI of the application under test at each step and before and after each step:
  • See on the UI where the mouse has clicked:
  • See the UI of the application under test over time:
  • Read the content of the network and console tabs from dev tools as they were at any point in the test run:
  • See how long each step in the test takes:
  • See the source code of the test that is executed at each step:
  • If a test fails I can read an error message that includes the name of the file and line number where the test failed:
  • Get metadata about your test:
  • Find the built-in locator for an element in the UI, I could not find this feature in the documentation. To find the built-in-locator for an element:
    • click on the circles next to Locator
    • click on the UI element in the screenshot that you would like to know the built-in-locator for
    • then the built-in locator will be displayed in the Locator tab:

If a Playwright test fails, Trace Viewer can help you:

  • see a screenshot of where the test failed
  • investigate the UI of the application under test at the point it failed and use the browser’s dev tools to investigate the failure
  • see the lines of test code in the test where the failure occurred
  • because you can share the URL from CI of the Trace View 

Trace Viewer can help you develop new tests because

  •  run locally by adding an option to the command line
  • it can show you the built-in locator for an element 
  • if a test fails it shows you where in the test it failed

I have automated tests with several tools and never found a feature as helpful as Playwright Trace Viewer. I will also likely find more ways that Trace Viewer provides valuable feedback over time. If you are considering automating tests with Playwright I would recommend investigating Playwright’s Trace Viewer because it may be the factor which persuades you to use Playwright!

Sometimes you need to assert that Playwright tests are not running too fast!

Tests that are automated in Playwright run fast, which is great. However, sometimes they run so fast that assertions need to be used to stop the test being flaky.

Playwright uses Auto-waiting to help it run tests fast. Auto-waiting means that Playwright performs several checks before making an action, for example, it checks that an element is visible before clicking on it. I have found two situations in which steps in Playwright tests ran so quickly that they caused tests to be flaky. An assertion, that checks that an action has been completed, can be added to a test to prevent this from occurring. The assertion controls the flow of the test so that an action is completed before the next action is executed.

If you automate a test for a configuration with optional items, the button to save those choices is enabled while the user makes their optional configuration choices. After a configuration option has been checked the ‘save’ button will pass the checks that auto-waiting makes and Playwright can click the ‘save’ button before the chosen configuration option has been saved. The test flakes if the optional configuration option has not been saved. So that the test did not flake I added an assertion that checks that the optional element has been selected before the test clicks on save, and the test no longer flakes. An example of this would be:

const locator = page.getByLabel('Subscribe to newsletter');

await locator.click();

await expect(locator).toBeChecked();

await page.getByRole(‘button’{ name: /save/i });

I also found that when Playwright selects an option from a select element, the application may not have completed selecting the option before Playwright makes the next action. The next option, say a button, passes the test for auto-waiting and so Playwright clicks on it before the application completes selecting the option. If the correct option has not been selected the test flakes. To prevent the test flaking I added an assertion that checks that the correct option has been selected before the test makes the next action. An example of this would be: 

await page.getByLabel('Choose').selectOption('Test.xsls');

const locator = page.getByLabel('Spreadsheet');

await expect(locator).toHaveText('Test.xsls');

await page.getByRole(‘button’{ name: /continue/i });

Asserts are a good way to make a test wait for an action to complete because an assert waits for its condition to be true instead of waiting for a fixed time. 

Playwright uses auto-waiting to wait the minimum time before making an action. Sometimes it is necessary to use an assertion to make the test wait for an action to complete before it takes the next action.

Get insights from “The World of W. Edwards Deming” by Cecelia S. Kilian

Cecelia S. Kilian was W. Edwards Deming‘s long-term secretary. She created this book which contains her memories, and many curated documents from Deming. I read this book with the Profound Deming Book Club and it made me think about why I am interested in Deming. The details of Deming’s life are interesting, however I am interested in him because I want the same things he did. I want companies to be successful businesses by continually improving the quality of their products and services. 

The most interesting chapters in the book are Deming’s notes on what he taught Japanese management in 1950 at lectures which he was invited to give by the Japanese Union of Science and Engineering. The book includes notes on his lecture at Mt. Hakone. If I understand the history of the idea I understand the idea more deeply. I found it fascinating to read in the notes of lectures from nearly 75 years ago many ideas that are used today. These points are taken from the chapters about Deming’s 1950 lectures :

  • Building quality into a product: “You can not inspect quality into a product”. ”You must build in quality” [1]
  • The use of control charts to analyse processes, as is done in many industries today. [1]
  • User testing, which he describes as consumer research.” Consumer research is communication between the manufacturer and users and potential users” of the product or service”. [3]
  • The plan-do-study-act cycle. Today the plan-do-study-act cycle describes a software development team’s iterations[4] and “the Toyota Production System .. embodies the learning cycle of ” the plan-do-study-act cycle [5].
  • New product development with the plan-do-study-act cycle. To begin the development “of a product on a pilot scale and to build up its production on a sound economic basis, only as fast as market conditions indicate, re-designing the product from time to time in the light of consumer needs and reactions”. Today, we would probably describe that form of new product development as ‘lean startup’.[6]
  • Viewing product development as a system. He uses a flow diagram to describe production as a system, showing how  “improvement of quality envelopes the entire production line”[7]. The only key change I would make to the diagram to make it relevant to software development is to have data as input instead of raw materials.
  • A non-blame culture. We should not blame an individual for a fault because the fault is probably common to the system. Fixing these common causes of faults is an important way to improve quality.[8]
  • Quality attributes, which Deming called quality-characteristics. [9]

It is worth considering how engineering, including testing, would have been different without these lectures. Would the Toyota Production System have been the same? And if Toyota was not the same, would we have had lean and agile?

Another useful chapter in the book is an appreciation of Deming’s “friend, mentor and colleague” Walter A. Shewhart. [10]

The book also includes moving recollections of family life, childhood memories,  Deming’s sheet music, a description of being introduced to the Emperor of Japan, and accounts of his meetings with the Toyoda and Ishikawa families.

If you are interested in Deming as a person this book is for you because it enables you to get to know Deming as a human being. If you would like to learn from his notes on what he taught in Japan in 1950 you will find this book useful because it has some chapters which describe the lectures he gave in Japan.

References

[1] The World of W. Edwards Deming by Cecelia S. Kilian (1992, p70)

[2] The World of W. Edwards Deming by Cecelia S. Kilian (1992, p63)

[3] The World of W. Edwards Deming by Cecelia S. Kilian (1992, p67)

[4] Scrum: The Art of Doing Twice the Work in Half the Time by Jeff Sutherland (2015, p36)

[5] The Toyota Way by Jeffrey K. Liker (2004, p263)

[6] The World of W. Edwards Deming by Cecelia S. Kilian (1992, p69)

[7] The World of W. Edwards Deming by Cecelia S. Deming (1992, p24)

[8] The World of W. Edwards Deming by Cecelia S. Deming (1992, p80)

[9] The World of W. Edwards Deming by Cecelia S. Deming (1992, p62)

[10] The World of W. Edwards Deming by Cecelia S. Deming (1992, p87)

The Pesticide Paradox is a reason to do exploratory testing

Whichever technique is used to create tests they will contain assumptions about the nature of bugs. Each technique targets a different set of bugs. If development teams react to bug reports by fixing the bugs that have been reported and by finding and fixing similar bugs, then running the same tests is unlikely to find bugs.[1] 

This is “called the ‘pesticide paradox’ after the agricultural phenomenon, where bugs such as the boll weevil build up tolerance to pesticides, leaving you with the choice of ever-more powerful pesticides followed by ever-more powerful bugs or an altogether different approach”. [1]

The “Pesticide Paradox” is more relevant now than when Beizer wrote about it in 1995 because today unit testing is widely used and every bug fix should have unit tests to ensure that it does not reoccur. 

Automated tests are run over and over again so the ‘Pesticide Paradox’ does raise questions about the role of automated tests. There are lots of reasons to automate tests but in my experience, automated tests are unlikely to find many bugs. 

A reason to do exploratory testing is the ‘Pesticide Paradox’. We need to “learn, create and use new tests based on new techniques to catch”[1] bugs and “exploratory testing involves scouting around the areas”[2] that tests do not cover.

References

[1] Black-box testing by Boris Beizer (1995, p9)

[2] Explore It! By Elizabeth Hendrickson (2013, p5)

A review of ”The Checklist Manifesto How to Get Things Right” by Atul Gawande

As testers, we often use checklists, such as a Definition of Done, so I was drawn to this book when it was suggested at the Profound Deming Bookclub. 

Atul Gawande tells the story of how he was asked by the World Health Organisation(WHO) to develop a global programme to reduce avoidable deaths and harm from surgery and used a checklist to do so.  He used knowledge from other professions to create the checklist.

Ineptitude and ignorance are causes of failure. We can reduce failures caused by ineptitude by applying “the knowledge we have consistently and correctly”. [1]

As a result of a test flight for a Flying Fortress that resulted in an air crash, a checklist was created for pilots. The checklist enabled the plane to be flown safely.[2] A checklist reduced infections in Michigan’s hospitals [3]. The construction industry has checklists that specify communication tasks which make everyone work as part of a team that takes each other’s concerns into account. [4]

Checklists “are not comprehensive how-to guides”. “They are quick simple tools aimed to buttress the skills of expert professionals”[5]

The first checklist that Gawande created for the WHO was too long, and lessons were taken from the aviation industry to improve it. A new checklist for the WHO was tested in eight institutions around the world. The test showed that the checklist was successful. The team was concerned that this was due to the Hawthorne Effect, but they could prove that it was not. The checklist “aimed for a team conversation” [6] and it had improved communication. The improvement in communication was the key to the improved results.[7] The two-minute WHO checklist helped teams embrace a culture of teamwork and discipline[8]. The WHO checklist has saved and is saving, many lives. 

Some people may feel that checklists are “beneath us”. If that is how you feel you should remember that Captain Chesley  B. “Sully” Sullenberg used checklists to save lives when birds hit his plane over New York.[9]

The WHO checklist was created by drawing on the experience of professionals from more than one industry. Checklists can improve communication and help teamwork. This book can be used to help testers create checklists by learning from the experience of others, for instance by creating checklists that help to improve communication within their team.

References

[1] The Checklist Manifesto How to Get Things Right by Atul Gawande (2011, p10)

[2] The Checklist Manifesto How to Get Things Right by Atul Gawande (2011, p34)

[3] The Checklist Manifesto How to Get Things Right by Atul Gawande (2011, p44)

[4] The Checklist Manifesto How to Get Things Right by Atul Gawande (2011, p65)

[5] The Checklist Manifesto How to Get Things Right by Atul Gawande (2011, p128)

[6] The Checklist Manifesto How to Get Things Right by Atul Gawande (2011, p137)

[7] The Checklist Manifesto How to Get Things Right by Atul Gawande (2011, p156)

[8] The Checklist Manifesto How to Get Things Right by Atul Gawande (2011, p160)

[9] The Checklist Manifesto How to Get Things Right by Atul Gawande (2011, p173)

Drive out fear to improve psychological safety

We need psychologically safe workplaces.  Testers, and all other members of development teams, need to feel safe enough to be able to ask questions and express opinions about the project we are working on. 

The need for psychological safety is not a new issue. W. Edwards Deming saw it as an important issue. Drive out fear is one of Demings’ 14 Points for Management which he published in 1982[1]. He first spoke about his 14 Points for Management at a conference in Tokyo in 1978[2]. The early versions of his Points for Management, when he spoke in Japan, did not include Drive Out Fear because Japanese managers did not need to be told to Drive Out Fear. [3]

He wrote that “fear takes on many faces. A common denominator is loss from impaired performance and padded figures”[1]. No one can put in their best performance unless they feel secure[4]. Examples of fear include[5]:

  • I am afraid I will lose my job
  • I am afraid that I may not always have an answer when my boss asks something
  • I am afraid to admit a mistake
  • My boss believes in fear

Drive out fear is one of Deming’s Points for Management. If you are working in a climate of fear then it is not your fault, it is the responsibility of management. Instead of command and control management, “Deming directed managers to build trust throughout the organisation.” [6]

“Management in authority will struggle over every one of the” 14 Points for Management [7], including Drive Out Fear. Deming wrote that the Plan-Do-Study-Act Cycle will be helpful as a procedure to follow for improvement[8]. Management can use the Plan-Do-Study-Act cycle to drive out fear and improve psychological safety. They can plan an initiative to improve psychological safety, such as providing training on how to facilitate meetings. Then “do” the initiative, study its results and then act on what they learned from the study.

Innovation in the twenty-first century is being supported by Deming’s 14 Points for Management, including Drive Out Fear, for example, they are the roots of DevOps.[9]

Psychological safety is important for everyone at work. It has particular importance for testers because we need to be able to ask questions. Drive out fear is a good example of how Deming’s 14 Points for Management provides a framework to create a great place for testers, and everyone else, to work.

References

[1] Out of the Crisis by W. Edwards Deming (1982, p59)

[2] Quality or Else by Lloyd Dobyns and Clare Crawford-Mason(1991, p102)

[3] The Deming Dimension by Henry R. Neave (1990, p37)

[4] Out of the Crisis by W. Edwards Deming (1982, p58)

[5] Out of the Crisis by W. Edwards Deming (1982, p60)

[6] Out of the Crisis by W. Edwards Deming (1982, p88)

[7] Out of the Crisis by W. Edwards Deming (1982, p86)

[8] Deming’s Journey to Profound Knowledge by John “Botchagalupe” Willis with Derek Willis (2023, p200)

[9] Deming’s Journey to Profound Knowledge by John “Botchagalupe” Willis with Derek Willis (2023, p163)

Further Information:

A review of “Understanding Variation The Key to Managing Chaos” by Donald J. Wheeler

This book is both insightful and useful. It was recommended to me by members of the Deming Profound Book Club. Wheeler describes how to create control charts and analyse processes using control charts. This book can help you start to use control charts to analyse data from your test and development processes.

Walter Shewhart invented control charts when he was working for Bell Telephones. For the last one hundred years managers and teams have been using control charts. They are used in many industries, but only sometimes used in software development. Testers can use control charts to provide a useful way to analyse the results of performance tests. 

The use of goals and targets to analyse data is contrasted with the use of control charts. Goals and targets are described as “the Voice of the Customer” because it defines what you want. Control charts are described as the “Voice of the Process” because they define what you will get from a system. [1]

Control charts contain time series data, a central line as a visual reference, and upper and lower control limits. Wheeler describes how to calculate the control limits from the variation in the time series data. 

All variation is characterised by control charts as either predictable (within the control limits and so in statistical control) or unpredictable (outside the control limits and so out of statistical control). [3]

A major advantage of “the Control Chart Approach filters out noise by the construction of the control limits. Signals are indicated by points which fall outside the control limits or by obvious non-random patterns of variation around the central line” [4]

Wheeler uses control charts to tell stories about the economy and companies. Each story throws light on a different aspect of the valuable insights provided by using control charts. These stories include sections about how to analyse events that occur rarely and how a company’s performance is affected by an improvement plan.

I used this book to help me create control charts using Geckoboard Datasets AP, the product that I am testing. I have learned so much about the product I test and about how to use control charts by creating control charts. I would encourage you to use this book to start experimenting by creating control charts and so gain insights from your process.

I would also like to thank Rob Park for his help when I had questions about creating and analysing control charts.

References

[1] Understanding Variation The Key to Managing Chaos by Donald J. Wheeler (1993, p79)

[2] Understanding Variation The Key to Managing Chaos by Donald J. Wheeler (1993, p23)

[3] Understanding Variation The Key to Managing Chaos by Donald J. Wheeler (1993, p112)

[4] Understanding Variation The Key to Managing Chaos by Donald J. Wheeler (1993, p30)

Testing qualities not quality

To help me test I used to find it useful to think about what quality is for the application that I am testing.  “There has been a tendency to conceive of quality as indicating the goodness of an object.”[1] There are many aspects to quality and I have found that this idea of quality is ”too indefinite for practical purposes”[1].  I now think about the qualities of the application, not its quality.

Walter Shewhart worked to improve the quality of telephones and wrote that “every conceptual “something” is really a group of conceptions in more elementary form. The minimum number of conceptions required to define an object may be called the qualities thereof”. “In this sense a thing has qualities not a quality”[2]. Shewhart’s approach to quality is useful because it prompts us to ask what are the qualities of the software that we are testing. 

Shewhart quoted Jevons[1] who wrote that “two fragments of rock may differ entirely in outward form, yet they may have the same colour, hardness, and texture. Flowers which agree in colour may differ in odour. The mind learns to regard each object as an aggregate of qualities, and acquires the power of dwelling at will upon one or other of those qualities to the exclusion of the rest”.[3]

When we are testing we need to find the qualities that differentiate things that appear to be the same. Testing can be, as Jevons suggested, like observing two fragments of rock that have similarities. One fragment could be our understanding of how the product should be and the other fragment could be the product that we are testing. The two fragments of rock could also be two versions of the application that have the same backend but different UIs. We need to find the differences between the two and discovering the qualities of each helps to do that. Performance, testability or usability could all be qualities of the product that we are testing.

The practice of analysing the qualities of a product led to the use of quality attributes for testing. 

Defining the qualities of the application under test will also help with defining the quality criteria used when defining quality coverage, as in Melissa Fisher’s recent blog post: Test coverage — how about quality coverage?

Asking questions about qualities helps me break down the functionality into smaller batches for testing. I can test one quality at a time, for example, I can test accessibility.  This approach can also help me gain insight into one or more of the qualities and give me new paths for testing through the application.

References

[1] Economic control of quality of manufactured product by Walter Shewhart (1931, p55)

[2] Economic control of quality of manufactured product by Walter Shewhart (1931, p56)

[3] The principles of science : a treatise on logic and scientific method by William Stanley Jevons (1913, p24 )

Developing your listening skills is really useful

We all know that speaking up and getting your point over at a meeting is important. However, I am sure that we have all taken part in meetings when everyone is so keen to speak that we do not listen to one another. 

It is not only important to speak, it is also important to listen. If people speak but do not listen then there is no conversation. If people do not listen to each other we can’t learn from each other and they cannot cooperate to achieve a goal. 

If people are talking and are not listening some people will find it difficult to get to speak because everyone else is busy speaking. Listening to each other will help improve inclusion because it will help everyone to speak at meetings. 

Sometimes it is hard for us to listen because, instead of listening, we are waiting to speak. I use a technique to help me listen. I have learned to ask myself what is going to be the last thing that someone is going to say. This technique helps me to listen to every word someone is saying and not to interrupt the person speaking. 

Asking myself what will be the last thing that someone says on a subject during a discussion also helps me when I speak because I can take into account what others have said. 

Listening is an important skill for testers. We need to build a rapport with developers and listening to them will help. We also need to hear every point of view on a subject because we may hear something that will help us test. We need to listen to stakeholders to know what they expect and hope for. We also need the team to work cooperatively because teams that work cooperatively achieve more.

This podcast from Amy Gallo via Harvard Business Review is a great resource with useful tips for improving your listening skills: Practice Your Active Listening Skills.

Design a site like this with WordPress.com
Get started