TIL: more about using zero in testing

Three testers walk into a bar…and we all know that they order, one beer, zero beer and minus one beer! We also all know that they need to order zero beers because the behaviour of zero can be different from the behaviour of other numbers.  We all test with zeros for this reason. Recently I have learned something more about the behaviour of zero that is helping me to test..

I am doing Typescript tutorials to help me automate tests and recently learned more about behaviour with zeros which helped me to understand the cause of a bug. I have recently learned about conditional narrowing, which can be used to check the type of a value in TypeScript. If the && operator or the || operator are used for truthiness narrowing then zero is coerced to false. The && and || operators treat zero in the same way as are NaN, “”(the empty string), 0n (the bigint version of zero), null and undefined. This means that truthiness narrowing does not treat zero as a number, it treats zero as false

A friend of mine used this knowledge and found a bug in which the user could enter zero but was not able to save the zero that they had entered. The application could not save zero because its logic said that zero was false so was not a number.

Zero is a curious thing and I am using what I have just learned about to help me test. 

Additional Learning Resources:

Test using Quality Characteristics\Factors\Attributes that you create.

It can be useful to define aspects of a product that describe the quality of the application that you are testing. I have found examples of doing this in seven different decades. Sometimes these aspects have been put into sets, lists or groups and have sometimes been called quality characteristics, sometimes quality factors and at other times they have been called quality attributes. These sets, lists and groups have also been used in different ways to improve quality.

Walter Shewhart wrote in 1931 about the “conception of the Quality of a Thing as a set of characteristics[1]”, and in 1939 wrote that the “aim is to produce a sequence of objects having a specified quality characteristic within some specified limits”[2]. The quality characteristics of the objects would be controlled by the use of control charts. 

In 1950 in Japan in his lecture at Mt Hakone, W. Edwards Deming spoke about “communication between buyer and the seller of the quality-characteristics of a product is provided by the same statistical techniques that are used for improved testing, production and inspection”[3]. The statistical technique he was suggesting is that of the control chart. The quality characteristics he spoke about would have been for physical products. 

The Western Electric Statistical Quality Control Handbook was first published in 1956 and describes the Statistical Quality Control process developed by Shewhart that used control charts. The Handbook says that  “the word “Quality” means much more than the goodness or badness of a product. It refers to the qualities or characteristics of the thing or process being studied”[4].

A report for the USAF in 1977 listed quality factors under headings in groups[5], for example under the heading testability it lists simplicity, modularity, instrumentation and self-descriptiveness. The report also recommends which factors should be measured[6], and how to measure them.[6]

Attributes to specify quality were defined in 1988 as “a quality concept or resource concept which describes a system quantitatively” [7]. These attributes are to be “specified and controlled throughout the project”[7]. The attributes could be expressed as a hierarchy, for example:

  • System performance
    • Weekly work-load capacity
    • System availability
    • Repose time to questions[8]

In 1998 the list of quality factors from the 1977 report is used as a list of quality attributes. “These attributes include both subjective and objective measures and are only a subset of the total attributes of software programs”, and “the applicable elements should be selected by the quality professional for application to each ..program”[7].

A list of eighteen quality attributes was listed in 2002 under Testing Techniques, with the advice “to determine whether a feature is defective, ask yourself how you would prove that it lacks or violates one of these attributes”[10]

Seven product dimensions, which included a quality attribute dimension, were taken from business analysis to help describe quality in 2015. It is “helpful to consider all seven dimensions during release, feature and iteration planning”[11]

Quality professionals have been using quality characteristics/factors/attributes of a product to describe the quality of products and services for a long time. I have not given an example of their use from the 2020s, it is up to us to find how best to use quality characteristics/factors/attributes to help us with the testing challenges we face today!

Thank you to Dennis Sergent for his help with this blog post. 

References

[1] Economic Control of Quality of Manufactured Product by Walter Shewhart (1931, p55)

[2] Statistical Method from the Viewpoint of Quality Control by Walter Shewhart with a foreword by W. Edwards Deming (1986 – originally published in 1939, p7)

[3] The World of W. Edwards Deming by Cecelia S. Kilian (1992, p62)

[4] Western Electric Statistical Process Control Handbook (1982, p2)

[5] FACTORS IN SOFTWARE QUALITY Concept and Definitions of Software Quality By Jim A. McCall, Paul K. Richards CD Cene, F. Walters (1977, p44)

[5] FACTORS IN SOFTWARE QUALITY Concept and Definitions of Software Quality By Jim A. McCall, Paul K. Richards CD Cene, F. Walters (1977, p38)

[6] FACTORS IN SOFTWARE QUALITY Concept and Definitions of Software Quality By Jim A. McCall, Paul K. Richards CD Cene, F. Walters (1977, p62)

[7] Principles of Software Engineering Management by Tom Gilb (1988, p134)

[8] Principles of Software Engineering Management by Tom Gilb (1988, p136)

[9] Computer Applications to Quality Systems by Frederic I. Orkin and Danliel Olivier in Jurans Quality Handbook 5th Edition edited by J.M. Juran and A. Blanton Godfrey (1998, p10.7)

[10] Lessons Learned in Software Testing by Cem Kaner, James Bach, Bret Pettichord (2002, p60)

[11] More Agile Testing by Janet Gregory and Lisa Crispin (2015, p130)

How long should an automated test pack take to run? And what should we do about it?

How long should an automated test pack take to run?

Some people say that a certain number of minutes is too long, and others say that a different number of minutes is too long. How can we determine what is “too long”? And what can we do about it?

What is an acceptable length of time for a test pack to run depends on several factors. Your team’s cycle time will affect how long it is acceptable for a test pack to run. Cycle time can be defined as a development process “ that transforms identified customer needs into delivered customer value at a reliable, repeatable cadence”[1] If your team is practising continuous deployment or continuous delivery and has a cycle time of a few hours there will be pressure for automated test packs to run more quickly than if your team makes monthly or quarterly releases. Teams that practice continuous deployment or continuous delivery work in small batches so the test packs will need to be small batches too.

Another factor that affects how long it is acceptable for a test pack to run is whether the length of time the tests take to run is a constraint on development. In some situations, a test pack of thirty tests that runs in fifteen minutes may be an enabler and in other situations, it may be a constraint. An example of this would be; if the test pack that you are concerned about takes no longer to run than other test and deployment processes then it is unlikely to be a constraint, however, if it takes much longer than other test or deployment processes then it may be a constraint on how developers work.

What can the test engineer responsible for the test pack do about the length of time it takes to run the test pack? 

The engineer who is responsible for the test pack should not be defensive about the test pack, instead, they should lead the discussion about how long it takes the test packs to run. They can, for instance, go to developer meetings and create a dialogue with developers by talking about how they are developing the test pack. This dialogue should include how long it takes for the pack to run. If the test automation engineer is involved in a dialogue they will find out how the test packs are affecting developers. They can also tell developers what they are doing to make the tests run faster and discuss with developers ideas as to how to improve the running of the test packs. Testers and developers should then use this dialogue to start to cooperate on improving test packs, including their running times.

What is an acceptable time for a test pack to run depends on several factors. We need to understand our development system and cooperate with our development team to improve test packs. 

References

[1] Implementing Lean Software Development: From Concept to Cash by Mary and Tom Poppendiek (2006,p98)

A theory of management for improvement of quality vs a quality improvement plan, which helps us more?

What can we learn from comparing Deming’s 14 Points for Management and Crosby’s 14-Point Quality Improvement Plan? Which will help us more to improve quality?

W. Edwards Deming first presented his 14 Points at a conference in 1978 in Tokyo[1] and published his 14 Points for Management in 1982[2]. Philip B. Crosby published his 14-point Quality Improvement Plan in 1979[3]. I have not found anything to suggest that their both having 14 points/ steps is anything but a coincidence.

Deming’s 14 Points for Management [2]Crosby’s 14-Step Quality Improvement Plan [3]
1. To create constancy of purpose towards improvement of product and service, with the aim to become competitive and stay in business and to provide jobs
2 .Adopt the new philosophy. We are in a new economic age. Western management must awaken to the challenge, must learn their responsibilities, and take on leadership for change. 
3. Cease dependence on inspection to achieve quality.
4. End the practice of awarding business on the basis of price tag. Instead, minimise total cost. Move toward a single supplier for any one item., on a long-term relationship of loyalty and trust.
5.  Improve constantly and forever the system of production and service, to improve quality and productivity, and thus constantly decrease costs.
6. Institute training on the job.
7. Institute leadership. The aim of supervision should be to help people … do a better job. 
8. Drive our fear, so that everyone may work effectively for the company.
9. Break down barriers between departments. People in research, design, sales and production must work as a team.
10. Eliminate slogans, exhortations and targets for the workforce..such exhortations only create adversarial relationships.
11. Eliminate management by numbers..substitute leadership.
12. Remove barriers that prevent people from taking pride in their work
13. Institute a vigorous programme of education and self-improvement.
14. Put everyone in the company to work to accomplish the transformation. The transformation is everybody’s work.		1. Management Commitment –  to make it clear where management stands on quality
2. The Quality Improvement Team –  to run the quality improvement programme
3. Quality Measurement – to provide a display of current and potential nonconformance problems in a manner that permits objective evaluation and corrective action
4. The Cost of Quality – to define the ingredients of the cost of quality and explain its use as a management tool
5. Quality Awareness – to provide a method of raising the personal concern felt by all personnel in the company toward the conformance of the product or service and the quality reputation of the company
6. Corrective Action  – to provide a systematic method of resolving forever the problems that are identified through previous action steps
7. Zero Defects Planning – to examine the previous activities that must be conducted in preparation for formally launching the Zero Defects programme.
8. Supervisor Traning – to define the type of training that supervisors need to actively carry out their part of the quality improvement programme
9. Zero Defects Day – to create an event that will let all employees realise through a personal experience that there has been a change.
10. Goal Setting – to turn pledges and commitments into action by encouraging individuals to establish improvement goals for themselves and their groups.
11. Error Cause Removal – to give the individual employee a method of communicating to management the situations that make it difficult for the employee to meet the pledge to improve.
12. Recognition – to appreciate those who participate
13. Quality Councils – to bring together professional quality people for planned communication on a regular basis.
14. Do it over again – to emphasise that the quality improvement program never ends

What do Deming’s points and Crosby’s steps have in common? 

  • both aim to help businesses improve the quality of their product or service
  • both were part of the Quality Movement
  • both aim to prevent errors
  • both view quality as management’s responsibility

What are the key differences between Deming’s points and Crosby’s steps?

  •  Deming described his 14 Points as a “theory of management for improvement of quality, productivity, and competitive position”[4]. In contrast, Crosby’s steps are a quality improvement program comprising 14 steps.
  • Deming’s 14 points are a coherent philosophy, each point of the 14 points is implicit in all the other points. An example would be training (point 6) will be ineffective unless barriers that prevent people from taking pride in their work are removed (point 12). Crosby’s plan is a series of steps rather than points that are implicit in each other.
  • The role of management is different:
    •  Deming’s points include instituting leadership so that the role of a manager ”is a coach and counsel, not a judge”[5]. In contrast, Crosby’s plan has a more traditional hierarchical view of management’s role in which it is management who runs the Quality Improvement Programme. 
    • The manager’s role in Deming’s Points includes driving out fear, this means that managers take responsibility for psychological safety. Staff working in a company using Crosby’s plan would probably need to use the Error Cause Removal step to raise issues relating to psychological safety.
  • Deming’s points include developing people’s potential through education and training for all staff, whereas Crosby’s plan only requires training for supervisors.
  • In Deming’s points the whole company, including its suppliers, is seen as a system in which everyone should cooperate, and the manager’s role is to help everyone work effectively. Crosby’s plan puts responsibility on individual members of staff, for example, on Zero Defects Day individual staff are asked to sign pledges to improve quality.
  • Deming’s Points include cooperating across company departments and between the company and its suppliers to improve quality. Crosby’s steps do not include cooperation.
  • Crosby‘s plan sets goals and looks at costs, whereas Deming’s points ask for constant improvement.
  • “Building quality in” instead of “inspecting quality in” is covered by Deming’s point 3. Crosby has a step on Corrective Action to resolve problems.
  • Crosby aims for Zero Defects, whereas Deming aims for continual improvement.
  • Deming’s 14 Points for Management are the roots of DevOps[6] and so are part of innovation in the twenty-first century. I am not aware of any twenty-first century innovation based on Crosby’s 14 Steps.

We should note that Deming also used his System of Profound Knowledge, and the plan-do-study-act cycle to help businesses. Crosby also used a Quality Management Maturity Grid.

Deming’s ideas are powerful “on cooperation and human potential”[6]. We need cooperation and the realisation of human potential to improve quality and I wouldn’t want to work anywhere that was very different to Deming’s 14 Points for Management.

References

[1] Quality or Else by Lloyd Dobyns and Clare Crawford-Mason (1991, p102)

[2] Out of the Crisis by W. Edwards Deming (1982, p23)

[3] Quality is Free by Philip P. Crosby (1979, p137)

[4] Out of the Crisis by W. Edwards Deming (1982, p19)

[5] The New Economics by W. Edwards Deming (1994, p126)

[6] Deming’s Journey to Profound Knowledge by John “Botchagalupe” Willis with Derek Willis (2023, p163)

Additional Resources

  • Deming NEXT – eLearning from the Deming Institute with a free two-week trial
  • Deming’s Journey to Profound Knowledge by John “Botchagalupe” Willis with Derek Willis  –  Chapter 19 gives contemporary true-to-life examples for each of Deming’s 14 Points. 

Gaining a knowledge of React is helping me automate tests

React is an open-source User Interface (UI) JavaScript library created by Facebook. I am doing some React tutorials to help me better understand the UI and what I am learning is helping me to automate tests.

The React tutorials I am doing are here: https://react.dev/learn. The tutorials include exercises that can be done in a code sandbox, so you do not need your own development environment to do the exercises. I have gained a better understanding of the course content by doing the exercises. I have done some exercises in my dev environment and some in the code sandbox.

I have learned that React apps are made out of components. I am now learning about React components and understand that if there is a bug in a component the bug may be present wherever the component is used. I have also learned about CSS classes, and how they can be used to create a similar appearance throughout the app. 

I now better understand the developers when they talk about “props” and “lifting state up”.  This helps me contribute to discussions at planning meetings.

The React tutorials have given me a better understanding of the UI that I am automating tests for. When I have had a problem automating an action I have been able to look in the repo and work out how to interact with the React component that I am trying to automate. On one occasion the DOM element was nested and I found that by reading the code I could see which element handled an onClick event, and so could automate the click that I wanted.

Doing the React tutorials is making me more of a ‘T’ shaped tester. I have deep testing skills but I also now have a growing understanding of React, and this knowledge of the UI is helping me automate tests.

Writing test automation standards is a journey, not a destination

I have started to automate tests in Playwright with TypeScript.  I am now writing standards for the tests. I view writing the standards as a journey because I am adding to the standards as I learn more.

 I have started to create these standards because:

  • It helps me get my thoughts in order
  • It helps me learn from my mistakes because I won’t repeat errors.
  • I can always add to the standards, such as when I started to use describe blocks and needed a naming convention for them.
  • The standards help to share knowledge.
  • Standards facilitate collaborative working because they enable test automation engineers to work in a consistent way
  • I want to be able to maintain the tests easily

I have found lots of useful advice in testing and programming literature:

  • “Any organization that wants to move fast will have standards in place” [1].
  • Standards can be grouped under headings such as naming conventions, coding standards, user interaction conventions, file structures, configuration management practices and tools [1].
  • Standardise is one of the Five S’s and standards help us “reduce complexity over time to improve ease of maintenance” [2] when developing software.
  • It is best to adopt naming conventions at the start of developing the test suite in order to avoid a free-for-all which will make it difficult to find scripts and files as the test suites get larger.[3]
  • “All standards should be simple to follow and make maintainability easier”[4].
  • Naming conventions improve the readability and maintainability of your Playwright Testing framework [5]

I have also learned from colleagues in a book club that it is useful to say why the standard is needed because at a later date, its need may not be obvious and it could be removed. 

Writing standards for test automation is a journey not a destination because I am learning and putting that learning into the standards. 

If you have more sources of information about writing test automation standards please let me know. 

References

[1] Implementing Lean Software Development by Mary and Tom Poppendieck (2007, p193)

[2] Implementing Lean Software Development by Mary and Tom Poppendieck (2007, p192)

[3] Software Test Automation by Mark Fewster and Dorothy Graham (1999, p197)

[4] Agile Testing by Lisa Crispin and Janet Gregory (2009, p227)

[5] Using naming conventions to improve the readability and maintenability of your Playwright Testing framework by Boyana Staneva (2023)

Why I don’t record automated tests

Recently it was pointed out to me that Playwright has a test generator and that I could use it to record tests. Many automated testing tools include a tool or extension that can be used to record tests. I don’t find them useful. 

My experience of using tools to record tests is that they record steps that work if they are played back immediately, but only if they are played back immediately. The recorded steps will be flaky. Every test automation tool I have used has needed additional steps to control the test flow. The recorded steps tend, to either need additional steps to wait for an element to be present or additional steps to wait for an action to finish. 

Test automation code should also be modularised, and recorded steps are not modularised.

The test automation engineer needs to create libraries of modules or functions, possibly in a Page Object Model, so that the code can be easily maintained and tests developed quickly. Steps that have been recorded need to be refactored into modules or functions. It is quicker to write the module or function than to record steps and refactor them.

If you record the steps you don’t learn how to develop tests yourself. Recently I was developing a test with Playwright and could not get the test to do what I wanted with built-in locators. I tried using CSS selectors instead of the built-in locators and could not get them to do what I wanted. So, I read the code in the front-end repo and worked out which element of the DOM I needed to interact with and which action I should use. While working out how to interact with the DOM I learned a lot about the application I am testing and about Playwright. If I had used the recorder all I would have learned would have been how to use the recorder.

The main reason to not use a recorder is that you want to develop your test automation skills. Instead of using a recorder I used the learning resources to learn how to automate tests. I now have more skills and am continuing to learn about test automation. 

I started test automation by using free resources like these:

There are many other great resources too.

A review of “Black-Box Testing” by Boris Beizer

In “Black-Box Testing” Boris Beizer describes a number of useful techniques that can be used for black-box testing. He defines black-box testing as “the testing you do when you put yourself in the software user’s place to verify that it behaves the way it’s supposed to”.[1] Each testing technique is described in a chapter which has the same structure: synopsis, vocabulary, the relations and the model, the technique, application considerations, summary and self-evaluation quiz. IRS forms are used to create examples of each technique.

The first chapter gives an introduction to graphs and relations, subsequent chapters are on testing techniques.

Control-flow graphs are “introduced as the basic model for test design”[2]. Beizer takes us through how to model the functionality being tested as a control-flow graph, select the test paths, and sensitise a graph with data.

Bugs often accompany loops [3] and loops can be tested if the loops are depicted in a graph. The critical test values that are used to control the loop can be described in a graph. Deterministic loops, which for example may handle a paycheck for one to 20,00 employees[4] can be tested as can non-deterministic loops, which for example may search for a file of unknown length[5].

Data flow can be tested by data flow graphs. There are similarities between control flow graphs and data flow graphs because ordinary processing is similar.[6]

“The transaction flow graphs can used in system testing of on-line applications and batch-processing software” [7] The technique pays attention to queues and loops. 

Domain testing “replaces the common heuristic method of testing extreme values and limit values of inputs” [8]. This chapter provides a way of testing where a domain boundary is missing an upper or lower value. It can be useful to read this chapter in conjunction with the chapter in The Art of Software Testing that describes boundary value analysis because they both discuss how to test the same types of values.

Syntax testing is used for “testing command-driven software”[9].

The “finite-state machine model is an excellent model for testing menu-driven applications”[10]. A finite-state machine consists of state, transitions, inputs and outputs and is depicted by state graphs.[11]

Over time tests become less effective at finding bugs. This is described as the “Pesticide paradox”, and testers have to learn to create new tests. [12] 

Black-box testing is a very useful book. I have a well-thumbed copy of the book, which I use as a reference when I need to use one of the techniques Beizer describes.

References

[1] Black-Box Testing by Boris Beizer (1995, pxi)

[2] Black-Box Testing by Boris Beizer (1995, p38)

[3] Black-Box Testing by Boris Beizer (1995, p70)

[4] Black-Box Testing by Boris Beizer (1995, p38)

[5] Black-Box Testing by Boris Beizer (1995, p70)

[6] Black-Box Testing by Boris Beizer (1995, p91)

[7] Black-Box Testing by Boris Beizer (1995, p122)

[8] Black-Box Testing by Boris Beizer (1995, p146)

[9] Black-Box Testing by Boris Beizer (1995, p178) 

[10] Black-Box Testing by Boris Beizer (1995, p204) 

[11] Black-Box Testing by Boris Beizer (1995, p205) 

[12] Black-Box Testing by Boris Beizer (1995, p9)

Are you testing output and outcome?

There is a discussion in the DORA Community about “the problem with DORA metrics is that they focus on maximizing outputs rather than outcomes”.  Some people have argued that Dora metrics will reward teams with a high number of commits, which would be considered ‘output’ but not reward teams whose commits benefit customers, which would be considered ‘outcome’.

This discussion could also apply to testing. 

We need to test the output. This could be described as testing that we are ‘building the thing right’. Testing should include static analysis, unit testing and regression testing as these things all test that we have ‘built the thing right’. However, does testing sometimes concentrate too much on testing output?

We also need to test the outcome. We need to test that the results of the software change will create a positive outcome for the customers, that is that we are ‘building the right thing’. Testing should also include consideration of the outcome of the software change. 

To have confidence that the outcome of a software change will be successful we need to test the requirements. We need to test the requirements of the work so that any misunderstandings in the requirements are found before the new software is released.

Many bugs are caused by ‘breakdowns in information handling’ [1]. These errors occur when information about the required software change is communicated to the development team and within the development team. If we use only the teams’ understanding of the required change we may only be testing the output because we are only testing that ’the thing has been built right’ according to the specification. 

Another way to test that we are ‘building the right thing’ and testing for a positive outcome is to test from the customer’s point of view. I try to understand the customer’s point of view.  I attend user testing regularly so that I see how customers use the product. I also use the Theory of Jobs to be Done to help me understand customers’ use of our product.

Does testing sometimes concentrate on testing output?

References

[1] The Art of Software Testing by Glenford J. Myers (1979, p103)

A review of “Deming’s Journey to Profound Knowledge” by John “Botchagalupe” Willis with Derek Lewis

W. Edwards Deming is often referred to as the “grandfather of quality”, and this new well-researched book about him and his philosophy gives us many new and useful insights. 

The book’s author, John Willis, is one of the people who created DevOps, and he says that to understand the roots of DevOps you need to read Deming’s 14 Points for Management [1]. In this book, John Willis describes how Deming’s ideas developed and how they provide the lens that helps so many successful businesses today.

Deming’s environment helped to shape his philosophy. Deming was an intern at Bell Telephone’s Hawthorne Works during the ‘Hawthorne experiments’. This ultimately led to one of the four elements of his System of Profound Knowledge: psychology [2]. Walter Shewhart also worked at Hawthorne Works but Deming only got to know him well when their paths crossed later. Shewhart gave us Statistical Process Control to improve the manufacturing process.  He also gave us, what is often known as, the Deming Cycle. This cycle provides managers with a ‘tool to help them track bad quality, make a change and see if it improves product quality’[3]. Deming wrote in 1984 that it may take us another fifty years to appreciate what Shewhart gifted us[4].

When Deming helped to rebuild the Japanese economy after World War Two “he found a culture of intrinsically motivated employees. This culture was something he’d vaguely glimpsed at Hawthorne Works”[5]. The book contains fascinating insights into how Japan developed some of the key ideas and practices that became “lean manufacturing”. The Toyota Way inspired “lean manufacturing”, which is a product of Deming’s ideas and Japanese business culture practices[5].

‘Demingmania’ occurred after Clare Crawford-Mason made the documentary ‘If Japan Can…Why Can’t We?’ in 1980[6]. Deming went on to help the Ford Motor Company become the most profitable car maker in America[7]. 

The book contains many examples of the use of Deming’s philosophy, which include:

  • Deming helped John Waraniak, who used the Theory of Knowledge to create Deming Cycles, at General Motors’ Motorsports division to win IndyCar and NASCAR[8]. Waraniak went on to use systems thinking when developing a prototype self-driving car[9] and is now working with NATO to transfer car racing technology to military vehicles to support Ukraine[10].
  • Josh Corman uses Deming’s philosophy to improve cyber security, including in the process of creating and distributing COVID-19 vaccines [11].
  • Dr Doris Quinn used Deming’s philosophy to improve the quality of healthcare [12].
  • The chapter “Out of the Cyber Crisis” contains valuable examples of the use of each of Deming’s 14 Points for Management.

“We live in the age of the digital Cambrian explosion. Thanks, in no small part, to William Edwards Deming” [13]. We are working as testing professionals in the middle of this explosion.  I recommend reading this book to understand Deming’s philosophy, how it contributed to this digital explosion, and examples of how to use the philosophy today.

References

[1] Deming’s Journey to Profound Knowledge by John Willis with Derek Lewis (2023, p163)

[2] Deming’s Journey to Profound Knowledge by John Willis with Derek Lewis (2023, p15)

[3] Deming’s Journey to Profound Knowledge by John Willis with Derek Lewis (2023, p43)

[4] Deming’s Journey to Profound Knowledge by John Willis with Derek Lewis (2023, p193)

[5] Deming’s Journey to Profound Knowledge by John Willis with Derek Lewis (2023, p115)

[6] Deming’s Journey to Profound Knowledge by John Willis with Derek Lewis (2023, p129)

[7] Deming’s Journey to Profound Knowledge by John Willis with Derek Lewis (2023, p133)

[8] Deming’s Journey to Profound Knowledge by John Willis with Derek Lewis (2023, p136)

[9] Deming’s Journey to Profound Knowledge by John Willis with Derek Lewis (2023, p137)

[10] Deming’s Journey to Profound Knowledge by John Willis with Derek Lewis (2023, p138)

[11] Deming’s Journey to Profound Knowledge by John Willis with Derek Lewis (2023, p187)

[12] Deming’s Journey to Profound Knowledge by John Willis with Derek Lewis (2023, p139)

[12] Deming’s Journey to Profound Knowledge by John Willis with Derek Lewis (2023, p153)

Additional links:

Automated tests do not have to be flaky.

Example Deming Cycles for Playwright and Ghost Inspector

When I create or maintain an automated test I use a “learning loop” so that I learn from the test results. This enables me to create stable test packs that return accurate results. 

I automate tests with Playwright using TypeScript and with the low code tool Ghost Inspector. The “learning loop” is the same for both tools.

When I automate a test I use a plan-do-study-act learning loop:

  • I plan the test. This includes planning what functions and modules I will need, and how I will assert that the test has passed.
  • I then “do” the test automation. I automate the test using my plan.
  • I then study the test. I run the test and study the test results. I run the test several times, both on my workstation and in CI. I study the results to see if the test fails. If the test fails I study the test and the test result to work out why it failed. 
  • I then act and take the findings of my study into a new plan-do-study-act learning loop and plan the fix for the failure. 

If a test that is running in CI “flakes”, I use a plan-do-study-act learning loop to fix the test:

  • I plan the fix. I consider why has the test flaked and plan how to make it stable
  • Then I “do” the fix. I make the change to the test that I had planned
  • I then study the test. I the test both locally and in CI several times to see if the test passes consistently. I study the results of the test runs to see if the test is now stable and returning accurate results.
  • I then “act”. If the test still flakes I will go through the loop again. If the test now returns accurate results I will merge the fixed test so that it runs in CI.

Each time I go through the “learning loop” I learn about the application I am testing and the tool I am using to automate tests. I refer to the loop as a learning loop because I have learned so much by using it. A diagram of examples of this loop for both Playwright and Ghost Inspector is shown above.

This loop was created by Walter Shewhart to help improve quality in the manufacture of telephones and was popularised by W. Edwards Deming. It is usually known as the Deming Cycle and is widely used, for example in developing self-driving cars [1]. 

I am responsible for packs of hundreds of tests. If you use a “learning loop” when you create and maintain tests your tests will be stable and give accurate results. Automated tests do not have to be flaky.

References:

[1] Deming’s Journey to Profound Knowledge by John Willis and Derek Lewis (2023, p137)

Learning resources:

How a lone tester can work across teams

I am a lone tester and I work with two development teams. It is not possible to give both teams one hundred per cent of my time. If I don’t have a structure as to how to support both teams I can end up being pulled in multiple directions.

The structure I use is that I am a ‘chicken’ with one team and I am a ‘pig’ with the other team. If you are making a traditional British breakfast a chicken is helpful because it provides eggs, whereas the pig is committed because it provides bacon and sausages.

I am a ’chicken’ with one team. I am helpful. I go to stand up so I know what the team is working on. I offer advice and help, such as automating a test, if time allows.

I am a ‘pig’ with the other team. I am committed. I go to stand-up, planning, retros and any other meetings that the team has. I am involved in the team and support their work as a tester.

The metaphor of a ‘pig and a ‘chicken’ work, in that both teams know to what level I am available. I know also what work both teams are doing, can support them either as a ‘chicken’ or as a ‘pig’ and am able to cope with the workload from both teams. It is also a fun way of describing my relationship with the two teams.

Thank you to Rob Falla for the idea.

Some “new” practices are actually quite “old”

The Aurora 7 capsule

Recently I was fortunate to visit the Museum of Science and Industry in Chicago. The Henry Crown Space Center in the museum has the Aurora 7 capsule from NASA’s Project Mercury. It was piloted by Scott Carpenter and orbited the Earth in 1962. The capsule is only two meters long. It was a surprise to me that the capsule was so small. Please see my photo of it above. 

What is also surprising is that the software engineers on NASA’s Project Mercury used a number of techniques that we would associate with “agile”. They used iterative and incremental development[1], and “Project Mercury ran with very short (half-day) iterations that were time boxed. The development team conducted a technical review of all changes, and, interestingly, applied the Extreme Programming practice of test-first development, planning and writing tests before each micro-increment.”[1]

NASA’s Project Mercury ran from 1958 to 1963[2] and Jerry Weinberg was one of the software engineers on the project.[3] 

Some of the techniques of developing software that we can think of as being “new” are actually quite old.  

References

[1] Iterative and Incremental Development: A Brief History by Craig Larman and Victor R. Basili (2003)

[2] About Project Mercury NASA

[3] Agile Impressions by Gerald M. Weinberg (2017, p67)

Further Reading:

The Seedbed of a Quality Revolution

The sign for the retail park on the site of the Hawthorne Works

A tester’s role is not only to do the testing but also to improve quality. I visited the site of the former Hawthorne Works during a recent trip to Chicago because so many innovations in quality started there.  The Hawthorne Works was the Bell Telephone Laboratories site that manufactured the hardware for the first national telephone system in the USA.[1] It was the “Silicon Valley of ”[2] the early twentieth century and was the “seedbed of a quality revolution.”[3].

The site of the Hawthorne Works is now a retail park; the only part of the works that still stands is the water tower. I have used photos I took while visiting the site to illustrate this blog post.

Hawthorne Works water tower stands over retail units

Significant people who worked at Hawthorne Works and created philosophies and practices that have become central to quality are:

Deming referred to Shewhart as “the master”[1]. 

Deming’s philosophy is of great value as it helps me to work with development teams to improve the quality of work continually. I am enjoying reading John Willis’ new book “Deming’s Journey to Profound Knowledge, which shows how Deming’s life experiences, including those at the Hawthorne Works, influenced his philosophy.

That so many of the advances in quality made in the twentieth and twenty-first centuries are connected to the Hawthorne Works shows what a remarkable place it was.

References

[1] Architect of Quality : The Autobiography of Dr. Joseph M. Juran by Joseph M. Juran (2004, p91)

[2] Deming’s Journey to Profound Knowledge by John Willis and Derek Lewis (2023,  p46)

[3].Manufacturing the future. A History of Western Electric. by S.B. Adams and O.R. Butler (1999, p161)

[4] What is Total Quality Control? By Kaoru Ishikawa (1985, p14)

[5] Out of the Crisis by W. Edwards Deming (1986, p88)

[6] Quality or Else by Lloyd Dobyns and Clare Crawford-Mason (1991, p52)

[7] Deming’s Journey to Profound Knowledge by John Willis and Derek Lewis (2023, p79)

[8] “Deming to Devops: The Science Behind Devops” by John Willis (a five minute long video) 

[9] When Coffee and Kale Compete by Alan Klement (2018, p156)

[10] Out of the Crisis by W. Edwards Deming (1986, p168)

Further Reading

eLearning:

Hawthorne Works water tower is behind the retail units

Make test automation easier with better feedback for parameters with TypeScript

We can make it easier to automate tests by giving feedback if incorrect values are passed as parameters. Developers can get feedback when they try to pass an invalid parameter if I use union types when using TypeScript. I have been using TypeScript to automate tests with Playwright.

We often pass strings to functions as parameters. Sometimes this is because any string can be passed as a parameter. Other times there are only a limited number of strings that can be passed, such as the name of a country. If only a limited number of strings can be passed and we are using Typescript to write our tests, we can pass a union type instead of a string. 


A union type can be used to create a type that consists of only the strings that can be passed as parameters. Union types can be used when a type can be more than one type, for example, a union type could contain a string and a number. The types held in a union type can be literal types such as a string, for example, “cricket”. The union type is an ‘or’ type, so the type will be: ‘string1’ | ’string2’. An example of a union type could be defined in a .d.ts file like this:

The file containing the union type can be imported into your file like this:

The type can then be passed to the function checkBowlingAverage as the parameter bowler:

If a developer enters an invalid value such as ‘swxxing’ for the parameter bowler Visual Studio code will show the error:

If the developer’s mouse hovers overs over the invalid parameter, Visual Studio Code displays a message which shows the valid values. This makes it easy for the developer to correct the invalid parameter.

Execute Program has a lesson on Type Unions, and FrontendMasters has a lesson on Union Types. Both lessons have helped me get a better understanding of union types.

It is helpful if a larger group of people can automate tests and so it is useful to make it as easy as possible for tests to be automated. Using union types can help make it easier to automate tests because feedback is given if an incorrect value is passed.

Design a site like this with WordPress.com
Get started