TestAndAnalysis

“Go see, ask why, show respect”

Test analysts, test engineers, test leads and test managers need to understand customers so that our testing includes using the product as the customer uses it. To do this we need to learn how customers use the product by meeting the customer and seeing how they use it. Mr Fuji Cho, the former President of Toyota, is quoted as saying “Go see, ask why, show respect” [1].

“One of the key tenets of the Toyota Way is ‘genchi gembatsu’: ‘go to the source to find the facts’ This is commonly referred to as ‘go to gemba’”[1] “Gemba is a Japanese word that means ‘the place where something actually happens’”[2]

We should ‘go to gemba’ with purpose. We should define why we are going to gemba and take an intentional approach when seeing a customer. I have been reading Learning to Lead, Leading to Learn by Katie Anderson with the Profound Book Club and she gives these examples of purposes for going to gemba:

Checking work processes and outcomes
Validating data and assumptions with observable facts
Talking to the people involved [1]

Examples of ‘go to gemba’ that I have been part of are:

When I worked for a fashion retailer as a Test Lead I spent part of my first month working in a retail outlet where I met customers, used the till and handled deliveries. I learned about customers and the issues in running one of the company’s retail outlets. The knowledge I gained was helpful when I returned to the engineering team because it gave me a better understanding of the business and its customers.
When I worked for a start-up that had an annual user testing week all developers and testers were encouraged to attend part of the user testing to observe how customers used the product and learn from any issues customers had. Developers and testers saw when customers had difficulties and took this knowledge into their work developing the product. I took the knowledge I gained of how customers used the product into my work as Test Lead.

These ‘go to gemba’ activities gave me a deeper understanding of the business and its customers. This understanding helped me to focus testing on customers. “The customer is the most important part of the production line.”[3] Testers and the business gain so much from ‘go to gemba’. I encourage you to ‘go to gemba”!

References

[1] Learning to Lead, Leading to Learn by Katie Anderson (2021, Lesson 1 Communicate Purpose, Every Day and Every Way)

[2] The Toyota Way by Jeffrey K. Liker (2004, p224)

[3] Out of the Crisis by W. Edwards Deming (1986, p174)

Learning from CrowdStrike with Taguchi

The recent CrowdStrike incident is estimated to have “affected 8.5 million Windows devices” [1] and may have been “the worst cyber event in history” [1] How should we understand its impact on quality?

Genichi Taguchi’s definition of quality helps us understand how the CrowdStrike incident affected quality. He wrote that “quality is the loss a product causes to society after being shipped, other than any losses caused by its intrinsic functions”.[2] and that “loss should be restricted to two categories:

Loss caused by variability of function
Loss caused by harmful side effects”[3]

“Somebody must pay for this loss – Dr Taguchi called it a loss to society …. We all help to pay for a mistake, a breakdown, failure (bankruptcy) of a company, inept management.”[4]

The CrowdStrike incident can be seen as causing loss to society because it showed extreme variability of function and harmful side effects on “8.5 million Windows devices”[1].

Defining quality as the loss caused to society gives us an insight into the effects of an incident like CrowdStrike. The software that we test and develop also affects society. If a business or person that uses your software experiences problems due to a bug in our software that is a loss to society. Taguchi’s definition of quality shows that a measure of the quality of software is the loss it causes to society.

Taguchi won the Deming Prize, Japan’s highest award for quality, for his ‘loss function’ which enables the loss to society due to variability of quality to be quantified[5].

John Hunter gives practical advice on how to use Taguchi’s insight: “I have seen the concept of the Taguchi Loss Function used quite a bit. I have never actually seen any losses quantified and totaled and shown on a graph. I think focusing specifically on who suffers a loss and what that loss could be, can help. I think actually quantifying the losses to society can be daunting. So, while I see the value in framing the concept that way I think to actually get the losses quantified you are best served by starting with those closest to the process and then adding additional losses to those results” [6]

A lesson that we can learn from the CrowdStrike incident is that we should use Taguchi’s insight to consider how faults in the software we develop and test can cause losses to society, and through this consider how to avoid these faults.

References

[1] CrowdStrike IT outage affected 8.5 million Windows devices, Microsoft says

[2] Introduction to Quality Engineering by Genichi Taguchi (1986, p1)

[3] Introduction to Quality Engineering by Genichi Taguchi (1986, p2)

[4] The New Economics by W. Edwards Deming (1994, p218)

[5] Introduction to Quality Engineering by Genichi Taguchi (1986, p19)

[6] Taguchi Loss Function blog post by John Hunter

Additional resources

Taguchi’s Quality Engineering Handbook by Genichi Taguchi, Subir Chowdhury and Yuin Wu
Integration and the Taguchi Loss Function: Awaken Your Inner Deming (Part 13) an interview with Bill Bellows from the Deming Institute
Taguchi Loss Function definition by John Hunter

My new guiding principles are helping me to automate tests.

It is useful to have guiding principles on how to be a good employee, teammate and tester.

I work in teams that describe themselves as lean or agile and so I am interested in learning what lean and agile are. Learning about how lean and agile came about helps me understand them. John Willis has spoken about how the history of ideas in our field can be described as Deming ->Toyota->Lean->Agile. I have learned, and am learning, about many of the American roots of lean and agile such as the work of Deming and Shewhart. I am now learning about the Japanese roots of lean by reading Katie Anderson‘s book Learning to Lead, Leading to Learn with the Deming Profound Book Club. Her book examines the life and career of Toyota executive Isao Yoshino.
Yoshino sees direct parallels between the leadership precepts of Tokugawa and Toyota’s culture. Tokugawa was the first Shogun and achieved supremacy in 1600[1]. Precepts are general rules intended to regulate behaviour or thought [2]. The table below shows Togukawa’s precepts and Yoshino’s view of Toyota’s culture[1]:

Yoshino’s view of Toyota’s culture	Tokugawa’s leadership precepts
Steadiness and have a long-term view	Life is like a long journey with a heavy burden on your back. Don’t hurry
Diligence	One who regards inconvenience as natural will never be discontented. When you want more than you have, remember the days when you were in need
Learn from failure	If you only know what it is to conquer and don’t know what it is to be defeated, it will be harmful to you.
Patience	Patience is the base of being safe forever. Regard anger as your enemy
Leaders take responsibility	Blame yourself, not others.
Humility	Not being enough is better than being too much

Yoshino considered the foundational principles of Toyota’s culture to be influenced by Tokugawa’s precepts[1]. The Toyota Production System has influenced lean and agile software engineering, so Tokugawa’s precepts can be seen as one of the roots of Lean and Agile.

Tokugawa’s precepts still provide a useful guide today. They are a guide which can show how to deal with situations, for instance by being patient. They can also show when we could have done better, for instance by showing humility.

I have started using them as guiding principles and am interested to see what I learn. I have already learned how valuable humility is when pair programming, such as developing Playwright tests with a developer. Humility enabled us to listen to and learn from one another. Previously I had thought of humility as a virtue but now think of humility as being a principle that will help me be a better employee, teammate and tester.

I would like to thank Dennis Sergent for his helpful questions that have made me think about humility.

References:

[1] Learning to Lead, Leading to Learn by Katie Anderson (2021, Foundational Leadership Lessons)

[2] Kindle dictionary

Whoever you are, whatever you have achieved you should recognise the achievements of others

Dr Joseph Juran rose from poverty to be an internationally respected management consultant who specialised in quality. His work included popularising the use of the Pareto Principle and creating The Juran Trilogy. Juran focussed on the role of management in quality.

He wrote and contributed to many books including six volumes of Juran’s Quality Handbook.

He was a manager for Bell Telephones at the Hawthorne Works and later was a professor of industrial engineering at New York University, where he taught quality management.

After World War Two he worked in Japan and helped rebuild the Japanese economy. In 1981 he was awarded the Second Order of Sacred Treasure by the Emperor of Japan.

Juran consulted internationally, including working with Steve Jobs.

In 1992 he received the National Technology Medal from the President of the United States.

These are considerable achievements for which Juran is quite rightly respected.

However, he also let himself down by how he wrote about his peers. How we treat people is so important.

Walter Shewhart died in 1967 and the August 1967 edition of Industrial Quality Control was dedicated to the memory of Walter Shewhart “in view of his outstanding achievements”[1]. Juran contributed to that edition of the magazine by writing that Shewhart was “at that stage mainly impractical and unintelligible[2]” and was “a competent promoter”[2]. This is an extraordinary way to write about Shewhart, who had created control charts and the plan-do-study-act cycle and had just died. John Willis and Derek Lewis have made an entertaining podcast about Juran’s article: Legacy of Quality Control Pioneers.

Juran wrote his autobiography after W. Edwards Deming had died and in it, Juran wrote that Deming “believed his wishful thinking“[3] that the courses he gave in 1950 were “the dominant reasons for Japan’s emergence as the world quality leader”[3]. Instead of showing that his 1950 lectures were the dominant cause of Japan’s transformation Deming showed several reasons for Japan’s rise. Deming wrote, in addition to his own role in 1950, about the role of the Japanese in initiating the transformation of the Japanese economy. He wrote about the work of the Japanese Union of Scientists and Engineers (JUSE) in 1948 and 1949, and its later role in educating management, engineers and supervisors. He did write about his role in 1950 but he also wrote the roles of others including Ichiro Ishikawa, Kaoru Ishikawa, Ken’ichi Koyanagi and Joseph Juran.[4]

In Juran’s Quality Handbook 6th Edition, which was published after Deming had died, Juran wrote that Deming defined quality as “conformance to requirements”[5]. I can’t find a quote from Deming in which he defines quality in that way. Deming wrote that “quality can be defined only in terms of the agent”[6], which defines quality as being subjective.

When I searched to see if Shewhart or Deming criticised Juran, I found that Deming praised him. Deming wrote that Juran’s “masterful teaching gave Japanese management new insight into management’s responsibility for improvement of quality”.[7]

I have found so much value in Juran’s work, however, the way he wrote about his peers leaves me speechless. We should treat each other well. The lesson to me is that whoever you are, whatever you have achieved you should recognise the achievements of others.

Thank you to Rob Park for helping me organise my thoughts.

References

[1] Walter Shewhart ASQC 1967 – Quality Leadership (1967, p3)

[2] Walter Shewhart ASQC 1967 – Quality Leadership (1967, p50)

[3] Architect of Quality by Joseph Juran (2004, p302)

[4] Out of the Crisis by W. Edwards Deming (1982, p486)

[5] Juran’s Quality Handbook 6th Edition by Joseph E. DeFoe and Joseph Juran (2010, “How to think about quality”)

[6] Out of the Crisis by W. Edwards Deming (1982, p168)

[7] Out of the Crisis by W. Edwards Deming (1982, p489)

Use code reviews to have discussions about your test automation code

Learning from discussions originating from code reviews is helping me create a pack of automated tests using TypeScript and Playwright.

I have been developing a pack of Playwright tests with a Page Object Model. A simplified example of a page in the Page Object Model looked something like this:

A simplified example of a test:

When I created the pack I was pleased because I felt tests were easy to maintain and read. However, from a discussion following a code review, I learned how to write the test with fewer lines of code, making the tests easier to read. The functions should create page objects used in the test. This change would have two improvements:

Each test would contain fewer lines of code because fewer pages need to be imported and a constant does not need to be created at the start of the test.
When a test makes a search it returns a searchResultsPage, this means that the test reads more like a story.

The function search() now returns a searchResultsPage object. The simplified example of the Page Object Model now looks like this:

In the example test, the SearchResults Page is no longer imported, a searchResultsPage constant is no longer created at the start of the test and a searchResultsPage object is created when a call is made to search(). The simplified example of a test now looks like this:

The test pack I am developing has more than two Page Objects so this pattern reduces the number of lines in a test by an even greater amount than in the example. The tests are also now easier to read.

The advice from the code review and the following discussion helped me develop better tests by giving me a better pattern for automating tests. The important learning for me is not the better the code pattern but how code reviews can lead to discussions that help me improve my test automation code.

Two ways of learning that benefit testing

Testers are learning all the time. I have been reading John Dues’ new book Win-Win: W. Edwards Deming, the System of Profound Knowledge, and the Science of Improving Schools with the Profound Deming Book Club and have gained insights into different ways of learning.

“Moving from planning to doing is deductive learning and moving from doing to studying is inductive learning”[1]. I often learn in these two ways. Before I read John Dues’ book, I had not considered how different and useful each way of learning is.

“Deductive learning involves moving from a theory to the test of a theory” [1]. I often learn when I move from planning to doing, and in doing so I test the theory I had developed on how to do the work. I write a charter for some exploratory testing by writing and then learn while I am testing. I plan the automation of a test and then learn while automating the test. The charter for exploratory testing or the plan for test automation can be described as my theory.

“Inductive learning involves using results from the test to revise the theory” [1]. I also learn when moving from doing to studying. When I am reviewing my exploratory testing and test automation I am also reviewing the theory I had created on how to test or automate. Once I have stopped exploratory testing I review my testing and consider what went well, and what could have been better. I also learn from reviewing my test automation after I have automated a test and may have an insight that I want to add to my test automation standards.

Both ways of learning are useful. When I am doing exploratory testing I may find that I have forgotten to include something in my testing charter, this would be deductive learning. When I review the results of my exploratory testing I may find that most issues that I had found were in a particular area of functionality, this would be inductive learning.

Working in a Plan-Do-Study-Act cycle includes both types of learning. If you are working in a plan-do-study-act cycle you: plan your work; do the work; study the results of the work and then act on what you learned by taking your learning into a new Plan-Do-Study-Act cycle. John Dues includes a diagram that shows how a Plan-Do-Study-Act cycle moves between inductive and deductive learning:

Plan-Do includes moves from a theory to testing a theory and so is deductive learning.
Do-Study uses results to test a theory and so is inductive learning.

Testers need to be learning continually. Each iteration of a Plan-Do-Study-Act cycle contains deductive and inductive learning. Using a Plan-Do-Study-Act loop to organise your testing and test automation builds inductive and deductive learning into your testing and test automation.

References:

[1] Win-Win: W. Edwards Deming, the System of Profound Knowledge, and the Science of Improving Schools by John Dues (2023, chapter 1)ues (2023, chapter 1)

A Great Self-Organising Team

The SIGiST Summer 2024 Conference was a great success. The British Computer Society hosted the conference at its London office. We had nearly 200 delegates, which is more than at previous conferences. Over twenty speakers gave interesting and inspiring talks. It was great to have speakers for whom this was their first experience of speaking at a conference. We all learned from each other and had a great time! This year we also made some improvements to the conference:

To help drive out fear and create psychological safety we created a Code of Conduct
At the conference, there was a Prayer Room/Quiet Room
We hired artists to illustrate the conference
We had a panel discussion with Women In Test

SIGiST is part of the British Computer Society, which is a not-for-profit. Our committee members are all volunteers.

The committee is a self-organising team. Instead of the organisation of the conference moving “sequentially from phase to phase” it was “ born out of the team members’ interplay”[1]. We each took initiatives and supported each other’s work. A self-organising team, like the SIGiST Committee, works using a “‘holistic method – as in rugby, the ball gets passed within the team as it moves as a unit up the field” [1]. This approach does not only work for organising conferences. When I was part of an engineering management team in a start-up that made a successful exit, we worked in a similar way.

To be a self-organising team we created “processes, procedures, routines and norms that enable people to do their work easily and well”[2], this made good “social circuitry” [2] for the committee. Our ‘social circuitry’ consists of monthly committee meetings, WhatsApp conversations, regular Zoom catch-ups and using Basecamp for scheduling and sharing resources.

I would like to thank SIGiST committee members for their work, the conference speakers for giving inspiring presentations, the conference attendees for their great questions and conversations, Kerry Wear for supporting the work of the committee, Rae and Aysha for illustrating the conference and the staff at the BCS London office for their work on the day.

It would be great if you joined the British Computer Society and helped us make It good for society: https://www.bcs.org/membership-and-registrations/become-a-member/

References

[1] The New New Product Development Game by Hirotaka Takeuchi and Ikujiro Nonaka (Harvard Business Review 1986)

[2] Wiring the Winning Organization by Gene Kim and Steven Spear (2024, Preface)

How do you decide which tests to automate?

An end-to-end test pack needs to run quickly so that it does not slow developers, and at the same time provide useful feedback to the developers. This makes deciding which categories of tests to include in the test pack challenging.

A ten-minute video from Russ Ackoff has helped me better understand my decisions on one category of tests to automate and helps me explain my choices better.

Ackoff says that “the essential or defining properties of any system are properties of the whole which none of its parts have” [1]. He illustrates his point by referring to a car: “The essential characteristic of an automobile is that it can carry you from one place to another. No part of an automobile can do that. The wheel can’t, the motor can’t”[1]. He also uses human beings to illustrate his point: “You have certain characteristics the most important of which is life. None of your parts live. You have life. You can write, your hand can’t write….You can see, your eye can’t see.”.[1]

The application I test is a system like the systems Ackoff described. The application has “essential or defining properties” that none of its parts have. The application that I test shares information. One part of the application, such as the UI, can not share information. The application as a whole has the property of sharing information, not its parts.

A category of tests that should have end-to-end tests are tests for the “essential or defining properties” of the application. This is a useful analysis because if I define the application’s “essential or defining properties ” there will only be a limited number of tests and they will be for the key parts of the application. It is also a useful analysis because I can share this analysis and work with my team to define the “essential or defining properties” of the application.

References:[1] Beyond Quality Improvement Dr Russ Ackoff – a 10-minute video (1994) This presentation is from an event hosted to capture the Learning and Legacy of Dr. W. Edwards Deming by Clare Crawford-Mason and Lloyd Dobyns, the journalists who made “If Japan can..Why can’t we?”.

Who is responsible for quality? Is it the tester, or the team?

I have been reading John A. Dues’ new book Win-Win W. Edwards Deming, the System of Profound Knowledge, and the Science of Improving Schools with the Deming Profound Book Club. John Dues uses an equation to describe who is responsible for student performance[1]. This equation works as a useful analogy to describe who is responsible for quality.

There are development teams in which the tester is held responsible for quality. The tester is responsible for their work, which will affect quality, but quality results from the output of more than one individual’s skills and efforts. The factors that affect quality can be described in the equation below:

A + B+ C + D+ E + F = quality

If F represents the tester’s contribution to quality, what is the value of F? (what is the contribution of the tester?) We can’t give a value to F until we know the values of A, B, C, D and E. Factors that affect quality, are not controlled by the tester that could be represented by A, B, C, D and E include:

The quality of the documentation of the feature being tested
The time available for testing
Which team the tester is a member of
The choice of which feature is being worked on
Unit test coverage
Annual reviews system

Should the development team be responsible for quality if the tester does not control all the factors determining quality? The team controls more factors than the tester. This could be described in the equation below:

G + H + I + J + K + L = quality

If L represents the team’s contribution to quality, what is the value of L? (what is the contribution of the team?) We can’t give a value to L until we know the values for G, H, I, J, and K. Factors not controlled by the team that affect quality that could be represented by G, H, I, J and K include:

Team structure, such as does the team have a designer or product owner.
The team’s training budget
Deadlines for the release of the feature
The company’s ‘social circuitry’, that is its ‘overlay of the processes, procedures, routines and norms’. [2]
How the team’s performance will be evaluated

Neither the tester nor the team controls all the factors that affect quality so we should not blame a person or development team for an issue with the quality of the application under development. However, for a problem with product quality to be addressed constructively, the tester and the team should act responsibly. They should work together using a technique, such as the Five Whys, Ishikawa diagrams or a blameless post-mortem, to learn from the issue related to quality without blaming a person or team so that they can prevent the problem from re-occurring.

Thank you to Rob Park and the Deming Profound Book Club for helping me organise my thoughts.

References

[1] Win-Win W. Edwards Deming, the System of Profound Knowledge, and the Science of Improving Schools by John A. Dues (2023, p304)

[2] Wiring the Winning Organization by Gene Kim and Steven Spear (2024, p29) by Gene Kim and Steven Spear (2024, p29)

How to help your team complete their work and so have more time for testing

Testing can be hard, particularly when time is short because the team has a tight schedule. While working with Rob Falla I learned to use Critical Path Analysis to help my team deliver work on time, which helped me have more time for testing.

Sometimes it is hard to complete the work that a team has committed to in an iteration. It can be that a task that takes a long time overruns the iteration or that a task can not be started until another is finished. You may also find that more than one developer is working in the same areas and that this causes merge conflicts.

We used Critical Path Analysis to analyse all the cards we had taken into the iteration for their size and dependencies. We could then see if there were any ‘large’ cards. If there were any ‘large’ cards, we would need to start on them early in the iteration so that they were completed before the end of the iteration. We also could see which cards were dependent on other cards. If a card was dependent on another being completed then the card it was dependent on would need to be worked on first.

It may be useful to create a chart to help you with the analysis or it may be sufficient to use the analysis to order the team’s backlog. We always created a chart. The chart below is for the work a team that uses t-shirt sizing plans to do in its next iteration. Critical Path Analysis has shown that two streams of work have dependencies. Card 5 is a large card that is dependent on card 2 which is dependent on card 1. It also shows that card 4 is dependent on card 3. Cards 6 and 7 have no dependencies. The chart shows that, for the team to finish their work in the iteration, it would be best for the team to start work in their iteration by working on cards 1 and 3. After looking at the chart the team can also see whether they are likely to complete all the cards within the iteration, and if they need to revise their planned work so that they can complete it within the iteration.

I recently learned at the Profound Deming Book Club that this technique came from Eli Goldratt, and is also known as the Critical Path Method.

Critical Path Analysis helped the team I was a member of delivering quality software on time and gave us more time for testing. Techniques such as Critical Path Analysis are helpful because they help testing, help your team, and broaden your skill set.

Gain insights by using control charts to analyse your performance test results

On Friday 16 May 1924 Walter Shewhart gave his manager at Bell Telephone Laboratories a memo. The memo “suggested a way of using statistics to improve quality in telephones.[1]” Shewhart’s memo proposed using Statistical Process Control, including Control Charts for visualisation, to improve quality. Shewhart sparked “a revolution in quality control”[2] that can help us analyse the results of our performance tests.

Control charts visualise time series data along with the data’s average and upper and lower natural process limits. “Faults from fleeting events[3]” are described as special causes and appear outside a control chart’s upper and lower natural process limits. Faults of the system are described as common causes and appear on a control chart within the upper and lower natural process limits. The natural process limits are calculated from the variations in the time series data[4].

The above control chart shows UK inflation as it spiked post covid. The control chart shows that UK inflation had a fault outside the natural process limits and so had a special cause. Seeing whether the data from performance tests falls inside our outside natural process limits enables a tester to understand if the faults have a special cause ( a ‘fleeting event’) or a common cause (a fault of the system). This is very useful when analysing the results of performance tests because a fault due to a fleeting event should be treated differently from a fault with the system.

Eight patterns, known as ‘Nelson Rules’, can be used to analyse data variation within the chart’s natural process limits.

A control chart lets managers track variation and compare datasets. After the introduction of control charts managers could find and fix the causes of defects, rather than blame the staff for faults.

Control charts can also be called Process Behaviour Charts.

We now have one hundred years of learning from using control charts, which are now widely used. If you have a Fitbit you will be using control charts because the personal ranges for your health data are based on control charts. Here are links to guides to using them in different processes including DevOps automated governance, analysis of unit testing, load testing, customer satisfaction, health care, nuclear power, car manufacturing, education quality management system, managing stock levels, and marketing Campaigns.

Thank you to John Willis, Dennis Sergent, Rob Park and the Deming Profound Book Club members for helping me to gain a deeper understanding of control charts.

Here are some resources that will help you if you would like to use control charts to gain insights from your performance test results:

An API that can be used to create control charts

Geckoboard dataset API I am using Geckoboard dataset API to create control chart dashboards.

A One Day course at Coventry University:

An Introduction to Process Control

References

[1] Quality or Else by Lloyd Dobyns and Clare Crawford-Mason (1991, p52)

[2] Manufacturing the Future by Stephen B. Adams and Orville R. Butler (1999, p216)

[3] Out of the Crisis by W. Edwards Deming (1982, p314)

[4] Understanding Variation: The Key to Managing Chaos by Donald J. Wheeler (1993, p35)

How long will that test automation take?

Sometimes testers are asked how long it will take to automate a batch of tests. Planning how long your test automation should be simple, however, the plan will have missed some requirements.

There are three types of requirements[1]:

Known requirements – these are the requirements that have been clearly defined in planning meetings.
Overlooked requirements – we are all human, we need to allow for missing things so we need to recognise that we will have missed some requirements. The initial analysis could have overlooked functionality that requires an automated test.
Emergent requirements – these are requirements “that surface through the act of building the product.” [1]. Examples of emergent requirements in test automation could include:
- A feature of the testing tool does not work as expected, and the test automation engineer needs to find a new solution to a test automation problem.
- An element in the UI is difficult to interact with, for example, it is difficult to identify the element with a CSS selector, XPath or the test tools built-in locator.
- The testing tool does not support something that the test requires and you need to extend the capability of the testing tool using native code such as TypeScript.
- When you start to automate a new test you discover that a function needs to be refactored and as a result of refactoring the function, other tests also need to be refactored..

“We will never understand all the requirements of a story ahead of time”[2], this applies to test automation stories too.

Instead of creating a plan for test automation, it is better to create a test automation backlog. To make a backlog think about your vision for test automation and “start breaking down the things you need to execute that vision” [3]. “Don’t over plan, just estimate”[3], and create a list of the tests you want to automate. “The picture will change” [3]. “The reason for doing this type of planning is to create transparency within the organisation”[3].

References

[1] Agile Requirements Gathering: Three Types of Requirements by Mike Cohn (2020)

[2] Agile Testing by Janet Gregory and Lisa Crispin (2009, p133)

[3] Scrum: The Art of Doing Twice the Work in Half the Time by Jeff Sutherland (2015, p200)

Do Chromatic tests complement Playwright tests?

I am using Playwright to automate end-to-end tests, and have started to complement my Playwright tests with Chromatic tests.

The Playwright tests are really useful, but each test needs to run through several steps to create a scenario I want to test. Creating a Playwright test for each input variation for each part of the UI would be expensive. I would like a test automation tool to check a wide range of inputs for each element of the UI that does not require automating a large number of steps.

I have been introduced to Chromatic by a developer. Chromatic is a test tool that is made by the team behind Storybook. If your team uses Storybook you can use Chromatic to test for visual regressions in UI components. Chromatic tests can be described as “unit tests for visual states”.

Chromatic can be used to test each component in the UI. With Chromatic you can, for instance, create tests that check how a component handles the values that can be entered into an input field. Chromatic tests are like unit tests and do not require steps to set them up, also each test runs quickly so you can create Chromatic tests for all the inputs you would like without worrying that the test pack will take a long time to run. A Playwright test of each of these inputs would have the disadvantage of requiring automating several steps to set the test up before testing the inputs.

Chromatic tests can be written in javascript or typescript. An example test is below:

Chromatic tests can be written in more than one way:

Chromatic is a test automation tool that can complement Playwright, this is because I can write end-to-end tests in Playwright that test flows through the application and write “visual unit tests” in Chromatic that test a wide variety of inputs into individual UI components.

Cooperation helps to improve testing, helps testers and helps the company

Testers can help increase cooperation across the company, and cooperation will help us and the company.

Testers give feedback to developers when we test. We should also get feedback from other departments and customers on our work and the product we are testing. The feedback we get from cooperating with other departments brings the perspectives of customers and other departments into our testing. These additional perspectives help our testing, help us learn about the company and help break down barriers within the company.

When testers get feedback on the product from other departments, we also foster cooperation between them. Cooperation helps the company. If departments cooperate rather than compete against each other they will accomplish more. The company can be described as a system. If you improve one part of the system separately you can also destroy another part of the system.[1] Cooperation helps the whole system improve.

Cooperation also helps develop testers’ potential. Learning from other departments in the company will also help a tester if they would like to progress in their career because it will help them gain a broader perspective of how the company functions.

We can cooperate with developers and learn from them for example, I was introduced to GitHub Copilot by a developer. We can also learn from Developers how they perceive risks and then use what we have learned when we test.

“Everyone in engineering design, purchase of materials, testing materials and testing performance of a customer has a customer…Why not get acquainted with the customer?”[2] I have learned a lot about customers from talking to Customer Support\Success, and have taken that learning into my testing and the development team’s work. Talking to Salespeople is also useful. I once created a stakeholder map from conversations with the Sales Team and shared the information it contained about our customers across the company.

“Teamwork is sorely needed throughout the company. Teamwork requires one to compensate with their strength someone else’s weakness, for everyone to sharpen each other’s wit with questions”[2]. We can help improve teamwork across the company through discussions with people in different roles than our own. These discussions will help us learn from others in the company and we can take what we learn into our testing. We may, for instance, learn that a particular feature is often discussed between customers and customer support, this learning can be taken into our testing.

Cooperation is part of W. Edwards Deming’s theory of management. Point 9 of his 14 Points for Management is: We should “break down barriers between departments. People in research, design, sales and production must work in a team, to foresee problems of production and in use that may be encountered with the product or service.”[3]

Testers can improve cooperation across the company, which is good for testing, testers and the company.

References

[1] 2003 Ackoff Seminar Part 1 of 4 from the Deming Cooperative (2019)

[2] Out of the Crisis by W. Edwards Deming (1986, p62)

[3] Out of the Crisis by W. Edwards Deming (1986, p24)

Think slowly when you are testing And Think slowly when you are automating

I am interested in how thinking slowly can help me test and I have been reading Gene Kim and Steven Spear’s new book Wiring the Winning Organization: Liberating Our Collective Greatness Through Slowification, Simplification, and Amplification with the Deming Profound Book Club. Spear and Kim acknowledge how Daniel Kahneman, who recently passed away, influenced them when they show thinking slowly helps you improve. They also describe a tool that helps slow thinking.

We can think fast and we can think slowly. We think fast when we are under pressure. We use established habits, routines and heuristics to do this. Fast thinking has its limitations, for example, our biases may guide our decision-making. Fast thinking also does not allow time for us to learn. We need to think slowly so we can improve. [1]

We can apply this to testing. If we use ‘fast’ thinking when testing we may test in the way we always have. If we use ‘slow’ thinking when we test we can view testing as a learning process and learn more about the product we are testing.

If we use ‘fast’ thinking when we automate tests we may follow a known pattern for automating the tests. If we use ‘slow’ thinking when we automate tests we can learn a new way to automate a test or learn more about the application we are testing.

When we test and when we automate there are benefits in “Slowing ourselves down so that we can be more deliberative and self-reflective”[2]

Sometimes I have had to repeat a path of exploratory testing because I tested too quickly and missed something I meant to test. When I repeated the testing I worked through the path slowly tested more slowly and found more issues with the product. I thought slowly when I repeated the testing and found more issues because I was learning more and thinking through the consequences of the tests I was executing.

I have sometimes merged an automated test and then regretted it. This happened because I was thinking quickly and ‘completed’ automating the test using existing patterns. After I had merged and thought slowly about the test, I realised there was a better way to automate the test, and I refactored the test.

Spear and Kim recommend ‘slowification’ which they define as “shifting problem-solving from performance (operation, execution) back to practice (preparation) and planning”[2]. They say that “Dr. W. Edwards Demings learning cycle of Plan-Do-Study-Act (PDSA) is a tool to encourage slowification”[3]. Testers can use the Plan-Do-Study-Act cycle to slow down our testing and automating so that we need to learn more while testing and automating. We can:

Plan our testing or automation
Do our testing or automation
Study our testing or automation
Act on what we learn from the Study
And then repeat the cycle

If we use Plan-Do-Study-Act we will slow our thinking by studying what we are doing and acting on what we learn. We can use the Deming Cycle as a tool to help us slow our thinking because the cycle includes making time to study.

We can achieve more as testers if we slow our thinking and use the Deming Cycle to do so.

References

[1] Wiring the Winning Organization: Liberating Our Collective Greatness Through Slowification, Simplification, and Amplification by Gene Kim and Steven Spear (2024, Chapter 4)

[2] Wiring the Winning Organization: Liberating Our Collective Greatness Through Slowification, Simplification, and Amplification by Gene Kim and Steven Spear (2024, Chapter 3)

[3] Wiring the Winning Organization: Liberating Our Collective Greatness Through Slowification, Simplification, and Amplification by Gene Kim and Steven Spear (2024, Chapter 5)

Introductory guides to control charts:

A useful guide if you want to create a control chart:

A video introduction to control charts:

An API that can be used to create control charts

A One Day course at Coventry University: