TestAndAnalysis

First impressions of test automation with Playwright

I have recently automated some tests using Playwright. It is a free-to-use Node.js library created by Microsoft that is designed to be used to automate end-to-end testing.

Playwright supports chromium, WebKit and Firefox, and, by default, all tests are run on all three browsers. Tests can be run in headless mode. It supports, Java, .Net, JavaScript, TypeScript and Python. I am writing my tests in TypeScript and have used FrontendMasters TypeScript Fundamentals, v3 course to help me get up to speed with TypeScript.

Its first-party test runner is Playwright Test, although a number of test runners such as Jest\Jasmine and mocha can be used. Playwright Test feels familiar because it has describe blocks skip and other familiar functionality.

There are good resources for learning Playwright, which include free courses provided by both LambdaTest and Test Automation University.

One of Playwright’s strengths is its support for iframes. Playwright contains functionality that enables tests to interact with elements inside an iframe, giving it an advantage over many test automation frameworks. To explore how to test elements inside an iframe you can use Jonathan Thompson‘s blog post about how to use Playwrights support for iframes[1]. You can use `frame.getAttribute` and `frame.$$` to get the values of attributes in elements within an iframe, which helped my test automation. It is really helpful that you can wait for events inside an iframe, for example with frame.waitForFunction().

Playwright has good easy-to-use documentation, however, you may not find what you are looking for. If you can’t find the item you want in Playwrights documentation, search in google for it. I wanted to find a way to have some tolerance for changes in screenshots. When I searched for tolerance in the documentation I found nothing, but when I searched for it in google I found the PR for the change that created the functionality I wanted. The PR gave me all the information I needed.

Playwright is a useful addition to your test automation tools.

References

[1] Jonathan Thompson Handling Iframes using Python and Playwright

Resources

Playwright Documentation: https://playwright.dev/docs/intro
A Playwright sandbox: https://try.playwright.tech/
Playwright community: https://playwright.tech/
GitHub repo: https://github.com/microsoft/playwright
LambdaTest Playwright Certifications: https://www.lambdatest.com/certifications/
Test Automation University Course on Playwright: https://testautomationu.applitools.com/js-playwright-tutorial/

Using plan-do-study-act to improve testing

Testers and developers can use the Deming Cycle to improve the quality of their testing. The Deming Cycle was initially used in the manufacture of telephones and has had a big influence on software development. The cycle has four steps:

Plan – plan what you are going to do
Do – execute the planned work
Study – what went well, what went wrong, what did you learn?
Act – act on what you have learned and take this learning into the next planning step

The cycle should be repeated with the knowledge accumulated.

The Deming Cycle can be described as a flow diagram and the Deming Cycle is shown in the flow diagram above.

This cycle is used by testers and developers in their testing for example in TDD and in exploratory testing.

An example of using the Deming Cycle in testing is Test Driven Development (TDD). Adam Hawkins Small Batches podcast has an episode about the cycle[1], during the episode he describes how the Deming Cycle is used every day in TDD. He describes TDD’s use of the cycle as:

Plan – write the test
Do – write the code
Check – did the test pass?
Act – green tests refactor, red test repeat loop

How TDD uses the Deming Cycle is shown in the flow diagram below:

Elisabeth Hendrickson’s Test Heuristics sheet[2] includes “Demings Cycle” in its list of resources for testing. She is known for her book “Explore It!” about exploratory testing and it is useful to look at how exploratory testing uses the Deming Cycle:

Plan – write a test charter
Do – execute a test
Study – analyse the result of the test
Act – act on the result of the test, maybe raise a bug card, re-run the test or make another test

How exploratory testing uses the Deming Cycle is shown in the flow diagram below:

Deming was keen that the cycle was understood as plan-do-study-act and not plan-do-check-act. One of the reasons for this was that if you only checked what had been done you might miss something[3]. The study part of the cycle is important because we need to consider what we have learned from the test before we can decide on our next action. It is also important that we are given time to think about the result of the test, this is because which other tests we execute and how we report on the test depend on how we understand the result of the test.

The Deming Cycle can help us improve our testing because it enables us to study and learn from each test we make.

References:

[1] Adam Hawkins Small Batches podcast: PDCA (Plan-Do-Check-Act)

[2] Elisabeth Hendrickson’s Test Heuristics sheet

[3] Lloyd Dobyns and Clare Crawford-Mason Quality or Else

Further reading:
- The PDSA Cycle
- Out of the Crisis by W. Edwards Deming
Further viewing (you can log in and view the video for free):
- What did Deming’s work on quality do for us?
eLearning:
- DemingNext

How to create a simple performance test with Puppeteer

Creating your first performance test can seem daunting, but it need not be so. This blog post is a guide to resources that you can use to create a simple automated performance test.

A good tool to use to create a performance test is Puppeteer. Puppeteer is a node.js library that can be used to automate chrome. There is good documentation for puppeteer both on its website and helpful posts on discussion groups and other websites such as stackoverflow. There is no need to buy a load testing tool as puppeteer can be downloaded for free.

Puppeteer is a node library so if you have not already done so, you will need to install Node. Node can be downloaded from here: https://nodejs.org/en/download/

You will also need to install Puppeteer and instructions to install Puppeteer are here: https://www.npmjs.com/package/puppeteer

Puppeteer contains functions such as await page.click and page.waitForSelector which you can use to create an automated test. You should use puppeteer to automate the steps that you want a performance test for. You could use the puppeteer example of a functional test on GitHub to help you automate these steps, and you could also use the tutorials on https://pptr.dev/ to help you.

Once you have automated the steps that you want to test for performance, you then need to add code to the test that will give you a timing for the steps that you are interested in.

For a simple test you can also get the start time and end time of the steps in the test by using new Date() to create timestamps and then calculate the difference between the two timestamps to show how long the steps took:

let beginTime = new Date();

// steps for the test

let endTime = new Date();

let duration = endTime.getTime() – beginTime.getTime();

It is useful to add additional metrics to help interpret the results of the performance test. Chrome dev tools makes available via its api a number of metrics, such as heap size. These metrics can help you interpret the results of the performance test: https://chromedevtools.github.io/devtools-protocol/tot/Performance/#method-getMetrics. This blog post contains an example of how to get the chrome dev tools runtime performance metrcs: https://addyosmani.com/blog/puppeteer-recipes/

The node app that you have created with puppeteer can be deployed to the platform that your team uses and scheduled to run however often you require.

You will want to share the results of the performance tests. You can write the results of the performance tests to a Geckoboard dataset and this enables you to create a dashboard to display the results of the tests. Geckoboard dashboards can contain a range of visualisations and can be sent to either a Slack channel or an email address on a schedule or displayed on a TV.

Creating an automated performance test creates data that enables you to ask questions about why the performance of your application is as it is and explore further the performance of your application. I hope that this blog post helps you to create a performance test and explore the performance of the application that you are testing.

Resources:

Node
- https://nodejs.org/en/
Puppeteer:
Chrome dev tools metrics:
Geckoboard Datasets
- Geckoboard datasets API
- Geckoboard dashboards

Lessons Learned from being Programme Secretary of a testing conference

I have just been the Programme Secretary for BCS SIGiST’s conference “Testing, Diversity, AI”, and it seems useful to share some of my thoughts on organising the conference.

Firstly it was a rewarding thing to do. I have worked in testing for twenty years and it was good to give something back to the testing community. It was great to give a platform to testing professionals who want to speak at a conference. It was great to hear so many fantastic speakers. I also learned about publicity, conference organisation, and met great people through being Programme Secretary.

Networking is an important part of attending a conference. It was important to make time in the conference schedule for networking as it was good to network with attendees and speakers at the conference.

It is important that the speakers are representative of the population. I had aimed to have 50% women speakers and missed that target on the in-person stream. Next time I am Programme Secretary for a conference I will view 50% of the speakers’ places as for women and will only fill those places with women speakers.

It helped to have themes for the conference as this grouped talks, signalled to attendees what they would hear at the conference and helped publicity.

You should avoid clashing with other testing conferences as we are all trying to help the testing community and no one gains from two conferences happening at the same time.

Don’t get downhearted if things go wrong and obstacles appear, with your team you can overcome these issues. Due to the Queen’s funeral, we had to reschedule the conference, which had eighteen speakers across two streams, and we did so successfully.

The British Computer Society is a not-for-profit and its Specialist Interest Group in Testing is run by volunteers like myself. It was great to work together as a team to organise the event. My membership of The British Computer Society is really useful as it enables me to connect with so many people from across the tech sector, such as those I worked with to organise the conference: Kerry Wear, Nicola Martin, Adam Leon Smith, Jonathan Wright, Paul Mowat, and Andy Shaw. I would encourage you to join BCS too: https://www.bcs.org/membership-and-registrations/become-a-member/

The conference went well and I enjoyed the presentations and conversations.

I also have to say a big thank you to all the speakers, and to the BCS staff who helped put on the conference.

The main lesson learned was that being the Programme Secretary for a testing conference was a really positive experience!

We need to remove barriers to good work

If we want to make good quality software we need to remove barriers to good work.

Dr Deming‘s 14 Points for management were the basis for lessons for top management in Japan. Dr Deming said that “the 14 Points apply anywhere, to small organisations as well as to large organisations[1]”. Point 12 is to “remove barriers that rob people of pride of workmanship”. I would reword this point to be that “we need to remove barriers to good work”.

He believed that employees being able to take pride in good work improved staff retention, and improved the quality of the products being created.

One of the things that Deming described[1] as a barrier to good work was that employees can sometimes be made to feel like a commodity. He also wrote about how employees complained about being required to ship products that they knew were substandard. He was writing about employees who worked in factories. From reading testers what testers write on Twitter and LinkedIn I can see that, from time to time, these issues still occur in software development today.

Silos within companies that stop one group of employees from knowing about another group of employees’ work can lead to one team’s work creating problems for another team. Good teamwork across the company is another way to remove barriers to good work as teamwork will break down silos.

The annual merit rating was also something that Deming saw as a barrier to good work. He said that it “builds fear”[1]. Managers should be working to support and develop their staff rather than judging them once a year. Dr Deming recommended the work of Peter R. Scholtes as an alternative to performance appraisals.

There are other barriers to good work. In order to remove barriers to good work we need to support inclusion and diversity. Two examples of how lack of diversity creates a barrier to good work would be: if we work in teams that do not include women engineers then those teams are missing out on working with some fantastic engineers, and if teams are working with women engineers who do not feel that they are listened to then those teams are missing out on great engineering skills. I am pleased to say that where I work we actively address this issue.

What other barriers are there to good work?

Continue this discussion about diversity at the SIGiST Conference “Testing, Diversity, AI” on 16th November 2022, get your tickets here: https://www.bcs.org/membership-and-registrations/member-communities/software-testing-specialist-group/conferences/testing-diversity-ai-conference/

References

[1] W. Edward Deming Out of the Crisis

More Information:

eLearning from the Deming Institute: DemingNext
Book: Peter R. Scholtes The Leader’s Handbook

Using Ishikawa diagrams to improve quality

Cause-effect diagrams are a useful technique that can be used to improve quality. Glenford J. Myers wrote that “a weakness of boundary value analysis and equivalence partitions is that they do not explore combinations of input circumstances”[1]. A technique that can be used to explore and describe combinations of inputs to an issue is a cause-effect diagram.

A type of cause-effect graph I have found useful is the Ishikawa or Fishbone Diagram. The diagram takes its name from Professor Kaoru Ishikawa who invented the diagram[2]. It is also known as a fishbone diagram because it looks like a fish’s bones. An example diagram is shown above. The effect that you are concerned about is shown in the diagram on the fish’s head. The effect shown in the diagram could be a bug or a production incident that you want to analyse. The causes of the effect are shown on the fish bones, and sub-causes are shown as branches of the fish bones.

The diagrams were named Ishikawa Diagrams by Joseph Juran [3], and are an example of how he and W. Edwards Deming brought ideas from Japan to North America and Europe.

Ishikawa categorised the causes of the issue, that is being analysed, into five groups called the Five Ms:

Material. Everything that is consumed by the project such as raw materials.
Method. Existing procedures and information flow.
Mother Nature. The environment and context.
Machine. The necessary equipment for the project.
(Hu)manpower. The human resources involved in the project.

An Ishikawa diagram can be used to find the root cause of a production incident or another complex issue. A root cause of the effect being analysed should be written on a bone in the diagram, and each root cause should also be analysed to find its root cause. The root cause of one of the causes should be written as a branch to the bone.

Ishikawa diagrams can be used to improve quality as they can be used to identify the causes of an issue, and countermeasures can then be identified to remove the causes of the issue.

The diagram may be created when you are working with a team or when you are working alone. The team can create a diagram by suggesting causes which are then added to the diagram. To help find the causes of the effect being analysed, the Five Ms can be used as prompts to consider the factors they represent. An example would be to consider material, this could be used to consider all the third-party components used and (Hu)manpower could be used to consider what training people working on the project have had and need.

An advantage of using an Ishikawa diagram is that it is visual and so it is easy to review by looking over the causes identified on the diagram. This can make it easy to work iteratively on finding the causes of an issue. The diagram is easy to share as it is one diagram rather than a long document. Another advantage is that there is also no need for notes that have to be written up after a team creates a diagram as the diagram explains itself.

Two disadvantages of Ishikawa diagrams are that they can be unfamiliar and may not look like diagrams that people are used to using. The success of teams and individuals, who have not used Ishikawa diagrams previously, relies on their wanting to learn how to use fishbone diagrams. If you want the team to use Ishikawa diagrams then it is your role to persuade them to do so.

Learning how to use an Ishikawa diagram is a useful skill. If a team or individual learns how to use Ishikawa diagrams, they will likely use them again later in their career. Ishikawa created fishbone diagrams in 1943[2], so they have been in use for a long time and it is reasonable to expect them to be used for the foreseeable future. Fishbone diagrams are useful and are used widely. Janet Gregory and Lisa Crispin suggest using Ishikawa diagrams to visualise your way of thinking and to help with identifying risks[5]. Masaaki Imai lists fishbone diagrams as one of the Seven Statistical Tools involved in Kaizen problem-solving[4].

There are many templates available for creating Ishikawa diagrams, to find them just google “Ishikawa diagram templates”. You should be able to find a template for an application that you use.

It would be great to hear of your experiences of using Ishikawa diagrams to improve quality

References

[1] Glenford J. Myers The Art of Software Testing

[2]. 50MINUTES series Ishikawa Diagram: Anticipate and solve problems within your business (Management & Marketing)

[3] Kaoru Ishikawa What is Total Quality Control – The Japanese Way

[4] Masaaki Imai Kaizen the Key to Japan’s Competitive Success

[5] Janet Gregory and Lisa Crispin More Agile Testing

Testing conferences and events that I have participated in:

Date	Event	Organiser	Title of Talk	Video
3/14/2017	SIGiST Spring conference	SIGiST	The Spin Bowler and the Agile Tester
1/30/2018	Meetup	Djugl	The Spin Bowler and the Agile Tester
10/31/2019	P3X – People, Product & Process eXchange 2019	Skillsmatter	What did Deming’s work on quality do for us?	https://skillsmatter.com/skillscasts/14109-what-did-deming-s-work-on-quality-do-for-us
1/27/2020	Employability Talk	Middlesex University	A career as a software tester
10/15/2020	Test Export & Agile and Devops Expo 2020	Unicom Seminars	How Does Deming’s Work on Quality Help Us?
12/10/2020	Meetup	Heart of England Scrum User Group	So where did all this agile stuff come from?	https://qahiccupps.blogspot.com/2020/12/plan-do-something-act.html
1/12/2021	SIGiST webinar on AI	SIGiST	Host
2/11/2021	Agile, Devops and Testing: The evolving scene	Unicom Seminars	Evolving your career: Questions an interviewee should ask
4/6/2021	QA Global Summit	Geekle	Evolving your career: Questions a tester should ask at an interview to find a great job	https://geekle.us/video_cluster/1617882514964×989718131845365800?video=1617972304070×284295516163407870
6/5/2021	Global Sumit 2021	QA Tech Talks	Two Quality Frameworks from Deming That Help Testers
6/10/2021	Thursdaymatters Testing Stories	Skillsmatters	Test Lead in an agile transformation	https://skillsmatter.com/skillscasts/15099-test-lead-in-an-agile-transformation
6/17/2021	Testing Stories Conference	Sage Continuous Innovation	Test Lead in an agile transformation
7/7/2021	QAL Workshop	QA Lead	Empowering lean and agile testing using Deming’s quality frameworks
9/9/2021	P3X — People Product Process eXchange 2021	Skillsmatters	How The Theory of Jobs To Be Done Helps Me Hear Customers	https://skillsmatter.com/skillscasts/15107-how-the-theory-of-jobs-to-be-done-helps-me-hear-customers
9/28/2021	SIGiST webinar on mentoring	SIGiST	Host
10/12/2021	Testing Stories Webinar	SIGiST	Test Lead in an agile transformation
1/19/2022	SAGE Testing Tales	SAGE Continuous Innovation	How does Deming’s work on quality help us?
2/9/2022	Agile, DevOps and Testing: Methods and Tools Revisited	Unicom Seminars	Panel discussion on Agile experimentation and innovation
2/15/2022	QA Global Summit’22	Geekle	How The Theory of Jobs To Be Done helps focus testing on customers	https://geekle.us/video_cluster/1646659459977×515382583082352640?video=1646672195436×505416905441941500
4/28/2022	Dev Meetup	Developers Shore Community of Great Devs	How does Deming’s work on quality help us?	https://www.youtube.com/watch?v=wq00RUj151E
19/10/2022	QA Global Summit 22.2	Geekle	Committee Member and Speaker: How can agile teams understand quality?	https://geekle.us/stage/1666209694857×716475234076065800
18/11/2022	Testing, Diversity, AI	SIGiST	Programme Secretary and Speaker How does Deming’s work on quality help us?	https://www.bcs.org/events-calendar/2022/november/hybrid-event-testing-diversity-ai-conference-software-testing-sg/
30-31/05/2023	QA Global Summit 23	Geekle	Committee Member	https://events.geekle.us/qa23/
14/6/2023	BCS SIGiST Summer Conference 2023	SIGiST	Programme Secretary	https://www.bcs.org/membership-and-registrations/member-communities/software-testing-specialist-group/conferences/bcs-sigist-summer-conference-2023/
8/8/2023	<break>point conference	BrowserStack	Panelist for discussion on “Empowering testers with low code Automation”	https://www.browserstack.com/events/breakpoint
7/11/2023	The Evolution of QA : Bug Hunters of Tomorrow	THINK AGILE TN Agile Montréal Bugs Hunter QA Talks Community (QAT)	How does Deming’s work on quality help us?	https://www.youtube.com/watch?v=nAaVTgrZJwI
25/01/2024	Meetup	BCS South Yorkshire and Yorkshire Cyber Security Cluster	How does Deming’s work on quality help us?	https://www.youtube.com/watch?v=t52VyloOM_o
15/06/2024	Kolkata Meetup Group	BrowserStack	How does Deming’s work on quality help us?	https://www.meetup.com/browserstack-meetup-group-kolkata/events/300466717/
19/06/2024	BCS SIGiST Summer Conference 2024	SIGiST	Friday 16 May 1924: a new beginning for quality	https://www.youtube.com/watch?v=-uisrji_f7I&list=PLKBhokJ0qd3_mL02F2ZfhgZbBD8CIvfVf&index=18
10/02/2025	Talk	Ministry of Testing	Enhance your performance tests and more with process behaviour charts	https://www.ministryoftesting.com/talks/enhance-your-performance-tests-and-more-with-process-behaviour-charts?s_id=19011619
10/03/2025	Panel Discussion	Ministry of Testing	Exploring Systems Thinking – The Testing Planet News – Episode 09	https://www.ministryoftesting.com/the-testing-planet-sessions/exploring-systems-thinking-the-testing-planet-news-episode-09?s_id=19011620
19/06/2025	Conference	BCS SIGiST Summer Conference 2025	Programme Secretary and Host	https://www.bcs.org/events-calendar/2025/june/hybrid-sigist-finding-calm-in-chaos-applying-testing-to-a-changing-world
26/06/2025	Webinar	BrowserStack	How process behaviour charts empower testers	https://www.browserstack.com/webinars/test-reporting-analytics

How to deal with a complaint about quality

How a company responds to a complaint about quality needs careful consideration. A model that we can use to explore this issue is Dr Deming’s Red Beads Experiment. We can extend the Red Beads Experiment to include a complaint from customers. I explored the Red Beads Experiment in a previous blog post The blog post explained the steps in the experiment and the effect of the experiment on testers and quality.

In the Red Beads Experiment, the White Beads Company makes white beads and struggles to reduce the number of red beads in the white beads it sells. We can use the Red Beads Experiment to help us explore how to deal with complaints about quality by extending the Red Beads Experiment to include customers complaining to the White Beads Company’s Chief Executive about quality. The customers are unhappy with many red beads being delivered in their orders of white beads.

The White Beads Company holds its staff responsible for their performance in a process that they can not influence, so I would expect them to respond to customer complaints by blaming the staff. This will not help improve quality as the staff do not control the production process at the White Beads Company and cannot improve quality. Blaming the staff will also create an atmosphere of fear in the company which will make it harder to improve quality. If there is fear in the company people will not answer questions honestly as they will want to protect their job and their friends’ jobs and so quality will not improve.

The White Beads Company should use Deming’s System of Profound Knowledge (SoPK) to improve quality. Deming wrote that SoPK “provides a map of theory by which to understand the organisation that we work in”[2]. SoPK has four components:

An appreciation for a system – systems thinking should be used to understand the problem and find a solution to it
Knowledge of variation – there will always be variation in quality, and the company must learn to understand it
Theory of knowledge – the company should develop a theory of what will happen in the future based on their actions to improve quality
Psychology – the White Beads Company should understand individuals and how individuals interact as this will have an effect on the quality of the company’s output.

An understanding of psychology and systems thinking will help the White Beads Company move from being a company where managers “find and record failures” to one where they “remove the causes of failure”[1].

The White Beads Company can use systems thinking to identify that the company and its suppliers are a system that makes white beads. It can then work to improve the quality of its white beads by working with its suppliers to improve the quality of the raw materials being supplied to the White Beads Company.

Knowledge of the number of red beads in the raw materials will enable the company to evaluate the success of its initiatives to improve quality.

Improving the quality of white beads by improving the quality of raw materials, rather than blaming the staff, will help the company to drive out fear from the company. This will help people to work more effectively. Working with suppliers will also break down barriers which will create teamwork that will improve quality.

The company should then implement this plan using the plan-do-study-act cycle; first, it would plan the initiative, then it would carry out the plan, next, it would study the results of the planned work, and then act on what it found by taking its findings into another plan-do-study-act cycle.

Using SoPK, and working iteratively with a plan-do-study-act cycle will enable the White Beads Company to improve the quality of its product.

Are there other useful ways to extend the Red Beads Experiment?

Note:

I wrote a longer post about the Red Beads Experiment for LambdaTest Blogs, and John Willis, one of the founders of DevOps, wrote a positive blog post in response to my post. He said in his post that Deming would be disappointed that people had not extended his Red Beads Experiment. This blog post is a suggestion on how to extend the experiment in a way that sheds light on how to deal with complaints from customers about quality and how Deming’s philosophy can help.

References

[1] W. Edwards Deming Out of the Crisis

[2] W. Edwards Deming The New Economics

More Information:

These two books each give a full account of the “Red Beads Experiment”:
- W. Edwards Deming The New Economics
- William J. Latzko, W. Edwards Deming, David M. Saunders Four Days With Dr. Deming

eLearning from the Deming Institute: DemingNext contains a course on the Red Beads Experiment

Videos:
- Lessons from the Red Bead Experiment (30 minutes)
- The Red Beads Experiment (38 minutes)

A useful blog post about Deming:
- John Willis ED – Rediscovering a WWII Statistician& Why We Need His Ideas to Survive the Digital Age

Using the Five Whys to improve quality

The Five Whys is a technique for finding the root cause of a problem. Toyota developed this technique and it is now widely used, including in software development. I was introduced to the Five Whys by Tom Gilb as part of a course he ran on Lean QA and have used the Five Whys in several places where I have worked. Testers want to improve quality, the Five Whys is a technique that improves quality so it is useful for testers to be able to use it.

The Toyota engineer Taiichi Ohno would ask why five times to find the root cause of a problem as this would enable him to create and implement appropriate countermeasures[1]. Asking why five times usually will find a “soft” issue as the cause of the problem, for example:

Why did the customer’s file fail to load in production? Because there was a bug in the loader
Why was there a bug in the loader? Because a feature branch that contained the bug was deployed to production
Why was the feature branch with the bug deployed to production? Because it passed all the automated tests
Why did the feature branch pass all the automated tests? Because the database used by the integration tests had not been updated
Why was the database not updated? Because there was no process to update the database

In this example, the root cause of the problem is that the database used by automated tests had not been kept updated. The countermeasure is to create a process that keeps the database updated.

The Five Whys can be used to find the root cause of a problem. The example above shows it being used to find why a bug was introduced into production, the technique can also be used to find the cause of an incident in production or it can be used to find the root causes of multiple problems such as when Toyota used the Five Whys to reduce the noise of the Lexus engine[2]. Teams should use the Five Whys to keep asking until the root cause(s) are identified. In one of the examples in The Toyota Way[2] why has been asked six times. When using the Five Whys you may find the root cause after asking why four times, five times or more times.

A development team can meet and work through the Five Whys together as a team. It can be useful to vary how the team uses the Five Whys and a team can also work through the Five Whys individually and then discuss their findings as a team.

Finding the root cause of a problem is a constructive way of dealing with problems that does not involve blame and so helps teams to grow. When using the Five Whys individuals can feel that they are having to admit to a mistake and find this difficult. In order to give people confidence, it can be hopeful to read the prime directive when starting a Five Whys meeting. When first using the Five Whys it can also be helpful if senior members of the team l say that they make mistakes as this can empower other members of the team. It can be unhelpful if senior managers want to join a Five Whys meeting as people may feel inhibited from talking about errors that have been made if a senior manager is present.

The Five Whys is a positive way of dealing with problems that can contribute to creating psychological safety because it enables team members find a way to deal with problems without blame. A blame culture prevents a team from finding the real issues behind problems as people will want to protect themselves and their friends. People who have been working in a blame culture can find using the Five Whys invigorating and empowering.

Once a team has found the root cause of a problem they should adopt a countermeasure so that the problem does not re-occur. Implementing a countermeasure to prevent a problem from reoccurring will improve quality.

I would be very interested to hear your experiences of using the Five Whys.

References

[1] Lean Thinking: Banish Waste and Create Wealth in your Corporation James P. Womack and Daniel T. Jones

[2] The Toyota Way Jeffrey K. Liker

Testing needs to include the needs of internal and external customers

When we test we think about the users of the functionality and we include their needs in our testing. Creating categories of customers, such as internal and external, can help us understand our customers’ uses of the functionality.

Dr Joseph Juran advocated viewing customers as either internal or external customers. Juran was born in Romania and emigrated to the USA. He, like W. Edwards Deming, worked with Walter Shewhart and helped the Japanese economy recover from World War Two. He also taught quality control at New York University and worked with Steve Jobs.

Dr Juran said that identifying customers is the first step in planning for quality[1]. He defined a customer as “someone who is impacted by the product”. He also said that customers may be external or internal.

An external customer is a customer who is external to the company or organisation. They may be a paying client, government regulatory bodies, international regulatory bodies or the public.

Departments and people within a company often supply products to one another, these are internal customers. They are not customers in the traditional sense as they are not paying clients, but they are often called customers.

Juran recommends creating a flow diagram to help identify customers. The flow diagram can show the steps involved in the process and help identify customers[1].

He says that customers can also be divided into classes. If there are many customers the Pareto principle can be used to classify customers into the vital few and the trivial many as this helps you to focus your efforts where it will have the most impact.

It is important to consider how the functionality we are testing impacts internal customers as the functionality we are developing will impact others within the company. Examples of this would be that the functionality could create additional work for the customer service team if functionality is complex.Sales and marketing are also impacted by the features a team is working on and it is useful to view sales and marketing as customers so that they are kept informed of new features.

Juran’s way of looking at external customers is useful too as maybe a playing client uses your product to provide a service to someone else such as “the public”, and your testing can then also take that other person’s needs into account. The success of the application that you are testing may depend on the quality of the service being provided by your paying client to hi/her paying clients.

I find that this categorisation of customers is useful because it provides a structure that enables me to think about external customers like clients but also about internal customers like customer service.

References:[1] Juran’s Quality Handbook Fourth Edition

Should we practice continuous learning?

I enjoy working in an organisation where we are learning from the work we do. An example of this would be using retrospectives to enable the team to learn from their work and taking this knowledge forward to help the team.

Until recently I have called this approach continuous learning. It sounded right, as it felt to me that agile and lean were about doing things continuously, such as continuous deployment.

I recently spoke with Kevin Cahill, and during the conversation, I used the phrase continuous learning. He told me that he had used the phrase ‘continuous learning’ when talking with his grandfather, W. Edwards Deming. His grandfather had told him that when talking about learning he should use the word continual not continuous and that he should look up in the dictionary the meaning of the two words.

When I looked up the meaning of the two words on google found:

Continuous:

“forming an unbroken whole; without interruption.”

Continual:

“forming a sequence in which the same action or event is repeated frequently.”

There is an important difference between the two words. Continuous learning would mean learning without interruption. This style of learning would be mechanical with no let up and no time to think. Continual learning would involve repeated learning and allows time for study and reflection.

The difference between continuous and continual is not just semantics. It is about an approach to learning.

We use continuous to describe automated processes such as continuous integration and continuous deployment. These processes are carried out by machines and so can not benefit from studying or reflecting as we do

An example of continual learning would be, if we work in an iterative way using a plan-do-study-act cycle, we may have to wait a while before we can study the outcome of our work. Our learning is continual because we have had to wait, and during that time we may reflect on the work we did.

I now think of learning as being continual, not continuous.

I am learning about systems thinking to help me test

I want to improve my understanding of systems thinking as this will help my testing. My friend John Holmstrom recommended the work of Peter Senge and so I have just read “The Fifth Discipline” by Peter Senge. The book contains many useful insights about system thinking, learning and leadership. Some of the points about systems thinking that I have taken from the book to help me with testing are:

Systems thinking is a discipline for seeing wholes
Don’t be your role. Don’t say “I am my position”. I am a tester and I should see beyond my role by using systems thinking. This way I can see what processes are influencing the feature I am testing and what processes will be influenced by the feature I am testing. Understanding these influences enables me to bring them into my testing and so test the feature in new ways, for example, I can think about who the customers’ stakeholders are and how they use the feature.
The team I work in is not autonomous therefore it is not a closed system, so I should continually learn about how it is influenced by and influences other systems.
The diagrams in the book that showed how systems affect something can be used to create discussion about the systems that influence the success of a new feature. The illustrative diagram in this blog post is one I have created based on what I have seen in the book. The diagram shows how the benefits of the new feature are influenced by the customers existing systems and their use of the new feature.
We should not ignore how our team’s decisions affect others
Don’t blame external factors for failures as external factors are part of the system that you are part of.
Everyone shares responsibility

I have used systems thinking to help me think about how customers are using the product and I am going to continue to learn how to use systems thinking. There is so much more to learn and so much more in the book!

Being the Programme Secretary for a testing conference is an exciting experience

I have always gained a lot from attending conferences, and it has now been over two years since I attended a conference in person. I am now enjoying being Programme Secretary for BCS SIGiST’s Testing, Diversity, AI Conference.

Over the last two and a bit years I have attended, spoken at and organised virtual events. These have all been great, and I have learned a lot from the speakers and connected to people virtually. An in-person conference offers the additional possibilities of meeting face to face with all that brings. It will be great to meet people I have only met virtually in person.

The in-person conference is being held in central London so there is good transport access to the conference, and there is a remote stream for people who can not attend in person.

It will also be the first time that I have been Programme Secretary for a conference, and this is a role that is a great way to give something back to the testing community. I have enjoyed working with our committee to create an exciting programme with eighteen speakers. I am looking forward to meeting the speakers we have invited. We have speakers on artificial intelligence, accessibility testing, mental health, diversity, Cypress and contract testing. All the presentations are listed here: https://www.bcs.org/membership-and-registrations/member-communities/software-testing-specialist-group/conferences/testing-diversity-ai-conference/

BCS SIGiST committee members are volunteers and we have only met virtually so I am looking forward to meeting them too!

I am also enjoying working with the British Computer Society HQ who have been very helpful.

Lunch is included in the price of the in-person conference, and lunch will be a great opportunity to meet up with other testers!

The in-person conference will end with a networking event where we can share ideas and catch up with lots of great testing professionals.

We want to encourage young people to attend the conference and Under 30s can buy a ticket for the in-person conference for only £2

The conference has had to be postponed as unfortunately it was scheduled for the day of the Queen’s funeral.

The conference had to be postponed as unfortunately it was scheduled for the day of the Queen’s funeral.

The conference will now be on the 16th of November. How to book tickets and the schedule can be found on this link: https://www.bcs.org/membership-and-registrations/member-communities/software-testing-specialist-group/conferences/testing-diversity-ai-conference/

I hope to meet you at the conference!

How have approaches to quality changed over the past 20 years?

We can all learn a great deal from people who have influenced our industry. BCS SIGiST recently hosted a discussion with Mary and Tom Poppendieck about “How have approaches to quality changed over the past 20 years?”. Mary Poppendieck wrote books with her husband Tom about Lean Software Development based on her experience of applying the principles of the Toyota Production System to software development. The term “lean software development” originates from their books. Adam Leon Smith chaired the discussion which lasted an hour, and here are some highlights.

The discussion started with a conversation about what is quality, about how it relates to requirements. Mary questioned whether we should use the word integrity rather than quality. Mary and Tom spoke about their experience of how often specifications changed in manufacturing, and how often specifications change while creating products.

They spoke about how SpaceX is highly instrumented. This high level of instrumentation creates observability as the output of the instrumentation enables engineers to find the cause of test failures.

There was also a discussion about pull systems. Mary also spoke about how when she changed her factory from having a push system to having a pull system all the inventory disappeared. She also spoke about how better flow improved quality.

Covid has meant that we need to reconsider our attitude to remote work. The focus now needs to be on the outcome rather than the output. Tom spoke about how Open Source projects in the 1990s had this focus, so in that way, remote is not new. In these projects, people solved their own problems and took responsibility. You had to make sure that your contribution worked.

We need to make it difficult for mistakes to get into products and mosquitoes were used as a metaphor. When considering quality you need to think whether it is better to “swat all the mosquitoes in the house” or to “put up screens to keep the mosquitos out of the house”.

Tom spoke about how to get smart people to work together on complex problems. At SpaceX engineers are responsible for functions and teams include people who are good at finding problems. The whole team contributes to meeting the goal.

Tom Gilb was in the audience and asked a question about defect prevention.

I would recommend using the link below to listen to the whole discussion: https://www.bcs.org/events-calendar/2022/february/webinar-how-have-approaches-to-quality-changed-over-the-past-20-years/

How can we communicate what agile testers do?

A tester’s job is to test, well that is what our job title says, however, we add value in a number of ways to features that we work on. We need to be able to communicate how we add value.

Some time ago we had some new managers where I was working and I wanted to show the new managers how the testers added value. I needed to find a simple model that I could use to show what we did and I also wanted to involve the testers in showing how we add value.

The model needed to describe the role of an agile tester as we worked in agile teams and the testers were involved in every part of development. We were not testers who only tested work that was “finished”. We did a wide range of things to build quality into the product and so regarded ourselves as agile testers. I wanted the model to show the range of work that we did.

I searched and read but could not find a model that solved the problem for me. At the weekend I had a session with a spin bowling coach who showed a model to illustrate three aspects of spin bowling. I realised that you could describe a tester’s work as having three aspects and that I could adapt the model of spin bowling to be a model that described what testers do.

The spin bowling model enabled me to create a model that grouped testers’ skills in three categories: testing skills, soft skills, and hard skills. Hard skills would include work such as programming skills. I decided not to use the phrase “technical skills” to describe any one part of the tester’s skill set as testing skills and “hard skills” are both technical skills. In the model’s centre, I put a sentence describing what agile testers do. The model is shown above.

I discussed the model with the testers and together we added, under each heading, examples of the work we did. Under soft skills, we included things such as questioning and facilitating root cause analysis, and under testing skills, we included items such as boundary value analysis and equivalence partitioning. The items we wrote under the heading hard skills included test automation, Bash scripting, writing SQL and reading stack traces.

The model was useful in communicating to the managers what the testers did because it showed all that we did to add value and it fitted on one screen.

I have used the model more than once and found that it successfully communicated the range of activities that agile testers do. Please leave comments on this post on how you think the model can be improved.