How to deal with a complaint about quality

How a company responds to a complaint about quality needs careful consideration. A model that we can use to explore this issue is Dr Deming’s Red Beads Experiment. We can extend the Red Beads Experiment to include a complaint from customers. I explored the Red Beads Experiment in a previous blog post The blog post explained the steps in the experiment and the effect of the experiment on testers and quality. 

In the Red Beads Experiment, the White Beads Company makes white beads and struggles to reduce the number of red beads in the white beads it sells. We can use the Red Beads Experiment to help us explore how to deal with complaints about quality by extending the Red Beads Experiment to include customers complaining to the White Beads Company’s Chief Executive about quality. The customers are unhappy with many red beads being delivered in their orders of white beads. 

The White Beads Company holds its staff responsible for their performance in a process that they can not influence, so I would expect them to respond to customer complaints by blaming the staff. This will not help improve quality as the staff do not control the production process at the White Beads Company and cannot improve quality. Blaming the staff will also create an atmosphere of fear in the company which will make it harder to improve quality. If there is fear in the company people will not answer questions honestly as they will want to protect their job and their friends’ jobs and so quality will not improve. 

The White Beads Company should use Deming’s System of Profound Knowledge (SoPK) to improve quality. Deming wrote that SoPK “provides a map of theory by which to understand the organisation that we work in”[2]. SoPK has four components:

  • An appreciation for a system – systems thinking should be used to understand the problem and find a solution to it
  • Knowledge of variation – there will always be variation in quality, and the company must learn to understand it
  • Theory of knowledge – the company should develop a theory of what will happen in the future based on their actions to improve quality
  • Psychology –  the White Beads Company should understand individuals and how individuals interact as this will have an effect on the quality of the company’s output. 

An understanding of psychology and systems thinking will help the White Beads Company move from being a company where managers “find and record failures” to one where they “remove the causes of failure”[1].

The White Beads Company can use systems thinking to identify that the company and its suppliers are a system that makes white beads. It can then work to improve the quality of its white beads by working with its suppliers to improve the quality of the raw materials being supplied to the White Beads Company. 

Knowledge of the number of red beads in the raw materials will enable the company to evaluate the success of its initiatives to improve quality.

Improving the quality of white beads by improving the quality of raw materials, rather than blaming the staff, will help the company to drive out fear from the company. This will help people to work more effectively. Working with suppliers will also break down barriers which will create teamwork that will improve quality.

The company should then implement this plan using the plan-do-study-act cycle; first, it would plan the initiative, then it would carry out the plan, next, it would study the results of the planned work, and then act on what it found by taking its findings into another plan-do-study-act cycle. 

Using SoPK, and working iteratively with a plan-do-study-act cycle will enable the White Beads Company to improve the quality of its product. 

Are there other useful ways to extend the Red Beads Experiment?

Note:

I wrote a longer post about the Red Beads Experiment for LambdaTest Blogs, and John Willis, one of the founders of DevOps,  wrote a positive blog post in response to my post. He said in his post that Deming would be disappointed that people had not extended his Red Beads Experiment. This blog post is a suggestion on how to extend the experiment in a way that sheds light on how to deal with complaints from customers about quality and how Deming’s philosophy can help.

References

[1] W. Edwards Deming Out of the Crisis

[2] W. Edwards Deming The New Economics

More Information:

  • eLearning from the Deming Institute: DemingNext contains a course on the Red Beads Experiment

Using the Five Whys to improve quality

The Five Whys is a technique for finding the root cause of a problem. Toyota developed this technique and it is now widely used, including in software development. I was introduced to the Five Whys by Tom Gilb as part of a course he ran on Lean QA and have used the Five Whys in several places where I have worked. Testers want to improve quality, the Five Whys is a technique that improves quality so it is useful for testers to be able to use it. 

The Toyota engineer Taiichi Ohno would ask why five times to find the root cause of a problem as this would enable him to create and implement appropriate countermeasures[1]. Asking why five times usually will find a “soft” issue as the cause of the problem, for example:

  1. Why did the customer’s file fail to load in production? Because there was a bug in the loader
  2. Why was there a bug in the loader? Because a feature branch that contained the bug was deployed to production
  3. Why was the feature branch with the bug deployed to production? Because it passed all the automated tests 
  4. Why did the feature branch pass all the automated tests? Because the database used by the integration tests had not been updated
  5. Why was the database not updated? Because there was no process to update the database

In this example, the root cause of the problem is that the database used by automated tests had not been kept updated. The countermeasure is to create a process that keeps the database updated.

The Five Whys can be used to find the root cause of a problem. The example above shows it being used to find why a bug was introduced into production, the technique can also be used to find the cause of an incident in production or it can be used to find the root causes of multiple problems such as when Toyota used the Five Whys to reduce the noise of the Lexus engine[2]. Teams should use the Five Whys to keep asking until the root cause(s) are identified. In one of the examples in The Toyota Way[2] why has been asked six times. When using the Five Whys you may find the root cause after asking why four times, five times or more times. 

A development team can meet and work through the Five Whys together as a team. It can be useful to vary how the team uses the Five Whys and a team can also work through the Five Whys individually and then discuss their findings as a team. 

Finding the root cause of a problem is a constructive way of dealing with problems that does not involve blame and so helps teams to grow. When using the Five Whys individuals can feel that they are having to admit to a mistake and find this difficult.  In order to give people confidence, it can be hopeful to read the prime directive when starting a Five Whys meeting. When first using the Five Whys it can also be helpful if senior members of the team l say that they make mistakes as this can empower other members of the team. It can be unhelpful if senior managers want to join a Five Whys meeting as people may feel inhibited from talking about errors that have been made if a senior manager is present. 

The Five Whys is a positive way of dealing with problems that can contribute to creating psychological safety because it enables team members find a way to deal with problems without blame. A blame culture prevents a team from finding the real issues behind problems as people will want to protect themselves and their friends. People who have been working in a blame culture can find using the Five Whys invigorating and empowering. 

Once a team has found the root cause of a problem they should adopt a countermeasure so that the problem does not re-occur. Implementing a countermeasure to prevent a problem from reoccurring will improve quality.

I would be very interested to hear your experiences of using the Five Whys.

References

[1] Lean Thinking: Banish Waste and Create Wealth in your Corporation James P. Womack and Daniel T. Jones

[2] The Toyota Way Jeffrey K. Liker

Testing needs to include the needs of internal and external customers

When we test we think about the users of the functionality and we include their needs in our testing. Creating categories of customers, such as internal and external, can help us understand our customers’ uses of the functionality. 

Dr Joseph Juran advocated viewing customers as either internal or external customers. Juran was born in Romania and emigrated to the USA. He, like W. Edwards Deming, worked with Walter Shewhart and helped the Japanese economy recover from World War Two. He also taught quality control at New York University and worked with Steve Jobs. 

Dr Juran said that identifying customers is the first step in planning for quality[1]. He defined a customer as “someone who is impacted by the product”. He also said that customers may be external or internal. 

An external customer is a customer who is external to the company or organisation. They may be a paying client, government regulatory bodies, international regulatory bodies or the public.

Departments and people within a company often supply products to one another, these are internal customers. They are not customers in the traditional sense as they are not paying clients, but they are often called customers. 

Juran recommends creating a flow diagram to help identify customers. The flow diagram can show the steps involved in the process and help identify customers[1].

He says that customers can also be divided into classes. If there are many customers the Pareto principle can be used to classify customers into the vital few and the trivial many as this helps you to focus your efforts where it will have the most impact.

It is important to consider how the functionality we are testing impacts internal customers as the functionality we are developing will impact others within the company. Examples of this would be that the functionality could create additional work for the customer service team if functionality is complex.Sales and marketing are also impacted by the features a team is working on and it is useful to view sales and marketing as customers so that they are kept informed of new features. 

Juran’s way of looking at external customers is useful too as maybe a playing client uses your product to provide a service to someone else such as “the public”, and your testing can then also take that other person’s needs into account. The success of the application that you are testing may depend on the quality of the service being provided by your paying client to hi/her paying clients.

I find that this categorisation of customers is useful because it provides a structure that enables me to think about external customers like clients but also about internal customers like customer service. 

References:[1] Juran’s Quality Handbook Fourth Edition

Should we practice continuous learning?

I enjoy working in an organisation where we are learning from the work we do. An example of this would be using retrospectives to enable the team to learn from their work and taking this knowledge forward to help the team. 

Until recently I have called this approach continuous learning. It sounded right, as it felt to me that agile and lean were about doing things continuously, such as continuous deployment.

I recently spoke with Kevin Cahill, and during the conversation, I used the phrase continuous learning. He told me that he had used the phrase ‘continuous learning’ when talking with his grandfather, W. Edwards Deming. His grandfather had told him that when talking about learning he should use the word continual not continuous and that he should look up in the dictionary the meaning of the two words.

When I looked up the meaning of the two words on google found: 

Continuous:

“forming an unbroken whole; without interruption.”

Continual:

“forming a sequence in which the same action or event is repeated frequently.”

There is an important difference between the two words. Continuous learning would mean learning without interruption. This style of learning would be mechanical with no let up and no time to think. Continual learning would involve repeated learning and allows time for study and reflection.

The difference between continuous and continual is not just semantics. It is about an approach to learning.  

We use continuous to describe automated processes such as continuous integration and continuous deployment. These processes are carried out by machines and so can not benefit from studying or reflecting as we do

 An example of continual learning would be, if we work in an iterative way using a plan-do-study-act cycle, we may have to wait a while before we can study the outcome of our work. Our learning is continual because we have had to wait, and during that time we may reflect on the work we did.

I now think of learning as being continual, not continuous.

I am learning about systems thinking to help me test

I want to improve my understanding of systems thinking as this will help my testing. My friend John Holmstrom recommended the work of Peter Senge and so I have just read “The Fifth Discipline” by Peter Senge. The book contains many useful insights about system thinking, learning and leadership. Some of the points about systems thinking that I have taken from the book to help me with testing are:

  • Systems thinking is a discipline for seeing wholes
  • Don’t be your role. Don’t say “I am my position”. I am a tester and I should see beyond my role by using systems thinking. This way I can see what processes are influencing the feature I am testing and what processes will be influenced by the feature I am testing. Understanding these influences enables me to bring them into my testing and so test the feature in new ways, for example, I can think about who the customers’ stakeholders are and how they use the feature.
  • The team I work in is not autonomous therefore it is not a closed system, so I should continually learn about how it is influenced by and influences other systems.
  • The diagrams in the book that showed how systems affect something can be used to create discussion about the systems that influence the success of a new feature. The illustrative diagram in this blog post is one I have created based on what I have seen in the book. The diagram shows how the benefits of the new feature are influenced by the customers existing systems and their use of the new feature. 
  • We should not ignore how our team’s decisions affect others
  • Don’t blame external factors for failures as external factors are part of the system that you are part of.
  • Everyone shares responsibility

I have used systems thinking to help me think about how customers are using the product and I am going to continue to learn how to use systems thinking. There is so much more to learn and so much more in the book!

Further Reading:

Being the Programme Secretary for a testing conference is an exciting experience

I have always gained a lot from attending conferences, and it has now been over two years since I attended a conference in person. I am now enjoying being Programme Secretary for BCS SIGiST’s Testing, Diversity, AI Conference. 

Over the last two and a bit years I have attended, spoken at and organised virtual events. These have all been great, and I have learned a lot from the speakers and connected to people virtually. An in-person conference offers the additional possibilities of meeting face to face with all that brings. It will be great to meet people I have only met virtually in person.

The in-person conference is being held in central London so there is good transport access to the conference, and there is a remote stream for people who can not attend in person.

It will also be the first time that I have been Programme Secretary for a conference, and this is a role that is a great way to give something back to the testing community. I have enjoyed working with our committee to create an exciting programme with eighteen speakers. I am looking forward to meeting the speakers we have invited. We have speakers on artificial intelligence, accessibility testing, mental health, diversity, Cypress and contract testing. All the presentations are listed here: https://www.bcs.org/membership-and-registrations/member-communities/software-testing-specialist-group/conferences/testing-diversity-ai-conference/

 BCS SIGiST committee members are volunteers and we have only met virtually so I am looking forward to meeting them too!

I am also enjoying working with the British Computer Society HQ who have been very helpful.

Lunch is included in the price of the in-person conference, and lunch will be a great opportunity to meet up with other testers!

The in-person conference will end with a networking event where we can share ideas and catch up with lots of great testing professionals.

We want to encourage young people to attend the conference and Under 30s can buy a ticket for the in-person conference for only £2

The conference has had to be postponed as unfortunately it was scheduled for the day of the Queen’s funeral.

The conference had to be postponed as unfortunately it was scheduled for the day of the Queen’s funeral.

The conference will now be on the 16th of November. How to book tickets and the schedule can be found on this link: https://www.bcs.org/membership-and-registrations/member-communities/software-testing-specialist-group/conferences/testing-diversity-ai-conference/

I hope to meet you at the conference!

How have approaches to quality changed over the past 20 years?

Mary and Tom Poppendieck

We can all learn a great deal from people who have influenced our industry. BCS SIGiST recently hosted a discussion with Mary and Tom Poppendieck about “How have approaches to quality changed over the past 20 years?”. Mary Poppendieck wrote books with her husband Tom about Lean Software Development based on her experience of applying the principles of the Toyota Production System to software development. The term “lean software development” originates from their books. Adam Leon Smith chaired the discussion which lasted an hour, and here are some highlights. 

The discussion started with a conversation about what is quality, about how it relates to requirements. Mary questioned whether we should use the word integrity rather than quality. Mary and Tom spoke about their experience of how often specifications changed in manufacturing, and how often specifications change while creating products.

They spoke about how SpaceX is highly instrumented. This high level of instrumentation creates observability as the output of the instrumentation enables engineers to find the cause of test failures.

There was also a discussion about pull systems. Mary also spoke about how when she changed her factory from having a push system to having a pull system all the inventory disappeared. She also spoke about how better flow improved quality.  

Covid has meant that we need to reconsider our attitude to remote work. The focus now needs to be on the outcome rather than the output. Tom spoke about how Open Source projects in the 1990s had this focus, so in that way, remote is not new. In these projects, people solved their own problems and took responsibility. You had to make sure that your contribution worked.

We need to make it difficult for mistakes to get into products and mosquitoes were used as a metaphor. When considering quality you need to think whether it is better to “swat all the mosquitoes in the house” or to “put up screens to keep the mosquitos out of the house”.

Tom spoke about how to get smart people to work together on complex problems. At SpaceX engineers are responsible for functions and teams include people who are good at finding problems. The whole team contributes to meeting the goal.

Tom Gilb was in the audience and asked a question about defect prevention.

I would recommend using the link below to listen to the whole discussion: https://www.bcs.org/events-calendar/2022/february/webinar-how-have-approaches-to-quality-changed-over-the-past-20-years/

How can we communicate what agile testers do?

A tester’s job is to test, well that is what our job title says, however, we add value in a number of ways to features that we work on.  We need to be able to communicate how we add value.

Some time ago we had some new managers where I was working and I wanted to show the new managers how the testers added value. I needed to find a simple model that I could use to show what we did and I also wanted to involve the testers in showing how we add value. 

The model needed to describe the role of an agile tester as we worked in agile teams and the testers were involved in every part of development. We were not testers who only tested work that was “finished”. We did a wide range of things to build quality into the product and so regarded ourselves as agile testers. I wanted the model to show the range of work that we did.

I searched and read but could not find a model that solved the problem for me. At the weekend I had a session with a spin bowling coach who showed a model to illustrate three aspects of spin bowling. I realised that you could describe a tester’s work as having three aspects and that I could adapt the model of spin bowling to be a model that described what testers do.

The spin bowling model enabled me to create a model that grouped testers’ skills in three categories: testing skills, soft skills, and hard skills. Hard skills would include work such as programming skills. I decided not to use the phrase “technical skills” to describe any one part of the tester’s skill set as testing skills and “hard skills” are both technical skills. In the model’s centre, I put a sentence describing what agile testers do. The model is shown above.

I discussed the model with the testers and together we added, under each heading, examples of the work we did. Under soft skills, we included things such as questioning and facilitating root cause analysis, and under testing skills, we included items such as boundary value analysis and equivalence partitioning. The items we wrote under the heading hard skills included test automation, Bash scripting, writing SQL and reading stack traces. 

The model was useful in communicating to the managers what the testers did because it showed all that we did to add value and it fitted on one screen.

I have used the model more than once and found that it successfully communicated the range of activities that agile testers do. Please leave comments on this post on how you think the model can be improved.

Don’t ask “Why did the tester miss that bug?”

“Why did the tester miss that bug?” is a question I have heard in many places but it is the wrong question. W. Edwards Deming’s philosophy shows that if something has gone wrong we need to look at the process and not blame the individual. A tester can be working in a system that the tester can not change and that creates software that contains bugs. If there are bugs we need to look at the system that produced them.

The “Red Beds Experiment” was regularly conducted by Deming. He first ran the “Red Beads Experiment” in Japan in 1950. The book ”Four Days with Dr Deming” gives an example of how the Yawata Steel Company benefited from changing its process as a result of seeing the “Red Beads Experiment”. In the 1980s and early 1990s, Deming ran the ”Red Beads Experiment” in the USA as part of his seminars for executives.

In the “Red Beads Experiment” Deming asked members of the audience at his seminar to act as Willing Workers, a Secretary, and Inspectors in a factory that makes white beads. Unfortunately, there are red beads, as well as white beads, in the raw material that is delivered to the factory. The Willing Workers are to remove the white beads with a paddle and are given detailed instructions on how to do so. The Inspectors count how many red beads have been removed on the paddle and the Secretary records the results. Each scoop of the paddle by a worker represents a day’s work. Unfortunately, because the raw material contains both red and white beads each scoop of the paddle also contains red and white beads. On day four management is disappointed at the number of red beads being collected in the paddles and develops a plan. The plan recommends keeping only the best workers, so the workers with the worst results that day are laid off. The remaining staff are warned that unless results improve the factory will be closed. On day five the results have not improved as red beds are still being scooped up with white beads, so the factory is closed.

In “The New Economics” Deming lists fourteen lessons from the experiment. One of the lessons is that the Willing Workers were victims of the process as they could not, under the manager’s rules, improve their performance.

Software quality and testers can be victims of the process, like the Willing Workers.  We should not ask “Why did the tester miss that bug?”, we should instead ask how to improve the process.

More Information:

  • eLearning from the Deming Institute: DemingNext contains a course on the Red Beads Experiment

State transition diagrams help to define tests

State Transition Diagram of sports matches

Identifying what states exist in functionality and how changes in state take place enables testers to identify tests that should be executed. State transition diagrams provide simple notation for summarising states and how changes in states take place. I recently used a state transition diagram on a project and it helped me identify the tests to execute.

To create a state transition diagram you need to first identify objects that may be in one of several states depending on what events have occurred. An event causes an action which may change the state of the object. 

The above state transition diagram shows the states of sports matches and the transitions between them. The notation used in the diagram above is:

  • Circles represent to the states that the system may be in
  • Labelled arrowed lines from states show events that cause actions
  • Text on a line represents actions that are taken when events occur
  • Arrowed lines from actions show a change to another state

The state transition diagram identifies for us the states of matches and the changes that cause matches to change state. We can then  test all of the states and test all the actions that cause the matches to change state. This gives us a level of test coverage and so gives us confidence in our testing.

Many charting tools that support the creation of state transition diagrams have free plans.I used Lucidchart for the diagram in this post. 

I find that creating a state transition diagram is an iterative process from which I learn about the functionality I am to test. Whenever I come back to the diagram I review how the diagram represents the functionality and this helps me get a better understanding of the functionality that is to be tested. As I get a deeper understanding of the functionality I update the diagram and in this way creating a state transition diagram helps me gain a fuller appreciation of the functionality my team is developing. 

The diagram can also be shared and reviewed. Sharing the diagram helps collaboration within your team. Something I learned from using a state transition diagram on a recent project was that the diagram is easier for the rest of the team to use if, instead of having one crowded diagram, you break the functionality down into several clear diagrams.

Further reading: Computer Science 4th Edition by C. S. French

Three Things That Have Helped Me in My Testing Career

The Ministry of Testing Bloggers Club has asked bloggers to write a post that describes t three things that have helped them in their testing career. Many things have helped me in my testing career, and here are three of them:

  1. The Testing Community

The Testing Community has helped me throughout my testing career. I started working as a tester over twenty years ago. Back then I remember there being two testing newsletters and BCS SIGiST conferences in Grosvenor Square. The conferences and newsletters introduced me to other testers and to resources that helped me develop my testing skills.One way I have found conferences useful is that I go to the conference with a problem I was trying to solve and ask people if they have the same problem. I have often found that other testers have a similar problem, and that a discussion with them helped to find a solution. 

Today there are so many more organisations and forums for testers, such as the Ministry of Testing. I am now working as a lone tester and being able to share ideas with other testers is a great help and support for me. The online discussions, the newsletters and and events all help me learn new ideas and validate those that I have developed.

During lockdown the on-line testing community was a great help by providing ideas, discussion and support. 

So, I would like to say a big thank you to all the testers who organise events, speak at events, take part in online discussions, write blogs, write courses, write articles, review articles, write newsletters, contribute to forums, make podcasts, make videos, mentor people wanting to become testers and generally support the testing community.

  1. Continual Learning

Testers need to be hungry to learn new things, and I enjoy learning new skills. There are fundamental principles in testing such as “building quality in”, rather than trying to test quality in. Reading books by W. Edwards Deming has helped me get a better understanding of these principles. How these principles can be applied is something which I continue to explore and learn about.

We also need to keep learning new ways of working and new techniques. I am continuing to learn how to use new tools, expand our technical knowledge, and develop our soft skills .

  1. Great Colleagues

Great colleagues mean that you can get so many positive things from work. I have learned so much from people I work with and continue to do so. I have learned new testing skills, new programming skills, learned about agile and lean, learned about CI/CD and learned new soft skills. 

Which programming languages should a tester learn?

It is helpful for a tester to be able to program as it enables a tester to get a better understanding of the application they are testing. Being able to code also helps create test data, automate tests and build a better relationship with developers.

But which programming language(s) should a tester learn? Over the last year or so I have learned some javascript. This has helped me work with front end developers, automate tests  and create test data. To help backend developers I used Node.js to automate tests for the API, but was this the best thing to do?

Javascript is used by frontend developers. If I want to support developers I should use the programming language that they use and not use Node.js. Our backend developers use golang and for this reason I have started to learn golang. I am using the go track on exorcism.org to learn golang. Exercism is free, and I like its style of learning. As an exercism.org student you write solutions to exercises, and the exercise passes if the tests for the exercise passes. It is sort of TDD style learning! You can also ask exercism.org for a mentor.

I have also created PR’s for my solutions to the exercises and got feedback from backend developers on the golang code that I have written. This has been really useful as it shifts my perspective from just getting the solution to the exercise to pass the tests to a perspective of trying to write good code.

Other free tools that are helping me to learn golang are:

Golang is quite different to javascript as it is a compiled language that is statically typed, and I am learning how these differences affect developing code. Learning golang is expanding my technical knowledge and is beginning to help me to work with backend developers.

Learning golang will also enable me to read the code that I am testing and pinpoint where in the code a bug is occurring.

It is great that where I work gives me time and support to learn new skills.

Being able to code in javascript has helped me as a tester, being able to code in golang will help me too when I work with backend developers to test API’s and create test data.

Testing messaging with a decision table

Decision Table

Recently I was testing the messages sent from an app to the user when I noticed that the messages were dependent on the interaction of three conditions. I used a decision table to analyse the messaging because decision tables show how many tests are required to test something that is affected by a group of conditions . 

In an earlier post I gave the steps for creating a decision table.

I had thought that decision tables would be useful for testing API endpoints. I have learned that testing messaging would be another use for decision tables. In the scenario I was testing there were three conditions. I found that all the three conditions could be true or false and that the message contained varied depending on whether each of the three conditions were true or false. Analysing this problem was most easily done with a decision table as the actions defined by the table would be the tests for the messages the app creates. 

Decision tables have one row for each condition. The number of rules in a decision table is 2 to the power of the number of conditions. There were three conditions so I would need eight columns for the rules. The actions in the table would cover all the test cases. What I needed to do was ensure that I had listed the conditions correctly as the decision table would then show me how many actions were required to test the messages. See the image above for an example of the decision table.

When I had created a decision table and I could execute the tests in the table. The decision table would ensure that I defined tests for each of the possible variations of the three conditions.

I shared my decision table with my team to help them in their work as I don’t want to find bugs, I want to help prevent them. 

Using a decision table had a number of advantages. It ensured that the tests covered all the functionality as I knew that I had covered all the conditions. It also gave me something I could share with the developers and help them in their developer testing which would mean that there are less bugs for me to find.

Further Reading: Computer Science 4th Edition by C. S. French

A review of “Artificial Intelligence and Software Testing”

“Artificial intelligence and Software Testing” is edited by Adam Leon Smith and has recently been published by the British Computer Society. The book covers both using Artificial Intelligence (AI) to test software and testing AI systems. The book will help testers who want to learn about AI and testing. 

Rex Black writes in the introduction about how AI is a wave of change and that we need to test AI in order to learn if we can trust AI. Testing AI provides new challenges as AI changes its behaviour in response to testing, and we need to rethink many aspects of testing.

In the first chapter Adam Leon Smith writes about AI trustworthiness and quality. He writes about the importance of being able to trust AI, about quality problems such as understanding how an AI system optimises itself, and also gives useful examples.

Quality and bias are the subjects of the second chapter by James Davenport. The chapter is an exceptionally good explanation of the issues relating to bias in AI. The chapter covers the difference  between bias and fairness and issues with bias in AI.The examples given in the chapter about bias in AI and in everyday life give a deeper picture of the issues relating to bias. 

The fourth chapter is about testing machine learning (ML) systems. In this chapter Adam Leon Smith writes about the role of the tester, the nature of machine learning systems and what testing metrics and testing techniques are appropriate for ML systems. The role of a tester in ML is described in a way that a tester in a team that does not use AI can relate to. The examples of testing metrics and techniques are different to those in non AI projects and so are really interesting.

Jeremias Rößler writes about AI based test automation in the next chapter. He writes about the problems in using AI for test automation and also covers AI being used to generate unit tests and UI tests. 

Ontologies for software testing are described by Joanna Isabelle Olszewska in the sixth chapter. She explains what ontologies are, how they relate to AI and gives examples of several ontologies for software testing.

In the seventh chapter Jonathan Wright describes a shift-right approach to testing called “digital twin testing” which he used when collaborating with MIT. This approach was used when Jonathan led the QA and testing efforts for the COVID Pathcheck Foundation.


Many testers are now thinking about how to use AI to assist testing and how to test and this book will help them. Reading this book is a great way to become better informed about AI and software testing. You can get your copy from: https://shop.bcs.org/store/221

I have changed my mind about…Test Plans and Test Documentation

When I started as a tester I learned that test documentation, such as test plans, needed to gain approval from the Director and could be regarded as a project deliverable. I have learned that this is not helpful and that test documentation should support testing and the development.

I remember that, before testing started, we wrote detailed test plans for each project. This documentation included test plans that conformed to IEE standards and all the documentation was written in Word docs. These plans needed the approval of the director but I am not sure that anyone ever read them apart from the testers. We used the plans to test the new project but getting the directors approval for the documentation was really very important. The testers discussed whether test documentation was a project deliverable as these documents had to be produced for every project and needed the approval of the director. I found that as a project progressed I learned more about the project and that this learning was not included in the documentation as the documentation had already been written. As I explored the functionality I frequently found bugs from tests I ran that were not included in the tests I had written in the test plans. 

Now I only create documentation if it helps testing and development. I have learned to have test documentation that supports testing as this way I can better support the development of the feature I am testing. I create test documentation such as mind maps, spreadsheets, decision tables, flow charts and state transition diagrams that help me analyse the features being developed. This type of documentation helps me test and can help the developers, designers and product owners too. I also create test strategy documents for a number of reasons which include that writing the strategy enables me to clarify my thoughts and enables me to share ideas. The test plans I create now are to help me test and to help me collaborate with the other members of my team. Documentation is no longer thought of as a project deliverable. The deliverable is a high quality product and the test documentation helps the team to achieve this.

I have changed my mind about documentation.It should not be about gaining the directors approval or being a project deliverable. Documentation should be about supporting testing and development because that way it can help improve quality.

Create your own website, free on WordPress.com.Create your own website.
Start free