Why does testing take so long?

Sometimes testers are asked ‘why does testing take so long?’ The question should not make testers feel defensive. We should always be looking to improve our test process. Testers can use a technique from Toyota to answer the question and improve our process. 

Testers want their testing to flow easily from one task to the next, whether exploratory testing or test automation. Testers don’t want to be delayed or held up but sometimes, something causes delays. It could be that configuring the automation environment is fiddly, or it could be that getting an answer on how a piece of functionality should work holds up your testing or it could be something else.

The problems holding up your testing are problems and you want to remove so that your testing can flow smoothly. Removing the problem that is slowing testing the most should be where to start because it will have the biggest impact. 

To help you choose which problem you should work on removing first you could use the Lean metaphor of rocks slowing the flow of a river. Taiicho Ohno at Toyota wanted to “focus the process improvement efforts”[1] at Toyota because they would have the biggest effect on improving the flow of work. He used the size of rocks in a river as a metaphor to enable him to find the biggest problem to remove: “rocks are the problems disturbing the flow. There are many rocks at the bottom of the river and it takes time and effort to remove them. The question is which rocks are important to remove. The answer is given by reducing the water level; those rocks which emerge above the water are the ones that should be removed.”[1]

Testers want the process of testing to flow smoothly. Testing can be viewed as a flowing river, and the problems that are slowing the flow of testing can be viewed as the rocks that are slowing the river’s flow. If testing is viewed in this way the ‘rocks’ are the problems that are causing testing to take so long.

The ‘rocks’ that are slowing testing should be identified. An experiment can be conducted, as Ohno suggested, in which the amount of testing is reduced so to see which problems are affecting the flow of testing. Once identified, the problems can be analysed to find the largest ‘rock’ which ‘emerges above the water’. Toyota used the Five Why’s to identify the root causes of problems that were slowing the flow of work.[1] 

The testers may not have the authority to make the changes required to remove the largest rock which is slowing the flow of testing. Management should help the testers plan how to remove the largest ‘rock’ so that testing flows more smoothly and does not take so long. The largest ‘rock’ can then be removed and testing will flow more smoothly. 

I am using this approach to reduce the time taken to run automated test packs by analysing the tests to find the ‘rocks’ slowing them down. I am looking for the largest ‘rock’ so that it can be removed.

Testers should find the problems that slow testing and focus improvement efforts on removing the largest problem.

I’d like to thank the Profound Book Club for introducing me to Standing on the shoulders of Giants by Eli Goldratt.

References

[1] Standing on the shoulders of Giants by Eli Goldratt (2006, p6)

I am a student of testing

I have been a tester for twenty years. I have learned to think of myself as a student of testing and I am a better tester for it.

I am a tester and, as a tester, I learn nearly every day. Over the last week, I have learned some new aspects of the Playwright API. I have also learned more about feedback loops and systems from a lecture by Donella Meadows.

I am also a qualified cricket coach and ‘a student of the game’. The phrase ‘a student of the game’ is something I like about cricket because it shows the humility to learn. The phrase “ a student of the game” is particularly powerful when used to refer to international cricketers because it shows that they still have the humility to learn.

Being a “student of the game” requires humility because learning requires humility. In this sense, humility is not a virtue. It is also not about being humble as a way of being ‘good’. Humility is a principle because it is a necessity for learning.  

Testers need the humility to learn continually because testing is about learning. When we are testing we are learning about the product we are testing, our customers, testing, test automation and more. We need humility to learn from the people we work with, customers and the testing community. 

I am a student of testing.

I would like to thank Dennis Sergent for getting me to think about this.

Do outages have to be the new normal?

Yesterday I was using a testing tool and it had an outage. Today I was automating a test when a third party had an outage and delayed my test automation. Even the third party’s Status page was not functioning. Social media was full of people complaining about the outage. Outages impacted two consecutive days of my work. This made me wonder if outages are the new normal. The following day my news feeds were full of a story of a bank whose customers could not access their accounts due to an outage.

What has happened to software quality? Companies are under pressure to cut costs and so they improve productivity. Some companies have cut their QA function. 

The objective of a strategy to improve productivity is to improve productivity which does not, in itself, improve quality. After changes to improve productivity, what percentage of changes to production or released to users result in degraded service?

Improving quality creates a chain reaction that improves productivity[1]. If quality improves there are fewer bugs and outages to be fixed. If fewer bugs and outages are being fixed, productivity improves and new features can be added more quickly. If more new features are being added, and less time is spent fixing bugs, then each new feature costs less. If you can add features at a lower cost, you can capture the market.

W. Edwards Deming used improving quality as a strategy to help rebuild the Japanese economy after World War Two and later to make Ford the most profitable car manufacturer in America[2]. 

Improving quality is a proven business strategy which also improves productivity. “Improvement of quality begets naturally and inevitably improvement of productivity”[3]. Outages should not be the new normal for your customers if your company’s strategy is to improve quality (and your company’s productivity will improve too).

Testing professionals can help companies strategies to improve quality because they know how to:

Testing professionals can help you improve quality, which creates a chain reaction that improves productivity. Outages do not have to be the new normal.

References

[1] Out of the Crisis by W. Edwards Deming (1986, p1)

[2] Deming’s Journey to Profound Knowledge by John “Botchaglaupe” Willis with Derek Lewis (2023, p133)

[3] Out of the Crisis by W. Edwards Deming (1986, p2)

Make your Playwright tests run faster by using the Playwright API to wait

There are times when automating a test in Playwright that the test needs to wait because the test will flake if it does not wait for something such as an event. It can be, for example, that you are waiting for a navigation to complete. 

Tests can be made to wait with a ‘wait’ for a given period of time, say five seconds.  If this is done the tests will always wait for that period of time, whether or not it is necessary and the tests will be slower than they need to be. 

It is better to have tests wait for something specific such as a locator to be rendered, an event or an API call because the test waits for the locator or event. If a test uses this type of wait, it only waits as long as necessary for the specified condition to be met. Also, the tests should not flake and run more quickly.

I have found the Typescript functions in the Playwright API described below to be useful ways of waiting when writing Playwright tests:

Waiting for a locator – waitFor()

The function waitFor() can be chained to a locator to wait for that locator to be rendered, for example:

I have found waitFor() useful when waiting for navigation to complete because the test will wait for the specified locator.

Arguments for the state of the locator being waited for and for the length of the timeout can be passed to waitFor() for the state that is being waited for: https://playwright.dev/docs/api/class-locator#locator-wait-for

Waiting for an API call – waitForResponse()

A test may need to wait for the response from an API call. This can be done with waitForResponse(). In this example, waitForReponse is waiting for a response after click().

The API call that the test needs to wait for can be found in the network tab of Dev Tools and is then passed as a parameter to waitForResponse(). The function waitforReponse() returns the matched response.

Arguments can be passed to WaitforResponse() for the predicate and the timeout: https://playwright.dev/docs/api/class-page#page-wait-for-response

Waiting for an event – waitForEvent()

Sometimes the test is not waiting for an API call or the presence of a locator, it is waiting for an event, such as ‘requestfinished’ or ‘domcontentloaded’. WaitForEvent waits for the event to fire and returns a truthy value. In the example below the test is waiting for a ‘domcontentloaded’ event that occurs after navigate().

Arguments can be passed to waitForEvent() for the event, predicate and timeout: https://playwright.dev/docs/api/class-page#page-wait-for-event

Waiting for a truthy value – waitForFunction()

If a test needs to wait for a truthy value, such as the value returned by querySelectorAll() you can use waitForFunction();

The function will wait up to the length of time specified in the timeout for truthy to be returned. This example returns truthy when length is greater than zero and falsey when length is zero.


Arguments can be passed to waitForFunction() for the function, evaluation argument, polling and timeout: https://playwright.dev/docs/api/class-page#page-wait-for-function

Conclusion

This post describes the functions that I am using to make tests wait. The Playwright API contains more functions that can be used for waiting than I have shown above. If you find that the functions in this post do not meet your needs it is worth exploring the Playwright API to find different ways of making tests wait. 

It is easy to get in the habit of using waits in our test that wait for a given number of seconds. However, waiting for a given number of seconds is a habit that will slow your tests down. 

If you use functions like those above to make your tests wait, the tests will only wait as long as necessary for the condition to be met and so run faster. 

“The Purpose of Analysis is Insight”

Testers want to analyse the product and process to improve quality. Once every engineering team had a statistician, that is not the case today. Testers can help their team by using statistics, such as control charts, to do analysis. There are a lot of resources to help us use statistics, for example, this blog post is based on a chapter from Understanding Variation: The Key to Managing Chaos. Control charts are a form of statistics that provide insight into processes and were part of the philosophy that W. Edwards Deming used to rebuild the Japanese economy and turn Ford into America’s most profitable car manufacturer.
A few years ago I was Test Lead in a company that treated each bug it needed to fix as an incident. Each time we fixed a bug we did root cause analysis with the Five Why’s, and I kept a line chart showing the count of how often we needed to fix a bug. The chart could have looked something like this:

Line chart of the count of bugs needing to be fixed

The line chart shows that there is never more than one bug at a time and bugs become more frequent towards the end of the year. The line chart does not provide deep insight. 

I can display the count of bugs in a control chart. The control chart has three additional lines, the average (mean) plus upper and lower limits, which are calculated from the time series data. Viewing the count of bugs in a control chart gives me a little more insight. The chart shows that the time series stays within the upper and lower limits, which means that the number of bugs is under statistical control.

Control Chart of the count of bugs needing to be fixed

A bug being fixed in this company is rare data. Instead of counting the number of bugs fixed, I can measure the average number of bugs fixed in a time period. I calculated the number of bugs per month in a spreadsheet as below:

I created an XMR chart with this data. An XMR chart consists of an X chart and an MR chart. An X chart is a control chart and is called an X chart because x̄ is the symbol for average. The MR chart contains the moving ranges. The moving ranges are the differences between the data points in the time series in the X chart.

XMR charts are relatively easy to create. They are based on the time series data in the X chart. Moving ranges are the differences between the data points in the X chart time series data and can be calculated using an abs function. The averages in both charts can be calculated using an average function. The upper and lower limits in the X chart and the upper limit in the moving ranges are calculated using a simple equation on Pages 40-41 in Understanding Variation: The Key to Managing Chaos.

The X chart shows that the number of bugs per month rose during the year, then fell and that it rose close to the upper limit towards the end of the year. The upper and lower limits are not targets or goals that have been added to the chart, they have been calculated from the data and so are the voice of the process. If the data points go outside the limits the process is said to be out of statistical control. The Moving Ranges chart shows that the rate of increase rose twice.

XMR Chart of the rate at whcih bugs need to be fixed

The analysis provided by the XMR chart provides insight regarding the rate at which bugs need to be fixed. 

The X chart shows that the rate of bugs needing to be fixed is increasing. The increases are occurring within the upper and lower limits. Changes in the time series outside the upper and lower limits are due to special causes, that is fleeting events. Change within the upper and lower limits are due to common causes that is they are the faults of the system. The increase in the rate at which bugs need to be fixed is within the upper and lower limits and so is a fault of the system. 

The Moving Ranges (MR) chart shows two dates when the increases occur. 

The company can use this analysis to decide if it wants to find and address the causes of the increases. It may decide that because the rate of fixing bugs is under statistical control it has more important issues to deal with. However, if it does want to find and address the causes of the rate of increase the analysis provided by the XMR chart will help.

“The purpose of analysis is insight”[3] and control charts provide useful insights. A better insight into rare data, such as the fixing of bugs for this company, is gained by analysing the rate of fixing bugs rather than by counting the bugs. “In general, counts are weaker than measurements”[2].

Testers can use XMR charts, including control charts, to enable their team to get insights into their product and processes.

Thank you Rob Park for prompting me to explore and write this.

Here are some resources that will help you if you would like to use XMR charts and control charts to gain insights:

Introductory guides to control charts:

A useful guide if you want to create an XMR or control chart:

A video introduction to control charts:

An API that can be used to create control charts

References

[1] Understanding Variation: The Key to Managing Chaos by Donald J. Wheeler (1993, p102)

[2] Understanding Variation: The Key to Managing Chaos by Donald J. Wheeler (1993, p104)

[3] Understanding Variation: The Key to Managing Chaos by Donald J. Wheeler (1993, p33)

What training did you get when you became a Test Manager?

From conversations I have had with friends in the UK and the US, I have found that it is a common experience for people not to receive management training when they first become test managers.

I received training as a manager and have learned that I was fortunate to do so.

A test manager or test lead is an important role. You are a line manager and are responsible for testing. When you become a test manager or test lead you get a new job title and new responsibilities, but, unfortunately, I think it is rare for you to receive training.  

It is interesting to contrast becoming a test manager with a voluntary role in sport. I managed my children’s cricket teams. Whilst in this role I had to complete a course on how to coach that took ten weekends plus two online courses. When I moved up a level as a cricket coach and became a volunteer Regional Manager for Middlesex CCC I needed to have completed this course. I also received continuing professional development and support from Middlesex.

When I became a test manager, I found the coaching knowledge I gained from cricket useful. I wrote these two blog posts about coaching techniques that are useful in 1:1’s, annual reviews and helping someone with a problem:

“Management needs training to learn about the company, all the way from incoming material to the customer” [1]. 

I have had a test management role in three companies and, in addition to my learning as a cricket coach, these resources have helped me understand the role of a manager/leader.  I hope that they are helpful to you:

I would like to thank RJ McDermott for prompting me to write this post. 

References

[1] Out of the Crisis by W. Edwards Deming (1986, p52)

A great resource for leaders in test and quality – A review of “Learning to Lead, Leading to Learn” by Katie Anderson

The Toyota leader Isao Yoshino learned many lessons during his career.  Katie Anderson shares these lessons in “Learning to Lead, Leading to Learn “,  which I have just read with the Profound Book Club. Toyota has made so many innovations in quality. If you are a Test\QE\QA Manager or Lead or want to become one this book will give you many insights that will help you build your own chain of learning to become a people-centred leader.

Originally Toyota made looms and this book is built around the warp and weft of Yoshino’s career. The warp threads of Yoshino’s career are the purpose of his career and the weft threads are the lessons of his career.

Each chapter finishes with reflection questions. These questions helped me think through how the lessons from the chapter can be applied to my work.

Toyota created the two-year ‘Kan-pro’ programme for all senior managers because all departments played a role in quality and success. The programme reinforced key management capabilities such as writing an A3 to make thinking visible. Yoshino was part of the task force that ran the programme. A key lesson he learned was that learning can not be delegated, managers should write their own A3’s and not delegate writing them.[1]

Yoshino’s career includes being involved in the success of NUMMI and the failure of Toyota’s water-ski boat business.

“The only secret to Toyota is its attitude to learning”[2]. Hansei is the key to learning. Hansei is the “check and adjust” part of the Plan-Do-Check-Act Cycle [2].

Yoshino learned that “No problem is a problem”. Leaders should hear “bad news” so people can work together to find solutions. He was praised when he gave the President of Toyota bad news about the water-ski boat business.[3]

The lessons from the failure of the ware-ski boat business are enlightening. Yoshino said that “failure isn’t failure if you learned something important you could never have learned elsewhere”[4].

Yoshino felt a key personal lesson from the failure of the water-ski business was his failure to use hoshin kanri, Toyota’s strategic planning and checking process. Under pressure, he developed the document alone. He skipped “engaging his people in conversation and input”[3]

I would like to thank the Profound Book Club for the great discussions that we had as we read the book. I learned a great deal not just from the book but also from the discussions. 

Leaders in testing need to be learning continually. Katie Anderson finishes the book with these words: “Never stop learning. Never stop leading. Never stop weaving a life of purpose”[5]

What a useful book!

References

[1] Learning to Lead, Leading to Learn by Katie Anderson (2020, Case Study #1)

[2] Learning to Lead, Leading to Learn by Katie Anderson (2020, Introduction)

[3] Learning to Lead, Leading to Learn by Katie Anderson (2020, Phase #4)

[4] Learning to Lead, Leading to Learn by Katie Anderson (2020, Part 11)

[5] Learning to Lead, Leading to Learn by Katie Anderson (2020, Conclusion)

Further reading:

“Go see, ask why, show respect”

Test analysts, test engineers, test leads and test managers need to understand customers so that our testing includes using the product as the customer uses it. To do this we need to learn how customers use the product by meeting the customer and seeing how they use it. Mr Fuji Cho, the former President of Toyota, is quoted as saying “Go see, ask why, show respect” [1].

 “One of the key tenets of the Toyota Way is ‘genchi gembatsu’: ‘go to the source to find the facts’ This is commonly referred to as ‘go to gemba’”[1] “Gemba is a Japanese word that means ‘the place where something actually happens’”[2]

We should ‘go to gemba’ with purpose. We should define why we are going to gemba and take an intentional approach when seeing a customer. I have been reading Learning to Lead, Leading to Learn by Katie Anderson with the Profound Book Club and she gives these examples of purposes for going to gemba:

  • Checking work processes and outcomes
  • Validating data and assumptions with observable facts
  • Talking to the people involved [1]

Examples of ‘go to gemba’ that I have been part of are:

  • When I worked for a fashion retailer as a Test Lead I spent part of my first month working in a retail outlet where I met customers, used the till and handled deliveries. I learned about customers and the issues in running one of the company’s retail outlets. The knowledge I gained was helpful when I returned to the engineering team because it gave me a better understanding of the business and its customers. 
  • When I worked for a start-up that had an annual user testing week all developers and testers were encouraged to attend part of the user testing to observe how customers used the product and learn from any issues customers had. Developers and testers saw when customers had difficulties and took this knowledge into their work developing the product. I took the knowledge I gained of how customers used the product into my work as Test Lead.

These ‘go to gemba’ activities gave me a deeper understanding of the business and its customers. This understanding helped me to focus testing on customers. “The customer is the most important part of the production line.”[3]  Testers and the business gain so much from ‘go to gemba’. I encourage you to ‘go to gemba”!

References 

[1] Learning to Lead, Leading to Learn by Katie Anderson (2021, Lesson 1 Communicate Purpose, Every Day and Every Way)

[2] The Toyota Way by Jeffrey K. Liker (2004, p224)

[3] Out of the Crisis by W. Edwards Deming (1986, p174)

Learning from CrowdStrike with Taguchi

The recent CrowdStrike incident is estimated to have “affected 8.5 million Windows devices” [1] and may have been “the worst cyber event in history” [1] How should we understand its impact on quality?

Genichi Taguchi’s definition of quality helps us understand how the CrowdStrike incident affected quality. He wrote that “quality is the loss a product causes to society after being shipped, other than any losses caused by its intrinsic functions”.[2] and that “loss should be restricted to two categories:

  1. Loss caused by variability of function
  2. Loss caused by harmful side effects”[3]

“Somebody must pay for this loss – Dr Taguchi called it a loss to society …. We all help to pay for a mistake, a breakdown, failure (bankruptcy) of a company, inept management.”[4]

The CrowdStrike incident can be seen as causing loss to society because it showed extreme variability of function and harmful side effects on “8.5 million Windows devices”[1].

Defining quality as the loss caused to society gives us an insight into the effects of an incident like CrowdStrike. The software that we test and develop also affects society. If a business or person that uses your software experiences problems due to a bug in our software that is a loss to society. Taguchi’s definition of quality shows that a measure of the quality of software is the loss it causes to society.

Taguchi won the Deming Prize, Japan’s highest award for quality, for his ‘loss function’ which enables the loss to society due to variability of quality to be quantified[5].

John Hunter gives practical advice on how to use Taguchi’s insight: “I have seen the concept of the Taguchi Loss Function used quite a bit. I have never actually seen any losses quantified and totaled and shown on a graph. I think focusing specifically on who suffers a loss and what that loss could be, can help. I think actually quantifying the losses to society can be daunting. So, while I see the value in framing the concept that way I think to actually get the losses quantified you are best served by starting with those closest to the process and then adding additional losses to those results” [6]

 A lesson that we can learn from the CrowdStrike incident is that we should use Taguchi’s insight to consider how faults in the software we develop and test can cause losses to society, and through this consider how to avoid these faults. 

References

[1] CrowdStrike IT outage affected 8.5 million Windows devices, Microsoft says

[2] Introduction to Quality Engineering by Genichi Taguchi (1986, p1)

[3] Introduction to Quality Engineering by Genichi Taguchi (1986, p2)

[4] The New Economics by W. Edwards Deming (1994, p218)

[5] Introduction to Quality Engineering by Genichi Taguchi (1986, p19)

[6] Taguchi Loss Function blog post by John Hunter

Additional resources

My new guiding principles are helping me to automate tests.

It is useful to have guiding principles on how to be a good employee, teammate and tester.

I work in teams that describe themselves as lean or agile and so I am interested in learning what lean and agile are. Learning about how lean and agile came about helps me understand them. John Willis has spoken about how the history of ideas in our field can be described as Deming ->Toyota->Lean->Agile. I have learned, and am learning, about many of the American roots of lean and agile such as the work of Deming and Shewhart. I am now learning about the Japanese roots of lean by reading Katie Anderson‘s book Learning to Lead, Leading to Learn with the Deming Profound Book Club. Her book examines the life and career of Toyota executive Isao Yoshino. 
Yoshino sees direct parallels between the leadership precepts of Tokugawa and Toyota’s culture. Tokugawa was the first Shogun and achieved supremacy in 1600[1]. Precepts are general rules intended to regulate behaviour or thought [2]. The table below shows Togukawa’s precepts and Yoshino’s view of Toyota’s culture[1]:

Yoshino’s view of Toyota’s culture  Tokugawa’s leadership precepts
Steadiness and have a long-term viewLife is like a long journey with a heavy burden on your back. Don’t hurry
DiligenceOne who regards inconvenience as natural will never be discontented. When you want more than you have, remember the days when you were in need
Learn from failureIf you only know what it is to conquer and don’t know what it is to be defeated, it will be harmful to you.
PatiencePatience is the base of being safe forever. Regard anger as your enemy
Leaders take responsibilityBlame yourself, not others.
HumilityNot being enough is better than being too much

Yoshino considered the foundational principles of Toyota’s culture to be influenced by Tokugawa’s precepts[1]. The Toyota Production System has influenced lean and agile software engineering, so Tokugawa’s precepts can be seen as one of the roots of Lean and Agile. 

Tokugawa’s precepts still provide a useful guide today. They are a guide which can show how to deal with situations, for instance by being patient. They can also show when we could have done better, for instance by showing humility.

 I have started using them as guiding principles and am interested to see what I learn. I have already learned how valuable humility is when pair programming, such as developing Playwright tests with a developer. Humility enabled us to listen to and learn from one another. Previously I had thought of humility as a virtue but now think of humility as being a principle that will help me be a better employee, teammate and tester. 

I would like to thank Dennis Sergent for his helpful questions that have made me think about humility.

References:

[1] Learning to Lead, Leading to Learn by Katie Anderson (2021, Foundational Leadership Lessons)

[2] Kindle dictionary

Whoever you are, whatever you have achieved you should recognise the achievements of others

Dr Joseph Juran rose from poverty to be an internationally respected management consultant who specialised in quality. His work included popularising the use of the Pareto Principle and creating  The Juran Trilogy. Juran focussed on the role of management in quality. 

He wrote and contributed to many books including six volumes of Juran’s Quality Handbook.

He was a manager for Bell Telephones at the Hawthorne Works and later was a professor of industrial engineering at New York University, where he taught quality management.

After World War Two he worked in Japan and helped rebuild the Japanese economy. In 1981 he was awarded the Second Order of Sacred Treasure by the Emperor of Japan. 

Juran consulted internationally, including working with Steve Jobs. 

In 1992 he received the National Technology Medal from the President of the United States.

These are considerable achievements for which Juran is quite rightly respected. 

However, he also let himself down by how he wrote about his peers. How we treat people is so important.

Walter Shewhart died in 1967 and the August 1967 edition of Industrial Quality Control was dedicated to the memory of Walter Shewhart “in view of his outstanding achievements”[1]. Juran contributed to that edition of the magazine by writing that Shewhart was “at that stage mainly impractical and unintelligible[2]” and was “a competent promoter”[2]. This is an extraordinary way to write about Shewhart, who had created control charts and the plan-do-study-act cycle and had just died. John Willis and Derek Lewis have made an entertaining podcast about Juran’s article: Legacy of Quality Control Pioneers.

Juran wrote his autobiography after W. Edwards Deming had died and in it, Juran wrote that Deming “believed his wishful thinking“[3] that the courses he gave in 1950 were “the dominant reasons for Japan’s emergence as the world quality leader”[3]. Instead of showing that his 1950 lectures were the dominant cause of Japan’s transformation Deming showed several reasons for Japan’s rise. Deming wrote, in addition to his own role in 1950, about the role of the Japanese in initiating the transformation of the Japanese economy. He wrote about the work of the Japanese Union of Scientists and Engineers (JUSE) in 1948 and 1949, and its later role in educating management, engineers and supervisors. He did write about his role in 1950 but he also wrote the roles of others including Ichiro Ishikawa, Kaoru Ishikawa, Ken’ichi Koyanagi and Joseph Juran.[4]

In Juran’s Quality Handbook 6th Edition, which was published after Deming had died, Juran wrote that Deming defined quality as “conformance to requirements”[5]. I can’t find a quote from Deming in which he defines quality in that way. Deming wrote that “quality can be defined only in terms of the agent”[6], which defines quality as being subjective.

When I searched to see if Shewhart or Deming criticised Juran, I found that Deming praised him. Deming wrote that Juran’s “masterful teaching gave Japanese management new insight into management’s responsibility for improvement of quality”.[7] 

I have found so much value in Juran’s work, however, the way he wrote about his peers leaves me speechless. We should treat each other well. The lesson to me is that whoever you are, whatever you have achieved you should recognise the achievements of others.

Thank you to Rob Park for helping me organise my thoughts.

References

[1] Walter Shewhart ASQC 1967 – Quality Leadership (1967, p3)

[2] Walter Shewhart ASQC 1967 – Quality Leadership (1967, p50)

[3] Architect of Quality by Joseph Juran (2004, p302)

[4] Out of the Crisis by W. Edwards Deming (1982, p486)

[5] Juran’s Quality Handbook 6th Edition by Joseph E. DeFoe and Joseph Juran (2010, “How to think about quality”) 

[6] Out of the Crisis by W. Edwards Deming (1982, p168)

[7] Out of the Crisis by W. Edwards Deming (1982, p489)

Use code reviews to have discussions about your test automation code

Learning from discussions originating from code reviews is helping me create a pack of automated tests using TypeScript and Playwright. 

I have been developing a pack of Playwright tests with a Page Object Model. A simplified example of a page in the Page Object Model looked something like this:

A simplified example of a test:

When I created the pack I was pleased because I felt tests were easy to maintain and read. However, from a discussion following a code review, I learned how to write the test with fewer lines of code, making the tests easier to read. The functions should create page objects used in the test. This change would have two improvements:

  1. Each test would contain fewer lines of code because fewer pages need to be imported and a constant does not need to be created at the start of the test.
  2. When a test makes a search it returns a searchResultsPage, this means that the test reads more like a story.

The function search() now returns a searchResultsPage object. The simplified example of the Page Object Model now looks like this:

In the example test, the SearchResults Page is no longer imported, a searchResultsPage constant is no longer created at the start of the test and a searchResultsPage object is created when a call is made to search(). The simplified example of a test now looks like this:

The test pack I am developing has more than two Page Objects so this pattern reduces the number of lines in a test by an even greater amount than in the example. The tests are also now easier to read.

The advice from the code review and the following discussion helped me develop better tests by giving me a better pattern for automating tests. The important learning for me is not the better the code pattern but how code reviews can lead to discussions that help me improve my test automation code. 

Further reading about code reviews:

Implementing Lean Software Development: From Concept to Cash by Mary and Tom Poppendieck (2006, chapter 8 Quality)

Two ways of learning that benefit testing

Testers are learning all the time. I have been reading John Dues’ new book Win-Win: W. Edwards Deming, the System of Profound Knowledge, and the Science of Improving Schools with the Profound Deming Book Club and have gained insights into different ways of learning.

“Moving from planning to doing is deductive learning and moving from doing to studying is inductive learning”[1]. I often learn in these two ways. Before I read John Dues’ book, I had not considered how different and useful each way of learning is.

“Deductive learning involves moving from a theory to the test of a theory”  [1]. I often learn when I move from planning to doing, and in doing so I test the theory I had developed on how to do the work. I write a charter for some exploratory testing by writing and then learn while I am testing. I plan the automation of a test and then learn while automating the test. The charter for exploratory testing or the plan for test automation can be described as my theory.

“Inductive learning involves using results from the test to revise the theory” [1]. I also learn when moving from doing to studying.  When I am reviewing my exploratory testing and test automation I am also reviewing the theory I had created on how to test or automate. Once I have stopped exploratory testing I review my testing and consider what went well, and what could have been better. I also learn from reviewing my test automation after I have automated a test and may have an insight that I want to add to my test automation standards. 

Both ways of learning are useful. When I am doing exploratory testing I may find that I have forgotten to include something in my testing charter, this would be deductive learning. When I review the results of my exploratory testing I may find that most issues that I had found were in a particular area of functionality, this would be inductive learning.

Working in a Plan-Do-Study-Act cycle includes both types of learning. If you are working in a plan-do-study-act cycle you: plan your work; do the work; study the results of the work and then act on what you learned by taking your learning into a new Plan-Do-Study-Act cycle. John Dues includes a diagram that shows how a Plan-Do-Study-Act cycle moves between inductive and deductive learning:

  • Plan-Do includes moves from a theory to testing a theory and so is deductive learning.
  • Do-Study uses results to test a theory and so is inductive learning.

Testers need to be learning continually. Each iteration of a Plan-Do-Study-Act cycle contains deductive and inductive learning. Using a Plan-Do-Study-Act loop to organise your testing and test automation builds inductive and deductive learning into your testing and test automation. 

References:

[1] Win-Win: W. Edwards Deming, the System of Profound Knowledge, and the Science of Improving Schools by John Dues (2023, chapter 1)ues (2023, chapter 1)

A Great Self-Organising Team

The SIGiST Committee at the conference

The SIGiST Summer 2024 Conference was a great success. The British Computer Society hosted the conference at its London office. We had nearly 200 delegates, which is more than at previous conferences. Over twenty speakers gave interesting and inspiring talks. It was great to have speakers for whom this was their first experience of speaking at a conference. We all learned from each other and had a great time! This year we also made some improvements to the conference:

  • To help drive out fear and create psychological safety we created a Code of Conduct
  • At the conference, there was a Prayer Room/Quiet Room
  • We hired artists to illustrate the conference
  • We had a panel discussion with Women In Test

SIGiST is part of the British Computer Society, which is a not-for-profit. Our committee members are all volunteers. 

The committee is a self-organising team. Instead of the organisation of the conference moving  “sequentially from phase to phase” it was “ born out of the team members’ interplay”[1]. We each took initiatives and supported each other’s work. A self-organising team, like the SIGiST Committee, works using a “‘holistic method – as in rugby, the ball gets passed within the team as it moves as a unit up the field” [1]. This approach does not only work for organising conferences. When I was part of an engineering management team in a start-up that made a successful exit, we worked in a similar way.

To be a self-organising team we created “processes, procedures, routines and norms that enable people to do their work easily and well”[2], this made good “social circuitry” [2] for the committee. Our ‘social circuitry’ consists of monthly committee meetings, WhatsApp conversations, regular Zoom catch-ups and using Basecamp for scheduling and sharing resources.

I would like to thank SIGiST committee members for their work, the conference speakers for giving inspiring presentations, the conference attendees for their great questions and conversations, Kerry Wear for supporting the work of the committee, Rae and Aysha for illustrating the conference and the staff at the BCS London office for their work on the day.

It would be great if you joined the British Computer Society and helped us make It good for society: https://www.bcs.org/membership-and-registrations/become-a-member/

References

[1] The New New Product Development Game by Hirotaka Takeuchi and Ikujiro Nonaka (Harvard Business Review 1986)

[2] Wiring the Winning Organization by Gene Kim and Steven Spear (2024, Preface)

How do you decide which tests to automate?

An end-to-end test pack needs to run quickly so that it does not slow developers, and at the same time provide useful feedback to the developers.  This makes deciding which categories of tests to include in the test pack challenging.

A ten-minute video from Russ Ackoff has helped me better understand my decisions on one category of tests to automate and helps me explain my choices better.

Ackoff says that “the essential or defining properties of any system are properties of the whole which none of its parts have” [1]. He illustrates his point by referring to a car: “The essential characteristic of an automobile is that it can carry you from one place to another. No part of an automobile can do that. The wheel can’t, the motor can’t”[1]. He also uses human beings to illustrate his point: “You have certain characteristics the most important of which is life. None of your parts live. You have life. You can write, your hand can’t write….You can see, your eye can’t see.”.[1]

The application I test is a system like the systems Ackoff described. The application has “essential or defining properties” that none of its parts have. The application that I test shares information. One part of the application, such as the UI, can not share information. The application as a whole has the property of sharing information, not its parts. 

A category of tests that should have end-to-end tests are tests for the “essential or defining properties” of the application. This is a useful analysis because if I define the application’s “essential or defining properties ” there will only be a limited number of tests and they will be for the key parts of the application. It is also a useful analysis because I can share this analysis and work with my team to define the “essential or defining properties” of the application.

References:[1]  Beyond Quality Improvement Dr Russ Ackoff – a 10-minute video (1994) This presentation is from an event hosted to capture the Learning and Legacy of Dr. W. Edwards Deming by Clare Crawford-Mason and Lloyd Dobyns, the journalists who made “If Japan can..Why can’t we?”.

Design a site like this with WordPress.com
Get started