Writing test automation standards is a journey, not a destination

I have started to automate tests in Playwright with TypeScript.  I am now writing standards for the tests. I view writing the standards as a journey because I am adding to the standards as I learn more.

 I have started to create these standards because:

  • It helps me get my thoughts in order
  • It helps me learn from my mistakes because I won’t repeat errors.
  • I can always add to the standards, such as when I started to use describe blocks and needed a naming convention for them.
  • The standards help to share knowledge.
  • Standards facilitate collaborative working because they enable test automation engineers to work in a consistent way
  • I want to be able to maintain the tests easily

I have found lots of useful advice in testing and programming literature:

  • “Any organization that wants to move fast will have standards in place” [1].
  • Standards can be grouped under headings such as naming conventions, coding standards, user interaction conventions, file structures, configuration management practices and tools [1].
  • Standardise is one of the Five S’s and standards help us “reduce complexity over time to improve ease of maintenance” [2] when developing software.
  • It is best to adopt naming conventions at the start of developing the test suite in order to avoid a free-for-all which will make it difficult to find scripts and files as the test suites get larger.[3]
  • “All standards should be simple to follow and make maintainability easier”[4].
  • Naming conventions improve the readability and maintainability of your Playwright Testing framework [5]

I have also learned from colleagues in a book club that it is useful to say why the standard is needed because at a later date, its need may not be obvious and it could be removed. 

Writing standards for test automation is a journey not a destination because I am learning and putting that learning into the standards. 

If you have more sources of information about writing test automation standards please let me know. 

References

[1] Implementing Lean Software Development by Mary and Tom Poppendieck (2007, p193)

[2] Implementing Lean Software Development by Mary and Tom Poppendieck (2007, p192)

[3] Software Test Automation by Mark Fewster and Dorothy Graham (1999, p197)

[4] Agile Testing by Lisa Crispin and Janet Gregory (2009, p227)

[5] Using naming conventions to improve the readability and maintenability of your Playwright Testing framework by Boyana Staneva (2023)

Why I don’t record automated tests

Recently it was pointed out to me that Playwright has a test generator and that I could use it to record tests. Many automated testing tools include a tool or extension that can be used to record tests. I don’t find them useful. 

My experience of using tools to record tests is that they record steps that work if they are played back immediately, but only if they are played back immediately. The recorded steps will be flaky. Every test automation tool I have used has needed additional steps to control the test flow. The recorded steps tend, to either need additional steps to wait for an element to be present or additional steps to wait for an action to finish. 

Test automation code should also be modularised, and recorded steps are not modularised.

The test automation engineer needs to create libraries of modules or functions, possibly in a Page Object Model, so that the code can be easily maintained and tests developed quickly. Steps that have been recorded need to be refactored into modules or functions. It is quicker to write the module or function than to record steps and refactor them.

If you record the steps you don’t learn how to develop tests yourself. Recently I was developing a test with Playwright and could not get the test to do what I wanted with built-in locators. I tried using CSS selectors instead of the built-in locators and could not get them to do what I wanted. So, I read the code in the front-end repo and worked out which element of the DOM I needed to interact with and which action I should use. While working out how to interact with the DOM I learned a lot about the application I am testing and about Playwright. If I had used the recorder all I would have learned would have been how to use the recorder.

The main reason to not use a recorder is that you want to develop your test automation skills. Instead of using a recorder I used the learning resources to learn how to automate tests. I now have more skills and am continuing to learn about test automation. 

I started test automation by using free resources like these:

There are many other great resources too.

A review of “Black-Box Testing” by Boris Beizer

In “Black-Box Testing” Boris Beizer describes a number of useful techniques that can be used for black-box testing. He defines black-box testing as “the testing you do when you put yourself in the software user’s place to verify that it behaves the way it’s supposed to”.[1] Each testing technique is described in a chapter which has the same structure: synopsis, vocabulary, the relations and the model, the technique, application considerations, summary and self-evaluation quiz. IRS forms are used to create examples of each technique.

The first chapter gives an introduction to graphs and relations, subsequent chapters are on testing techniques.

Control-flow graphs are “introduced as the basic model for test design”[2]. Beizer takes us through how to model the functionality being tested as a control-flow graph, select the test paths, and sensitise a graph with data.

Bugs often accompany loops [3] and loops can be tested if the loops are depicted in a graph. The critical test values that are used to control the loop can be described in a graph. Deterministic loops, which for example may handle a paycheck for one to 20,00 employees[4] can be tested as can non-deterministic loops, which for example may search for a file of unknown length[5].

Data flow can be tested by data flow graphs. There are similarities between control flow graphs and data flow graphs because ordinary processing is similar.[6]

“The transaction flow graphs can used in system testing of on-line applications and batch-processing software” [7] The technique pays attention to queues and loops. 

Domain testing “replaces the common heuristic method of testing extreme values and limit values of inputs” [8]. This chapter provides a way of testing where a domain boundary is missing an upper or lower value. It can be useful to read this chapter in conjunction with the chapter in The Art of Software Testing that describes boundary value analysis because they both discuss how to test the same types of values.

Syntax testing is used for “testing command-driven software”[9].

The “finite-state machine model is an excellent model for testing menu-driven applications”[10]. A finite-state machine consists of state, transitions, inputs and outputs and is depicted by state graphs.[11]

Over time tests become less effective at finding bugs. This is described as the “Pesticide paradox”, and testers have to learn to create new tests. [12] 

Black-box testing is a very useful book. I have a well-thumbed copy of the book, which I use as a reference when I need to use one of the techniques Beizer describes.

References

[1] Black-Box Testing by Boris Beizer (1995, pxi)

[2] Black-Box Testing by Boris Beizer (1995, p38)

[3] Black-Box Testing by Boris Beizer (1995, p70)

[4] Black-Box Testing by Boris Beizer (1995, p38)

[5] Black-Box Testing by Boris Beizer (1995, p70)

[6] Black-Box Testing by Boris Beizer (1995, p91)

[7] Black-Box Testing by Boris Beizer (1995, p122)

[8] Black-Box Testing by Boris Beizer (1995, p146)

[9] Black-Box Testing by Boris Beizer (1995, p178) 

[10] Black-Box Testing by Boris Beizer (1995, p204) 

[11] Black-Box Testing by Boris Beizer (1995, p205) 

[12] Black-Box Testing by Boris Beizer (1995, p9)

Are you testing output and outcome?

There is a discussion in the DORA Community about “the problem with DORA metrics is that they focus on maximizing outputs rather than outcomes”.  Some people have argued that Dora metrics will reward teams with a high number of commits, which would be considered ‘output’ but not reward teams whose commits benefit customers, which would be considered ‘outcome’.

This discussion could also apply to testing. 

We need to test the output. This could be described as testing that we are ‘building the thing right’. Testing should include static analysis, unit testing and regression testing as these things all test that we have ‘built the thing right’. However, does testing sometimes concentrate too much on testing output?

We also need to test the outcome. We need to test that the results of the software change will create a positive outcome for the customers, that is that we are ‘building the right thing’. Testing should also include consideration of the outcome of the software change. 

To have confidence that the outcome of a software change will be successful we need to test the requirements. We need to test the requirements of the work so that any misunderstandings in the requirements are found before the new software is released.

Many bugs are caused by ‘breakdowns in information handling’ [1]. These errors occur when information about the required software change is communicated to the development team and within the development team. If we use only the teams’ understanding of the required change we may only be testing the output because we are only testing that ’the thing has been built right’ according to the specification. 

Another way to test that we are ‘building the right thing’ and testing for a positive outcome is to test from the customer’s point of view. I try to understand the customer’s point of view.  I attend user testing regularly so that I see how customers use the product. I also use the Theory of Jobs to be Done to help me understand customers’ use of our product.

Does testing sometimes concentrate on testing output?

References

[1] The Art of Software Testing by Glenford J. Myers (1979, p103)

A review of “Deming’s Journey to Profound Knowledge” by John “Botchagalupe” Willis with Derek Lewis

W. Edwards Deming is often referred to as the “grandfather of quality”, and this new well-researched book about him and his philosophy gives us many new and useful insights. 

The book’s author, John Willis, is one of the people who created DevOps, and he says that to understand the roots of DevOps you need to read Deming’s 14 Points for Management [1]. In this book, John Willis describes how Deming’s ideas developed and how they provide the lens that helps so many successful businesses today.

Deming’s environment helped to shape his philosophy. Deming was an intern at Bell Telephone’s Hawthorne Works during the ‘Hawthorne experiments’. This ultimately led to one of the four elements of his System of Profound Knowledge: psychology [2]. Walter Shewhart also worked at Hawthorne Works but Deming only got to know him well when their paths crossed later. Shewhart gave us Statistical Process Control to improve the manufacturing process.  He also gave us, what is often known as, the Deming Cycle. This cycle provides managers with a ‘tool to help them track bad quality, make a change and see if it improves product quality’[3]. Deming wrote in 1984 that it may take us another fifty years to appreciate what Shewhart gifted us[4].

When Deming helped to rebuild the Japanese economy after World War Two “he found a culture of intrinsically motivated employees. This culture was something he’d vaguely glimpsed at Hawthorne Works”[5]. The book contains fascinating insights into how Japan developed some of the key ideas and practices that became “lean manufacturing”. The Toyota Way inspired “lean manufacturing”, which is a product of Deming’s ideas and Japanese business culture practices[5].

‘Demingmania’ occurred after Clare Crawford-Mason made the documentary ‘If Japan Can…Why Can’t We?’ in 1980[6]. Deming went on to help the Ford Motor Company become the most profitable car maker in America[7]. 

The book contains many examples of the use of Deming’s philosophy, which include:

  • Deming helped John Waraniak, who used the Theory of Knowledge to create Deming Cycles, at General Motors’ Motorsports division to win IndyCar and NASCAR[8]. Waraniak went on to use systems thinking when developing a prototype self-driving car[9] and is now working with NATO to transfer car racing technology to military vehicles to support Ukraine[10].
  • Josh Corman uses Deming’s philosophy to improve cyber security, including in the process of creating and distributing COVID-19 vaccines [11].
  • Dr Doris Quinn used Deming’s philosophy to improve the quality of healthcare [12].
  • The chapter “Out of the Cyber Crisis” contains valuable examples of the use of each of Deming’s 14 Points for Management.

“We live in the age of the digital Cambrian explosion. Thanks, in no small part, to William Edwards Deming” [13]. We are working as testing professionals in the middle of this explosion.  I recommend reading this book to understand Deming’s philosophy, how it contributed to this digital explosion, and examples of how to use the philosophy today.

References

[1] Deming’s Journey to Profound Knowledge by John Willis with Derek Lewis (2023, p163)

[2] Deming’s Journey to Profound Knowledge by John Willis with Derek Lewis (2023, p15)

[3] Deming’s Journey to Profound Knowledge by John Willis with Derek Lewis (2023, p43)

[4] Deming’s Journey to Profound Knowledge by John Willis with Derek Lewis (2023, p193)

[5] Deming’s Journey to Profound Knowledge by John Willis with Derek Lewis (2023, p115)

[6] Deming’s Journey to Profound Knowledge by John Willis with Derek Lewis (2023, p129)

[7] Deming’s Journey to Profound Knowledge by John Willis with Derek Lewis (2023, p133)

[8] Deming’s Journey to Profound Knowledge by John Willis with Derek Lewis (2023, p136)

[9] Deming’s Journey to Profound Knowledge by John Willis with Derek Lewis (2023, p137)

[10] Deming’s Journey to Profound Knowledge by John Willis with Derek Lewis (2023, p138)

[11] Deming’s Journey to Profound Knowledge by John Willis with Derek Lewis (2023, p187)

[12] Deming’s Journey to Profound Knowledge by John Willis with Derek Lewis (2023, p139)

[12] Deming’s Journey to Profound Knowledge by John Willis with Derek Lewis (2023, p153)

Additional links:

Automated tests do not have to be flaky.

Example Deming Cycles for Playwright and Ghost Inspector

When I create or maintain an automated test I use a “learning loop” so that I learn from the test results. This enables me to create stable test packs that return accurate results. 

I automate tests with Playwright using TypeScript and with the low code tool Ghost Inspector. The “learning loop” is the same for both tools.

When I automate a test I use a plan-do-study-act learning loop:

  • I plan the test. This includes planning what functions and modules I will need, and how I will assert that the test has passed.
  • I then “do” the test automation. I automate the test using my plan.
  • I then study the test. I run the test and study the test results. I run the test several times, both on my workstation and in CI. I study the results to see if the test fails. If the test fails I study the test and the test result to work out why it failed. 
  • I then act and take the findings of my study into a new plan-do-study-act learning loop and plan the fix for the failure. 

If a test that is running in CI “flakes”, I use a plan-do-study-act learning loop to fix the test:

  • I plan the fix. I consider why has the test flaked and plan how to make it stable
  • Then I “do” the fix. I make the change to the test that I had planned
  • I then study the test. I the test both locally and in CI several times to see if the test passes consistently. I study the results of the test runs to see if the test is now stable and returning accurate results.
  • I then “act”. If the test still flakes I will go through the loop again. If the test now returns accurate results I will merge the fixed test so that it runs in CI.

Each time I go through the “learning loop” I learn about the application I am testing and the tool I am using to automate tests. I refer to the loop as a learning loop because I have learned so much by using it. A diagram of examples of this loop for both Playwright and Ghost Inspector is shown above.

This loop was created by Walter Shewhart to help improve quality in the manufacture of telephones and was popularised by W. Edwards Deming. It is usually known as the Deming Cycle and is widely used, for example in developing self-driving cars [1]. 

I am responsible for packs of hundreds of tests. If you use a “learning loop” when you create and maintain tests your tests will be stable and give accurate results. Automated tests do not have to be flaky.

References:

[1] Deming’s Journey to Profound Knowledge by John Willis and Derek Lewis (2023, p137)

Learning resources:

How a lone tester can work across teams

I am a lone tester and I work with two development teams. It is not possible to give both teams one hundred per cent of my time. If I don’t have a structure as to how to support both teams I can end up being pulled in multiple directions.

The structure I use is that I am a ‘chicken’ with one team and I am a ‘pig’ with the other team. If you are making a traditional British breakfast a chicken is helpful because it provides eggs, whereas the pig is committed because it provides bacon and sausages.

I am a ’chicken’ with one team. I am helpful. I go to stand up so I know what the team is working on. I offer advice and help, such as automating a test, if time allows.

I am a ‘pig’ with the other team. I am committed. I go to stand-up, planning, retros and any other meetings that the team has. I am involved in the team and support their work as a tester.

The metaphor of a ‘pig and a ‘chicken’ work, in that both teams know to what level I am available. I know also what work both teams are doing, can support them either as a ‘chicken’ or as a ‘pig’ and am able to cope with the workload from both teams. It is also a fun way of describing my relationship with the two teams.

Thank you to Rob Falla for the idea.

Some “new” practices are actually quite “old”

The Aurora 7 capsule

Recently I was fortunate to visit the Museum of Science and Industry in Chicago. The Henry Crown Space Center in the museum has the Aurora 7 capsule from NASA’s Project Mercury. It was piloted by Scott Carpenter and orbited the Earth in 1962. The capsule is only two meters long. It was a surprise to me that the capsule was so small. Please see my photo of it above. 

What is also surprising is that the software engineers on NASA’s Project Mercury used a number of techniques that we would associate with “agile”. They used iterative and incremental development[1], and “Project Mercury ran with very short (half-day) iterations that were time boxed. The development team conducted a technical review of all changes, and, interestingly, applied the Extreme Programming practice of test-first development, planning and writing tests before each micro-increment.”[1]

NASA’s Project Mercury ran from 1958 to 1963[2] and Jerry Weinberg was one of the software engineers on the project.[3] 

Some of the techniques of developing software that we can think of as being “new” are actually quite old.  

References

[1] Iterative and Incremental Development: A Brief History by Craig Larman and Victor R. Basili (2003)

[2] About Project Mercury NASA

[3] Agile Impressions by Gerald M. Weinberg (2017, p67)

Further Reading:

The Seedbed of a Quality Revolution

The sign for the retail park on the site of the Hawthorne Works

A tester’s role is not only to do the testing but also to improve quality. I visited the site of the former Hawthorne Works during a recent trip to Chicago because so many innovations in quality started there.  The Hawthorne Works was the Bell Telephone Laboratories site that manufactured the hardware for the first national telephone system in the USA.[1] It was the “Silicon Valley of ”[2] the early twentieth century and was the “seedbed of a quality revolution.”[3].

The site of the Hawthorne Works is now a retail park; the only part of the works that still stands is the water tower. I have used photos I took while visiting the site to illustrate this blog post.

Hawthorne Works water tower stands over retail units

Significant people who worked at Hawthorne Works and created philosophies and practices that have become central to quality are:

Deming referred to Shewhart as “the master”[1]. 

Deming’s philosophy is of great value as it helps me to work with development teams to improve the quality of work continually. I am enjoying reading John Willis’ new book “Deming’s Journey to Profound Knowledge, which shows how Deming’s life experiences, including those at the Hawthorne Works, influenced his philosophy.

That so many of the advances in quality made in the twentieth and twenty-first centuries are connected to the Hawthorne Works shows what a remarkable place it was.

References

[1] Architect of Quality : The Autobiography of Dr. Joseph M. Juran by Joseph M. Juran (2004, p91)

[2] Deming’s Journey to Profound Knowledge by John Willis and Derek Lewis (2023,  p46)

[3].Manufacturing the future. A History of Western Electric. by S.B. Adams and O.R. Butler (1999, p161)

[4] What is Total Quality Control? By Kaoru Ishikawa (1985, p14)

[5] Out of the Crisis by W. Edwards Deming (1986, p88)

[6] Quality or Else by Lloyd Dobyns and Clare Crawford-Mason (1991, p52)

[7] Deming’s Journey to Profound Knowledge by John Willis and Derek Lewis (2023, p79)

[8] “Deming to Devops: The Science Behind Devops” by John Willis (a five minute long video) 

[9] When Coffee and Kale Compete by Alan Klement (2018, p156)

[10] Out of the Crisis by W. Edwards Deming (1986, p168)

Further Reading

eLearning:

Hawthorne Works water tower is behind the retail units

Make test automation easier with better feedback for parameters with TypeScript

We can make it easier to automate tests by giving feedback if incorrect values are passed as parameters. Developers can get feedback when they try to pass an invalid parameter if I use union types when using TypeScript. I have been using TypeScript to automate tests with Playwright.

We often pass strings to functions as parameters. Sometimes this is because any string can be passed as a parameter. Other times there are only a limited number of strings that can be passed, such as the name of a country. If only a limited number of strings can be passed and we are using Typescript to write our tests, we can pass a union type instead of a string. 


A union type can be used to create a type that consists of only the strings that can be passed as parameters. Union types can be used when a type can be more than one type, for example, a union type could contain a string and a number. The types held in a union type can be literal types such as a string, for example, “cricket”. The union type is an ‘or’ type, so the type will be: ‘string1’ | ’string2’. An example of a union type could be defined in a .d.ts file like this:

The file containing the union type can be imported into your file like this:

The type can then be passed to the function checkBowlingAverage as the parameter bowler:

If a developer enters an invalid value such as ‘swxxing’ for the parameter bowler Visual Studio code will show the error:

If the developer’s mouse hovers overs over the invalid parameter, Visual Studio Code displays a message which shows the valid values. This makes it easy for the developer to correct the invalid parameter.

Execute Program has a lesson on Type Unions, and FrontendMasters has a lesson on Union Types. Both lessons have helped me get a better understanding of union types.

It is helpful if a larger group of people can automate tests and so it is useful to make it as easy as possible for tests to be automated. Using union types can help make it easier to automate tests because feedback is given if an incorrect value is passed.

I got such a buzz from the SIGiST23 testing conference!

Members of the BCS SIGiST Committee meeting with Rashik Parmar, the CEO of the British Computer Society at the conference
Members of the BCS SIGiST Committee meeting with Rashik Parmar, the CEO of the British Computer Society at the conference.

BCS SIGiST’s Summer Conference 2023 was a great success. We had more speakers and more attendees than at our previous conference. Everyone I spoke to enjoyed the conference and asked when the next one would be.

I got a real buzz from the day of the conference! It was great to hear so many great speakers and meet so many talented testing professionals!

SIGiST’s conference can be described as voluntarism, self-development and mutual development.

When Kaoru Ishikawa explained the basic ideas behind Quality Circles he referred to voluntarism, self-development and mutual development.[1]

The conference was about voluntarism. The BCS SIGiST committee which organised the conference are all volunteers, and all the speakers volunteered their time.

The conference was about self-development too. Each of us participating in the event wanted to learn and develop ourselves. We all learned from the presentations. When I speak at a conference I learn a lot from from preparing and giving my presentation. I am sure that the speakers at SIGiST23 also learned in this way. As the SIGiST Programme Secretary, I learned about the importance of reserve speakers. The conference programme was designed to give a balance of themes and a balance between male and female speakers. Our reserve speakers enabled this balance to be maintained when speakers had to withdraw. 

The conference was also about mutual development. We all learned from each other. We learned, not only from the presentations but also from the connections we made in the networking sessions. The exchange of ideas enabled us to gain broader perspectives on our work. 

I would like to thank the BCS SIGiST Committee members, the presenters, the BCS staff who supported the committee and the conference sponsors Keysight Technologies.

BCS SIGiST Committee:

Nicola Martin  Beth Clarke EngTech MBCS MIET Paul Mowat Adam Leon Smith FBCS Laveena Ramchandani Andrew Shaw MBCS Mihai Grigorescu Jonathon Wright Geoff Thompson

Conference Speakers:

Simon Prior Colin Hammond Divyeshkumar Patel Petros Plakogiannis Leela A.Putten Dan Ashby Nadia Soledad Cavalleri Beth Marshall Peter Johnson Dr Carl Adams Shajahan Pulikkal Hristina Koleva Arun Kumar Dutta Riya Dennis David Maynard Abigail Bangser Steve Mellor (MBCS) Paul Gerrard

The British Computer Society is a “not-for-profit”, which promotes wider social and economic progress through the advancement of information technology science and practice. I would encourage you to join: https://www.bcs.org/membership-and-registrations/become-a-member/

References

[1] What is Total Quality Control? The Japanese Way by Kaoru Ishikawa

A review of “The Art of Software Testing”

The Art of Software Testing is a book that I often refer to. The book contains valuable views on how to approach testing and clear descriptions of some test case design methodologies. It was first published in 1979 by Glenford J. Myers and an updated third edition was published in 2012. 

The book starts with “a self-assessment test” to define the test cases for a simple program and shows how difficult it is to define all the test cases for the program.

The chapter “The Psychology and the Economics of Testing” starts by discussing some definitions of testing, and argues that the most appropriate definition is that “testing is the process of executing a program with the intent of finding errors”[1]. The implications of this being the definition of testing are then explored, for instance, it means that a successful test is a test that fails. It is useful to consider how this definition of quality works in your context as this can help you develop your own testing philosophy.

Myers then discusses whether it is possible to test a program to find all of its errors. He does the maths and shows that it is not possible to exhaustively test a simple program using black-box or white-box testing. 

Ten testing principles are then defined. The tenth principle is that “testing is an extremely creative and intellectually challenging task”[2]. 

The key issue of testing is “What subset of all possible test cases has the highest possibility of detecting the most errors?”[3]. Several methodologies are described for designing test cases for both black-box and white-box testing, this includes useful descriptions of both equivalence partitioning and boundary value analysis.

“A software error occurs when the program does not do what its end user reasonably expects it to do”[4]. Testing designed to find these errors is described as high-order testing. These types of errors would not necessarily be found by unit testing.

The book also includes chapters about code inspections and walkthroughs, debugging and unit testing.

The 2012 edition contains several new chapters. 

It includes a new chapter on usability testing, which contains a useful list of questions that can be used to drive testing considerations[5].

There is a new chapter on testing in the agile environment, which contains an interesting overview of extreme programming and testing[6].

There are also new chapters on testing internet applications and mobile application testing.

The approach to testing described in The Art of Software Testing was relevant in 1979 and is still relevant today. That this approach has been relevant for so long means that it will continue to be relevant. This is a book that is very useful, and I would encourage all testers to read it. 

References:

[1]The Art of Software Testing, 3rd Edition by Glenford J. Myers, Corey Sandler and Tom Badgett (2012, p6)

[2]The Art of Software Testing, 3rd Edition by Glenford J. Myers, Corey Sandler and Tom Badgett (2012, p18)

[3]The Art of Software Testing, 3rd Edition by Glenford J. Myers, Corey Sandler and Tom Badgett (2012, p41)

[4]The Art of Software Testing, 3rd Edition by Glenford J. Myers, Corey Sandler and Tom Badgett (2012, p113)

[5]The Art of Software Testing, 3rd Edition by Glenford J. Myers, Corey Sandler and Tom Badgett (2012, p113)

[6]The Art of Software Testing, 3rd Edition by Glenford J. Myers, Corey Sandler and Tom Badgett (2012, p179)

Improve test coverage by parameterising tests with Playwright and other tools

Automated tests can be run multiple times with different values by parameterising tests. Parameterising your tests can improve test coverage because a value in the test can be parameterised which means that the test can iterate through a number of values. This enables one automated test to cover a number of test cases. You could, for example, write one test that checks the behaviour of the application on entering a text string, and then parameterise the test to run with a number of different text strings. 

Parameterised tests also have less code to maintain because one parameterised test covers multiple test cases. 

When a test has been parameterised it is also easier to see what tests have been created because you can review the array holding the parameters rather than read multiple test cases.

You can, for example, parameterise tests using Playwright.

When you parameterise a Playwright test you will need to parameterise the test’s name because the test’s name is used in the name of the screenshot. If the value of your parameter can be used in the test name and in the test you can use an array of strings to hold your parameters. Playwrights documentation shows you how to parameterise a test with an array of strings: https://playwright.dev/docs/test-parameterize.

If there is more than one value that you want to parameterise then you will need to hold the values in a data structure such as a typescript interface. The test can then use a for loop to iterate through an array of objects which means that you can change a number of values each time the test is called. A simple example in which three values are changed each time the test is run is shown below:

Other test automation tools offer similar functionality for parameterising tests to Playwright, for example:

How can you improve the testability of your product?

Teams whose focus includes testability are more likely to be high performing.[1] Good testability will “minimize testing costs”[2]. Testability can also be seen as one of the factors that determine quality.

When the testability of a feature is discussed people often ask what is meant by testability.

The clearest definition of testability I have found was in Juran’s Quality Handbook 5th Edition[3]. The definition was taken from a report by GEC for the USAF which defined testability as the “effort required to test a program to ensure it performs its intended function.” [4] 

There are a number of ways to improve testability. 

Several writers point out that considering testability during planning and design enables us to design testability into the feature we are developing. The report by GEC shows that is important to consider testability during design[5]. Others also see testability as relating to design: Kowalski describes testability as “a design characteristic”[6] and Beizer wrote that “we test software: to “achieve a testable design; a design that can be easily validated, falsified and maintained’ [7].

GEC’s report suggests instrumentation, simplicity, and self-descriptiveness as ways to improve testability[5].

Janet Gregory and Lisa Crispin wrote that “business-facing tests built with appropriate design patterns and ahead of any coding help the team achieve a testable design pattern”[8].

Improving visibility and control are ways to improve testability, and a way of doing this is to give all the team access to the source code [9]. 

Making commands easier to execute, such as via an API, is a way to improve control and so is a way to improve testability.[10].

How can you improve the testability of the feature that your team is developing? Here are some suggestions:

  • It is really helpful for development teams to discuss how they will test new features because they can identify the issues with testability for the feature. The team can then work out how to address these issues. 
  • Testability is a feature of design therefore a tester should be included in planning so that they can raise issues that will make testing easier and help the team create business-facing tests.
  • Difficulties in testing a feature can be raised in team retros to enable discussion and learning about them, and the finding of a solution.
  • Making architectural decisions that enable the development team to do most of its testing without requiring an integrated environment.[1] This could include creating the infrastructure required to test feature branches in production
  • The development team should ensure that it is easy to identify elements in the DOM in order to facilitate test automation via the UI.
  • If the team is working on an API the team needs to ensure that it facilitates automated API testing. An example of something that will help testability of an API is that if you can create an object via the API you should also be able to delete it via the API. 
  • The design of new features needs to include useful log messages, for example, through logging in Dev Tools.

The book Team Guide to Software Testability also has many suggestions, including exercises that you can do with your team to better understand how testability applies to your project. 

I am sure that there are other examples of ways that teams can improve testability, and it would be great to hear how your team is improving testability. 

References:

[1] Accelerate by Nicole Forsgren PhD, Jez Humble and Gene Kim (2018, p89)

[2] FACTORS IN SOFTWARE QUALITY Concept and Definitions of Software Quality

 By Jim A. McCall, Paul K. Richards CD Cene, F. Walters (1977, p155)

[3]  Juran’s Quality Handbook 5th Edition –  Computer Applications to Quality Systems by Fredric I. Orkin and Daniel Oliver (1998, Section 10)

[4] FACTORS IN SOFTWARE QUALITY Concept and Definitions of Software Quality

 By Jim A. McCall, Paul K. Richards CD Cene, F. Walters (1977, p32)

[5] FACTORS IN SOFTWARE QUALITY Concept and Definitions of Software Quality

 By Jim A. McCall, Paul K. Richards CD Cene, F. Walters (1977, p42)

[6] Juran’s Quality Handbook 5th Edition – Quality in Research and Development by Al C. Endres (1998, Section 19.29)

[7] Black-Box Testing by Boris Beizer (1995, p7)

[8] Agile Testing by Lisa Crispin and Janet Gregory (2009, p183)

[9] Lessons Learned in Software Testing by Cem Kaner, James Bach and Bret Pettichord (2002, p149)

[10] AUTOMATED TESTS: TESTABILITY by Kamil Grzybek (2023)

Why we should work in small batches – a story from Japan

“The ability to work and deliver in small batches is especially important” [1], which is why many development teams work in small batches. 

Scrum teams work in small batches by limiting the work they plan to do in a sprint. Kanban teams work in small batches by limiting the work in progress. Working in small batches is one of “the key principles at the heart of continuous delivery”[2], and so teams that practice continuous delivery or continuous deployment work in small batches.

I like a story in The Machine that Changed the World about Taiichi Ohno that shows some benefits that Toyota found from working in small batches.[3]

Back in the late 1940’s American car manufacturers used dies to stamp car parts from large sheets of steel using huge presses. Each press worked 24/7 and could produce over a million parts a year. A misalignment in the die would produce wrinkled parts and could result in the need for expensive repairs. Changing a die took a day and was done by a specialist team.[2]

At this time Toyota was a relatively small company. Taichi Ohno had a smaller budget than American car manufacturers, and this meant that he had to stamp virtually the whole car from a few second-hand presses. Through experimenting, by the late 1950s, he found that dies could be changed in three minutes by production line workers without any specialists. Ohno then “made an unexpected discovery- it actually cost less per part to make small batches of stamplings than to run off enormous lots”.[4]

There were two reasons for small batches to be cheaper. One reason was that Toyota did not have to manage a huge inventory of parts and the other reason was that mistakes showed up very quickly.[3]

Working in small batches meant that the workers in the stamping shop became more concerned with quality, and that there were less faulty parts.[4]

Toyota needed a skilled and motivated workforce to implement this system.[4]

I like the story because, despite being such an old story, the points it makes are still relevant today. The Toyota Production System has had a big influence on lean and agile and so has influenced how we work. 

I manage my manual testing and test automation in small batches, this enables me to be flexible in how I work and give quick feedback. I merge one automated test at a time into the main branch so I can focus on making it a good-quality test. Working in “small batches” also enables me to be more flexible. If I was automating more than one test and a developer wanted my time they would have to wait. However, as I am automating only one test I can quickly be available to help them. 

I work with a team that practices continuous deployment. We deploy in as small a batch as possible by deploying to production each merge to the main branch. We do not have to manage the inventory of a monthly release, faults are detected quickly and as a consequence, customers experience fewer software errors. 

References

[1] Accelerate by Nicole Forsgren, PhD, Jez Humble and Gene Kim (2018, p54)

[1] Accelerate by Nicole Forsgren, PhD, Jez Humble and Gene Kim (2018, p71)

[2] The Machine that Changed the World by James P. Womack, Daniel T. Jones and Daniel Roos (1990, p52)

[3] The Machine that Changed the World by James P. Womack, Daniel T. Jones and Daniel Roos(1990, p53)

Design a site like this with WordPress.com
Get started