Regular expressions can help you to do more with your automated tests

   “A regular expression defines a set of one or more strings of characters” [1]. Regular expressions (regex) can be used when developing automated tests, for example, by passing a regular expression as a parameter.

If you have not used a regular expression before the MDN Web docs are a good place to start learning about them.

In the examples below a regular expression is used as a function parameter in Playwright.

openMenu() has a regular expression as a parameter, and calls filter() which can take either a string or a regular expression as a parameter. openMenu() could have a string as a parameter but if the parameter is a RegExp we can pass a regular expression to filter (). If we pass a regular expression, instead of a string, as the parameter there are more ways we can use the parameter. 

Here are some examples of how we could pass a regular expression to the function:

/More/ will find an element that has text that contains More:

/^More/ will find an element that has text that starts with More

/More$/ will find an element that has text that ends with More

/More[a-z]/ will find an element that contains More followed by a character in the range a-z

If I had passed the string ’More’ to the function it would have clicked an element that contained More. Passing a regular expression as a parameter gives us more options. 

You can do much more in your automated tests with regular expressions than I have in the examples above. Regular expressions can become complicated, making them hard to read. Testing regular expressions using one of the regex testers listed below is useful. The regex testers not only enable you to test your regex but also explain how it works.

There are so many ways that a regular expression can be used that I find it useful to have points of reference to help. Below there is a list of resources that can be used to help create regular expressions. Please let me know if there are more resources that I have missed

Documentation

Regex Testers

University Course

Regular Expressions (Regex) Nanyang Technological University, Singapore

Blogs

Enhancing Playwright Test Automation with Regular Expressions by Cerosh Jacob

How to Define a Regex-Matched String Type in TypeScript ?

RegEx and Selenium by Tracy O’Connor

cy.contains and regular expressions by  Gleb Bahmutov

Book

A Practical Guide to the UNIX System by Mark G. Sobell (It is rather old, but has a useful appendix on Regular Expressions)

Thank you to Danny Dainton, Christian Bauman and Sebastian Stautz for their help

References
[1] A Practical Guide to the UNIX System by Mark G. Sobell (1995, p735)

How do testers assist organisational learning?

“Most organisations focus on the acquisition, processing of data and information….but data, information and knowledge are not all that can be learned, there is also understanding and wisdom”[1] Testers contribute to each of these types of learning. “These five types of mental content form a hierarchy of value; data have the least value, wisdom the most”[1]

The table below has a row for each type of learning. Each row contains a definition of that type of learning and an example of how testers contribute to it.

Type of LearningDefinitionsExample of testers contributing to organisational learning
DataConsists of symbols the properties of objects and events[1]The result of tests, whether they are manual or automated tests.
InformationConsists of data that has been processed[1]A report on the results of tests showing issues found in testing along with the priority and implications of each issue.
KnowledgeContained in instructions, answers to how to questions[1]The readme on automated tests that explains how to run the tests, or a wiki page on how to use a recognised testing technique.
UnderstandingContained in explanations, answers to why questions[1]A tester’s contribution to planning or retros in which they argue why something needs testing
WisdomIs concerned with the value of outcomes and effectiveness[1]Testers using the understanding they have gained from finding bugs to help their team identify and remove the causes of bugs.

Data, information, knowledge, and understanding are all concerned with efficiency. Wisdom is concerned with effectiveness. “Efficiency is concerned with doing things right; effectiveness is concerned with doing the right thing”[1]

Testers can help teams do things right and help teams do the right thing by helping to build organisational learning.

References
[1] Idealized Design Creating an Organisation’s Future by Russell L. Ackoff, Jason Magidson and Herbert J. Addison (2006, p202)

Using recognised techniques to create tests gives teams advantages

Creating tests using recognised techniques, such as boundary value analysis, enables teams to create more efficient tests and is a basis for learning about testing.

Recognised techniques for creating tests provide a way of analysing functionality that enables the creation of more efficient tests. Teams that use techniques to design tests for their code have advantages. Using a technique like a decision table to design your test gives good test coverage. A technique, like boundary value analysis, makes your tests efficient because you have no more tests than are needed to cover the functionality being tested. 

The value that is added by using recognised techniques to create tests also includes:

  • “Objectivity; guarantees a certain level of coverage linked to an identifiable process of achieving it.
  • Increase in the defect-finding capability.
  • The ability to reproduce tests.” [1]

Recognised techniques can be used in exploratory testing. While conducting exploratory testing it could become apparent that a recognised technique could help test an area of functionality. 

Automated tests can also become more efficient and have better coverage if the automated tests are designed using recognised techniques. 

Recognised techniques for creating tests are a “foundation for continuous improvement and employee empowerment”[2] because they are like “standard work” at Toyota: they are “stable, repeatable methods” [2] and they “capture the accumulated learning”[2]. It is also important to “allow creating and individual expression to improve upon the standard”[2] because this enables individuals and teams to learn. Improvements can then be incorporated “into the new standard”[2] so that “you can hand off the learning to the next person”[2]. 

“It is difficult to present a cookbook for boundary value analysis”[3] but we can add to the knowledge about it. Someone I know used boundary value analysis to refactor a suite of unit tests, reducing the number of tests while maintaining the same level of coverage. This knowledge was shared so that others could benefit. I recently learned more about boundary value analysis and shared it in a blog post.

Teams that use recognised techniques for creating tests have advantages, these teams will have more efficient tests for their projects, and knowledge of the techniques gives teams a “foundation for continuous improvement and employee empowerment”[2]. 

References

[1] The Testing Practitioner by Erik van Veenendal (2002, p201)

[2] The Toyota Way by Jeffery  K. Liker (2004, p38)

[3] The Art of Software Testing by Glenford J. Myers (1979, p. 50)

Resources that can be used to learn recognised techniques for designing tests include:

Great careers advice for testers – A Review of The Software Testers Journey by Nicola Lindgren and Vernon Richards

This book offers useful advice to testing professionals on how to build their careers. The advice in the book includes examples from testers on how they progressed in their careers. 

There are different career paths: people, technology and business. When you start your career you should consider which direction you want to go in. Thinking about who you want to help the most will help you. You can also look at who has ‘walked this path before’ and learn from their success. 

The differences between mentors, sponsors and coaches are described and useful advice on how to get the most from a mentoring relationship is given.

Testers need to acquire new skills and can do this by building new habits, and gives an example of how a tester improved their technical skills.

Software testers need to learn how to ‘manage up’. Overlooking the company’s goals is a common mistake when advancing your career. 

Feedback can be helpful or damaging. Testers should learn how to give feedback, and there are useful models on how to do this.

There are several examples of how to build meaningful connections, which helped testers to network their way to success.

People say that most jobs are not advertised. There are several routes through which you can apply for these jobs. You need to do research. The book contains useful advice on how to update your LinkedIn profile and how to use chatGPT.  There are also things to avoid doing when applying for these jobs.

One career option is to  ‘go independent’, if you are considering becoming your own boss there are several questions you should consider.

A tester can also take the skills they learned from being a tester and advance their career by moving to another field. Several former testers give examples of how they used skills they gained as testers to progress in other fields. 

Vernon Richards and Nicola Lindgren have written a book that will be useful to testing professionals. We all change jobs as we build our careers, and this book will help us in our career journeys. 

You can buy The Software Tester’s Journey on Leanpub and Amazon.

Ten reasons why you should lint your test automation code

A linter is a static analysis tool that scans source files and reports warnings and errors for the code it scans. I use a linter on the test automation code that I write. The linter tests the code I write without running the code. Here are ten reasons to use a linter on your test automation code:

  1. Linting will ensure that your code is consistent because the linter’s rules will check all the code against its rules. It will enforce coding standards and style conventions.
  2. Code that has been linted is easier to maintain because it is consistent and contains fewer errors.
  3. Linting is ‘testing early’ because, by running the linter locally, the linter will find issues in your code before it is merged.
  4. Linters can be included in CI so code written by all developers is linted without additional effort by a developer. 
  5. You can learn from linting errors because the linter will explain the error it is reporting. I have increased my knowledge of Typescript and Playwright my analysing the errors found by the linter,
  6. Using a linter will help you build a better relationship with developers because your code will be more consistent. 
  7. Using a linter identifies issues that should be fixed. If you do not use a linter the issues are still there and have not been fixed.
  8. Linters are simple to run locally either from the command line or from your IDE
  9. Linters give real-time feedback.
  10. There is a lot of help on how to install and configure linters, for example:
    1. Setting Up ESLint for Playwright Projects with TypeScript by Cerosh Jacob
    2. The Definitive Guide to API Test Automation With Playwright: Part 8 – Adding ESlint, Prettier, and Husky by Butch Mayhew
    3. ESlint documentation
    4. Cypress Automation Code Quality With Cypress Eslint & Prettier Plugins
    5. Use lint in SeleniumLibrary With Examples
    6. ESLint plugin for Playwright

Interpreting ‘quality’ in more than one way helps me uncover issues

Each project has different requirements, and each set of requirements makes its own demands on ‘quality’. On every project I work on, I find it useful to interpret ‘quality’ in more than one way. 

Interpreting quality is more helpful than defining it. A definition is definitive whereas you can have many interpretations. It is useful to see things from more than one angle. Using more than one interpretation of ‘quality’ helps diversify how I test each project. 

 “How one interprets the word ‘quality’ is important”[1]. How I interpret ‘quality’ affects how I test because when I am testing I look for issues covered by that interpretation of ‘quality’. I use more than one interpretation of ‘quality’ to help me test the project I am working on. Each interpretation pushes me to test differently and so uncover different issues.

“There is little agreement on what constitutes quality”[2]. Many senior figures, who have had significant careers, have had quite different interpretations of ‘quality’.

The table below contains various interpretations of quality and how they affect my testing:

Interpretation of qualityHow does this interpretation affect my testing?
“Quality can be defined only in terms of the agent.”
Out of the Crisis by W. Edwards Deming (1986, p168)		This helps me think about who is using the application I am testing.  They are agents so they have agency and I can think about who is using the application, and how and why they are using it. This interpretation helps me understand that quality is subjective because each agent will view quality differently. Thinking about the agents helps me to test from the perspective of different users of the application under test.
“A thing has qualities, not a quality”
 Economic control of quality of manufactured product by Walter Shewhart (1931, p56)		That an application has qualities, not a quality enables me to break down what quality is into quality attributes for the application under test. This helps me review what I have tested and find new areas to test.
“Quality is the loss a product causes to society after being shipped, other than any losses caused by its intrinsic functions”
Introduction to Quality Engineering by Genichi Taguchi (1986, p2)		If, when I test, I consider the cost of product failures to others, I am considering the impact of product failures on society.  An example of the loss to society would be the cost of customers’ time due to an outage. To help prioritise my testing I can consider which failures would affect customers, the company I work for, or wider society. 
“An article with good quality performs its intended functions without variability”
Introduction to Quality Engineering by Genichi Taguchi (1986, p2)		If there is variation in quality the customer will experience bad quality, for example, if page load time varies sometimes the load time will be too long. This interpretation of quality helps me think about which aspects of product quality, I can measure to understand quality an example would be performance.
“We must define quality as ‘conformance to requirements’”
Quality is Free by Philip B. Crosby (1979, p8)		If I used this definition of quality to help me test, I would miss issues not covered in the requirements. This interpretation of quality is a reminder to test more than ‘conformance to requirements’. In addition to testing the known requirements, my testing should help to discover overlooked and emergent requirements[3]. 
To achieve quality at speed you need:Excellent, rapid feedback, superb, detailed discipline
Implementing Lean Software Development by Mary and Tom Poppendieck (2007, Chapter 8 Quality)		This interpretation of quality makes me think about my and my team’s work. I need to give good feedback and so does my team. Also, my team and I need to be focused and disciplined in how we work. 

References

[1] What is Total Quality Control – The Japanese Way by Kaoru Ishikawa (1985, p45)

[2] Kaizen: The Key to Japan’s Competitive Sucess by Masaaki Imai (1986, pxxiii)

[3] Agile Requirements Gathering: Three Types of Requirements by Mike Cohn (2020)

Benefit from a richer Page Object Model with abstract classes and functions

The Page Object Model we create to support our automated tests can be described as a model of the application we are testing. We can make it a richer model using abstract classes and functions.

“classes, methods, and fields in TypeScript may be abstract.
An abstract method or abstract field is one that hasn’t had an implementation provided. These members must exist inside an abstract class, which cannot be directly instantiated.
The role of abstract classes is to serve as a base class for subclasses which do implement all the abstract members. When a class doesn’t have any abstract members, it is said to be concrete.” [1]

One feature of abstract classes is that instances can not be created of them. The base case classes in your Page Object Model probably should be abstract because instances of them can not be created in the tests and the abstract base class can include an abstract method which can then be implemented in sub-classes. 

Another feature of abstract classes is that they can contain abstract methods. If an abstract class contains an abstract method sub-classes must implement the method. This helps to create a consistent interface for the tests making them easier to read and maintain. 

If I wanted to create Playwright tests that test different search engines I would create an abstract class that contains an abstract method makeSearch(), as shown in the image below:

If I try to create an instance of SearchPage an error is returned because it is an abstract class:

 To test Google and Bing I created concrete classes that extend SearchPage:


The concrete classes must contain the method makeSearch() because they extend SeachPage. If they do not contain makeSearch() an error is returned:

When I created tests I created instances of the Google and Bing classes and called makeSearch() on the instances of both classes.

If there is a new requirement to test another search engine a new concrete class for the search engine can be created by extending the SearchPage and makeSearch() can be used to test the additional search engine.

This example uses an abstract class to make the POM richer and provide control over how tests are developed. Tests should be written using the GoogleSearch and BingSearch classes, not the SearchPage class. SearchPage is an abstract class so testers are prevented from creating an instance of the class SearchPage, but can create instances of GoogleSearch and BingSearch. 

The abstract class also contains an abstract function which its subclasses have to implement. The GoogleSearch and BingSearch classes both contain an implementation of the method makeSearch() which is used in the tests. The tests are easier to read because the tests for both search engines use makeSearch().

Abstract classes are a way to enrich the Page Object Model which provides control over how tests are written and also makes tests easier to read.

More information:

References

[1] Typescriptlang.org

To copy a testing process “is to invite disaster”

Sometimes, we hear about a company that creates high-quality software. The company has great testing and quality processes, so we think about copying their ideas to improve our process.

“If anyone were to study such a company without theory, i.e. without knowing what questions to ask,” they “would be tempted to copy the company”[1]. The company whose process you are tempted to copy may be successful due to good luck or coincidence[1].  Copying another company’s processes does not necessarily involve understanding the company you are tempted to copy or the organisation you work in. Deming warned that “to copy is to invite disaster”[1].

We should try to improve our test process, but not by copying. Before we decide how to improve our test process we need a “map or theory by which to understand the organisations that we work in“[2]. A lens that provides such a map can be found by considering these four factors:

  • Appreciation of a system.
    • We need to understand the system we are working in. To do this, we should use systems thinking. Systems thinking enables you to see the whole of your system. One way systems thinking helps me is by helping me understand the interdependent processes in the system. If you have not used systems thinking you can start your systems thinking journey.
  • Knowledge about variation.
    • Variation is all around us. For example, the number of incidents rises and falls, the performance of the application speeds and slows, the number of bugs changes, etc. Using control charts to understand these variations in data gives us insight, which helps me identify and share issues relating to quality.
  • Theory of knowledge.
    • Theory predicts future outcomes that fit observations of the past without failure and builds knowledge through revision. [3] When I am testing I develop theories and test them, and this way I learn about the product, customers, the company and testing.
  • Psychology.
    • We need to “understand people, interactions between people and circumstances”[4].  This helps me understand the team I work with and the other teams my work interacts with.

Deming called these four factors “A System of Profound Knowledge” because they give a deep understanding. You need not be an expert in all four factors to use the system. 

If we copy testing and quality processes from another company without fully understanding the company we work for, the processes we copy may not work for us and disaster may result.

If we use the System of Profound Knowledge, we can understand our company, its quality processes, and its testing processes and then work to improve them. 

References

[1] The New Economics by W. Edwards Deming (1994, p36)

[2] The New Economics by W. Edwards Deming (1994, p92)

[3] The New Economics by W. Edwards Deming (1994, p102)

[3] The New Economics by W. Edwards Deming (1994, p107)

Additional Resources

Generic functions help me to reduce the amount of Playwright test automation code

One of the challenges in writing automated tests is to have a low maintenance cost of the tests. Creating a library of functions in a Page Object Model helps because the test can reuse the functions in the library. Typescript generic functions within my Playwright Page Object Model also help me, in some situations, to reduce the amount of code the tests require. 

A generic function can be used with many types so one function can have many uses. The image below shows a class containing functions that log a player’s name and position in their team. The class includes a function that logs the name and position of a footballer and a second function that logs the name and position of a cricketer. The class also contains a generic function that logs the name and position of a player that can be used for a cricketer or a footballer. When a function is generic, it has a “type hole” shown with <> and must be filled by a type. The type in the function definitions below is shown by the letter T. The non-generic functions have a parameter of position that takes a type of either footballer or cricketer. The type of the parameter named position is passed to the generic function via T.

The types of footballer and cricketer are in the file lib.sport.d.ts shown below

Types are more powerful than strings because the types are validated, for example, if I try to give a footballer a cricketer’s position an error is returned:

When the functions are called in the test the type is passed to the function logPlayer(), see below:

When the test is executed the output from the two functions is the same:

The generic function logPlayer() can replace the functions logCricketer() and logFootballer() so I only need one function instead of two. This means that there is less code to maintain.

If the product being tested increases in scope, for example, if this application needs to support rugby players, you need not add a new function to support rugby players. You can, for example, add a rugby player type and use the logPlayer() function.

The generic function also creates a consistent interface, making the tests easier to read. The same logPlayer() function is used for every type of player.

Generic functions are a form of modelling. Cricketers and footballers are both types of players, so they can be modelled in a generic function that models cricketers and footballers as players. Where this form of modelling is useful generic functions are helpful.

This blog post contains a very simple example of a generic function. Generic functions can reduce the maintenance required for test automation by reducing the code you must maintain. Are there places in your test automation code where generic functions would help you?

I learned about generic functions on Execute Program: https://www.executeprogram.com/courses/typescript-basics/lessons/generic-functions

Why does testing take so long?

Sometimes testers are asked ‘why does testing take so long?’ The question should not make testers feel defensive. We should always be looking to improve our test process. Testers can use a technique from Toyota to answer the question and improve our process. 

Testers want their testing to flow easily from one task to the next, whether exploratory testing or test automation. Testers don’t want to be delayed or held up but sometimes, something causes delays. It could be that configuring the automation environment is fiddly, or it could be that getting an answer on how a piece of functionality should work holds up your testing or it could be something else.

The problems holding up your testing are problems and you want to remove so that your testing can flow smoothly. Removing the problem that is slowing testing the most should be where to start because it will have the biggest impact. 

To help you choose which problem you should work on removing first you could use the Lean metaphor of rocks slowing the flow of a river. Taiicho Ohno at Toyota wanted to “focus the process improvement efforts”[1] at Toyota because they would have the biggest effect on improving the flow of work. He used the size of rocks in a river as a metaphor to enable him to find the biggest problem to remove: “rocks are the problems disturbing the flow. There are many rocks at the bottom of the river and it takes time and effort to remove them. The question is which rocks are important to remove. The answer is given by reducing the water level; those rocks which emerge above the water are the ones that should be removed.”[1]

Testers want the process of testing to flow smoothly. Testing can be viewed as a flowing river, and the problems that are slowing the flow of testing can be viewed as the rocks that are slowing the river’s flow. If testing is viewed in this way the ‘rocks’ are the problems that are causing testing to take so long.

The ‘rocks’ that are slowing testing should be identified. An experiment can be conducted, as Ohno suggested, in which the amount of testing is reduced so to see which problems are affecting the flow of testing. Once identified, the problems can be analysed to find the largest ‘rock’ which ‘emerges above the water’. Toyota used the Five Why’s to identify the root causes of problems that were slowing the flow of work.[1] 

The testers may not have the authority to make the changes required to remove the largest rock which is slowing the flow of testing. Management should help the testers plan how to remove the largest ‘rock’ so that testing flows more smoothly and does not take so long. The largest ‘rock’ can then be removed and testing will flow more smoothly. 

I am using this approach to reduce the time taken to run automated test packs by analysing the tests to find the ‘rocks’ slowing them down. I am looking for the largest ‘rock’ so that it can be removed.

Testers should find the problems that slow testing and focus improvement efforts on removing the largest problem.

I’d like to thank the Profound Book Club for introducing me to Standing on the shoulders of Giants by Eli Goldratt.

References

[1] Standing on the shoulders of Giants by Eli Goldratt (2006, p6)

I am a student of testing

I have been a tester for twenty years. I have learned to think of myself as a student of testing and I am a better tester for it.

I am a tester and, as a tester, I learn nearly every day. Over the last week, I have learned some new aspects of the Playwright API. I have also learned more about feedback loops and systems from a lecture by Donella Meadows.

I am also a qualified cricket coach and ‘a student of the game’. The phrase ‘a student of the game’ is something I like about cricket because it shows the humility to learn. The phrase “ a student of the game” is particularly powerful when used to refer to international cricketers because it shows that they still have the humility to learn.

Being a “student of the game” requires humility because learning requires humility. In this sense, humility is not a virtue. It is also not about being humble as a way of being ‘good’. Humility is a principle because it is a necessity for learning.  

Testers need the humility to learn continually because testing is about learning. When we are testing we are learning about the product we are testing, our customers, testing, test automation and more. We need humility to learn from the people we work with, customers and the testing community. 

I am a student of testing.

I would like to thank Dennis Sergent for getting me to think about this.

Do outages have to be the new normal?

Yesterday I was using a testing tool and it had an outage. Today I was automating a test when a third party had an outage and delayed my test automation. Even the third party’s Status page was not functioning. Social media was full of people complaining about the outage. Outages impacted two consecutive days of my work. This made me wonder if outages are the new normal. The following day my news feeds were full of a story of a bank whose customers could not access their accounts due to an outage.

What has happened to software quality? Companies are under pressure to cut costs and so they improve productivity. Some companies have cut their QA function. 

The objective of a strategy to improve productivity is to improve productivity which does not, in itself, improve quality. After changes to improve productivity, what percentage of changes to production or released to users result in degraded service?

Improving quality creates a chain reaction that improves productivity[1]. If quality improves there are fewer bugs and outages to be fixed. If fewer bugs and outages are being fixed, productivity improves and new features can be added more quickly. If more new features are being added, and less time is spent fixing bugs, then each new feature costs less. If you can add features at a lower cost, you can capture the market.

W. Edwards Deming used improving quality as a strategy to help rebuild the Japanese economy after World War Two and later to make Ford the most profitable car manufacturer in America[2]. 

Improving quality is a proven business strategy which also improves productivity. “Improvement of quality begets naturally and inevitably improvement of productivity”[3]. Outages should not be the new normal for your customers if your company’s strategy is to improve quality (and your company’s productivity will improve too).

Testing professionals can help companies strategies to improve quality because they know how to:

Testing professionals can help you improve quality, which creates a chain reaction that improves productivity. Outages do not have to be the new normal.

References

[1] Out of the Crisis by W. Edwards Deming (1986, p1)

[2] Deming’s Journey to Profound Knowledge by John “Botchaglaupe” Willis with Derek Lewis (2023, p133)

[3] Out of the Crisis by W. Edwards Deming (1986, p2)

Make your Playwright tests run faster by using the Playwright API to wait

There are times when automating a test in Playwright that the test needs to wait because the test will flake if it does not wait for something such as an event. It can be, for example, that you are waiting for a navigation to complete. 

Tests can be made to wait with a ‘wait’ for a given period of time, say five seconds.  If this is done the tests will always wait for that period of time, whether or not it is necessary and the tests will be slower than they need to be. 

It is better to have tests wait for something specific such as a locator to be rendered, an event or an API call because the test waits for the locator or event. If a test uses this type of wait, it only waits as long as necessary for the specified condition to be met. Also, the tests should not flake and run more quickly.

I have found the Typescript functions in the Playwright API described below to be useful ways of waiting when writing Playwright tests:

Waiting for a locator – waitFor()

The function waitFor() can be chained to a locator to wait for that locator to be rendered, for example:

I have found waitFor() useful when waiting for navigation to complete because the test will wait for the specified locator.

Arguments for the state of the locator being waited for and for the length of the timeout can be passed to waitFor() for the state that is being waited for: https://playwright.dev/docs/api/class-locator#locator-wait-for

Waiting for an API call – waitForResponse()

A test may need to wait for the response from an API call. This can be done with waitForResponse(). In this example, waitForReponse is waiting for a response after click().

The API call that the test needs to wait for can be found in the network tab of Dev Tools and is then passed as a parameter to waitForResponse(). The function waitforReponse() returns the matched response.

Arguments can be passed to WaitforResponse() for the predicate and the timeout: https://playwright.dev/docs/api/class-page#page-wait-for-response

Waiting for an event – waitForEvent()

Sometimes the test is not waiting for an API call or the presence of a locator, it is waiting for an event, such as ‘requestfinished’ or ‘domcontentloaded’. WaitForEvent waits for the event to fire and returns a truthy value. In the example below the test is waiting for a ‘domcontentloaded’ event that occurs after navigate().

Arguments can be passed to waitForEvent() for the event, predicate and timeout: https://playwright.dev/docs/api/class-page#page-wait-for-event

Waiting for a truthy value – waitForFunction()

If a test needs to wait for a truthy value, such as the value returned by querySelectorAll() you can use waitForFunction();

The function will wait up to the length of time specified in the timeout for truthy to be returned. This example returns truthy when length is greater than zero and falsey when length is zero.


Arguments can be passed to waitForFunction() for the function, evaluation argument, polling and timeout: https://playwright.dev/docs/api/class-page#page-wait-for-function

Conclusion

This post describes the functions that I am using to make tests wait. The Playwright API contains more functions that can be used for waiting than I have shown above. If you find that the functions in this post do not meet your needs it is worth exploring the Playwright API to find different ways of making tests wait. 

It is easy to get in the habit of using waits in our test that wait for a given number of seconds. However, waiting for a given number of seconds is a habit that will slow your tests down. 

If you use functions like those above to make your tests wait, the tests will only wait as long as necessary for the condition to be met and so run faster. 

“The Purpose of Analysis is Insight”

Testers want to analyse the product and process to improve quality. Once every engineering team had a statistician, that is not the case today. Testers can help their team by using statistics, such as control charts, to do analysis. There are a lot of resources to help us use statistics, for example, this blog post is based on a chapter from Understanding Variation: The Key to Managing Chaos. Control charts are a form of statistics that provide insight into processes and were part of the philosophy that W. Edwards Deming used to rebuild the Japanese economy and turn Ford into America’s most profitable car manufacturer.
A few years ago I was Test Lead in a company that treated each bug it needed to fix as an incident. Each time we fixed a bug we did root cause analysis with the Five Why’s, and I kept a line chart showing the count of how often we needed to fix a bug. The chart could have looked something like this:

Line chart of the count of bugs needing to be fixed

The line chart shows that there is never more than one bug at a time and bugs become more frequent towards the end of the year. The line chart does not provide deep insight. 

I can display the count of bugs in a control chart. The control chart has three additional lines, the average (mean) plus upper and lower limits, which are calculated from the time series data. Viewing the count of bugs in a control chart gives me a little more insight. The chart shows that the time series stays within the upper and lower limits, which means that the number of bugs is under statistical control.

Control Chart of the count of bugs needing to be fixed

A bug being fixed in this company is rare data. Instead of counting the number of bugs fixed, I can measure the average number of bugs fixed in a time period. I calculated the number of bugs per month in a spreadsheet as below:

I created an XMR chart with this data. An XMR chart consists of an X chart and an MR chart. An X chart is a control chart and is called an X chart because x̄ is the symbol for average. The MR chart contains the moving ranges. The moving ranges are the differences between the data points in the time series in the X chart.

XMR charts are relatively easy to create. They are based on the time series data in the X chart. Moving ranges are the differences between the data points in the X chart time series data and can be calculated using an abs function. The averages in both charts can be calculated using an average function. The upper and lower limits in the X chart and the upper limit in the moving ranges are calculated using a simple equation on Pages 40-41 in Understanding Variation: The Key to Managing Chaos.

The X chart shows that the number of bugs per month rose during the year, then fell and that it rose close to the upper limit towards the end of the year. The upper and lower limits are not targets or goals that have been added to the chart, they have been calculated from the data and so are the voice of the process. If the data points go outside the limits the process is said to be out of statistical control. The Moving Ranges chart shows that the rate of increase rose twice.

XMR Chart of the rate at whcih bugs need to be fixed

The analysis provided by the XMR chart provides insight regarding the rate at which bugs need to be fixed. 

The X chart shows that the rate of bugs needing to be fixed is increasing. The increases are occurring within the upper and lower limits. Changes in the time series outside the upper and lower limits are due to special causes, that is fleeting events. Change within the upper and lower limits are due to common causes that is they are the faults of the system. The increase in the rate at which bugs need to be fixed is within the upper and lower limits and so is a fault of the system. 

The Moving Ranges (MR) chart shows two dates when the increases occur. 

The company can use this analysis to decide if it wants to find and address the causes of the increases. It may decide that because the rate of fixing bugs is under statistical control it has more important issues to deal with. However, if it does want to find and address the causes of the rate of increase the analysis provided by the XMR chart will help.

“The purpose of analysis is insight”[3] and control charts provide useful insights. A better insight into rare data, such as the fixing of bugs for this company, is gained by analysing the rate of fixing bugs rather than by counting the bugs. “In general, counts are weaker than measurements”[2].

Testers can use XMR charts, including control charts, to enable their team to get insights into their product and processes.

Thank you Rob Park for prompting me to explore and write this.

Here are some resources that will help you if you would like to use XMR charts and control charts to gain insights:

Introductory guides to control charts:

A useful guide if you want to create an XMR or control chart:

A video introduction to control charts:

An API that can be used to create control charts

References

[1] Understanding Variation: The Key to Managing Chaos by Donald J. Wheeler (1993, p102)

[2] Understanding Variation: The Key to Managing Chaos by Donald J. Wheeler (1993, p104)

[3] Understanding Variation: The Key to Managing Chaos by Donald J. Wheeler (1993, p33)

What training did you get when you became a Test Manager?

From conversations I have had with friends in the UK and the US, I have found that it is a common experience for people not to receive management training when they first become test managers.

I received training as a manager and have learned that I was fortunate to do so.

A test manager or test lead is an important role. You are a line manager and are responsible for testing. When you become a test manager or test lead you get a new job title and new responsibilities, but, unfortunately, I think it is rare for you to receive training.  

It is interesting to contrast becoming a test manager with a voluntary role in sport. I managed my children’s cricket teams. Whilst in this role I had to complete a course on how to coach that took ten weekends plus two online courses. When I moved up a level as a cricket coach and became a volunteer Regional Manager for Middlesex CCC I needed to have completed this course. I also received continuing professional development and support from Middlesex.

When I became a test manager, I found the coaching knowledge I gained from cricket useful. I wrote these two blog posts about coaching techniques that are useful in 1:1’s, annual reviews and helping someone with a problem:

“Management needs training to learn about the company, all the way from incoming material to the customer” [1]. 

I have had a test management role in three companies and, in addition to my learning as a cricket coach, these resources have helped me understand the role of a manager/leader.  I hope that they are helpful to you:

I would like to thank RJ McDermott for prompting me to write this post. 

References

[1] Out of the Crisis by W. Edwards Deming (1986, p52)

Design a site like this with WordPress.com
Get started