TestAndAnalysis

Reasons why you should write a test strategy

I wrote our test strategy after taking Michael Bolton’s Rapid Software Testing course using what I had learned on the course. I reviewed the strategy recently and one of the thoughts I had while reviewing it was why should you write a test strategy? There are many good reasons to write a test strategy and I have thought of these reasons:

To clarify your own thoughts – writing helps clarify your thoughts and this helps you to define your test strategy
To Including to update when things change – things change and if you have a test strategy you can review the strategy and update when things change
To review from time to time – it is useful to have a test strategy to review as sometimes when you review the strategy you can see how to make it better
To share ideas – if you have a test strategy you can share it
To create consensus – a test strategy should not be contentious, if you have written a test strategy you can discuss it with people you work with to create a consensus
To learn from others by creating it collaboratively – writing a test strategy is a chance to learn from others as you can ask others to help you and give their opinion on it
To engage developers in discussions about testing – asking developers to review the test strategy is a way to start discussion about testing. It is really useful to engage in conversation with developers about testing as you can learn what their problems are and how to help them.
To communicate upwards – writing a test strategy is a good way to communicate with more senior members of the company
To communicate to fellow team members – writing a test strategy is a good way to share with fellow team members your strategy for testing
To communicate with new starters – sharing your test strategy with new starters is a good way to
As a point of reference – when people ask what testing you do it is useful to be able to share the test strategy with them
For self development – to learn how to think strategically about testing
To get feedback – we all need feedback on what we do and once you have written a test strategy you can ask for feedback on it
To gain credibility – to show that you have a test strategy and that it is useful
To focus it – to be able to focus your test strategy for a particular project
To adapt it – to be able to adapt your test strategy for a particular project
To be accountable to the team – so that you can be accountable to your team for your test strategy

I am sure that there are more reasons to write a test strategy. What other reasons can you give for writing a test strategy?

Can decision tables help you test API endpoints?

I recently saw a comment about how testers should be able to use decision tables to aid their testing. I decided to refresh my understanding of them and these are the notes I created about them.

Decision tables are an analytical tool. They are used to describe logic and can be helpful in understanding the rules that make up functionality.

The example decision table below contains the rules for car insurance:

	Rules
Conditions	1	2	3	4
Is car is older than 5 years	Y	N	N	Y
driver has no accidents in the last 2 years	Y	N	Y	N
Actions
Discount of 10%			X	X
Discount of 20%	X
Decline insurance		X

Key	Y	Yes
N	No

A decision table should have the conditions and actions in the left hand column. The left hand column defines the rules. The table must also have the columns that contain the entries for the conditions and the entries for the actions to be taken for each combination of conditions. None of the columns that contain the entries for conditions should be the same.

A decision table is drawn from left to right. You should start by listing the conditions in the left hand column. Each of the rules columns should be unique. You can then enter the combinations of conditions in the rules columns. Once the rules columns are complete you can fill in the actions. You should follow each rule down and mark the action to be taken.

The table can help in understanding functionality because it enables you to see the effect of different combinations of conditions. You can then look for gaps, and issues and decide on the level of coverage that is appropriate for your testing.

Decision tables are less likely to be helpful when testing a UI as probably the UI you are testing is not so complex as to require decision tables to analyse it. However, decision tables are likely to be useful when analysing an API endpoint as you can use the table to analyse the effects of parameters on the return values of the endpoint.

I have used Computer Science 4th Edition by C.S French as a reference for this post.

Use Low Code as a springboard for learning

I have been using Ghost Inspector to automate tests that run through the UI for a few years. It is an automation tool which says “No coding required”. If you are not a programmer you can use Ghost Inspector to automate tests. You can also use Ghost Inspector to learn about development practices and coding.

Tests can be created using Ghost Inspector’s UI, you can insert a css selector or xpath next to “click” or “element is present”, so you do not need to be able to code.You can also create tests by importing other tests into your test. This means that, instead of creating the log in steps each time your tests log in to the app, you can create a test that logs in, and import this login test wherever you need to login. Creating a module like a “login test” means that if the login changes you only need to update the login test and not the steps wherever the tests login to the app. This really reduces the costs of maintaining your tests. It also makes it easier to create new tests as you can use the modules you have created to build new tests. If your tests are built from modules you have learned to use modularisation.

Variables can be used to hold values that you use in your tests, for example a variable could hold the value for a commonly used css selector. Variables can be used throughout your tests. If you do this and the selector changes, instead of having to update the selector wherever it occurs in your tests you only need to update the value held in the variable. This will reduce the cost of maintaining your tests. Variables can be scoped for the test, the test suite or for all your tests. Using variables can help you create test suites more quickly too as you can use the variable rather than having to find the css selector or xpath. Learning to use variables is a programming skill.

Ghost Inspector is a test tool that has a JavaScript sandbox and you can use this to learn JavaScript. JavaScript can be used to enhance your tests and there are lots of free resources that you can use to learn JavaScript.You can learn at your own pace and add JavaScript steps to your tests as you learn.

Test automation tools like Ghost Inspector do not require you to be able to write code and you can use it to learn development skills such as how to use modularisation and variables and to start writing JavaScript.

A quality costs scorecard

Juran’s Quality Control Handbook is a reference book for people who “are involved with” quality. The 4th Edition of the handbook includes an article by Frank M. Gryna about Quality Costs.

Gryna says that as companies began to extend the concept of identifying costs to quality they found that costs relating to quality were much larger than had been shown in accounting reports. They also found that the costs of poor quality were avoidable. Over time the meaning of “quality costs” came to mean the costs of poor quality.

Evaluating quality costs enables companies to quantify the size of the quality problem in a way that will have an impact and identifies major opportunities for cost reduction.

Quality Costs can be grouped into categories, which include:

Internal Failure Costs, the costs of defects found prior to the product being shipped, such as rework
External Failure Costs, the costs of defects found after the product is shipped, such as customer complaints

Some companies publish a scoreboard of quantifiable quality costs and I have created an example scoreboard on the tool that I test. You can see the board above.

These are the costs that I have included in the example scorecard:

Rework: Gryna includes rework in internal failure costs. He defines rework as “the costs of correcting defectives to make them fit for use.” This would be fixing bugs for a development team.
Partially Written Code: I have included the cost of partially written code as an internal failure cost as many development teams have projects that have not been completed, are in the code base and so are a cost.
Customer adjustment is listed as an external failure cost by Gryna. This is “the costs of investigation and adjustment of justified complaints attributable to defective product”. I have included these in my scorecard as customer service.
Customer churn, this is the percentage of customers who the company have lost as customers.
Production incidents. These are unintended events in production that reduce service to the customer

The scoreboard represents the quality costs of a company and can be used to illustrate the need for continuous improvement of quality.

Further reading: Juran’s Quality Control Handbook 4th Edition

One cycle to inspire them all

I have worked in lean or agile teams for a number of years and developing my knowledge of lean and agile has helped me. The Deming Cycle is something I have found helpful to understand because it is a part of agile and lean and it is also about continuous improvement of quality.

Engineering teams have been using the cycle to improve the quality of their work for nearly one hundred years so I expect to be using the cycle for the rest of my career.

W. Edwards Deming learnt the cycle from Walter Shewhart. When Deming was working to help rebuild Japan after World War Two he presented the cycle as the Shewhart Cycle but it went into use as the Deming Cycle.

There are four parts to the cycle:

Plan – plan what the team will do
Do – build and test
Study – What was wrong? What was right?
Act – change the way of working based on real results

The cycle is repeated and the team moves forward with the knowledge they have accumulated.

The reason to study is to learn how to improve tomorrow’s product. Everyone can take part and everyone can contribute ideas.

The diagram above shows all four parts of the cycle in a circle, so are all related to one another and there is no hierarchy.

Deming spoke about the cycle as a wheel that rolls along the line of “concepts regarding product quality” and the “sense of responsibility for product quality”. He also wrote about it under “continuous improvement of quality”.

The Deming Cycle has influenced cycles used in engineering. In “The Toyota Way” Jeffrey Liker said that it embodies the learning cycle in the Toyota Production System, and in this way it is part of Lean. In “The Art of Doing Twice the Work in Half the Time” Jeff Sutherland says that “it’s how… Scrum product development is done”, and this is a way that it is part of Agile. Other cycles can also be seen to have been influenced by the Deming Cycle.

It is helpful for us as testing professionals to understand the Deming Cycle because we mostly work in lean or agile teams and we want to improve quality.

Learning JavaScript is helping me to test

I am learning JavaScript because it helps me in so many ways when I am testing.

Many apps have JavaScript sandboxes that enable you to write some JavaScript that adds to your testing. Examples of where you can write JavaScript are the Tests tab in Postman, and browser dev tools which enable you to write JavaScript in the browser via the console or snippets.

I am learning how to use JavaScript to help me test, but so many of the resources for learning JavaScript are there to help you learn how to create a web application. I had to find resources to help me learn what I needed, and fortunately, there are free resources to help me learn how to use JavaScript as part of testing.

I became interested in learning JavaScript when I spent a day using Alan Richardson’s course on Automating the Browser. This taught me how to create bookmarkets which I now use for setting up tests where there are many steps such as filling in personal information on a new account. This meant that instead of spending my time on repetitive manual tasks I could run the bookmarket. I wanted to learn more JavaScript.

When I was creating tests in Postman I wanted to learn how to use the Chai assertion library so that I could make assertions on the HTTP requests I was making. I found the free course Introduction to Chai Assertions on the Test Automation University. This course enabled me to create useful tests for our API and I discovered the Test Automation University.

I found that Test Automation University is free and has a wide range of useful courses. There are three JavasScript Learning Paths: Web UI javaScript, Mobile javaScript and API JavaScript. I started by working my way through the Web UI path and then took other courses that I found useful. Through taking these courses I gained a basic knowledge of JavaScript.

The JavaScript I have learned is helping me with test automation. We use Ghost Inspector for automating tests through the UI and it contains a JavaScript sandbox so I started to use the JavaScript I had learned to enhance these tests.

If you get stuck with a programming problem you can also use openai.com’s ChatGPT to help you by asking it to generate JavaScript for you and then work out how the generated JavaScript works.

There were other tasks that I wanted to automate such as creating test data via an API. I decided to build on my knowledge of JavaScript and learn to use Nodejs for these tasks. I found a free course on Udemy: Node Js API development for Beginners. What I learned on the course enabled me to write a node app that loads test data.

Pairing with a developer is really helpful. When I had some more difficult code to write on a node app I paired with a developer and learned so much. It also meant that I was progressing in what I could do with JavaScript. Pairing has enabled me to develop automated tests for our API. I have also found code reviews of my code really useful as I learn so much from the developer who is doing the review.

JavaScript is helping me to test and I feel like I have only scratched the surface of what I can do with JavaScript. I am enjoying learning JavaScript and am continuing to learn as I am sure that what I learn will help me be a better tester.

Thank you to the developers who are helping me and thank you to the Test Automation University.

API Testing can enhance automated testing strategy

An automated test strategy will normally include tests that are automated through the UI, and unit tests. It is an advantage if an automated testing strategy also includes testing that is automated via the API.

Automated tests need to run quickly so that developers get fast feedback on their work. Tests that are automated via the UI can take a long time to run as a UI has to be fired up for each test, this means that it is difficult to use automated testing via UI to automate a large number of workflows as they will take too long to run. Tests that are automated via the API are integration tests just like tests that are automated via the UI, however API tests have the advantage that they run quickly as they do not require the user interface.

Unit tests run quickly and give confidence that individual components of the code do what they are supposed to but don’t help you have confidence that the whole application works as it should. Tests that are automated via the API run quickly like unit tests and help give confidence in the application because they are integration tests.

Tests that run through the UI can also require maintenance when the UI changes, for example, the css selectors that they use may have been changed. This creates a maintenance overhead when the UI changes. Tests that are automated via the UI can also be flaky because sometimes the UI can take longer to render or a test relies on a complex relative xpath or some other reason. Flaky tests can undermine developers’ confidence in automated testing and create maintenance work for whoever is maintaining the tests. Tests that are automated via the API do not have these maintenance issues as they do not use the UI.

Tests that are automated via the API have many advantages and can be used to enhance test automation strategy. They run quickly, can be used to increase integration test coverage by automating tests of many workflows, can give confidence in the application and do not have a large maintenance overhead.

Improving quality creates a chain reaction

W. Edwards Deming wrote that improving quality results in a chain reaction “lower costs, better competitive position, happier people…and more jobs”.

Deming worked in Japan after World War Two where he helped rebuild the Japanese economy. He received honours from the Emperor of Japan and the President of the USA. His work underpins agile, lean and our understanding of quality.

In “Out of the Crisis” Deming points out that improving quality increases productivity. He quotes research from Rikkyo University in Tokyo that says “when we improve quality we also improve productivity”. He also speaks to company employees who tell him that as quality improves productivity increases because there is less rework and less waste. Rework is work that you have to do again, an example would be fixing bugs. Bug fixing reduces productivity as we have to fix the bug rather than create new features. Another example of waste in software development would be an incident in production which forces engineers to stop their work on new features to fix the issue in production.

Deming says that improving quality “transfers” work time into “good product and better service”. The result of better quality is what he calls a chain reaction. He says that he wrote the chain reaction on the black board of every meeting with top management in Japan:

Improve quality-> productivity improves-> capture the market with better quality and lower price.

The costs of rework in software development mean that improving software quality causes the same positive chain reaction that lowers costs, improves productivity and helps capture the market.

What testing does your team do that is Lean?

We want to find bugs as early as possible so that the cost of fixing them is as little as possible.

The earlier a bug is found the cheaper it is to fix. If a bug is found and fixed in a specification before any code is written then it costs very little to fix. If a bug is found during development then it costs some development time to fix it. If a bug is found in production and fixed then the cost to fix it includes some development time, some customer service time and its effect on customers. A lean perspective expresses this in another way as lean is about reducing waste and improving the flow of work. Bugs are waste so finding bugs early is lean because it reduces waste, and so improves the flow of work.

The value of testing as early as possible was highlighted by Tom Gilb when I heard him speak about ‘Lean QA’ a few years ago. He spoke about how it is best to find issues as early as possible in development.

The advantages are clear in theory and it is helpful to be able to give examples. Tom Gilb spoke about inspecting specifications for software before code is written as an example of early testing and showed how many issues can be identified by inspecting the specifications. One way for testers to help with inspecting specifications is to review requirements with a developer before they start work as this may identify issues. It can also be useful to inspect any test assets, such as automated tests or test plans, as this may find dependencies on requirements that have issues.

Further viewing: https://www.gilb.com/Real-Quality-Assurance-from-an-Agile-and-Lean-Viewpoint

Three short stories about the Definition of Done

What does it mean when a card is put into the Done column? Done can mean that the coding is completed, it can also include that the code has been reviewed, that it includes unit tests and many other things. Teams sometimes create a “Definition of Done” to state exactly what “done” means and this can be a powerful way of improving quality.

A team I worked with that created a “Definition of Done” and did so to create a consensus across the engineering teams about what tasks need to be completed on a card before it could be put into the Done column. Team members put forward suggestions on changing the Definition when they felt it was appropriate, and what was included in the Definition was reviewed. This Definition of Done was part of a successful transition to improve quality and release features more quickly.

I heard about a company where the Definition of Done was negotiated as if it was a legal contract between management and the development teams. Once written the Definition was very difficult to update. The Definition was supposed to help improve quality. The status of the Definition as a type of contract showed the lack of trust that existed. The quality of the team’s software did not improve.

I also heard about a Team Lead whose team created a Definition of Done at a retro type meeting with all members of the team contributing to the discussion on what should be included. The discussion led to an in depth discussion of what needed to be completed before a card could be put into the Done column. This team worked together to deliver good quality new features quickly.

A Definition of Done can help teams deliver good quality software quickly. These stories show that using a Definition of Done can help a team improve quality if the Definition is agreed in a collaborative way and there is trust.

Further reading: Defining What Done Means

How can you work with your team to identify risks?

Identifying risks to their work helps development teams. Recently I spoke to a team who had talked through the ways in which their work could fail, and built on this to identify risks to their work.

The team adapted Failure Modes Effects Analysis (FMEA) to suit their needs. FMEA is a technique that is used in manufacturing to identify the modes of failure for a piece of work and calculate a number that represents the risk of each failure mode. A team using FMEA will, for each failure mode, assign a value for each of the following :

Severity (how severe is the failure)
Occurrence ( how likely the failure is to occur)
Detection (how likely it is that the failure will be detected)

And from these values calculate a value to represent the risk for each failure mode. This enables the team to identify the failure mode with the highest risk.

The team I heard about used a simplified version of FMEA. They found that identifying failure modes for their work would be useful, and that discussing severity, occurrence and detection for each failure mode helped them identify the risks to their project. They created a simplified spreadsheet, based on the one in the article below, with columns for: failure type, potential causes, potential impact and current process controls. However, assigning values and calculating risk for each failure mode felt to be more than was required for their project.

The team identified failure modes, entered them in the failure type column, and then completed the columns for each of the failure modes. The team were able to ensure that there were process controls in place for each failure mode, and identified an additional control to put in place. The team plans to revisit the process if they identify more failure modes for their work.

This simplified FMEA process gave the team confidence as it enabled the team to ensure that they had methods of reducing the risk of each failure mode of their work. The confidence the team gained enabled the team to move more quickly.

Further reading: https://www.isixsigma.com/tools-templates/fmea/fmea-quick-guide/

Quality is Free

Quality is Free is the name of a book by Philip B. Crosby in which he argues that quality is free and lays out a Zero Defects program to improve quality. Crosby’s ideas, along with Deming, Juran, and others, influenced the Quality Movement.

Crosby says that there is a cost of quality. The cost of quality comprises fourteen ingredients such as rework, warranty, engineering changes, and purchase order changes, and that these costs more than cover the costs of improving quality. He says that improving quality will reduce costs and so will increase profits without increasing sales.

The staff are not the cause of poor quality. Crosby argues that the staff are like a mirror and reflect the attitude of management towards quality.

Zero Defects is the name Crosby gives to his program to improve quality. Zero Defects is about preventing defects rather than fixing them and has the aim of getting work right the first time it is done. The promotional activities of the program such as giving every member of staff a badge saying “I’m for quality” have not aged well, but the Zero Defects program shows how a company can improve quality by removing the causes of defects. The program also includes teams setting their own goals for improving quality and that teams should work together on error cause removal.

A phrase used by Crosby in the book is “Quality is free, but requires effort”. He says this because, in the book, he shows that improving quality will cut costs and that effort is needed to improve quality.

Further reading:
A theory of management for improvement of quality vs a quality improvement plan, which helps us more?

How do you decide when to stop testing?

There is always more testing that can be done on a feature but there are reducing returns on testing a feature over time. At some point you need to stop testing, and the question as to when to stop is sometimes a difficult call. Also, sometimes there is more than one feature to be tested. Choosing when to stop testing the first feature and move to testing the second feature can be a difficult choice.

If I test a feature for a while I find that after a while I am finding less issues, and after a while longer I stop finding issues. At the point when you stop finding issues it makes sense to stop testing. Sometimes you don’t have the time to keep testing until you stop finding more issues and it makes sense to stop testing when you are finding only minor issues. In both situations I find that I find more issues at the beginning of testing than at the end and it is the slowing rate at which I find issues that prompts me to consider stopping testing.

That I find more issues, and more serious issues, early in testing than I do later is a pattern I have identified. The pattern is not unlike the Pareto Principle. Pareto was a nineteenth century Italian mathematician who observed that 80% and Italy’s land was owned by 20% of the population and found that this ratio applied in many other situations too. When I am testing it feels that 20% of the testing effort finds 80% of the issues.

One of the key principles of Joseph Juran, the American evangelist for quality, was to use the Pareto Principle. This meant identifying “the vital few and the trivial many” in order to focus on the key areas to improve quality. The Pareto Principle is the pattern that I observed in my testing where I found more issues earlier in my testing and less issues later in my testing. The “vital few” tests were those that I made earlier in my testing and the “trivial many” were the tests that I ran later and found less issues.

We need to know when to stop testing a feature and the Pareto Principle can help to decide when to stop. I stop testing when my testing ceases to find issues and I know that my testing has moved from the “vital few” tests to the “trivial many” tests.

Further Reading: https://www.juran.com/blog/a-guide-to-the-pareto-principle-80-20-rule-pareto-analysis/

The Quality Trilogy helps discussions with your team

We want to create quality features, products and services. Using a framework to think through how to achieve this is helpful as the framework ensures that vital points are covered. Testers can provide value by understanding these frameworks and using them to shape their own thinking in order to be able to prompt and/or lead discussions in their team that help to improve overall quality.

Joseph Juran was an American who helped Japan’s economy recover after World War 2 and later worked for companies such as Apple. He created The Quality Trilogy which is a framework that can be used for discussions with your team. The trilogy consists of:

Quality planning

This part of the trilogy can be described as designing the feature including the team understanding the customers needs. Quality planning helps me test because I want to understand the customer and for this reason I find it useful to have a discussion about what job the customer will be “hiring” the feature to do. It also helps to have a discussion at this early stage about what support for accessibility is appropriate for the new project.

Quality control

Tracking metrics and periodic checks are an important part of quality control because they tell us whether or not the feature we released is “working”. If there are no periodic checks in production that check that features are “working” then quality in production can fall without the company knowing. If quality in production falls this will affect customers who may leave or complain.

It is useful to discuss with the team how the new feature will be tested, including how we know the feature is working once it is deployed. Testing once the feature is deployed to production I think of this as “shift right” testing. As testers we need, not only, to “shift left” to identify issues before they become bugs and we also need to “shift right” to ensure that we know the feature continues to work after it is deployed. We need to ensure that the released product “is working” in production and continues to work in production. “Shift right” testing helps reduce the number of times a customer will surprise us by reporting bugs, reduces the time that bugs are live in production, enables us to plan fixes for bugs we find and will control quality by ensuring that the released feature “works”. If the discussion on testing does not happen then a new feature can be deployed to production that either does not work in production or starts to fail soon after it is deployed to production.

Quality improvement

Identifying areas where processes can be improved to improve quality is an important part of a team’s work. It is constructive to have a conversation about how the team could be improving quality, and it is also useful to have a conversation about what we can do to improve the quality of the new feature.

It is useful to have discussions within teams about quality and Juran’s Quality Trilogy gives a management framework in which we can have these discussions.

Thank you Jason Liggi for reviewing this blog entry

Further reading: Juran’s Quality Handbook