How long should an automated test pack take to run? And what should we do about it?

How long should an automated test pack take to run?

Some people say that a certain number of minutes is too long, and others say that a different number of minutes is too long. How can we determine what is “too long”? And what can we do about it?

What is an acceptable length of time for a test pack to run depends on several factors. Your team’s cycle time will affect how long it is acceptable for a test pack to run. Cycle time can be defined as a development process “ that transforms identified customer needs into delivered customer value at a reliable, repeatable cadence”[1] If your team is practising continuous deployment or continuous delivery and has a cycle time of a few hours there will be pressure for automated test packs to run more quickly than if your team makes monthly or quarterly releases. Teams that practice continuous deployment or continuous delivery work in small batches so the test packs will need to be small batches too.

Another factor that affects how long it is acceptable for a test pack to run is whether the length of time the tests take to run is a constraint on development. In some situations, a test pack of thirty tests that runs in fifteen minutes may be an enabler and in other situations, it may be a constraint. An example of this would be; if the test pack that you are concerned about takes no longer to run than other test and deployment processes then it is unlikely to be a constraint, however, if it takes much longer than other test or deployment processes then it may be a constraint on how developers work.

What can the test engineer responsible for the test pack do about the length of time it takes to run the test pack? 

The engineer who is responsible for the test pack should not be defensive about the test pack, instead, they should lead the discussion about how long it takes the test packs to run. They can, for instance, go to developer meetings and create a dialogue with developers by talking about how they are developing the test pack. This dialogue should include how long it takes for the pack to run. If the test automation engineer is involved in a dialogue they will find out how the test packs are affecting developers. They can also tell developers what they are doing to make the tests run faster and discuss with developers ideas as to how to improve the running of the test packs. Testers and developers should then use this dialogue to start to cooperate on improving test packs, including their running times.

What is an acceptable time for a test pack to run depends on several factors. We need to understand our development system and cooperate with our development team to improve test packs. 

References

[1] Implementing Lean Software Development: From Concept to Cash by Mary and Tom Poppendiek (2006,p98)

A theory of management for improvement of quality vs a quality improvement plan, which helps us more?

What can we learn from comparing Deming’s 14 Points for Management and Crosby’s 14-Point Quality Improvement Plan? Which will help us more to improve quality?

W. Edwards Deming first presented his 14 Points at a conference in 1978 in Tokyo[1] and published his 14 Points for Management in 1982[2]. Philip B. Crosby published his 14-point Quality Improvement Plan in 1979[3]. I have not found anything to suggest that their both having 14 points/ steps is anything but a coincidence.

Deming’s 14 Points for Management [2]Crosby’s 14-Step Quality Improvement Plan [3]
1. To create constancy of purpose towards improvement of product and service, with the aim to become competitive and stay in business and to provide jobs
2 .Adopt the new philosophy. We are in a new economic age. Western management must awaken to the challenge, must learn their responsibilities, and take on leadership for change. 
3. Cease dependence on inspection to achieve quality.
4. End the practice of awarding business on the basis of price tag. Instead, minimise total cost. Move toward a single supplier for any one item., on a long-term relationship of loyalty and trust.
5.  Improve constantly and forever the system of production and service, to improve quality and productivity, and thus constantly decrease costs.
6. Institute training on the job.
7. Institute leadership. The aim of supervision should be to help people … do a better job. 
8. Drive our fear, so that everyone may work effectively for the company.
9. Break down barriers between departments. People in research, design, sales and production must work as a team.
10. Eliminate slogans, exhortations and targets for the workforce..such exhortations only create adversarial relationships.
11. Eliminate management by numbers..substitute leadership.
12. Remove barriers that prevent people from taking pride in their work
13. Institute a vigorous programme of education and self-improvement.
14. Put everyone in the company to work to accomplish the transformation. The transformation is everybody’s work.		1. Management Commitment –  to make it clear where management stands on quality
2. The Quality Improvement Team –  to run the quality improvement programme
3. Quality Measurement – to provide a display of current and potential nonconformance problems in a manner that permits objective evaluation and corrective action
4. The Cost of Quality – to define the ingredients of the cost of quality and explain its use as a management tool
5. Quality Awareness – to provide a method of raising the personal concern felt by all personnel in the company toward the conformance of the product or service and the quality reputation of the company
6. Corrective Action  – to provide a systematic method of resolving forever the problems that are identified through previous action steps
7. Zero Defects Planning – to examine the previous activities that must be conducted in preparation for formally launching the Zero Defects programme.
8. Supervisor Traning – to define the type of training that supervisors need to actively carry out their part of the quality improvement programme
9. Zero Defects Day – to create an event that will let all employees realise through a personal experience that there has been a change.
10. Goal Setting – to turn pledges and commitments into action by encouraging individuals to establish improvement goals for themselves and their groups.
11. Error Cause Removal – to give the individual employee a method of communicating to management the situations that make it difficult for the employee to meet the pledge to improve.
12. Recognition – to appreciate those who participate
13. Quality Councils – to bring together professional quality people for planned communication on a regular basis.
14. Do it over again – to emphasise that the quality improvement program never ends

What do Deming’s points and Crosby’s steps have in common? 

  • both aim to help businesses improve the quality of their product or service
  • both were part of the Quality Movement
  • both aim to prevent errors
  • both view quality as management’s responsibility

What are the key differences between Deming’s points and Crosby’s steps?

  •  Deming described his 14 Points as a “theory of management for improvement of quality, productivity, and competitive position”[4]. In contrast, Crosby’s steps are a quality improvement program comprising 14 steps.
  • Deming’s 14 points are a coherent philosophy, each point of the 14 points is implicit in all the other points. An example would be training (point 6) will be ineffective unless barriers that prevent people from taking pride in their work are removed (point 12). Crosby’s plan is a series of steps rather than points that are implicit in each other.
  • The role of management is different:
    •  Deming’s points include instituting leadership so that the role of a manager ”is a coach and counsel, not a judge”[5]. In contrast, Crosby’s plan has a more traditional hierarchical view of management’s role in which it is management who runs the Quality Improvement Programme. 
    • The manager’s role in Deming’s Points includes driving out fear, this means that managers take responsibility for psychological safety. Staff working in a company using Crosby’s plan would probably need to use the Error Cause Removal step to raise issues relating to psychological safety.
  • Deming’s points include developing people’s potential through education and training for all staff, whereas Crosby’s plan only requires training for supervisors.
  • In Deming’s points the whole company, including its suppliers, is seen as a system in which everyone should cooperate, and the manager’s role is to help everyone work effectively. Crosby’s plan puts responsibility on individual members of staff, for example, on Zero Defects Day individual staff are asked to sign pledges to improve quality.
  • Deming’s Points include cooperating across company departments and between the company and its suppliers to improve quality. Crosby’s steps do not include cooperation.
  • Crosby‘s plan sets goals and looks at costs, whereas Deming’s points ask for constant improvement.
  • “Building quality in” instead of “inspecting quality in” is covered by Deming’s point 3. Crosby has a step on Corrective Action to resolve problems.
  • Crosby aims for Zero Defects, whereas Deming aims for continual improvement.
  • Deming’s 14 Points for Management are the roots of DevOps[6] and so are part of innovation in the twenty-first century. I am not aware of any twenty-first century innovation based on Crosby’s 14 Steps.

We should note that Deming also used his System of Profound Knowledge, and the plan-do-study-act cycle to help businesses. Crosby also used a Quality Management Maturity Grid.

Deming’s ideas are powerful “on cooperation and human potential”[6]. We need cooperation and the realisation of human potential to improve quality and I wouldn’t want to work anywhere that was very different to Deming’s 14 Points for Management.

References

[1] Quality or Else by Lloyd Dobyns and Clare Crawford-Mason (1991, p102)

[2] Out of the Crisis by W. Edwards Deming (1982, p23)

[3] Quality is Free by Philip P. Crosby (1979, p137)

[4] Out of the Crisis by W. Edwards Deming (1982, p19)

[5] The New Economics by W. Edwards Deming (1994, p126)

[6] Deming’s Journey to Profound Knowledge by John “Botchagalupe” Willis with Derek Willis (2023, p163)

Additional Resources

  • Deming NEXT – eLearning from the Deming Institute with a free two-week trial
  • Deming’s Journey to Profound Knowledge by John “Botchagalupe” Willis with Derek Willis  –  Chapter 19 gives contemporary true-to-life examples for each of Deming’s 14 Points. 

Gaining a knowledge of React is helping me automate tests

React is an open-source User Interface (UI) JavaScript library created by Facebook. I am doing some React tutorials to help me better understand the UI and what I am learning is helping me to automate tests.

The React tutorials I am doing are here: https://react.dev/learn. The tutorials include exercises that can be done in a code sandbox, so you do not need your own development environment to do the exercises. I have gained a better understanding of the course content by doing the exercises. I have done some exercises in my dev environment and some in the code sandbox.

I have learned that React apps are made out of components. I am now learning about React components and understand that if there is a bug in a component the bug may be present wherever the component is used. I have also learned about CSS classes, and how they can be used to create a similar appearance throughout the app. 

I now better understand the developers when they talk about “props” and “lifting state up”.  This helps me contribute to discussions at planning meetings.

The React tutorials have given me a better understanding of the UI that I am automating tests for. When I have had a problem automating an action I have been able to look in the repo and work out how to interact with the React component that I am trying to automate. On one occasion the DOM element was nested and I found that by reading the code I could see which element handled an onClick event, and so could automate the click that I wanted.

Doing the React tutorials is making me more of a ‘T’ shaped tester. I have deep testing skills but I also now have a growing understanding of React, and this knowledge of the UI is helping me automate tests.

Writing test automation standards is a journey, not a destination

I have started to automate tests in Playwright with TypeScript.  I am now writing standards for the tests. I view writing the standards as a journey because I am adding to the standards as I learn more.

 I have started to create these standards because:

  • It helps me get my thoughts in order
  • It helps me learn from my mistakes because I won’t repeat errors.
  • I can always add to the standards, such as when I started to use describe blocks and needed a naming convention for them.
  • The standards help to share knowledge.
  • Standards facilitate collaborative working because they enable test automation engineers to work in a consistent way
  • I want to be able to maintain the tests easily

I have found lots of useful advice in testing and programming literature:

  • “Any organization that wants to move fast will have standards in place” [1].
  • Standards can be grouped under headings such as naming conventions, coding standards, user interaction conventions, file structures, configuration management practices and tools [1].
  • Standardise is one of the Five S’s and standards help us “reduce complexity over time to improve ease of maintenance” [2] when developing software.
  • It is best to adopt naming conventions at the start of developing the test suite in order to avoid a free-for-all which will make it difficult to find scripts and files as the test suites get larger.[3]
  • “All standards should be simple to follow and make maintainability easier”[4].
  • Naming conventions improve the readability and maintainability of your Playwright Testing framework [5]

I have also learned from colleagues in a book club that it is useful to say why the standard is needed because at a later date, its need may not be obvious and it could be removed. 

Writing standards for test automation is a journey not a destination because I am learning and putting that learning into the standards. 

If you have more sources of information about writing test automation standards please let me know. 

References

[1] Implementing Lean Software Development by Mary and Tom Poppendieck (2007, p193)

[2] Implementing Lean Software Development by Mary and Tom Poppendieck (2007, p192)

[3] Software Test Automation by Mark Fewster and Dorothy Graham (1999, p197)

[4] Agile Testing by Lisa Crispin and Janet Gregory (2009, p227)

[5] Using naming conventions to improve the readability and maintenability of your Playwright Testing framework by Boyana Staneva (2023)

Why I don’t record automated tests

Recently it was pointed out to me that Playwright has a test generator and that I could use it to record tests. Many automated testing tools include a tool or extension that can be used to record tests. I don’t find them useful. 

My experience of using tools to record tests is that they record steps that work if they are played back immediately, but only if they are played back immediately. The recorded steps will be flaky. Every test automation tool I have used has needed additional steps to control the test flow. The recorded steps tend, to either need additional steps to wait for an element to be present or additional steps to wait for an action to finish. 

Test automation code should also be modularised, and recorded steps are not modularised.

The test automation engineer needs to create libraries of modules or functions, possibly in a Page Object Model, so that the code can be easily maintained and tests developed quickly. Steps that have been recorded need to be refactored into modules or functions. It is quicker to write the module or function than to record steps and refactor them.

If you record the steps you don’t learn how to develop tests yourself. Recently I was developing a test with Playwright and could not get the test to do what I wanted with built-in locators. I tried using CSS selectors instead of the built-in locators and could not get them to do what I wanted. So, I read the code in the front-end repo and worked out which element of the DOM I needed to interact with and which action I should use. While working out how to interact with the DOM I learned a lot about the application I am testing and about Playwright. If I had used the recorder all I would have learned would have been how to use the recorder.

The main reason to not use a recorder is that you want to develop your test automation skills. Instead of using a recorder I used the learning resources to learn how to automate tests. I now have more skills and am continuing to learn about test automation. 

I started test automation by using free resources like these:

There are many other great resources too.

A review of “Black-Box Testing” by Boris Beizer

In “Black-Box Testing” Boris Beizer describes a number of useful techniques that can be used for black-box testing. He defines black-box testing as “the testing you do when you put yourself in the software user’s place to verify that it behaves the way it’s supposed to”.[1] Each testing technique is described in a chapter which has the same structure: synopsis, vocabulary, the relations and the model, the technique, application considerations, summary and self-evaluation quiz. IRS forms are used to create examples of each technique.

The first chapter gives an introduction to graphs and relations, subsequent chapters are on testing techniques.

Control-flow graphs are “introduced as the basic model for test design”[2]. Beizer takes us through how to model the functionality being tested as a control-flow graph, select the test paths, and sensitise a graph with data.

Bugs often accompany loops [3] and loops can be tested if the loops are depicted in a graph. The critical test values that are used to control the loop can be described in a graph. Deterministic loops, which for example may handle a paycheck for one to 20,00 employees[4] can be tested as can non-deterministic loops, which for example may search for a file of unknown length[5].

Data flow can be tested by data flow graphs. There are similarities between control flow graphs and data flow graphs because ordinary processing is similar.[6]

“The transaction flow graphs can used in system testing of on-line applications and batch-processing software” [7] The technique pays attention to queues and loops. 

Domain testing “replaces the common heuristic method of testing extreme values and limit values of inputs” [8]. This chapter provides a way of testing where a domain boundary is missing an upper or lower value. It can be useful to read this chapter in conjunction with the chapter in The Art of Software Testing that describes boundary value analysis because they both discuss how to test the same types of values.

Syntax testing is used for “testing command-driven software”[9].

The “finite-state machine model is an excellent model for testing menu-driven applications”[10]. A finite-state machine consists of state, transitions, inputs and outputs and is depicted by state graphs.[11]

Over time tests become less effective at finding bugs. This is described as the “Pesticide paradox”, and testers have to learn to create new tests. [12] 

Black-box testing is a very useful book. I have a well-thumbed copy of the book, which I use as a reference when I need to use one of the techniques Beizer describes.

References

[1] Black-Box Testing by Boris Beizer (1995, pxi)

[2] Black-Box Testing by Boris Beizer (1995, p38)

[3] Black-Box Testing by Boris Beizer (1995, p70)

[4] Black-Box Testing by Boris Beizer (1995, p38)

[5] Black-Box Testing by Boris Beizer (1995, p70)

[6] Black-Box Testing by Boris Beizer (1995, p91)

[7] Black-Box Testing by Boris Beizer (1995, p122)

[8] Black-Box Testing by Boris Beizer (1995, p146)

[9] Black-Box Testing by Boris Beizer (1995, p178) 

[10] Black-Box Testing by Boris Beizer (1995, p204) 

[11] Black-Box Testing by Boris Beizer (1995, p205) 

[12] Black-Box Testing by Boris Beizer (1995, p9)

Are you testing output and outcome?

There is a discussion in the DORA Community about “the problem with DORA metrics is that they focus on maximizing outputs rather than outcomes”.  Some people have argued that Dora metrics will reward teams with a high number of commits, which would be considered ‘output’ but not reward teams whose commits benefit customers, which would be considered ‘outcome’.

This discussion could also apply to testing. 

We need to test the output. This could be described as testing that we are ‘building the thing right’. Testing should include static analysis, unit testing and regression testing as these things all test that we have ‘built the thing right’. However, does testing sometimes concentrate too much on testing output?

We also need to test the outcome. We need to test that the results of the software change will create a positive outcome for the customers, that is that we are ‘building the right thing’. Testing should also include consideration of the outcome of the software change. 

To have confidence that the outcome of a software change will be successful we need to test the requirements. We need to test the requirements of the work so that any misunderstandings in the requirements are found before the new software is released.

Many bugs are caused by ‘breakdowns in information handling’ [1]. These errors occur when information about the required software change is communicated to the development team and within the development team. If we use only the teams’ understanding of the required change we may only be testing the output because we are only testing that ’the thing has been built right’ according to the specification. 

Another way to test that we are ‘building the right thing’ and testing for a positive outcome is to test from the customer’s point of view. I try to understand the customer’s point of view.  I attend user testing regularly so that I see how customers use the product. I also use the Theory of Jobs to be Done to help me understand customers’ use of our product.

Does testing sometimes concentrate on testing output?

References

[1] The Art of Software Testing by Glenford J. Myers (1979, p103)

A review of “Deming’s Journey to Profound Knowledge” by John “Botchagalupe” Willis with Derek Lewis

W. Edwards Deming is often referred to as the “grandfather of quality”, and this new well-researched book about him and his philosophy gives us many new and useful insights. 

The book’s author, John Willis, is one of the people who created DevOps, and he says that to understand the roots of DevOps you need to read Deming’s 14 Points for Management [1]. In this book, John Willis describes how Deming’s ideas developed and how they provide the lens that helps so many successful businesses today.

Deming’s environment helped to shape his philosophy. Deming was an intern at Bell Telephone’s Hawthorne Works during the ‘Hawthorne experiments’. This ultimately led to one of the four elements of his System of Profound Knowledge: psychology [2]. Walter Shewhart also worked at Hawthorne Works but Deming only got to know him well when their paths crossed later. Shewhart gave us Statistical Process Control to improve the manufacturing process.  He also gave us, what is often known as, the Deming Cycle. This cycle provides managers with a ‘tool to help them track bad quality, make a change and see if it improves product quality’[3]. Deming wrote in 1984 that it may take us another fifty years to appreciate what Shewhart gifted us[4].

When Deming helped to rebuild the Japanese economy after World War Two “he found a culture of intrinsically motivated employees. This culture was something he’d vaguely glimpsed at Hawthorne Works”[5]. The book contains fascinating insights into how Japan developed some of the key ideas and practices that became “lean manufacturing”. The Toyota Way inspired “lean manufacturing”, which is a product of Deming’s ideas and Japanese business culture practices[5].

‘Demingmania’ occurred after Clare Crawford-Mason made the documentary ‘If Japan Can…Why Can’t We?’ in 1980[6]. Deming went on to help the Ford Motor Company become the most profitable car maker in America[7]. 

The book contains many examples of the use of Deming’s philosophy, which include:

  • Deming helped John Waraniak, who used the Theory of Knowledge to create Deming Cycles, at General Motors’ Motorsports division to win IndyCar and NASCAR[8]. Waraniak went on to use systems thinking when developing a prototype self-driving car[9] and is now working with NATO to transfer car racing technology to military vehicles to support Ukraine[10].
  • Josh Corman uses Deming’s philosophy to improve cyber security, including in the process of creating and distributing COVID-19 vaccines [11].
  • Dr Doris Quinn used Deming’s philosophy to improve the quality of healthcare [12].
  • The chapter “Out of the Cyber Crisis” contains valuable examples of the use of each of Deming’s 14 Points for Management.

“We live in the age of the digital Cambrian explosion. Thanks, in no small part, to William Edwards Deming” [13]. We are working as testing professionals in the middle of this explosion.  I recommend reading this book to understand Deming’s philosophy, how it contributed to this digital explosion, and examples of how to use the philosophy today.

References

[1] Deming’s Journey to Profound Knowledge by John Willis with Derek Lewis (2023, p163)

[2] Deming’s Journey to Profound Knowledge by John Willis with Derek Lewis (2023, p15)

[3] Deming’s Journey to Profound Knowledge by John Willis with Derek Lewis (2023, p43)

[4] Deming’s Journey to Profound Knowledge by John Willis with Derek Lewis (2023, p193)

[5] Deming’s Journey to Profound Knowledge by John Willis with Derek Lewis (2023, p115)

[6] Deming’s Journey to Profound Knowledge by John Willis with Derek Lewis (2023, p129)

[7] Deming’s Journey to Profound Knowledge by John Willis with Derek Lewis (2023, p133)

[8] Deming’s Journey to Profound Knowledge by John Willis with Derek Lewis (2023, p136)

[9] Deming’s Journey to Profound Knowledge by John Willis with Derek Lewis (2023, p137)

[10] Deming’s Journey to Profound Knowledge by John Willis with Derek Lewis (2023, p138)

[11] Deming’s Journey to Profound Knowledge by John Willis with Derek Lewis (2023, p187)

[12] Deming’s Journey to Profound Knowledge by John Willis with Derek Lewis (2023, p139)

[12] Deming’s Journey to Profound Knowledge by John Willis with Derek Lewis (2023, p153)

Additional links:

Automated tests do not have to be flaky.

Example Deming Cycles for Playwright and Ghost Inspector

When I create or maintain an automated test I use a “learning loop” so that I learn from the test results. This enables me to create stable test packs that return accurate results. 

I automate tests with Playwright using TypeScript and with the low code tool Ghost Inspector. The “learning loop” is the same for both tools.

When I automate a test I use a plan-do-study-act learning loop:

  • I plan the test. This includes planning what functions and modules I will need, and how I will assert that the test has passed.
  • I then “do” the test automation. I automate the test using my plan.
  • I then study the test. I run the test and study the test results. I run the test several times, both on my workstation and in CI. I study the results to see if the test fails. If the test fails I study the test and the test result to work out why it failed. 
  • I then act and take the findings of my study into a new plan-do-study-act learning loop and plan the fix for the failure. 

If a test that is running in CI “flakes”, I use a plan-do-study-act learning loop to fix the test:

  • I plan the fix. I consider why has the test flaked and plan how to make it stable
  • Then I “do” the fix. I make the change to the test that I had planned
  • I then study the test. I the test both locally and in CI several times to see if the test passes consistently. I study the results of the test runs to see if the test is now stable and returning accurate results.
  • I then “act”. If the test still flakes I will go through the loop again. If the test now returns accurate results I will merge the fixed test so that it runs in CI.

Each time I go through the “learning loop” I learn about the application I am testing and the tool I am using to automate tests. I refer to the loop as a learning loop because I have learned so much by using it. A diagram of examples of this loop for both Playwright and Ghost Inspector is shown above.

This loop was created by Walter Shewhart to help improve quality in the manufacture of telephones and was popularised by W. Edwards Deming. It is usually known as the Deming Cycle and is widely used, for example in developing self-driving cars [1]. 

I am responsible for packs of hundreds of tests. If you use a “learning loop” when you create and maintain tests your tests will be stable and give accurate results. Automated tests do not have to be flaky.

References:

[1] Deming’s Journey to Profound Knowledge by John Willis and Derek Lewis (2023, p137)

Learning resources:

How a lone tester can work across teams

I am a lone tester and I work with two development teams. It is not possible to give both teams one hundred per cent of my time. If I don’t have a structure as to how to support both teams I can end up being pulled in multiple directions.

The structure I use is that I am a ‘chicken’ with one team and I am a ‘pig’ with the other team. If you are making a traditional British breakfast a chicken is helpful because it provides eggs, whereas the pig is committed because it provides bacon and sausages.

I am a ’chicken’ with one team. I am helpful. I go to stand up so I know what the team is working on. I offer advice and help, such as automating a test, if time allows.

I am a ‘pig’ with the other team. I am committed. I go to stand-up, planning, retros and any other meetings that the team has. I am involved in the team and support their work as a tester.

The metaphor of a ‘pig and a ‘chicken’ work, in that both teams know to what level I am available. I know also what work both teams are doing, can support them either as a ‘chicken’ or as a ‘pig’ and am able to cope with the workload from both teams. It is also a fun way of describing my relationship with the two teams.

Thank you to Rob Falla for the idea.

Some “new” practices are actually quite “old”

The Aurora 7 capsule

Recently I was fortunate to visit the Museum of Science and Industry in Chicago. The Henry Crown Space Center in the museum has the Aurora 7 capsule from NASA’s Project Mercury. It was piloted by Scott Carpenter and orbited the Earth in 1962. The capsule is only two meters long. It was a surprise to me that the capsule was so small. Please see my photo of it above. 

What is also surprising is that the software engineers on NASA’s Project Mercury used a number of techniques that we would associate with “agile”. They used iterative and incremental development[1], and “Project Mercury ran with very short (half-day) iterations that were time boxed. The development team conducted a technical review of all changes, and, interestingly, applied the Extreme Programming practice of test-first development, planning and writing tests before each micro-increment.”[1]

NASA’s Project Mercury ran from 1958 to 1963[2] and Jerry Weinberg was one of the software engineers on the project.[3] 

Some of the techniques of developing software that we can think of as being “new” are actually quite old.  

References

[1] Iterative and Incremental Development: A Brief History by Craig Larman and Victor R. Basili (2003)

[2] About Project Mercury NASA

[3] Agile Impressions by Gerald M. Weinberg (2017, p67)

Further Reading:

The Seedbed of a Quality Revolution

The sign for the retail park on the site of the Hawthorne Works

A tester’s role is not only to do the testing but also to improve quality. I visited the site of the former Hawthorne Works during a recent trip to Chicago because so many innovations in quality started there.  The Hawthorne Works was the Bell Telephone Laboratories site that manufactured the hardware for the first national telephone system in the USA.[1] It was the “Silicon Valley of ”[2] the early twentieth century and was the “seedbed of a quality revolution.”[3].

The site of the Hawthorne Works is now a retail park; the only part of the works that still stands is the water tower. I have used photos I took while visiting the site to illustrate this blog post.

Hawthorne Works water tower stands over retail units

Significant people who worked at Hawthorne Works and created philosophies and practices that have become central to quality are:

Deming referred to Shewhart as “the master”[1]. 

Deming’s philosophy is of great value as it helps me to work with development teams to improve the quality of work continually. I am enjoying reading John Willis’ new book “Deming’s Journey to Profound Knowledge, which shows how Deming’s life experiences, including those at the Hawthorne Works, influenced his philosophy.

That so many of the advances in quality made in the twentieth and twenty-first centuries are connected to the Hawthorne Works shows what a remarkable place it was.

References

[1] Architect of Quality : The Autobiography of Dr. Joseph M. Juran by Joseph M. Juran (2004, p91)

[2] Deming’s Journey to Profound Knowledge by John Willis and Derek Lewis (2023,  p46)

[3].Manufacturing the future. A History of Western Electric. by S.B. Adams and O.R. Butler (1999, p161)

[4] What is Total Quality Control? By Kaoru Ishikawa (1985, p14)

[5] Out of the Crisis by W. Edwards Deming (1986, p88)

[6] Quality or Else by Lloyd Dobyns and Clare Crawford-Mason (1991, p52)

[7] Deming’s Journey to Profound Knowledge by John Willis and Derek Lewis (2023, p79)

[8] “Deming to Devops: The Science Behind Devops” by John Willis (a five minute long video) 

[9] When Coffee and Kale Compete by Alan Klement (2018, p156)

[10] Out of the Crisis by W. Edwards Deming (1986, p168)

Further Reading

eLearning:

Hawthorne Works water tower is behind the retail units

Make test automation easier with better feedback for parameters with TypeScript

We can make it easier to automate tests by giving feedback if incorrect values are passed as parameters. Developers can get feedback when they try to pass an invalid parameter if I use union types when using TypeScript. I have been using TypeScript to automate tests with Playwright.

We often pass strings to functions as parameters. Sometimes this is because any string can be passed as a parameter. Other times there are only a limited number of strings that can be passed, such as the name of a country. If only a limited number of strings can be passed and we are using Typescript to write our tests, we can pass a union type instead of a string. 


A union type can be used to create a type that consists of only the strings that can be passed as parameters. Union types can be used when a type can be more than one type, for example, a union type could contain a string and a number. The types held in a union type can be literal types such as a string, for example, “cricket”. The union type is an ‘or’ type, so the type will be: ‘string1’ | ’string2’. An example of a union type could be defined in a .d.ts file like this:

The file containing the union type can be imported into your file like this:

The type can then be passed to the function checkBowlingAverage as the parameter bowler:

If a developer enters an invalid value such as ‘swxxing’ for the parameter bowler Visual Studio code will show the error:

If the developer’s mouse hovers overs over the invalid parameter, Visual Studio Code displays a message which shows the valid values. This makes it easy for the developer to correct the invalid parameter.

Execute Program has a lesson on Type Unions, and FrontendMasters has a lesson on Union Types. Both lessons have helped me get a better understanding of union types.

It is helpful if a larger group of people can automate tests and so it is useful to make it as easy as possible for tests to be automated. Using union types can help make it easier to automate tests because feedback is given if an incorrect value is passed.

I got such a buzz from the SIGiST23 testing conference!

Members of the BCS SIGiST Committee meeting with Rashik Parmar, the CEO of the British Computer Society at the conference
Members of the BCS SIGiST Committee meeting with Rashik Parmar, the CEO of the British Computer Society at the conference.

BCS SIGiST’s Summer Conference 2023 was a great success. We had more speakers and more attendees than at our previous conference. Everyone I spoke to enjoyed the conference and asked when the next one would be.

I got a real buzz from the day of the conference! It was great to hear so many great speakers and meet so many talented testing professionals!

SIGiST’s conference can be described as voluntarism, self-development and mutual development.

When Kaoru Ishikawa explained the basic ideas behind Quality Circles he referred to voluntarism, self-development and mutual development.[1]

The conference was about voluntarism. The BCS SIGiST committee which organised the conference are all volunteers, and all the speakers volunteered their time.

The conference was about self-development too. Each of us participating in the event wanted to learn and develop ourselves. We all learned from the presentations. When I speak at a conference I learn a lot from from preparing and giving my presentation. I am sure that the speakers at SIGiST23 also learned in this way. As the SIGiST Programme Secretary, I learned about the importance of reserve speakers. The conference programme was designed to give a balance of themes and a balance between male and female speakers. Our reserve speakers enabled this balance to be maintained when speakers had to withdraw. 

The conference was also about mutual development. We all learned from each other. We learned, not only from the presentations but also from the connections we made in the networking sessions. The exchange of ideas enabled us to gain broader perspectives on our work. 

I would like to thank the BCS SIGiST Committee members, the presenters, the BCS staff who supported the committee and the conference sponsors Keysight Technologies.

BCS SIGiST Committee:

Nicola Martin  Beth Clarke EngTech MBCS MIET Paul Mowat Adam Leon Smith FBCS Laveena Ramchandani Andrew Shaw MBCS Mihai Grigorescu Jonathon Wright Geoff Thompson

Conference Speakers:

Simon Prior Colin Hammond Divyeshkumar Patel Petros Plakogiannis Leela A.Putten Dan Ashby Nadia Soledad Cavalleri Beth Marshall Peter Johnson Dr Carl Adams Shajahan Pulikkal Hristina Koleva Arun Kumar Dutta Riya Dennis David Maynard Abigail Bangser Steve Mellor (MBCS) Paul Gerrard

The British Computer Society is a “not-for-profit”, which promotes wider social and economic progress through the advancement of information technology science and practice. I would encourage you to join: https://www.bcs.org/membership-and-registrations/become-a-member/

References

[1] What is Total Quality Control? The Japanese Way by Kaoru Ishikawa

A review of “The Art of Software Testing”

The Art of Software Testing is a book that I often refer to. The book contains valuable views on how to approach testing and clear descriptions of some test case design methodologies. It was first published in 1979 by Glenford J. Myers and an updated third edition was published in 2012. 

The book starts with “a self-assessment test” to define the test cases for a simple program and shows how difficult it is to define all the test cases for the program.

The chapter “The Psychology and the Economics of Testing” starts by discussing some definitions of testing, and argues that the most appropriate definition is that “testing is the process of executing a program with the intent of finding errors”[1]. The implications of this being the definition of testing are then explored, for instance, it means that a successful test is a test that fails. It is useful to consider how this definition of quality works in your context as this can help you develop your own testing philosophy.

Myers then discusses whether it is possible to test a program to find all of its errors. He does the maths and shows that it is not possible to exhaustively test a simple program using black-box or white-box testing. 

Ten testing principles are then defined. The tenth principle is that “testing is an extremely creative and intellectually challenging task”[2]. 

The key issue of testing is “What subset of all possible test cases has the highest possibility of detecting the most errors?”[3]. Several methodologies are described for designing test cases for both black-box and white-box testing, this includes useful descriptions of both equivalence partitioning and boundary value analysis.

“A software error occurs when the program does not do what its end user reasonably expects it to do”[4]. Testing designed to find these errors is described as high-order testing. These types of errors would not necessarily be found by unit testing.

The book also includes chapters about code inspections and walkthroughs, debugging and unit testing.

The 2012 edition contains several new chapters. 

It includes a new chapter on usability testing, which contains a useful list of questions that can be used to drive testing considerations[5].

There is a new chapter on testing in the agile environment, which contains an interesting overview of extreme programming and testing[6].

There are also new chapters on testing internet applications and mobile application testing.

The approach to testing described in The Art of Software Testing was relevant in 1979 and is still relevant today. That this approach has been relevant for so long means that it will continue to be relevant. This is a book that is very useful, and I would encourage all testers to read it. 

References:

[1]The Art of Software Testing, 3rd Edition by Glenford J. Myers, Corey Sandler and Tom Badgett (2012, p6)

[2]The Art of Software Testing, 3rd Edition by Glenford J. Myers, Corey Sandler and Tom Badgett (2012, p18)

[3]The Art of Software Testing, 3rd Edition by Glenford J. Myers, Corey Sandler and Tom Badgett (2012, p41)

[4]The Art of Software Testing, 3rd Edition by Glenford J. Myers, Corey Sandler and Tom Badgett (2012, p113)

[5]The Art of Software Testing, 3rd Edition by Glenford J. Myers, Corey Sandler and Tom Badgett (2012, p113)

[6]The Art of Software Testing, 3rd Edition by Glenford J. Myers, Corey Sandler and Tom Badgett (2012, p179)

Design a site like this with WordPress.com
Get started