Difference between revisions of "Acceptance Tests"

From MSEC
Jump to navigation Jump to search
Access restrictions were established for this page. If you see this message, you have no access to this page.
 
(9 intermediate revisions by 2 users not shown)
Line 1: Line 1:
''This is a preview version of '''MSEC'''. The full version is available to [https://www.patreon.com/continuousdelivery Patreon supporters] at the Gemini level and above.''


In [https://lisacrispin.com/2011/11/08/using-the-agile-testing-quadrants/ Brian Marick's Test Quadrant] model, [[Acceptance Tests]] are  '''"business facing"''' tests that '''"support the team"'''.
''[https://www.patreon.com/continuousdelivery Click here for details] on how to join the growing '''MSEC''' community.''  
<br/>
----


The [[Acceptance Stage]] is the first step in the [[Acceptance Phase]] of a [[Deployment Pipeline]], and an important one.  
In [https://lisacrispin.com/2011/11/08/using-the-agile-testing-quadrants/ Brian Marick's Test Quadrant] model, [[Access Full MSEC via Patreon|Acceptance Tests]] are  '''"business facing"''' tests that '''"support the team"'''.


[[Acceptance Tests]] represent one of the key 'gates' in making progress with our [[Release Candidate]]s. Once the [[Acceptance Tests]] pass, we know that the system...
The [[Access Full MSEC via Patreon|Acceptance Stage]] is the first step in the [[Acceptance Phase]] of a [[Deployment Pipeline]], and an important one.
 
[[Acceptance Tests]] represent one of the key 'gates' in making progress with our [[Access Full MSEC via Patreon|Release Candidate]]s. Once the [[Acceptance Tests]] pass, we know that the system...


<blockquote>"Does what our users want the system to do"</blockquote>
<blockquote>"Does what our users want the system to do"</blockquote>


This is the first time that we evaluate our software in the form that it will be deployed into [[Production]].
This is the first time that we evaluate our software in the form that it will be deployed into [[Access Full MSEC via Patreon|Production]].


==What are Acceptance Tests?==
==What are Acceptance Tests?==
Line 23: Line 28:
==How to Write Acceptance Tests==
==How to Write Acceptance Tests==


The most effective way to create an [[Acceptance Tests|Acceptance Test]] is to write an [[Executable Specification]], that describes the desired behaviour of the new piece of software, before we write any code. [[Acceptance Tests]] focus only on what the system/code should do; and say nothing about how it does it.  
The most effective way to create an [[Acceptance Tests|Acceptance Test]] is to write an [[Access Full MSEC via Patreon|Executable Specification]], that describes the desired behaviour of the new piece of software, before we write any code. [[Acceptance Tests]] focus only on what the system/code should do; and say nothing about how it does it.  


We do this for every change that we intend to make. These specifications guide the development process: we work to fulfil these [[Executable Specification|Executable Specifications]] as we carry out lower level, [[Test Driven Development]] testing until the specification is met.
We do this for every change that we intend to make. These specifications guide the development process: we work to fulfil these [[Access Full MSEC via Patreon|Executable Specifications]] as we carry out lower level, [[Access Full MSEC via Patreon|Test Driven Development]] testing until the specification is met.


We make the scenarios that these tests capture atomic, and don’t share test-data between test cases. Each test-case starts from the assumption of a running, functioning system, but one that contains no data.
We make the scenarios that these tests capture atomic, and don’t share test-data between test cases. Each test-case starts from the assumption of a running, functioning system, but one that contains no data.
Line 31: Line 36:
The [[Acceptance Tests]] are business facing, i.e written from the perspective of external consumers of the system. [[Acceptance Tests]] are designed to support programming and guide the development of code to meet the users' need.  
The [[Acceptance Tests]] are business facing, i.e written from the perspective of external consumers of the system. [[Acceptance Tests]] are designed to support programming and guide the development of code to meet the users' need.  


These functional, whole-system tests are difficult to get right. It helps to consciously work to separate concerns in our testing, to allow the system under test to change, without invalidating the tests that evaluate its behaviour. For more detail on that, see the [[Four-Layer Approach]] to '''Acceptance Testing'''.
These functional, whole-system tests are difficult to get right. It helps to consciously work to separate concerns in our testing, to allow the system under test to change, without invalidating the tests that evaluate its behaviour. For more detail on that, see the [[Access Full MSEC via Patreon|Four-Layer Approach]] to '''Acceptance Testing'''.




==Automating the Acceptance Stage==
==Automating the Acceptance Stage==


We aim to eliminate the need for manual regression testing, and automate any and all repeatable processes, in the [[Deployment Pipeline]]. Manual processes are slow, unreliable and expensive. There is a role for [[Manual Testing]], and we should use human beings where they can have the best effect - in creative processes and in qualitative assessments. We use our computers to carry out routine, repeatable tasks. They are much more efficient and reliable than we are, for that kind of work.  
We aim to eliminate the need for manual regression testing, and automate any and all repeatable processes, in the [[Deployment Pipeline]]. Manual processes are slow, unreliable and expensive. There is a role for [[Access Full MSEC via Patreon|Manual Testing]], and we should use human beings where they can have the best effect - in creative processes and in qualitative assessments. We use our computers to carry out routine, repeatable tasks. They are much more efficient and reliable than we are, for that kind of work.  


As well as testing the code, we can test the configuration and deployment.  
As well as testing the code, we can test the configuration and deployment.  


By using the same (or as close as possible) mechanisms, tools and techniques to deploy into test environments, as we will use when we get to [[Production]], we can get a high level of confidence that everything will work together. By the time we get to [[Production]], everything has been tested together multiple times, we have reduced the likelihood of ‘unpleasant surprises’, and we have a high level of confidence that everything will work.
By using the same (or as close as possible) mechanisms, tools and techniques to deploy into test environments, as we will use when we get to [[Access Full MSEC via Patreon|Production]], we can get a high level of confidence that everything will work together. By the time we get to [[Access Full MSEC via Patreon|Production]], everything has been tested together multiple times, we have reduced the likelihood of ‘unpleasant surprises’, and we have a high level of confidence that everything will work.


[[Acceptance Tests]] can act as a kind of whole system super-integration test. If we assemble all the components that represent a [[Deployable Unit of Software]] and evaluate them together, if the [[Acceptance Tests]] pass, we know that they are configured appropriately, work together and can be deployed successfully.
[[Acceptance Tests]] can act as a kind of whole system super-integration test. If we assemble all the components that represent a [[Access Full MSEC via Patreon|Deployable Unit of Software]] and evaluate them together, if the [[Acceptance Tests]] pass, we know that they are configured appropriately, work together and can be deployed successfully.


==Scaling Up==
==Scaling Up==
Line 55: Line 60:


:* Incorporate [[Acceptance Tests]] into the development process from the start.
:* Incorporate [[Acceptance Tests]] into the development process from the start.
:* Create an [[Executable Specification]] for the desired behaviour of each new piece of software before starting on the code.
:* Create an [[Access Full MSEC via Patreon|Executable Specification]] for the desired behaviour of each new piece of software before starting on the code.
:* Think of the least technical person who understands the problem-domain, reading the Acceptance Tests. The tests should make sense to that person.
:* Think of the least technical person who understands the problem-domain, reading the Acceptance Tests. The tests should make sense to that person.
:* Create a new [[Acceptance Tests|Acceptance Test]] for every [[Acceptance Criteria]] for every [[User Story]].
:* Create a new [[Acceptance Tests|Acceptance Test]] for every [[Access Full MSEC via Patreon|Acceptance Criteria]] for every [[Access Full MSEC via Patreon|User Story]].
:* Make it easy to identify [[Acceptance Tests]] and differentiate them from other sorts of tests.
:* Make it easy to identify [[Acceptance Tests]] and differentiate them from other sorts of tests.
:* Automate control of test environments, and [[Control the Variables]], so the tests are reproducible.
:* Automate control of test environments, and [[Access Full MSEC via Patreon|Control the Variables]], so the tests are reproducible.
:* Make it easy for development teams to run [[Acceptance Tests]] and get results, by automating deployment to the test environment and automating control of test execution.
:* Make it easy for development teams to run [[Acceptance Tests]] and get results, by automating deployment to the test environment and automating control of test execution.
:* Automate the collection of results, so developers can easily get the answer to the question "Has this Release Candidate passed [[Acceptance Testing]]?".
:* Automate the collection of results, so developers can easily get the answer to the question "Has this Release Candidate passed [[Acceptance Testing]]?".
Line 68: Line 73:
<youtube width=330 height=160>Ju50D11EIoE</youtube>&nbsp;<youtube width=330 height=160>9P5WG8CkPrQ</youtube>&nbsp;<youtube width=330 height=160>JDD5EEJgpHU</youtube>
<youtube width=330 height=160>Ju50D11EIoE</youtube>&nbsp;<youtube width=330 height=160>9P5WG8CkPrQ</youtube>&nbsp;<youtube width=330 height=160>JDD5EEJgpHU</youtube>


[[File:Screenshot 2021-09-22 at 17.46.19.png|link=https://courses.cd.training|330px]]
[[File:Screenshot 2021-09-22 at 17.46.19.png|link=https://courses.cd.training/courses/acceptance-testing-webinar|330px]]


<br/>
<br/>
----
----
''This is a preview version of '''MSEC'''. The full version is available to [https://www.patreon.com/continuousdelivery Patreon supporters] at the Gemini level and above.''
''[https://www.patreon.com/continuousdelivery Click here for details] on how to join the growing '''MSEC''' community.''
----
<span style="color:white;background:#3399ff;padding:4px"><big>'''[[Main Page|<span style="color:white">Home</span>]]&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[Main Page|<span style="color:white">< Previous</span>]]&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[Access Full MSEC via Patreon|<span style="color:white">Next ></span>]]'''</big></span>
<!--
<span style="color:white;background:#3399ff;padding:4px"><big>'''[[Main Page|<span style="color:white">Home</span>]]&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[Main Page|<span style="color:white">< Previous</span>]]&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[Executable Specification|<span style="color:white">Next ></span>]]'''</big></span>
<span style="color:white;background:#3399ff;padding:4px"><big>'''[[Main Page|<span style="color:white">Home</span>]]&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[Main Page|<span style="color:white">< Previous</span>]]&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[Executable Specification|<span style="color:white">Next ></span>]]'''</big></span>
<big>'''<span style="color:white;background:#61BA38;padding:4px">[[Main Page|<span style="color:white">Home</span>]]&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[Three Types of Test in TDD|<span style="color:white">< Previous</span>]]&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[Getting Started with TDD|<span style="color:white">Next ></span>]]</span>'''</big>
-->

Latest revision as of 15:13, 17 January 2023

This is a preview version of MSEC. The full version is available to Patreon supporters at the Gemini level and above.

Click here for details on how to join the growing MSEC community.


In Brian Marick's Test Quadrant model, Acceptance Tests are "business facing" tests that "support the team".

The Acceptance Stage is the first step in the Acceptance Phase of a Deployment Pipeline, and an important one.

Acceptance Tests represent one of the key 'gates' in making progress with our Release Candidates. Once the Acceptance Tests pass, we know that the system...

"Does what our users want the system to do"

This is the first time that we evaluate our software in the form that it will be deployed into Production.

What are Acceptance Tests?

Effective Acceptance Tests:

  • Are written from the perspective of an external user of the system
  • Evaluate the system in life-like scenarios
  • Are evaluated in production-like test environments
  • Interact with the System Under Test (SUT) through public interfaces (no back-door access for tests)
  • Focus only on WHAT the system does, not HOW it does it
  • Are part of a systemic, strategic approach to testing and Continuous Delivery

How to Write Acceptance Tests

The most effective way to create an Acceptance Test is to write an Executable Specification, that describes the desired behaviour of the new piece of software, before we write any code. Acceptance Tests focus only on what the system/code should do; and say nothing about how it does it.

We do this for every change that we intend to make. These specifications guide the development process: we work to fulfil these Executable Specifications as we carry out lower level, Test Driven Development testing until the specification is met.

We make the scenarios that these tests capture atomic, and don’t share test-data between test cases. Each test-case starts from the assumption of a running, functioning system, but one that contains no data.

The Acceptance Tests are business facing, i.e written from the perspective of external consumers of the system. Acceptance Tests are designed to support programming and guide the development of code to meet the users' need.

These functional, whole-system tests are difficult to get right. It helps to consciously work to separate concerns in our testing, to allow the system under test to change, without invalidating the tests that evaluate its behaviour. For more detail on that, see the Four-Layer Approach to Acceptance Testing.


Automating the Acceptance Stage

We aim to eliminate the need for manual regression testing, and automate any and all repeatable processes, in the Deployment Pipeline. Manual processes are slow, unreliable and expensive. There is a role for Manual Testing, and we should use human beings where they can have the best effect - in creative processes and in qualitative assessments. We use our computers to carry out routine, repeatable tasks. They are much more efficient and reliable than we are, for that kind of work.

As well as testing the code, we can test the configuration and deployment.

By using the same (or as close as possible) mechanisms, tools and techniques to deploy into test environments, as we will use when we get to Production, we can get a high level of confidence that everything will work together. By the time we get to Production, everything has been tested together multiple times, we have reduced the likelihood of ‘unpleasant surprises’, and we have a high level of confidence that everything will work.

Acceptance Tests can act as a kind of whole system super-integration test. If we assemble all the components that represent a Deployable Unit of Software and evaluate them together, if the Acceptance Tests pass, we know that they are configured appropriately, work together and can be deployed successfully.

Scaling Up

Acceptance Tests take time and can be expensive to develop, maintain and run. We need enough tests to spot unexpected problems, but we don't, usually, have unlimited resources. We can design the test infrastructure to run multiple tests in parallel, within available resources, having regard to which tests take the longest amount of time.

We can use sampling strategies, and grow the number and complexity of Acceptance Tests as the software grows. In this way, we achieve test coverage as required to determine the releasability of the software, and not as a goal in itself.

The aim should be to allow developers to add any test that they need, but also for developers to care enough to think about the cost, in time, of each test. Again we have the trade-off between thoroughness and speed. Both matter, but ‘slow and thorough’ is as bad as 'fast and sketchy’!

Tips for Writing Acceptance Tests

  • Incorporate Acceptance Tests into the development process from the start.
  • Create an Executable Specification for the desired behaviour of each new piece of software before starting on the code.
  • Think of the least technical person who understands the problem-domain, reading the Acceptance Tests. The tests should make sense to that person.
  • Create a new Acceptance Test for every Acceptance Criteria for every User Story.
  • Make it easy to identify Acceptance Tests and differentiate them from other sorts of tests.
  • Automate control of test environments, and Control the Variables, so the tests are reproducible.
  • Make it easy for development teams to run Acceptance Tests and get results, by automating deployment to the test environment and automating control of test execution.
  • Automate the collection of results, so developers can easily get the answer to the question "Has this Release Candidate passed Acceptance Testing?".
  • Don't chase test coverage as a goal - good coverage comes as a side-effect of good practice, but makes a poor target.
  • Leave room to scale up the number and complexity of Acceptance Tests, as the software grows in complexity.

Further Viewing

  

Screenshot 2021-09-22 at 17.46.19.png



This is a preview version of MSEC. The full version is available to Patreon supporters at the Gemini level and above.

Click here for details on how to join the growing MSEC community.


Home     < Previous     Next >