Paul Gerrard paulgerrardconsultingcom gerrardconsultingcom Overview What is Agile Test Strategy Project Profiling Test Strategy as Agile Interventions Test Automation Whats Left Summary ID: 635512
Download Presentation The PPT/PDF document "Agile Test Strategy @ paul_gerrard" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Agile Test Strategy
@paul_gerrard
Paul
Gerrard
paul@gerrardconsulting.com
gerrardconsulting.comSlide2
Overview
What is Agile Test Strategy?Project Profiling(Test Strategy as) Agile InterventionsTest AutomationWhat’s Left?SummaryQ&A
Intelligent Definition and AssuranceSlide
2Slide3
What is Agile Test Strategy?
Agile Strategy – an oxymoron?Slide4
Agile Test Strategy
What do we mean by this?AGILE Test Strategy – how to create a test strategy in an Agile way?AGILE Test Strategy – a test strategy for an Agile project?
We’ll look at how we created an Agile approach to strategy, but we’ll spend more time on strategy for an Agile project.Intelligent Definition and Assurance
Slide 4Slide5
Google “Agile Test Strategy”
There are plenty of recipes out thereMost offer a selection of techniques but don’t provide much guidance on how to blend themWe need to know how to make choices, not just know what choices existStrategy is a thought process, not a document
Although you might document the ideas for reuse, as a reminder or ‘for the record’.
Intelligent Definition and AssuranceSlide 5Slide6
Agile governance
Governance is the act of governing. It relates to decisions that define expectations, grant power, or verify performanceWikipedia
Define expectations – DEFINITION of needGrant power
– DELEGATION to a project teamVerify performance – ASSURANCE of solution.
Intelligent Definition and AssuranceSlide 6Slide7
Strategy helps you decide what to do
The strategy presents some decisions that can be made ahead of timeDefines the process or method or information that will allow decisions to be made (in project)Sets out the principles (or process) to follow for uncertain situations or unplanned events
In a structured/waterfall environment, most questions answered off-the-shelf – “A-style, ready for anything”In an Agile environment – might have some ready-made policies but we manage scope and adapt (mostly C?)
Intelligent Definition and AssuranceSlide 7Slide8
Contexts of Test
Strategy
TestStrategy
Risks
GoalsConstraintsHuman resource
Environment
Timescales
Process
(lack of?)
Contract
Culture
Opportunities
User involvement
Automation
De-Duplication
Early Testing
Skills
Communication
Axioms
ArtefactsSlide9
Traditional v Agile test strategy
Traditional – structured, goal/risk-drivenIdentify stakeholders; what are their goals?Product risk analysisAllocate risks/goals to test stagesFormulate test stage definitions (entry/exit criteria, environments, tools etc. etc.Agile – interventionist, consensus-driven
Project profiling to set the testing themeIdentify testing interventions in the Agile processTest policy overlays the process; catches exceptions
Intelligent Definition and AssuranceSlide 9Slide10
Project Profiling
Select a profile for your project first, then choose the aspects of test strategy that suite your projectSlide11
Template-driven? Bah!
So this is just a template copy and edit process?Won’t you always end up with the same document?Profiling doesn’t need to be prescriptiveNo need to write a document if you don’t need toBut if company policy or common sense dictates certain approaches, save yourself some timeCreate a set of deeper, more detailed questions to be answered (Pocketbook)
Profilers are really just checklists: heuristic guidelines designed to help you make choices and trade-offs.
Intelligent Definition and AssuranceSlide 11Slide12
Cerise
Orange
Green
Test Plan Items
Product Risks
Project Profiler
Risk Profiler
Project Plan
Test
Assurance
Project Manager
Business, Project Team
and Boards
Consultation
Blue
Unknowns
Using
a Project
Profiler to Derive a Test Strategy and Project
Plan
(A government client example)
The Project Profiler (with Test Assurance) helps Project Managers to:
Select a project style that fits (Waterfall or Agile)
Identify the product risks that need testing
Identify test activities to include in project plans
Carefully define the scope of the project
Environment
Story Guideline
Tools
Incident Mgt.
Waterfall
Test Strategy
SCRUM/Agile
Test
Strategy
Intelligent Definition and Assurance
12Slide13
Project profiling process
Task
1
Have the Information you need to hand
2Project Profiler (section 3):Select the statements that best match your project context. The Blue column indicates that you need more information – consult your stakeholders, team or relevant Board(s).3Generic Risk Profiler (section 4):Consider the generic project risks – which are significant for your project? Can testing help?4Product Risk Profiler (Section 5):Consider the functional and non-functional risks that most concern your stakeholders – should they be tested?
5
Actions and Test Activities (Section 6):Consider the actions that prompt your ‘next steps’ and the test activities that should be incorporated into your project plan.
6
Create your Test Strategy from the Test Strategy Framework Template
7
Incorporate the activities from stage 5 and identified in 6 into your Project Plan.
Intelligent Definition and Assurance
13Slide14
Project Profiler (part of section 3)
Project Aspect
Cerise
Orange
GreenBlue
Responsibility for Acceptance
Users will take responsibility for UAT; they have UAT experience
Users will be responsible for UAT but have no test experience
Users will take part in UAT or witness tests at critical periods, and will review the outcome
Users are unwilling/unable to take part in UAT; reluctant to make the acceptance decision or not known
Requirements (Sources of Knowledge)
New system replaces a well-understood existing system; users have clear vision of system goals and prefer to document their requirements up-front
Users want to collaborate to jointly define requirements and meet them incrementally or iteratively
Users put the onus of requirements elicitation on the project; requirements and the solution will evolve
Inexperienced users who are unable or unwilling to collaborate with requirements gathering
Requirements Stability
New system is a functional replacement of an existing system or a well-defined process (requirements can be fixed early on)
New system replaces an existing system with enhancements or an established (but not necessarily documented process)
New system supports a new business need; business process exists but will change/evolve; users have experience of requirements
New system supports a new business need; business process is not yet known; users have no experience or requirements
Visibility, Formality
High visibility/risk to general public; formal progress reporting required at board level; fixed scope and deliverables; formal approvals and sign-offs
High visibility/risk to business; formal progress reporting required; some defined deliverables, some deliverables will emerge/evolve; some approvals and sign-offs
Relatively low business-risk; informal progress reporting is acceptable; partial solution may suffice, incremental/iterative delivery
Potentially, high visibility, high risk project, uncertain impact on the business
External Dependencies
More than one or new external suppliers responsible for development (and supplier testing)
Single, known supplier responsible for development (and supplier testing)
In-house development, no external dependencies
Dependencies on external suppliers, their responsibilities or competence not yet known
Etc.
Etc.
Etc.
Etc.
Etc.
Intelligent Definition and Assurance
14Slide15
Project types - examples
Cerise
Structured, waterfall style of project (and includes COTS projects)
Orange
Iterative/prototyping style of project using SCRUM in a formal way and having dedicated resources for the Business Analyst and Tester roles.GreenA project using SCRUM in a less formal way but not having dedicated resources for the Business Analyst and/or the Tester roles.BlueBlue column statements describe where there is insufficient information available to identify the style of project and the recommendation must be that some further investigation is required.
Intelligent Definition and Assurance
15Slide16
(Test Strategy as)Agile Interventions
I’m using Scrum/Sprint terminology, but you don’t have to of courseSlide17
Interventions (a government client example)
On the following slides, we highlight 8 interventionsSome are test phases, but some aren’t
No.
Activity
When?1Story ChallengeAs stories are added to the Product Backlog2Story DefinitionAs stories are added to a Sprint Backlog
These activities are repeated for each Sprint iteration
3
Daily Stand-UpOnce per day during the Sprint
4
Story Refinement
Occurs throughout the Sprint as new information emerges
5
Developer Testing
Occurs throughout the Sprint as the developer codes the stories
6
Integration (and incremental System) Testing
During and at the end of each sprint, including the final sprint
7
System Testing
At the end of each sprint, including the final sprint
8
User Acceptance Testing
At the end of each sprint, including the final sprint
9
Non-functional Testing and Pre-Production Testing
Expected to take place on an as-needs basis.
Intelligent Definition and Assurance
Slide
17Slide18
Integration into
Existing Code baseAutomated testing
New Code
8. User Test
7. System TestSprint 1Developed Stories
Developed Stories
Developed Stories
Sprint 3
Sprint 2
Sprint Backlog
Sprint Backlog
Sprint Backlog
Story Challenge
Suggest ‘what-ifs’ to challenge new stories and define story headlines
Increasing Scope of Sys. Test and UAT
Increasing Scope of Integration, System and Users Testing
2. Story Definition
Introduce scenarios to enhance the Acceptance Criteria
Complete Tests after Final Sprint
Project Level Test Activities
(This diagram shows three
sprints,
but there could be more or fewer)
6. Integration Test
6. Integration Test
6. Integration TestSlide19
Integration into
Existing Code baseAutomated testing
New Code
8. User Test
7. System TestSprint 1Developed Stories
Developed Stories
Developed Stories
Sprint 3
Sprint 2
Sprint Backlog
Sprint Backlog
Sprint Backlog
Story Challenge
Suggest ‘what-ifs’ to challenge new stories and define story headlines
Increasing Scope of Sys. Test and UAT
Increasing Scope of Integration, System and Users Testing
2. Story Definition
Introduce scenarios to enhance the Acceptance Criteria
Complete Tests after Final Sprint
Project Level Test Activities
(This diagram shows three sprints, but there could be more or fewer)
6. Integration Test
6. Integration Test
6. Integration TestSlide20
Integration into
Existing Code baseAutomated testing
New Code
8. User Test
7. System TestSprint 1Developed Stories
Developed Stories
Developed Stories
Sprint 3
Sprint 2
Sprint Backlog
Sprint Backlog
Sprint Backlog
Story Challenge
Suggest ‘what-ifs’ to challenge new stories and define story headlines
Increasing Scope of Sys. Test and UAT
Increasing Scope of Integration, System and Users Testing
2. Story Definition
Introduce scenarios to enhance the Acceptance Criteria
Complete Tests after Final Sprint
Project Level Test Activities
(This diagram shows three sprints, but there could be more or fewer)
6. Integration Test
6. Integration Test
6. Integration TestSlide21
Integration into
Existing Code baseAutomated testing
New Code
8. User Test
7. System TestSprint 1Developed Stories
Developed Stories
Developed Stories
Sprint 3
Sprint 2
Sprint Backlog
Sprint Backlog
Sprint Backlog
Story Challenge
Suggest ‘what-ifs’ to challenge new stories and define story headlines
Increasing Scope of Int. Sys. and UAT
Increasing Scope of Integration, System and Users Testing
2. Story Definition
Introduce scenarios to enhance the Acceptance Criteria
Complete Tests after Final Sprint
Project Level Test Activities
(This diagram shows three sprints, but there could be more or fewer)
6. Integration Test
6. Integration Test
6. Integration TestSlide22
Integration into
Existing Code baseAutomated testing
New Code
8. User Test
7. System TestSprint 1Developed Stories
Developed Stories
Developed Stories
Sprint 3
Sprint 2
Sprint Backlog
Sprint Backlog
Sprint Backlog
Story Challenge
Suggest ‘what-ifs’ to challenge new stories and define story headlines
Increasing Scope of Int. Sys. and UAT
Increasing Scope of Integration, System and Users Testing
2. Story Definition
Introduce scenarios to enhance the Acceptance Criteria
Complete Tests after Final Sprint
Project Level Test Activities
(This diagram shows three sprints, but there could be more or fewer)
6. Integration Test
6. Integration Test
6. Integration TestSlide23
Daily Scrum
Stand-Up
Meeting
24 Hours
2-4 Weeks
Backlog tasks
expanded
by team
Potentially Shippable
Product increment
Product backlog
As prioritised by Product Owner
Sprint Backlog
4. Story Refinement
Refine scenarios to enhance story definition, create system tests as stories, as required
6) Integration/System Testing
Incorporate automated unit tests into the CI regime.
On weekly basis and at end of Sprint, deploy to System test environment and tester runs system tests.
3. Daily Stand-Up
Report anomalies found, stories tested, amended, created
5) Developer Testing
Private ad-hoc tests and build/run automated unit tests
Test Activities in the Sprint
Intelligent Definition and Assurance
23Slide24
Daily Scrum
Stand-Up
Meeting
24 Hours
2-4 Weeks
Backlog tasks
expanded
by team
Potentially Shippable
Product increment
Product backlog
As prioritised by Product Owner
Sprint Backlog
4. Story Refinement
Refine scenarios to enhance story definition, create system tests as stories, as required
6) Integration/System Testing
Incorporate automated unit tests into the CI regime.
On weekly basis and at end of Sprint, deploy to System test environment and tester runs system tests.
3. Daily Stand-Up
Report anomalies found, stories tested, amended, created
5) Developer Testing
Private ad-hoc tests and build/run automated unit tests
Test Activities in the Sprint
Intelligent Definition and Assurance
24Slide25
Daily Scrum
Stand-Up
Meeting
24 Hours
2-4 Weeks
Backlog tasks
expanded
by team
Potentially Shippable
Product increment
Product backlog
As prioritised by Product Owner
Sprint Backlog
4. Story Refinement
Refine scenarios to enhance story definition, create system tests as stories, as required
6) Integration/System Testing
Incorporate automated unit tests into the CI regime.
On weekly basis and at end of Sprint, deploy to System test environment and tester runs system tests.
3. Daily Stand-Up
Report anomalies found, stories tested, amended, created
5) Developer Testing
Private ad-hoc tests and build/run automated unit tests
Test Activities in the Sprint
Intelligent Definition and Assurance
25Slide26
Daily Scrum
Stand-Up
Meeting
24 Hours
2-4 Weeks
Backlog tasks
expanded
by team
Potentially Shippable
Product increment
Product backlog
As prioritised by Product Owner
Sprint Backlog
4. Story Refinement
Refine scenarios to enhance story definition, create system tests as stories, as required
6) Integration/System Testing
Incorporate automated unit tests into the CI regime.
On weekly basis and at end of Sprint, deploy to System test environment and tester runs system tests.
3. Daily Stand-Up
Report anomalies found, stories tested, amended, created
5) Developer Testing
Private ad-hoc tests and build/run automated unit tests
Test Activities in the Sprint
Intelligent Definition and Assurance
26Slide27
4. Story Refinement (example definition)
Objectives
To define acceptance criteria for all stories that are included in a Sprint as they are worked on by development
To define scenarios that describe the tests and expected behaviours of the System
Improve understanding of the requirement and communicate anomalies to developersTo identify System Tests that exercise functionality of multiple stories that can be system tested in this sprintTo assure the completeness for stories in the current SprintWhat’s being tested?
Stories to be included in the current Sprint
Deliverables
Refined story definitions with defined acceptance criteria and scenarios, where appropriate
System tests
Responsibilities (Orange)
Tester – challenges stories by suggesting potential scenarios, new stories, story merges and splits; performs ad-hoc testing with/on behalf of developers; assures completeness of stories.
Developers – considers stories, evaluates impact on development
Product Owner or Analyst – collates feedback and decisions on stories
Product Owner – approves changes to stories, accepts completeness of stories
Scrum Master – monitors progress; evaluates impact on resources and schedules
Responsibilities (Green)
Not performed in Green projects
Baseline
Story Guideline (reference 3)
Entry Criteria
On commencement of the Sprint
Exit Criteria
When all stories within a Sprint are considered complete
Queries, anomalies, discrepancies and inconsistencies have been eliminated or explained
System Tests appropriate to the Sprint have been defined
Definition of acceptance is agreed with Product Owner
Intelligent Definition and Assurance
27Slide28
Test Automation
Could you create an Agile Test Strategy without automation?Slide29
Brian Marick’s Testing quadrants
Intelligent Definition and Assurance
29Slide30
Test Automation Pyramid – Lisa Crispin’s version (Google others)
Pyramid reflects the relative numbers of tests
Focus on unit/componentAcceptance of “Services”
GUI are end-to-endManual checking the exception?Intelligent Definition and Assurance
30Slide31
GUI Test Framework
GUI Test Tool
Browser
Inter/Intranet
HTTP/SWeb ServerApp. ServerDB ServerHTTP/S
HTTP Driver
Stories/
Scenarios
Unit Test Framework
Test Code
HTTP/S
Unit Test Framework
Test Code
API
4.
Programmers write low level HTTP GET/POST calls through a driver that simulates a browser
3.
Non-Technical testers write scripts
Tools Experts write interface
2.
Technical Testers code scripts directly
1.
Programmers write unit tests or execute embedded unit tests using a unit test framework to test components
Where do you automate?Slide32
Distributed testing
Use business stories and scenarios/acceptance criteria to validate requirementsReuse those stories to feed ‘Acceptance-Driven Development’ BDD/TDD processAutomated tests are an Anti-Regression tacticAutomated tests don’t replicate manual tests; think of them as a ‘change trip-wire’ that triggers an alarm, if tripped.
Intelligent Definition and Assurance32Slide33
Deriving scenarios to test
To understand feature scope?To get stakeholder to accept?To validate the requirement?To estimate the work to build this feature?To system test this feature?To unit test this feature?
Scenarios are created to meet several goals
Story Challenge
Story RefinementStory DefinitionIteration PlanningSystem TestingDeveloper TestingIntelligent Definition and Assurance33Slide34
What’s Left?Slide35
Other aspects of test policy
Definitions (of done etc.)Incident managementTest automationStory format e.g. GherkinEnvironment request and managementRegression testing (at what levels)Test deliverables and documentation.
Intelligent Definition and Assurance
Slide 35Slide36
The Three Amigos
Business AnalystLiaises and manages stakeholders and their needsTransforms business requirements into specification (at multiple levels)DeveloperScopes, designs, builds, tests and delivers featuresTesterChallenges the thinking on the project
Performs ‘Assurance in the small’.Intelligent Definition and Assurance
Slide 36Slide37
The tester’s contribution to testing
_________ feature/story acceptance criteria_________ the developers to unit test (auto)_________ feature/story acceptance_________ acceptance test_________ service/UI level automationScope: from low-level detail to system integrationLiaison with
integration testers and feedbackFill in the blanks yourself; negotiate with your team.
Intelligent Definition and AssuranceSlide 37Slide38
SummarySlide39
Close
Agile test strategy has its place and many aspects of test can be pre-definedImportantly, we use a principled approach to deal with the unexpectedProject profiling can helpConsider testing as interventions, rather than test phasesTest automation in the context of Specification by Example, requirements validation, BDD, TDD.
Intelligent Definition and Assurance
Slide 39Slide40
Agile Test Strategy
@paul_gerrard
Paul
Gerrard
paul@gerrardconsulting.com
gerrardconsulting.com