Silent Killer of DevOps Ingo Philipp Businesses must continuously exploit digital technologies to both create new sources of customer value and increase operational agility ID: 816155
Download The PPT/PDF document "Traditional Testing The" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Traditional TestingThe Silent Killer of DevOps
Ingo
Philipp
Slide2Businesses must continuously exploitdigital technologies to both
create
new sources
of
customer value
and
increase operational agility in service of customers
.
©
Forrester
Research
Slide3SoftwareDigitalDisruption
Disrupt
Get
Disrupted
Across industries,
companies face the
challenge of
software
-
led
transformation
Slide4PlanCodeVersionBuild
Provision
Test
Deploy
Monitor
Identify
Needs
Receive
Money
Build
Solution
Accelerate
Value Exchange
Fail
Fast
›› Fail
Cheap
›› Fail
Forward
Slide5Requirements
Design
Implementation
Testing
Acceptance
Deployment
Inception
Elaboration
Construction
Transition
Development
Operations
DevOps
Agile
Event
Horizon
Digital
Singularity
t
ime to react on change
Incremental
Rational Unified Process
4
Agile
Scrum, Kanban
2
Continuous
DevOps
1
6
+
Big Bang
Waterfall
Development
Response
Flexible
Framework
Rigid
Rules
Following a Plan
Contract Negotiation
Copious Documentation
Processes & Tools
Individuals
& Interactions
Working
Software
Customer C
ollaboration
Responding to
Change
Revolution
Evolution
Slide6It’s not the strongest that survive, nor the most intelligent, but the one most responsive to change
Charles
Darwin
Because software
is not written, it is
rewritten
Slide780
%
Manual
Testing
2
0
%
Automated
Testing
The
reliance on
manual testing
is the
top technical challenge
in
app
development
World Quality Report, 2015/16
Test
automation
requires
developers
UI
Load
API
Present
Traditional Testing
Slide8+
85
%
Automated
Testing
Future
Continuous Testing
Exploratory
Tests
Automated UI
Tests
API
Tests
Orchestrated Service
Virtualization
Load
Tests
80
%
Manual
Testing
2
0
%
Automated
Testing
Present
Traditional Testing
Slide9100
%
0
%
Sprint
Start
Sprint
End
API
UI
Start testing
early
,
s
hift
left
test automation
Degree of
Completion
Time
Efficiency
Gain
+
4
x
Creation
+
6
x
Maintenance
+
20
x
Execution
Slide10*voke, market snapshot report on service virtualization
-
2012
On average, organizations require access to 33
systems
for development or testing.
18
Average # of systems with
unrestricted access
96
% of
testers
have
restricted
test lab access
Testing
Present
Enterprise
system landscapes are alike
disease gene
networks
.
Welcome to the
tester’s
hell
.
Slide11Automation is the dictate of today. It doesn’tmake testing easy, it makes testing possible.
Lesson
Learned
© Wolfgang
Platz
Slide12A
B
Visiting
London
~
100 Billion Billion Billion Billion Billion Billion Billion
Combinations
=
50
Slide13Testing harder isn’t theanswer, testing
smarter
is!
Lesson
Learned
© Wolfgang
Platz
Slide14LowRiskHigh
Risk
Medium
Risk
80
%
20
%
Business Risk
Coverage
Test
Cases
The time
needed
for
testing is
infinitely larger
than
the time
available
.
Critical
L
imit
Have the right test cases.
Do the
right
things.
Slide15R
isks
Performance
Issue
Usability
Issue
Functional
Issue
Stability
Issue
Reliability
Issue
Scalability
Issue
Coherence
Issue
Understandability
Issue
Testability
Issue
Convenience
Issue
Security
Issue
Accessibility
Issue
Slide16Testing is exactly likewashing a pig. Because it’s messy. It has no rules. No clear beginning, middle
, or
end. It’s kind of a pain in the ass, and when you’re done you’re not sure if the pig is really clean or even why you were washing a pig in the first place.
© Luke
Sullivan
Slide17That’s what we could possibly know
That’s what we
know
Knowledge Gap
That’s
what we
check
Problem
© Richard
Bradshaw
Closing the
gap between
what
we
know
and
what
we
don’t know
Testing
Purpose
Exploratory
Testing
Confirmatory
Testing
Slide18Mechanical TestingProcess pre-defined data in pre-designed steps
Monitor
Known Risks
C
onfirm what you already know
Low
Information Value
Repeat what you have already learned
High
Risk Coverage
Easy to scale because it’s parallelizable
Change Detector
Confirmatory
Testing
« Demonstrate your
depth
of
knowledge »
Exploratory
Testing
« Demonstrate your
breadth
of knowledge »
Intelligent
Testing
Create new test ideas based on what you have learned
Analyze
Potential Risks
F
ocus on the things you don’t know
High
Information
Value
Learn something new
Low
Risk Coverage
Hard to scale continuously because it relies on humans
Problem Detector
© M.
Bolton
,
C.
Kaner
,
J.
Bach
Slide19Change Detector
Checking
« Requires
Processing
»
Exploratory
Testing« Demonstrate your
breadth
of knowledge »
Intelligent
Testing
Create new test ideas based on what you have learned
Analyze
Potential Risks
F
ocus on the things you don’t know
High
Information
Value
Learn something new
Low
Risk Coverage
Hard to scale continuously because it relies on humans
Problem Detector
Evaluate a product by applying
algorithmic
decision rules to specific observations of a product
© M.
Bolton
,
C.
Kaner
,
J.
Bach
Slide20Change Detector
Checking
« Requires
Processing
»
Exploring
« Requires Thinking
»
Problem Detector
Evaluate a product by applying
algorithmic
decision rules to specific observations of a product
© M.
Bolton
,
C.
Kaner
,
J.
Bach
Evaluate a
product
by
learning
about
it
through
exploration
and
experimentation
Slide21Checking« Requires Processing
»
Exploring
« Requires
Thinking
»
Problem Detector
Evaluate a
product
by
learning
about
it
through
exploration
and
experimentation
Verify through
Instructions
Pay attention to
Deviations
Create
Test Cases
Follow
Procedure
Examine
Requirements
Factory
Process
© Rich
Rogers
Slide22Checking« Requires Processing
»
Exploring
« Requires
Thinking
»
© Rich
Rogers
Verify through
Instructions
Pay attention to
Deviations
Create
Test Cases
Follow
Procedure
Examine
Requirements
Factory
Process
Investigate through
Experiments
Pay attention to
Oracles
Create
Test Ideas
Follow
Clues
Examine
Risks
Adaptive
Investigation
Slide23CheckingEfficient Confirmatory TestingExploringEffective Exploratory Testing
Testing
Thorough
Testing
+
=
Agile
Testing Equation
© Elisabeth
Hendrickson
Slide24If this doesn’t make an impact, thenits absence doesn’t make a
difference
Our
Proof
© Trey
Smith
Slide25100
%
Manual
Testing
0
%
Automated
Testing
Testing
Present
There’s a way to do it
better
.
F
ind it!
Project Charter
11
Manual
Testers
4755
Manual
Test Cases
?
Unknown
Coverage
10
Weeks
Execution
Slide26Risk Coverage OptimizationTest Data Management
UI/API Test Automation
Continuous Integration
&
Distributed Execution
48
Hours
Execution
Single
Agent
8
Hours
Execution
Multiple
Agents
11
Manual
Testers
4755
Manual
Test
Cases
?
Unknown
Coverage
10
Weeks
Execution
89
%
Business Risk
Coverage
1193
Manual
Test
Cases
37
%
Business Risk
5
Weeks
Execution
2.5
Weeks
Execution
92
%
Automated
Test Cases
53
%
Business Risk
Coverage
Smoke
Testing
34
Minutes
Execution
Smoke
Testing
Test Case Count
7
%
75
%
Redundancy
50
%
Effort Test Data
64
%
API Tests
Slide278HoursExecution
Multiple
Agents
72
%
Critical Defect
Reduction
Production Defects
89
%
Business Risk
Coverage
1193
Manual
Test Cases
5
Weeks
Execution
2.5
Weeks
Execution
92
%
Automated
Test Cases
34
Minutes
Execution
Smoke
Testing
53
%
Business Risk
Coverage
Smoke
Testing
Test Case Count
7
%
Test Case
Design
&
Exploratory
Testing
64
%
API Tests
Risk Coverage Optimization
Test Data Management
UI/API Test Automation
Continuous Integration
&
Distributed Execution
Slide28The future doesn’t just happen,it gets happened, so make it happen
Our
Conclusion
© Paul
Tagliablue
Slide29QuestionsBecause answers exist only to questions...
Ingo
Philipp
?