Driving Test Automation through the Continuous Deployment Process - PowerPoint Presentation

Slide1

Driving Test Automation through the Continuous Deployment Process

Adam

Guida

Slide2

Introduction

Adam

Guida

Senior QA Automation DeveloperWork History: 2011 – Present: Deutsche Bank Global Technologies Cary, NC2006-2011: lulu.com - Raleigh, NCEmail: guidadic@gmail.com

2

Slide3

Automated Testing is a Liability

Unless you have

Stable

Test Environments Deployment AutomationStrong Communication CollaborationTechnical Expertise

Slide4

The Truth About Automated Testing

Automated Testing:

When done correctly, it can significantly increase the repeatability and effectiveness of testing efforts

Augments classical test methodology but does NOT replace manual testingIncreases regression consistency Reduces tediumAllows human testers a greater focus on new and creative ways of breaking thingsHowever:Automated testing is ineffective without an approach that considers all facets of the development automation ecosystem

Slide5

Purpose of this lecture

Continuous Automation in QA

Test Strategies

Best PracticesEnvironmentsTools & TechnologiesCommunication

Lessons Learned

Poorly written

t

est

automation isn’t sustainable

Writing good tests alone

isn’t enough

Slide6

Continuous AutomationPractices

Continuous

Integration (CI)

A practice where code is automatically compiled and unit tested on every developer code check-in Continuous DeploymentIf CI phase passes, application(s) are automatically deployed to one or more pre-production environments, triggering additional layers of automated testing

Slide7

Build + Unit Test

App SCM

Automated

Testing

Deploy

Automation Pipeline

Report

Test Environment

Test SCM

DEV & QA

Continuous Automation

public class Feature {

private

int

1;

public void

doSomething

(){

if (a == b){

return true;

}

Feature

: Test A and B are True

Scenario

: Verify that A Equals B

Given

I open the “Feature”

When

I click on “A” and “B”

Then

the result is

“

…….”

Dude, I’m having a problem getting this test working. The web service is returning a 404

Let me take a look. I see! You’re calling the wrong resource. I’ll IM you the correct name

My bad! I’ll check-in a fix for that right away.

Hey, can you take a look at the last run? Looks like a regression test failed.

Unit test failed! I’ll take a look at what broke and try again

This one is my fault. Looks like I forgot to adjust some tests after that last build. I’ll check-in a fix & restart the automation

Sweet! All tests passed!

Awesome! Let’s grab some lunch!

Latest build is up on CI for testing!

Slide8

Establish partnerships & comradery

with your development team

Treat QA & Developers as a single team

Close low-level knowledge gaps in functional testingDevelop workflows for how DEV & QA address test failures as a teamFoster a culture that values cross-team learning, exchange of knowledge and professional growthContinuous Automation Collaboration

Slide9

Continuous AutomationBest Practices Demo

Slide10

Automated Test Suite should be run as often as possibleAutomated tests should run……………………...automatically

On every successful automated deployment

Automation executes using the fewest clicks possible

Start Small and Simple – It won’t all happen overnightContinuous AutomationBest Practices

Slide11

Automation Test Team:Strategies

Delegation -

Separate the

technical from test responsibilities for a given taskIncreases efficiency and focus by reducing context switchingPair-testing - Pair technical & non-technical testers together for greater impactIncreases technical experience and promotes mentorship of non-technical testers Increases usability & readability of tests Reporting - Test Lead/Manager - keep a close eye on automated test reporting. If tests start failing, seek to understand the underlying causes, then try to assist test team by removing roadblocksUnderstanding daily/weekly trends in test results helps guide risk assessments

Slide12

Continuous AutomationBest Practices

Treat Your Test Suite like an Enterprise Application

Automated testing *IS* a form of programming and bound by the same laws as software development

Strive to be the best programmer you can & adhere to software development best practicesUse a Source Control Management toolUnit Test your test automation (Yes, that means testing your tests)Practice Code reviewsSeek architectural guidance and input from your development teamDefect tracking (automated tests apps get bugs too!)

Slide13

Automated Test Strategies

The Perfect Automated Test

Fast –

Speed impacts every aspect of a quality automation suiteReusable – Reuse reduces code complexity and increases simplicity of a test suiteRelevant – Test indicate a problem we care about and is not redundantIndependent – Is self sufficient and does not depend on the ordering of other testsFocused – Tests a specific aspect of the application without being too broadClear – Determines a pass or failure result automatically, ideally narrowing down the exact cause of the failureUnderstandable

– Is written in such a way that documents behavior other stake-holders can understandFlexible – Change is the only constant

– write tests in ways that can handle changes within reason

Slide14

System Integration / End-to-End –

Provides the most comprehensive test coverage at the cost of speed and complexity.

Unit –

Isolated testing of individual classes, methods and functionsComponent – Tests are isolated to the component itself. Inputs/Outputs are simulated through test adapters or mockAutomated Test Strategies

Managing the Complexity Versus Coverage Tradeoff

Slide15

Automated Test Strategies

Example of Component Testing

Marketing Aggregator

Order

Database

Web Server

Browser

Product

Database

User

Database

WEB SERVICE CALL

TEST 2

TEST 3

MOCK AGGREGATOR

TEST 1

MOCK DATABASE

H2

Data Transformer Testing

Web Service Testing

UI Testing

End-2-End

Slide16

Diversify your test portfolio using a combination of Unit

,

Component

and System testingPromote Unit Testing w/ Devs - allows you to start trusting some basic elements of the system, so you can focus on more challenging test scenariosMinimize System/End-to-End tests - to fewer high-level smoke tests based on areas of importanceMaximize your Component Testing - focus on lower level functionality, API’s and data model interactions

Use Test Adapters – for both greater control & reliability of the application input/outputs

Automated Test Strategies

Managing the Complexity Versus Coverage Tradeoff

Slide17

Choosing the right Tools & Technologies

Don’t Reinvent the Wheel

– someone has already solved your problem

Carefully Evaluate All Technologies – Require experience prior to adoption. Invest time in evaluating the pros & cons of a variety of different solutions. Build POC’s and give stakeholders (developers) a role in decision makingStraightforward Integration – Utilize tools which accept open formats for read/write – ensure each tool can fully integrate with others as necessaryActive Community of Users – Favor technologies where an active online support community exists over managed (paid) support solutions

Slide18

TechnologiesProgramming Language

Rule of thumb

: Test Automation should be based on the same programming language of your application

Enables sharing of development resources, support and contributionShared libraries and other resources between testing & devShould be capable of utilizing a highly functional IDE’s such as Eclipse, Intellij, Visual Studio, etc

Slide19

Tools: IDEs (Integrated Development Environment)

Intellij

IDEA /

RubyMinesFully integrated support for CucumberSyntax highlightingCode Editing/RefactoringSupports many languages Build-in database browserDebugger

Slide20

Tools: Continuous Integration Server

CI (

Continous

Integration Server) More than just a build agentManages the end-to-end flow of your automation pipelines and reporting Is both the glue and front-end that binds the continuous deployment, testing and reporting processes

Slide21

Tools: VMs - Virtual Machines

VirtualBox

Virtualized OS’s

LinuxWindowsAndroidOS XContained sandbox environmentsBackup/Restore, share and take snapshots of your VM environments

Slide22

Create and configure lightweight, reproducible, and portable development environmentsEasy to setup/tear down and deploy to a standardize test environment

Works with

VirtualBox

and other virtual environment productsSupport integrated into Intellij IDEA via plug-inTools:

Slide23

Help automate aspects of environment provisioning & deployment

Install & Configure Servers & other resources

Helps facilitate continuous deployment of your application

Many great frameworks to choose from - each has it’s own set of strength & weaknessTools: Deployment Orchestration

Slide24

24

Tools & Technology:

Database Management

Treats SQL like version-able code XML, JSON, YAML and generic SQL syntax supports multiple database technologies

Flexible runtime: Maven Plugin and standalone support

Slide25

Final Thoughts

Automation is an investment, that when managed effectively, provides a particular set of benefits which increase efficiency and repeatability of testing.

AUTOMATION IS NOT A BENEFIT IN ITSELF

Treat automation as a 1st class citizen of your projectIf you’d like a copy of this presentation, please send me an email at:guidadic@gmail.com