Presented by Tom Chapel Thomas J Chapel MA MBA Tchapelcdcgov Chief Evaluation Officer 4046392116 Centers for Disease Control and Prevention Agenda 1 The why and how ID: 692931
Download Presentation The PPT/PDF document "“ Mixed Methods in Program Evaluation..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
“
Mixed Methods in Program Evaluation”Presented by Tom Chapel
Thomas J. Chapel, MA, MBA Tchapel@cdc.gov
Chief Evaluation Officer 404-639-2116
Centers for Disease Control and PreventionSlide2
Agenda
1. The why and how of mixed methods:RationaleOptionsChallengesCriteria for making choices
2. Apply points to some simple examplesSlide3
Ensure use and share lessons learned
Gather credible evidence
Engage stakeholders
Describe the program
Focus the evaluation design
Justify conclusions
STEPS
Standards
Utility
Feasibility
Propriety
Accuracy
CDC’s Evaluation Framework
The Standards apply
especially
when we’re trying to make data collection choices.Slide4
CDC’s Evaluation Standards
The Standards provide a quick and easy way to identify the 2 or 3 best data collection choices for this evaluation.
Standards
Utility
Feasibility
ProprietyAccuracySlide5
CDC’s Evaluation Framework
Not “Collect data”Not “Analyze data”Rather… “Gather credible evidence”
Ensure use and share lessons learned
Gather credible evidence
Engage stakeholders
Describe the program
Focus the evaluation design
Justify conclusions
STEPS
Standards
Utility
Feasibility
Propriety
AccuracySlide6
Steps 1-3 Help You Focus Design
And Data Collection ChoicesAfter the first 3 steps of the Evaluation Framework, we know which evidence will work for these stakeholders in this situation.
Qualitative data?
Quantitative data?
Randomized control trials?
Performance
measures?Slide7
CDC’s Evaluation Standards
The Evaluation Standards help us narrow down our data collection choices to the handful of methods that will work for this evaluation at this time.
Standards
Utility
Feasibility
ProprietyAccuracySlide8
Mixed Methods
Data collection methods that will work for this evaluation at this time sometimes means surveys or focus groups.But sometimes there is no one best way.
The best choice would be a combination of methods or “mixed methods”. Slide9
Six (Most) Common Ways to Collect DataSlide10
How Standards Inform the Choice of Methods
Consider the context :How soon do I need the results? What resources can I use?
Are there any ethical issues to consider?
Standards
Utility
Feasibility
Propriety
Accuracy
Standards
Utility
Feasibility
Propriety
AccuracySlide11
How Standards Inform the Choice of Methods
Also consider the content :Sensitivity of the issue
Standards
Utility
Feasibility
Propriety
Accuracy
Standards
Utility
Feasibility
Propriety
AccuracySlide12
How Standards Inform the Choice of Methods
Also consider the content :“The Hawthorne Effect” Will the act of being observed cause someone to distort their response?
Standards
Utility
Feasibility
Propriety
Accuracy
Standards
Utility
Feasibility
Propriety
AccuracySlide13
How Standards Inform the Choice of Methods
Also consider the content :Validity Reliability
Standards
Utility
Feasibility
Propriety
Accuracy
Standards
Utility
Feasibility
Propriety
AccuracySlide14
Mixed Methods Address Concerns
Key Concept:Regardless of the method, when there are validity and reliability concerns, often using more than one method-- i.e., mixed methods--will help.Slide15
Mixed Methods: Definition
“The combination of at least one qualitative and at least one quantitative component in
a single research project or program.”
(Bergman 2008)Slide16
Use Complementary Methods
Mixed methods is:A combination of methods that has complementary strengths and non-overlapping
weaknesses.
The purpose is to
supplement
or
complement
the validity and reliability of the information.Slide17
Strengths of Quantitative Methods
Strengths of quantitative methods:Require less time than qualitative methodsCost lessPermit researcher controlQuantitative data is considered to be “scientific”Easier to explain validity and reliabilityEasily amenable to statistical analysesSlide18
Strengths of Qualitative Methods
Choose qualitative methods when you are trying to:Explore or describe a phenomenonSlide19
Strengths of Qualitative Methods
Choose qualitative methods when you are trying to:Look for induction (i.e., “surprise”)Slide20
Strengths of Qualitative Methods
Choose qualitative methods when you are trying to:Identify patternsSlide21
Strengths of Qualitative Methods
Qualitative data can help you understand not just “what” but “WHY”.Slide22
When to Use Mixed Methods
1. Corroborationbetter understanding; more credibility“triangulation” – measuring the same thing from several different viewpoints
2.
Clarificationtrying to understand why we got this resultSlide23
When to Use Mixed Methods
Mixed Methods are most commonly used for:3. Explanation – similar to clarificationwant to know the “why” or “what” behind the situation
4. Exploration – similar to explanationcharting new territory
trying to observe patternsexamine different situations and varying
results to induce patternsSlide24
Number of Project Facets Reported via Each Data
Collection MethodSource: Gregory Guest, PhDSlide25
Number of
Project Facets Reported via Each Data Collection Method
This is an example of
using a qualitative
method (site visits) to corroborate a quantitative method (surveys).
The result was increased validity of the data.
Source: Gregory Guest, PhDSlide26
Which to Choose?How do you choose which methods to use?
Which method comes first, the quantitative or the qualitative?You have a lot of flexibility in these decisions.Slide27
Parallel or ConcurrentMixed Methods
For “parallel” or “concurrent” mixed methods,quantitative and qualitative data collection happen at the same time.QUANTITATIVE
QUALITATIVESlide28
SequentialMixed Methods
For “sequential” mixed methods, either quantitative or qualitative data collection can happen first.QUANTITATIVE
QUALITATIVE
QUANTITATIVE
QUALITATIVE
ORSlide29
Example of Sequential Mixed Methods to Corroborate Data
In this case, the qualitative method (site visits) was used to corroborate the quantitative (survey) method and the results were different.
QUALITATIVE
QUANTITATIVESlide30
Mixed Methods Is Your ChoiceYou are never
required to use mixed methods.However, you may choose to use mixed methods when:you have some indication that a single method may give you incorrect data.a single method may give you an incorrect perception of reality.Slide31
Mixing Methods During Data Analysis
Qualitative data (focus groups, observations, secondary data, etc.) can be converted to “numbers” via quantitative techniques like content analysis. This is also a mixed method design approach.Slide32
Mixing Methods During Data Analysis
Qualitative data can be very complex. Examining qualitative data with quantitative techniques helps to identify or validate patterns or themes.Slide33
Deciding When To Use
Mixed Methods and HowKey Concept:Using mixed methods is a deliberate design decision. You use it when you don’t trust the data from any single method.
The reason for your uncertainty determines the methods you choose to mix and the order in which you use them. Slide34
Example 1
Concurrent DesignSlide35
Example 2
Explanatory Sequential DesignSlide36
Example 3
Exploratory Sequential DesignSlide37Slide38
Design Options SummaryYou mix quantitative and qualitative methods in a different order depending on the presenting problem:
Validate results Slide39
Design Options SummaryYou mix quantitative and qualitative methods in a different order depending on the presenting problem:
Validate resultsExplain the unexpected Slide40
Design Options SummaryYou mix quantitative and qualitative methods in a different order depending on the presenting problem:
Validate resultsExplain the unexpectedExplore new themes Slide41
Selected Resources(Page 1 of 2)Caracelli
, V. and J. Greene (eds.). 1997. Advances in Mixed-Method Evaluation: The Challenges and Benefits of Integrating Diverse Paradigms. San Francisco, CA: Jossey-Bass. Creswell, J. and V. Plano Clark. 2010. Designing and Conducting Mixed Methods Research, 2nd edition. Thousand Oaks, CA. Sage Publications.Morse, J. and L. Niehaus. 2009. Mixed Method Design: Principles and Procedures. Walnut Creek, CA. Left Coast Press. Slide42
Selected Resources(Page 2 of 2)
Johnson, R. Burke, and L. Christensen. Evaluation Methods. 2008. www.southalabama.edu/coe/bset/johnson/Plano Clark, V. and J. Creswell. 2008. The Mixed Methods Reader. Thousand Oaks, CA: Sage Publications. Teddlie, C. and Tashakkori, A. 2009. Foundations of Mixed Methods Research: Integrating Quantitative and Qualitative Approaches in the Social and Behavioral Sciences. Thousand Oaks, CA. Sage Publications. Slide43
Recommended ResourceCreswell, J. and V. Plano Clark. 2010.
Designing and Conducting Mixed Methods Research, 2nd edition. Thousand Oaks, CA. Sage Publications.Slide44
The Community Tool BoxCommunity Tool Box http://ctb.ku.edu
Chapter 37, Section 5. Collecting and Analyzing DataSlide45
End “Mixed Methods”
Return to Webinar 4: Gathering Data, Developing Conclusions, and Putting Your Findings to UseReturn to Evaluation Webinars home page