Getting Started and Engaging Your Stakeholders Presented by Leslie Fierro and Carlyn Orians Getting Started and Engaging Your Stakeholders Leslie A Fierro MPH TKCIS Contractor NCEHCDC ID: 707709
Download Presentation The PPT/PDF document "Program Evaluation Webinar Series Part 2..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Program Evaluation Webinar Series Part 2:
“Getting Started and
Engaging Your Stakeholders”
Presented by: Leslie
Fierro
and
Carlyn
Orians
Slide2
Getting Started and Engaging Your StakeholdersLeslie A.
Fierro, MPHTKCIS Contractor NCEH/CDC
Let6@cdc.gov
Carlyn
Orians
, MA, Battelle Centers for Public Health Research and Evaluationorians@battelle.org
Presented November 20, 2008Slide3
AgendaDefinition of evaluation.
Difference between evaluation & research.
CDC’s Evaluation Framework. The who, why, when & how of stakeholders.
Tangible examples of engaging stakeholders.Slide4
What is the Purpose of Evaluation?Slide5
Evaluation: Definition 1
Evaluation is the systematic investigation of the merit, worth, or significance of an object.
--
Michael ScrivenSlide6
Evaluation: Definition 2
Evaluation is the systematic assessment of the operation and/or the outcomes of a program or policy, compared to a set of explicit or implicit standards as a means of contributing to the improvement of the program or policy.
--
Carol Weiss Slide7
Evaluation: Definition 3
Evaluation is the systematic collection of information about the activities, characteristics, and outcomes of programs to make judgments about the program, improve program effectiveness, and/or inform decisions about future programming.
-- Michael Patton Slide8
The Common Element
Note that all 3 of these statements share a common element. ..
…they all define evaluation as
a systematic and formalized endeavor.Slide9
Summative Evaluations Summative evaluations seek to judge a program by asking, “Should this project be…” cancelled? continued?
expanded?Slide10
Formative EvaluationsFormative evaluations seek to
use the evaluation findings. They ask the question:“Is the program being conducted as planned?”“Is the program doing well?”
Formative evaluations seek to
improve programs or policies.Slide11
The CDC DefinitionEvaluation is the systematic collection of information about the activities, characteristics and outcomes of the program to make judgments about the program, improve program effectiveness and/or inform decisions about future program development. Slide12
Research vs. EvaluationSpecific to
Research
Specific to
Evaluation
Research and evaluation share methodologies
but ask different questions.
CommonalitiesSlide13
The Research ContextSlide14
Research Findings
test and improve theories.develop generalizable theories.
theories apply across different
settings, people, and times.
Research is conducted to:Slide15
The Evaluation ContextSlide16
Evaluation FindingsEvaluation results:
are not usually generalizable. focus on a specific situation.
evaluation of a single program in a particular context.Slide17
Research vs. EvaluationEvaluation asks:
“Is this program working?”
Research asks:
“Will this program work across
multiple settings?”Slide18
The Role of Researchers
Researchers:
Play a single role as content experts.Slide19
The Role of Researchers
Researchers:
Play a single role as content experts.
Identify gaps in current knowledge.Slide20
The Role of Researchers
Researchers:
Play a single role as content experts.
Identify gaps in current knowledge.Derive their own questions.Slide21
The Role of Researchers
Researchers:
Play a single role as content experts.
Identify gaps in current knowledge.Derive their own questions.
Perform their own research.Slide22
The Role of Evaluators
Evaluators:Play multiple roles—facilitator, educator, scientific expert, etc.Slide23
The Role of Evaluators
Evaluators:Play multiple roles—facilitator, educator, scientific expert, etc.
Involve stakeholders.Slide24
The Role of Evaluators
Evaluators:Play multiple roles—facilitator, educator, scientific expert, etc.
Involve stakeholders.
Collaborate to identify and prioritize questions.Slide25
“Research seeks to prove
, evaluation seeks to
improve.
”
M.Q. Patton
In a Nutshell…Slide26
The Findings Must be Useful
To improve a program, the findings must be useful! How?
The stakeholders must be involved.
The questions must be relevant to the program.The findings must be credible to key stakeholders. Slide27
Getting Started and Engaging Your StakeholdersThe Who, When, Why, and How
of Stakeholder Involvementin CDC’s Evaluation FrameworkSlide28
“Who” are the Stakeholders?
Stakeholders are:
people and/or organizations
that are
interested in the program
, are
interested in the results
of the evaluation and/or
have a stake
in what will be done with the results of the evaluation.Slide29
Examples of StakeholdersSlide30
Each Program is DifferentDevelop a list of stakeholders at the start of any evaluation activity. Slide31
Which Stakeholders Matter Most?
Review your list of stakeholders and think strategically about these questions:
“Who do we
need to…”Slide32
Which Stakeholders Matter Most?
Review your list of stakeholders and think strategically about
these questions: “Who
do we need to
…”
enhance credibility?Slide33
Which Stakeholders Matter Most?
Review your list of stakeholders and think strategically about these questions:
“Who do we
need to…”
enhance credibility?
implement program changes?Slide34
Which Stakeholders Matter Most?
Review your list of stakeholders and think strategically about these questions:
“Who do we
need to…”
enhance credibility?
implement program changes?
advocate for changes?Slide35
Which Stakeholders Matter Most?
Review your list of stakeholders and think strategically about these questions:
“Who do we
need to…”
enhance credibility?
implement program changes?
advocate for changes?fund
, authorize, or expand the program
?Slide36
“Why” Engage Stakeholders?
Engaging stakeholders is an important part of the CDC Evaluation Framework.
Involving stakeholders may be a requirement of your program.Slide37
“Why” Engage Stakeholders?
Engaging stakeholders is an important part of the CDC Evaluation Framework
Stakeholders will add credibility.
If you want the results to be acted upon, they must be credible.Slide38
“Why” Engage Stakeholders?
Engaging stakeholders is an important part of the CDC Evaluation Framework.
Stakeholders will add credibility.
Stakeholders may have resources
to help.
Stakeholders may be able to contribute data, skills, analytical skills, etc. Slide39
“Why” Engage Stakeholders?
Engaging stakeholders is an important part of the CDC Evaluation Framework.
Stakeholders will add credibility.
Stakeholders may have resources to help.
Stakeholders may be critical to implementing
or advocating for action based on the results. Slide40
“Why” Engage Stakeholders?
Engaging stakeholders is an important part of the CDC Evaluation Framework.
Stakeholders will add credibility.
Stakeholders may have resources to help.
Stakeholders may be critical to implementing or advocating for action based on the results.
You will build trust and understanding
among program constituents.Involving stakeholders helps to reduce fear of the evaluation process.Slide41
“Why” Would Stakeholders Want to be Involved?
Get answers to their questions.Slide42
“Why” Would Stakeholders Want to be Involved?
Get answers to their questions.
Learn about evaluation.Slide43
“Why” Would Stakeholders Want to be Involved?
Get answers to their questions.
Learn about evaluation.
Influence the design and methods.Slide44
“Why” Would Stakeholders Want to be Involved?
Get answers to their questions.
Learn about evaluation.
Influence the design and methods.Protect their constituents.Slide45
“Why” Would Stakeholders Want to be Involved?
Get answers to their questions.
Learn about evaluation.
Influence the design and methods.Protect their constituents.
Motivated to help program succeed.Slide46
CDC’s Evaluation Framework
Ensure use and share lessons learned
Gather credible evidence
Engage stakeholders
Describe the program
Focus the evaluation design
Justify conclusions
STEPS
Standards
Utility
Feasibility
Propriety
AccuracySlide47
“When” and “How” Can You Engage Stakeholders?
You can use the CDC Evaluation Framework to engage stakeholders in:identifying and prioritizing evaluation questions,
selecting credible sources, and
developing a plan for action based on evaluation results.Slide48
“When” and “How” Can You Engage Stakeholders?
You can engage stakeholders in every step of the evaluation process.
But… you don’t have to engage stakeholders in all these ways in every evaluation.Slide49
“How” Depends on…
Evaluator preference.Slide50
“How” Depends on…
Evaluator preference.
Stakeholder preference.Slide51
“How” Depends on…
Evaluator preference.
Stakeholder preference.
Resources.Slide52
“How” Depends on…
Evaluator preference.
Stakeholder preference.
Resources.Degree of trust or threats to credibility.Slide53
“How” Depends on…
Evaluator preference.
Stakeholder preference.
Resources.Degree of trust or threats to credibility.
If there is a high degree of mistrust, engage stakeholders early in the evaluation process. This helps ensure that the results are viewed as credible and are acted upon. Slide54
CDC’s Evaluation Framework
Ensure use and share lessons learned
Gather credible evidence
Engage stakeholders
Describe the program
Focus the evaluation design
Justify conclusions
STEPS
Standards
Utility
Feasibility
Propriety
AccuracySlide55
Who should be involved?
Develop list of potential stakeholders.
Decide which stakeholders are the most important to include.
CDC’s Framework Step 1
Ensure use and share lessons learned
Gather credible evidence
Engage stakeholders
Describe the program
Focus the evaluation design
Justify conclusions
STEP 1
Standards
Utility
Feasibility
Propriety
Accuracy
Engage stakeholdersSlide56
CDC’s Framework Step 2
Do stakeholders share a vision of what the program does and its intended outcomes?
Diverse views?
Similar views?
Engage them early in the process.
Ensure use and share lessons learned
Gather credible evidence
Engage stakeholders
Describe the program
Focus the evaluation design
Justify conclusions
STEP 1
Standards
Utility
Feasibility
Propriety
Accuracy
Describe the programSlide57
CDC’s Framework Step 3
What are the most pressing and important evaluation questions for stakeholders?
What questions do they need answered to be able to take action?
What methods are available and preferred?
What will be considered “credible evidence”?
Ensure use and share lessons learned
Gather credible evidence
Engage stakeholders
Describe the program
Focus the evaluation design
Justify conclusions
STEP 1
Standards
Utility
Feasibility
Propriety
Accuracy
Focus the evaluation designSlide58
CDC’s Framework Step 4
Gather evidence stakeholders will find credible.
Ensure use and share lessons learned
Gather credible evidence
Engage stakeholders
Describe the program
Focus the evaluation design
Justify conclusions
STEP 1
Standards
Utility
Feasibility
Propriety
Accuracy
Gather credible evidenceSlide59
CDC’s Framework Step 5
How do diverse stakeholders interpret the findings?
May engage stakeholders in the analysis.
Perhaps solicit their interpretation before results are finalized.
Ensure use and share lessons learned
Gather credible evidence
Engage stakeholders
Describe the program
Focus the evaluation design
Justify conclusions
STEP 1
Standards
Utility
Feasibility
Propriety
Accuracy
Justify conclusionsSlide60
CDC’s Framework Step 6
Which stakeholders can play a role in disseminating results or acting on findings?
This is a
critical step
for stakeholder involvement.
Ensure use and share lessons learned
Gather credible evidence
Engage stakeholders
Describe the program
Focus the evaluation design
Justify conclusions
STEP 1
Standards
Utility
Feasibility
Propriety
Accuracy
Ensure use and share lessons learnedSlide61
Getting Started and Engaging Your Stakeholders
Tangible Examples ofStakeholder Involvementin the Evaluation ProcessSlide62
Example #1 – Asthma and Home Environment in Low-Income Apts
Intervention to improve indoor environment for children with asthma via following activities
:Slide63
Example #1 – Asthma and Home Environment in Low-Income Apts
Intervention to improve indoor environment for children with asthma via following activities:
Provide education and training for apartment owners, building inspectors, maintenance vendors, and tenants regarding asthma triggers and housing codes.Slide64
Example #1 – Asthma and Home Environment in Low-Income Apts
Intervention to improve indoor environment for children with asthma via following activities:
Provide education and training for apartment owners, building inspectors, maintenance vendors, and tenants regarding asthma triggers and housing codes.
Work with city officials to enhance existing housing code.Slide65
Example #1 – Asthma and Home Environment in Low-Income Apts
Intervention to improve indoor environment for children with asthma via following activities:
Provide education and training for apartment owners, building inspectors, maintenance vendors, and tenants regarding asthma triggers and housing codes.
Work with city officials to enhance existing housing code.
Promote smoke-free housing.Slide66
CDC’s Evaluation Framework
Ensure use and share lessons learned
Gather credible evidence
Engage stakeholders
Describe the program
Focus the evaluation design
Justify conclusions
STEPS
Standards
Utility
Feasibility
Propriety
AccuracySlide67
Step 1: Engage the Stakeholders
Low-Income Housing Scenario
Who are the stakeholders?
Slide68
Step 1: Engage the Stakeholders
Low-Income Housing Scenario
Who are the stakeholders?
affected families Slide69
Step 1: Engage the Stakeholders
Low-Income Housing Scenario
Who are the stakeholders?
affected families
those who receive training (e.g., owners, inspectors, maintenance personnel, tenants)Slide70
Step 1: Engage the Stakeholders
Low-Income Housing Scenario
Who are the stakeholders?
affected families
those who receive training (e.g., owners, inspectors, maintenance personnel, tenants)
trainers Slide71
Step 1: Engage the Stakeholders
Low-Income Housing Scenario
Who are the stakeholders?
affected families
those who receive training (e.g., owners, inspectors, maintenance personnel, tenants)
trainers
city officials responsible for the housing code
Slide72
Step 1: Engage the Stakeholders
Low-Income Housing Scenario
Who are the stakeholders?
affected families
those who receive training (e.g., owners, inspectors, maintenance personnel, tenants)
trainers
city officials responsible for the housing code
smoke-free advocates
Slide73
Step 1: Engage the Stakeholders
Low-Income Housing Scenario
Who are the stakeholders?
affected families
those who receive training (e.g., owners, inspectors, maintenance personnel, tenants)
trainers
city officials responsible for the housing code
smoke-free advocates
Stakeholders may include advocates as well as those resistant to change .
It’s important to include a mix of views so you don’t get surprised at the end.Slide74
Step 2: Describe the Program
Low-Income Housing Scenario
Obtain stakeholders’ views on:
Slide75
Step 2: Describe the Program
Low-Income Housing Scenario
Obtain stakeholders’ views on:
What is
the current indoor environment?
Slide76
Step 2: Describe the Program
Low-Income Housing Scenario
Obtain stakeholders’ views on:
What is
the current indoor environment?
What changes are both desired and seen as realistic? Slide77
Step 2: Describe the Program
Low-Income Housing Scenario
Obtain stakeholders’ views on:
What is
the current indoor environment?
What changes are both desired and seen as realistic?What outcomes do they think are most important?
Slide78
Step 2: Describe the Program
Low-Income Housing Scenario
Obtain stakeholders’ views on:
What is
the current indoor environment?
What changes are both desired and seen as realistic?What
outcomes do they think are most important?Which activities contribute to which outcomes?
Slide79
Step 2: Describe the Program
Low-Income Housing Scenario
Obtain stakeholders’ views on:
What is
the current indoor environment?
What changes are both desired and seen as realistic?Which outcomes do they think are most important?
Which activities contribute to which outcomes?
(You could use a logic model
here… Stay tuned for the next Webinar in this series…)
Slide80
Step 3: Focus the Evaluation Design
Low-Income Housing Scenario
What questions do the
stakeholders
think are the most important to ask?Slide81
Step 3: Focus the Evaluation Design
Low-Income Housing Scenario
What questions do the
stakeholders
think are the most important to ask?
Should the evaluation focus on changes to the housing code? Slide82
Step 3: Focus the Evaluation Design
Low-Income Housing Scenario
What questions do the
stakeholders
think are the most important to ask?
Should the evaluation focus on changes to the housing code?
Focus on trigger reduction? Which triggers? Slide83
Step 3: Focus the Evaluation Design
Low-Income Housing Scenario
What questions do the
stakeholders
think are the most important to ask?
Should the evaluation focus on changes to the housing code?
Focus on trigger reduction? Which triggers? What is the best design? Slide84
Step 3: Focus the Evaluation Design
Low-Income Housing Scenario
What questions do the
stakeholders
think are the most important to ask?
Should the evaluation focus on changes to the
housing code? Focus on trigger reduction? Which triggers?
What is the best design?
Is a pre-post design best? Slide85
Step 3: Focus the Evaluation Design
Low-Income Housing Scenario
What questions do the
stakeholders
think are the most important to ask?
Should the evaluation focus on changes to the housing code?
Focus on trigger reduction? Which triggers? What is the best design?
Is a pre-post design best?
Should there be a control group? Slide86
Step 3: Focus the Evaluation Design
Low-Income Housing Scenario
What questions do the
stakeholders
think are the most important to ask?
Should the evaluation focus on changes to the housing code?
Focus on trigger reduction? Which triggers? What is the best design?
Is a pre-post design best?
Should there be a control group?
Are observational methods appropriate?
Slide87
Step 4: Gather Credible Evidence
Low-Income Housing Scenario
What role can stakeholders play?Slide88
Step 4: Gather Credible Evidence
Low-Income Housing Scenario
What role can stakeholders play?
Are there stakeholders that can facilitate access to residences and/or translate if necessary?Slide89
Step 4: Gather Credible Evidence
Low-Income Housing Scenario
What role can stakeholders play?
Are there stakeholders that can facilitate access to residences and/or translate if necessary?
Can the trainers help collect data before, during, or after each training? Other opportunities for data collection?Slide90
Step 4: Gather Credible Evidence
Low-Income Housing Scenario
What role can stakeholders play?
Are there stakeholders that can facilitate access to residences and/or translate if necessary?
Can the trainers help collect data before, during, or after each training? Other opportunities for data collection?
What information will the stakeholders need to act? For example, will policy makers be more influenced by information on environmental conditions, asthma symptoms, or medical care usage? Slide91
Step 4: Gather Credible Evidence
Low-Income Housing Scenario
What role can stakeholders play?
Are there stakeholders that can facilitate access to residences and/or translate if necessary?
Can the trainers help collect data before, during, or after each training? Other opportunities for data collection?
What information will the stakeholders need to act? For example, w
ill policy makers be more influenced by information on environmental conditions, asthma symptoms, or medical care usage? Data in what form? Pictures? Stories? Numerical data?Slide92
Step 5: Justify Conclusions
Low-Income Housing Scenario
How do stakeholders view success? What does “success” look
like to them?Slide93
Step 5: Justify Conclusions
Low-Income Housing Scenario
How do stakeholders view success? What does “success” look
like to them?
Do stakeholders have alternative interpretations of the initial findings?Slide94
Step 5: Justify Conclusions
Low-Income Housing Scenario
How do stakeholders view success? What does “success” look
like to them?
Do stakeholders have alternative interpretations of the initial findings?
Do they perceive barriers you might have missed?Slide95
Step 6: Ensure Use and Share Lessons Learned
Low-Income Housing Scenario
Can stakeholders help disseminate evaluation findings?Slide96
Step 6: Ensure Use and Share Lessons Learned
Low-Income Housing Scenario
Can stakeholders help disseminate evaluation findings?
Is there an owner’s newsletter? A tenant meeting? A neighborhood association?Slide97
Step 6: Ensure Use and Share Lessons Learned
Low-Income Housing Scenario
Can stakeholders help disseminate evaluation findings?
Is there an owner’s newsletter? A tenant meeting? A neighborhood association?
Is information needed in a different form to prompt action? A report?
An executive brief? An oral presentation?Slide98
Step 6: Ensure Use and Share Lessons Learned
Low-Income Housing Scenario
Can stakeholders help disseminate evaluation findings?
Is there an owner’s newsletter? A tenant meeting? A neighborhood association?
Is information needed in a different form to prompt action? A report?
An executive brief? An oral presentation?
Is there interest beyond the community involved in the intervention? Slide99
What if you ignored stakeholders?
If you ignore the stakeholders, potential pitfalls include…Slide100
What if you ignored stakeholders?
If you ignore the stakeholders, potential pitfalls include…Slide101
What if you ignored stakeholders?
If you ignore the stakeholders, potential pitfalls include…Slide102
What if you ignored stakeholders?
If you ignore the stakeholders, potential pitfalls include…Slide103
What if you ignored stakeholders?
If you ignore the stakeholders, potential pitfalls include…Slide104
What if you ignored stakeholders?
If you fail to involve the stakeholders…
…your evaluation may not lead to action!Slide105
105
Example #2 – Care Coordination Across Health Systems
Intervention to provide and integrate care coordination and case management for high-risk children with asthma. Slide106
106
Example #2 – Care Coordination Across Health Systems
Intervention to provide and integrate care coordination and case management for high-risk children with asthma.
Involves standardizing protocols across care systems, including:
Medicaid HMOs
Home nursing agencies
Health departmentsSlide107
107
Example #2 – Care Coordination Across Health Systems
Intervention to provide and integrate care coordination and case management for high-risk children with asthma.
Involves standardizing protocols across care systems, including:
Medicaid HMOs
Home nursing agencies
Health departments
The goal: to prove success and thus convince
insurers to continue reimbursing case
management services.Slide108
Engage the Stakeholders
Housing ScenarioHealthcare Scenario
affected families
those who receive training
trainers
city officials
smoke-free advocates
Who
are
the stakeholders
in this
scenario?Slide109
Engage the Stakeholders
Housing ScenarioHealthcare Scenario
affected families
those who receive training
trainers
city officials
smoke-free advocates
insurersSlide110
Engage the Stakeholders
Housing ScenarioHealthcare Scenario
affected families
those who receive training
trainers
city officials
smoke-free advocates
insurers
case managersSlide111
Engage the Stakeholders
Housing ScenarioHealthcare Scenario
affected families
those who receive training
trainers
city officials
smoke-free advocates
insurers
case managers
familiesSlide112
Engage the Stakeholders
Housing ScenarioHealthcare Scenario
affected families
those who receive training
trainers
city officials
smoke-free advocates
insurers
case managers
families
health system administratorsSlide113
Focus the Evaluation Design
Housing ScenarioHealthcare Scenario
What changes are both desired and seen as realistic?
What
outcomes do they think are most important?
Which activities contribute to which outcomes?
What outcome questions?Slide114
Focus the Evaluation Design
Housing ScenarioHealthcare Scenario
What changes are both desired and seen as realistic?
What
outcomes do they think are most important?
Which activities contribute to which outcomes?
Is this intervention sustainable?Slide115
Focus the Evaluation Design
Housing ScenarioHealthcare Scenario
What changes are both desired and seen as realistic?
What
outcomes do they think are most important?
Which activities contribute to which outcomes?
Is this intervention sustainable?
What is the cost of doing this kind of case management?Slide116
Gather Credible Evidence
Housing ScenarioHealthcare Scenario
Data in what form?
Pictures? Stories?
Numerical data?
Stakeholders that can facilitate access ?
Trainers help collect data ?
What type of data do they need to make a decision?Slide117
Gather Credible Evidence
Housing ScenarioHealthcare Scenario
Data in what form?
Pictures? Stories?
Numerical data?
Stakeholders that can facilitate access ?
Trainers help collect data ?
Is
quantitative data more highly valued
than
qualitative data
?Slide118
Gather Credible Evidence
Housing ScenarioHealthcare Scenario
Data in what form?
Pictures? Stories?
Numerical data?
Stakeholders that can facilitate access ?
Trainers help collect data ?
Is
quantitative data more highly valued
than
qualitative data
?
Do the data need to show cost savings?Slide119
Gather Credible Evidence
Housing ScenarioHealthcare Scenario
Data in what form?
Pictures? Stories?
Numerical data?
Stakeholders that can facilitate access ?
Trainers help collect data ?
Be sensitive to the stakeholders’ interests and time constraints.Slide120
Gather Credible Evidence
Housing ScenarioHealthcare Scenario
Data in what form?
Pictures? Stories?
Numerical data?
Stakeholders that can facilitate access ?
Trainers help collect data ?
Be sensitive to the stakeholders’ interests and time constraints.
Be aware of and consider the political environment.Slide121
Example #3 – Daycare Education
Intervention to train childcare providers to identify triggers and to manage children with asthma. Slide122
Example #3 – Daycare Education
Intervention to train childcare providers to identify triggers and to manage children with asthma.
Intervention reaches out to:
Large daycare centers
Licensed home daycare programs
Participation is encouraged through continuing education creditsSlide123
Example #3 – Daycare Education
Intervention to train childcare providers to identify triggers and to manage children with asthma.
Intervention reaches out to:
Large daycare centers
Licensed home daycare programs
Participation is encouraged through continuing education credits
Long-term goal: to expand beyond pilot providers to larger community.Slide124
Engage the Stakeholders
Housing ScenarioDaycare Scenario
affected families
those who receive training
trainers
city officials
smoke-free advocates
Who
would be
the stakeholders in this
scenario?Slide125
Engage the Stakeholders
Housing ScenarioDaycare Scenario
affected families
those who receive training
trainers
city officials
smoke-free advocates
trainers
daycare providers
licensing boards
familiesSlide126
Focus the Evaluation Design
Housing ScenarioDaycare Scenario
What changes are both desired and seen as realistic?
What
outcomes do they think are most important?
Which activities contribute to which outcomes?
What outcome questions?Slide127
Focus the Evaluation Design
Housing ScenarioDaycare Scenario
What changes are both desired and seen as realistic?
What
outcomes do they think are most important?
Which activities contribute to which outcomes?
behavior change
environmental outcomes
evidence to support decision to expand beyond pilotSlide128
Focus the Evaluation Design
Housing ScenarioHealthcare Scenario
Focus on changes to the housing code?
Trigger reduction?
Should there be a control group?
Observational methods?
What design
mix?Slide129
Focus the Evaluation Design
Housing ScenarioHealthcare Scenario
Focus on changes to the housing code?
Trigger reduction?
Should there be a control group?
Observational methods?
Consider pre-post design.
Consider control group design.Slide130
Gather Credible Evidence
Housing ScenarioHealthcare Scenario
Data in what form?
Pictures? Stories?
Numerical data?
Stakeholders that can facilitate access ?
Trainers help collect data ?
What
data collection methods?Slide131
Gather Credible Evidence
Housing ScenarioHealthcare Scenario
Data in what form?
Pictures? Stories?
Numerical data?
Stakeholders that can facilitate access ?
Trainers help collect data ?
Success stories may be important .Slide132
Getting Started and Engaging Your StakeholdersTangible Examples of
Stakeholder Involvement inApplying the Evaluation StandardsSlide133
Evaluation Standards Apply to Every Step
Ensure use and share lessons learned
Gather credible evidence
Engage stakeholders
Describe the program
Focus the evaluation design
Justify conclusions
STEPS
Standards
Utility
Feasibility
Propriety
AccuracySlide134
The Evaluation Standards
Standards
Utility
Feasibility
Propriety
AccuracySlide135
The Evaluation Standards
There are actually 30 evaluation standards grouped into four categories.
A complete list of the standards is published in CDC’s Framework for Program Evaluation in Public Health.
(See link under “Learning Aids”.)Slide136
Why Use Standards?Standards provide a way to:
Make difficult decisions when designing and implementing an evaluation.Judge the quality of an evaluation.
Determine where an evaluation can be better balanced.Slide137
The “Utility” Standard
Who needs the evaluation information and what information do they need?
Identify affected stakeholders.
Collect data that addresses important
questions.
Ensure the data is responsive to the needs of the stakeholders.Slide138
The “Feasibility” Standard
How much money, time, and
effort can we put into this?
Keep the burden of data collection
as low as possible
.Consider the political interests of the stakeholders.
Ensure that the evaluation is cost effective.Slide139
The “Propriety” Standard
What steps need to be taken for the evaluation to be ethical?
Ensure the ethical treatment of evaluation participants.
Ensure the ethical treatment of those who commissioned the evaluation.
Ensure the ethical treatment of those affected by the findings
of the
evaluation.Slide140
The “Accuracy” Standard
What design will lead to accurate
information?
Clearly and accurately document the program.
Gather data in a way that produces reliable information
.
Analyze and report
the data in a systematic and unbiased way.
Assure that the source of the information is defensible.Slide141
CDC’s Asthma Control Program
CDC’s Asthma Control Program funds 35 states and territories to
implement statewide asthma control programs using a public health perspective.
Their approach involves:Engaging, enhancing, and maintaining relationships with partners.
Developing, improving, and conducting asthma surveillance activities.Designing and implementing interventions with partners. Slide142
Example #4 - Asthma Program Monitoring System
This example involves collecting information for the purpose of evaluation across multiple sites. Slide143
Example #4 - Asthma Program Monitoring System
Purpose: Develop a strategic, systematic approach to collecting information about our program activities, progress, and accomplishments across all funded states.Slide144
Employing the Framework Model
Step 1: Involve the stakeholdersCDC Air Pollution and Respiratory Health Branch.
Representatives
of funded State Asthma Programs.
Step 2: Describe the Program
Develop logic models.Step 3: Focus the evaluation designIdentify and prioritize evaluation questions.
Step 4: Gather credible evidence
Create data collection instrument.Slide145
Developing the Data Collection Instrument
In this example, developing the data collection instrument was a highly collaborative endeavor aided by the
program evaluation standards.Slide146
The Iterative Process
CDC internal workgroup identifies core information needs and drafts survey instrument.State workgroup members review instrument, pilot test, provide pilot data and comments.
Create mock report to illustrate type of information obtained from instrument.
CDC internal workgroup discusses comments and mock report.Modify instrument as necessary.Slide147
Applying the StandardsHow were the evaluation standards employed in this example?
How was “engaging the stakeholders” important in applying each of the standards?Slide148
The “Utility” Standard
Who are the intended users of the information?Why do they need the information?
CDC Air Pollution and Respiratory Health Branch
State partners
Examine progress. Identify
promising patterns.Share successful practices
Identify areas for improvement.
Compare and contrast
program designs and performance. Slide149
The Utility Standard in Action
Determined what information the CDC needed to do their job and how they will use the information.Engaged states in discussions about how to make the information useful for them.
Developed mock reports to facilitate conversations with stakeholders about how the data could be used.Slide150
The “Feasibility” Standard
How much money, time, and
effort can we put into this?
Remained conscious of how much time this data collection
will require of the states
.Identified areas where processes could be streamlined.
Recognized that there will be less time for other programmatic activities.Slide151
The Feasibility Standard in Action
Carefully selected “need-to-know” questions.Asked states:
What information do you already have?
How feasible is it for you to obtain?How long does it take to locate information?
How long does it take to
fill out the data collection instrument?Always kept the burden of data collection as low as possible.Slide152
The Feasibility Standard in Action
The content of the data collection instrument was a careful balance between feasibility and utility.
High priority data requiring higher burden was balanced by eliminating lower priority questions.Slide153
The “Propriety” Standard
What steps need to be taken for the evaluation to be ethical?
Program monitoring
systems can be used to generate performance measures
.
The data collected could have significant implications.
Understand that the information collected can affect the stakeholders’ programs.Slide154
The Propriety Standard in Action
Used a highly collaborative process.Fostered open, honest dialogue about expectations and concerns.
Remained
highly responsive to stakeholder concerns.Modified instrument to address stakeholder concerns.Slide155
The “Accuracy” Standard
What design will lead to accurate
information?Be certain that the information collected is
valid and reliable.If the information is inconsistent or irrelevant to the concept, reconsider whether to collect it.Slide156
Balancing the StandardsWhen balancing feasibility vs. accuracy
consider the effect on propriety.Slide157
The Accuracy Standard in Action
Involved states in: Pilot tests and reviews.
Discussions about “guessing”.
Analyzing the quality of the information provided.R
evising questions or collecting information that would help to improve accuracy. Decisions to discard some questions.Slide158
What We Have Reviewed Today
The goal of evaluation and role of stakeholders. Who to engage as stakeholders.
Why it is important to engage stakeholders. When and how to engage stakeholders.
Stakeholder involvement across all steps of CDC framework.Stakeholder involvement in fulfilling the evaluation standards.
Examples: Single and multiple sitesSlide159
Thank You for Joining Us!Leslie Fierro: Let6@cdc.gov
Carlyn Orians: Orians@Battelle.org