Is It Working Yet? Evaluating and Creating Policy Changes for Complex Initiatives - PowerPoint Presentation

Slide1

Is It Working Yet? Evaluating and Creating Policy Changes for Complex Initiatives

Dr. Karen Linkins, IBHP, Tides Center

Dr. Benjamin Miller, University of Colorado

Dr. Lynda Frost, Hogg Foundation for Mental Health

Dr. Becky Hayes Boober, Maine Health Access FoundationSlide2

1) Become familiar with strategies to evaluate a complex health initiative;2) Explore strategies for advocating with policy makers;

3) Understand how to use data related to quality health care interventions to create compelling messages;

4) Gain insights on policy development and leveraging; and5) Share lessons learned and practical tools.

Session ObjectivesSlide3

Performance Accountability MeasuresHow much did we do?

How well did we do it?

Is anyone better off? Friedman, M. (2005).

Trying hard is not good enough: How to produce measurable improvements for customers and communities.

FPSI Publishing.

Mark Friedman QuestionsSlide4

How much did we do?

# Customers served (by customer characteristic)

How well did we do it?

% Common Measures

% Activity-specific Measures

Is Anyone Better Off?

# Skills/Knowledge

#Attitude/Opinion

# Behavior

# Circumstance

#Improved Health Outcomes

Is Anyone Better Off?

% Skills/Knowledge

% Attitude/Opinion

% Behavior

% Circumstance

% Improved Health OutcomesSlide5

Who are our “customers”?How can we measure if our “customers” are better off?

How can we measure if we are delivering services well?

How are we doing on the most important of these measures?Who are the partners who have a role to play in doing better?What works to do better, including no-cost and low-cost ideas?

What do we propose to do?

Performance Accountability QuestionsSlide6

Did we treat you well?Did we help you with your problem?

“Customer” SatisfactionSlide7

Performance Accountability QuestionsPopulation Accountability Questions

Evaluation FocusSlide8

What is the story behind these data?

What are the stories that can influence policy?Slide9

Fighting fragmentation at the level of innovation: Advancing the field of integrated primary care

Benjamin F. Miller,

PsyDDirector of the Office of Integrated Healthcare Research and PolicyDepartment of Family Medicine

University of Colorado Denver School of MedicineSlide10

The problem(s)

Sometimes in the face of innovation we lose sight of our ultimate goal – to change healthcare.

We focus on the problems rather than recognize what is working.

We focus on meeting immediate needs (e.g. financial) rather than plan for long term success.

We slip into “protective mode” and forget why we started the innovation to begin with.

We stop seeing the other innovators around us and focus on ourselves rather than the larger community or larger field. Slide11

State of the field

But firstSlide12

Brilliance

Brilliance

Brilliance

Brilliance

BrillianceSlide13

What we do (models)What data we collect (clinical)What we call ourselves (integrated)

What we need for sustainability (money)

Who we talk to (ourselves)What we want (change)

Fragmentation as a Parallel Process Slide14

Measuring integrated mental health (what is that exactly?)There is no gold standard “tool”

Consistency across sites (e.g., documenting mental health diagnosis)

The evidence is lacking and the field is in need of knowledge around the “elements”HUGE scopeFinancial sustainability (or the business case)

What’s the problem?Slide15

Range

Mental and Physical Health Multimorbidity

Coordination of mental and physical health treatment plans

Severe Mental Illness and/or Substance Abuse

Full coordination with specialty care

Medical issues with psychosocial barriers to care

Psychosocial Support Services

Medical issues requiring behavioral or psychological intervention

Behavior Change Education & Evidence-Based Treatments

Mental Health and Substance Use Presentations

Mental health treatment plan

Example Targeted Service Response

Mental Health PresentationSlide16

Two pots of moneyWorkarounds are often viewed as the solution We don’t know what we don’t know (but we think we know what we don’t know)

Turf wars and bad feelings

What’s the problem? – the money issueSlide17
Slide18

What we need to consolidate (or integrate)

Clinical data

LanguageFinancial data

What we measure

How we track and measure what we do

Better community connections and state to state connections (and collaborations)

Shared and consistent evaluation plans for integration projects Slide19

Studying time

Case studySlide20

Miller, B. F., B. Teevan, et al. (2011). "The importance of time in treating mental health in primary care."

Families, systems & health :

The

journal of collaborative family healthcare

29

(2): 144-145

.Slide21

Time spent with patientTime spent with other providersAssigning monetary amounts to time (and or patient volume)

Assessing changes in time and volume

Assessing value and outcomesLearning about what patients use more time and benefit from integrated initiatives

What can be tracked and learnedSlide22

Studying screening

Case studySlide23

Phillps

, R. L., B. F. Miller, et al. (2011). "Better Integration of Mental Health Care Improves Depression Screening and Treatment in Primary Care."

American Family Physician

84

(9): 980.Slide24

Number of patients identifiedNumber of patients treatedNumber of patients who improve from treatment

Comparing rates of identification to rates of diagnosis (accuracy)

Using screening tools repeatedly for treatment tracking What can be tracked and learnedSlide25

We must

In summarySlide26

Be heardKnow what policy solutions can help lead to sustainability (including financial)Begin to collect some of the

same

dataMake sure our data are put into the medical record in such a way it can be extracted Have an entity

that can pull it all together

Be compelling, be accurate, be timely

non-negotiable?Slide27

Benjamin.miller@ucdenver.edu

@miller7

occupyhealthcare.net Thank youSlide28

Evaluating Complex Initiatives: Lessons Learned for Sustaining Change and Influencing Policy

Karen W. Linkins, PhD

Project Director

Integrated Behavioral Health Project

Tides CenterSlide29

Systems

Change: Key Goal of Complex Initiatives

“Change is disturbing when it is done to us, but exhilarating when it is done by us” (Elizabeth Moss

Kanter

, Professor, Harvard Business School)

Many different definitions of systems change exist, but they share common elements: policies and practices, resources, relationships, power and

decision-making,

values, attitudes, skills,

governance, and supportive policies and reforms.

Systems

change is dynamic, developmental, non-linear, and

complex.

The target of change is the system, not the

individual.Slide30

Definition of Systems Change

System change is defined as: changes in organizational culture, policies and procedures within

and

across organizations that enhance or streamline access, and reduce or eliminate barriers to needed services by target populations. Slide31

What does sustainable systems change look like in integrated care?

Changes that endure beyond the funded project that lead to any or all of the following:

Increased Access

Improved Quality

Enhanced Efficiency

Increased Consumer EmpowermentSlide32

Examples of Systems Changes

Systems Change

Example

Increased Access

Changes in clinic operational policies (e.g., electronic open scheduling and wait time monitoring, expanding specialty staffing (

telepsychiatry

))

Improved Quality

Improve provider capacity to meet patient needs by learning new skills and knowledge through distance learning

Enhanced Efficiency

Data sharing across PC and BH providers to increase identification and care coordination

Increased Consumer Empowerment

Access to personal health record; use of technology to facilitate client support groupsSlide33

Factors in Designing Evaluations of Complex, Systems Change Initiatives

Stakeholder interests

Initiative goals, including desired outcomes and impactsHow findings will be used, e.g.:Educate policy makers

Disseminate best practices

Change local systems and policiesSupport sustainability plans and garner new funding sources

Available resources for the evaluationSlide34

Different stakeholders are interested in different outcomes

Providers:

Individual patient outcomes, panel management

Clinics/Clinic Systems:

Population health management, administrative metrics (e.g., cycle times, provider productivity, patient and provider satisfaction), billing, culture change

Policy Makers:

Cost and other administrative metrics

Community:

Prevention, community health and wellness, healthy behaviors, consumer engagement

Foundations:

Alignment with strategic priorities, return on investment, grantee accountabilitySlide35

CDC Evaluation Framework

Step 1: Engage stakeholders


Step 2: Describe the program
Step 3: Focus the evaluation design
Step 4: Gather credible evidence
Step 5: Justify conclusions


Step

6: Ensure use and share lessons learnedSlide36

Key questions to Guide Evaluation Design (CDC)

What will be evaluated? (program, context)

What aspects of the program will be considered in assessing program performance?What standards (i.e., type or level of performance) must be reached for the program to be considered successful?

What evidence will be used to indicate how the program has performed?

How will the lessons learned be used to improve public health effectiveness? Slide37

Evaluation Design Considerations

Design types: experimental, quasi-experimental, and observational designs.

No design is better or best in all circumstances.Design and methods should be matched to the interests of targeted stakeholders (e.g., foundation, grantees, policymakers). Slide38

Considerations (cont.)

Design drives what counts as evidence, how data are gathered, what claims can be made, who needs to be involved, and what data management systems are needed.

Mixed method designs are most effective because each method has biases and limitations.During the course of an evaluation, methods might need to be revised or modified. Slide39

Challenges and Threats to Evaluating Complex Initiatives

Complex initiatives require significant investments of time, resources and energy to create common ground for change.

Programs often become so focused on immediate implementation issues (client “fixes”), the long-term vision for systems change becomes lost or deferred.

Balancing the funder’s need for accountability/rigor in reporting with developing and maintaining authentic relationships with grantees.Slide40

Challenges and Threats (cont.)

Data collection must be

relevant.Data should not be collected unless they are

shared and fed back

to those responsible for collection.Evaluation should be clearly connected to longer term outcomes.

Failure to do so limits buy-in, understanding, and a greater sense of accountabilitytothe process.Slide41

Case Example: Integrated Care Initiative

Initiative Goals: Create

a more responsive and integrated system

of

care to increase access and reduce costs for individuals with co-morbid conditions (MH & chronic conditions)

Patient focused

Address patients’ needs, improve health outcomes

Reduce reliance on ED resources for care that is more effectively provided in less costly, community-based settings

System Focused

Reduce ED volume

and

diversion time, and avoidable

inpatient use

Encourage financing and policies that promote coordinated

, cross system,

multidisciplinary care and integration of servicesSlide42

Stakeholders Influencing Evaluation Process

Foundations

Project Officers/Program Staff

Policy Staff

Evaluation Staff

Program Office

Evaluation Team

Grantees & Collaboratives

Community-Based Organizations

Hospitals

Public Health, Housing/Homeless Programs, Mental Health, Substance Abuse, MediCal, Criminal Justice

Oversight

GroupSlide43

Evaluation Design

Participatory approach

Three evaluation phases of the evaluationPlanningImplementation ProcessOutcomes and Promising Practices (“What Works”)Multi-level, pre-post designSlide44

Enrolled TP Clients

Outcomes

Service utilization

Costs

Organizations

Policies and practices

Data systems

MOUs

Changes in services

County System

Data systems

Financing

Collaborations

New services

Restructuring

State Level

Laws and regulations

Budget and financing

Implementation Grants

(e.g., Intensive Case Management)

Structure

Intensity

Other Activities

Meetings/Convenings

Other activities

Broader FUI Initiative

Policy papers

Other activities

Intermediate Outcomes/Changes

Interventions

Long Range Impacts

Service Delivery Change

Client-based: Compare enrolled clients & TP at beginning and end of grant period (utilization and cost)

System-based: MIS analysis of changes in the patterns of service utilization and costs system wide

Planning Grants

Broad Systems Change

County

State

Frequent Users Initiative

TP = Target PopulationSlide45

Evaluation Outcomes

Measures

C

ost and

utilization (ED, inpatient and other systems as available)

Clinical measures of health and functioning

S

tability

(e.g., income and insurance enrollment)

S

ervice

intensity (frequency and duration)

Strength of partnerships and collaborations

Policy and systems change (evidence of improved coordination, streamlined access, permanent policy changes to address/eliminate barriers)Slide46

Evaluation Challenges

Participatory orientation

Balancing research rigor with “what’s reasonable and feasible” – selecting outcome measures and data collection strategies that

matched

capacity and didn’t over burden staff

Developing and maintaining meaningful stakeholder participation (on-going communication)

Establishing and maintaining trust of programs to ensure buy-in and data integrity

Defining/

operationalizing

multi-level outcomes

Ensuring evaluation findings

aligned

with and

relevant

to information needs of various stakeholders – at the “right time”Slide47

Evaluation Challenges (cont.)

Client centered interventions: challenge of programs/ models balancing individual client “fixes” vs. permanent programmatic and systems

changeData accuracy and

consistency

Data availability and linkage capability

Mis

-match of Foundation and Grantee Goals -- Foundations

wanted

systems and policy change, but funded local interventionsSlide48

Despite the Challenges . . .

Findings were compelling and rigorous enough to use for policy development (Medicaid Waiver and other legislation).

The combination of quantitative and cost data, as well as qualitative process and outcome data created a strong and policy relevant story of sustainable systems change.Slide49

Data stories can influencepublic policy.

Lynda Frost

Director of Planning and ProgramsHogg Foundation for Mental HealthSlide50

20 years of research on collaborative care model framed grant program on integrated healthcareLarge conference highlighted research and grantees’ work

Grantees engaged in advocacy around reimbursement, other issues

Evaluation of grant program gathered state-specific outcome data and identified barriers to implementation

Background Research and

Evidence-Based PracticesSlide51

Foundation convened key stakeholders to identify barriers to implementationOne stakeholder lobbied for creation of “Integration of Health and Behavioral Health Workgroup”

Legislation mandated broad group of appointed workgroup members

Resulting report described “best practices” and recommended next steps

“Best Practices” in Policy, Training

and Service DeliverySlide52

Foundation signed agreement with DHHS Office of Minority Health to examine integrated healthcare

as a means of eliminating health disparities in racial and ethnic minority populations and persons with limited English

proficiencyDeveloped consensus report drawing on practice-based evidenceHeld large conference to share results; OMH will release report with other national reports

Collaborative Agreement Around Practice-Based EvidenceSlide53

Maine Experience: Embedding Integrated Care

Becky Hayes Boober

Program OfficerMaine Health Access FoundationSlide54

Maine Health Access Foundation—2 storiesMiddle of the night sentence embedded in budget shifting hospital-based outpatient BH care from Section 45 to Section 65.

“Medical Care - Payments to Providers 0147

Initiative: Reduces funding from reducing reimbursement for outpatient substance abuse and mental health services to MaineCare Section 65 rates effective July 1, 2012.”

More ExamplesSlide55

Know what is happening (rule making draft)Take action (Work with DHHS to slow process)Explore alternatives

Partner (Maine Hospital Association and legislators)

MonitorResolutionSlide56

IFS Committee InvitationBudget shortagesMessaging is important (Endowment is 1/10 of 1% of what is spent annually in Maine on health care costs)

MeHAF Story 2Slide57

Build relationships, partnerships.Be proactive.

Tell a compelling story.

Human element (sans drama)DataCost effectivenessResulting outcomes

Embed into other key endeavors.

Identify key leverage points (employers)

MeHAF Advocacy StrategiesSlide58

Maine’s Medical Home Movement

540

Maine Primary Care Practices

26 Maine PCMH Pilot Practices

20 Pilot Phase 2 Practices

14 FQHCs CMS APC Demo

82 NCQA PCMH Recognized Practices

100

MaineCare

Health Home Practices

Payers

:

Medicare

Medicaid

Commercials (Anthem, Aetna, HPHC)

Payer

:

Medicare

Payer

:

MedicaidSlide59

BH HIT Support

and grant

ACA

Embedding Integrated Care into Maine’s Transforming Health System Reforms

540

Maine Primary Care

Practices; 53 Community

Behavioral Health Agencies;

30

SA Agencies

26 Maine PCMH Pilot Practices + 20 new

14 FQHCs CMS APC Demo (Medicare)

82 NCQA PCMH Recognized Practices

~100 MaineCare HH Practices??

Beacon

Payment reform grants; ACOs

Community Care Teams

DHHS Value-based contracting

SAMHSA Health Home

ACOs:

Pioneer and Employer-Based

Section 1703

FQHC expansion

AHRQ Academy

MeHAF IC grants, TASlide60

The Kid’s good.

The New Yorker.

March 21, 2011Slide61

Policy Development:Less Silver Bullet;

More Silver Buckshot.

What are your experiences?Slide62

In your small group, select a policy change you would like to see happen. Develop a messaging plan.

Creating Policy MessagesSlide63

What compelling human interest stories will build the case? What data do you have that will help build a compelling story? What data do you still need? How will you get it? Present it?

How will you involve patients/families?

Who are potential partners (current and needed)?Questions to ConsiderSlide64

Share 1 key idea about messaging.Share 1 key strategy for influencing policy, using data/stories.

SharingSlide65

Benjamin.miller@ucdenver.edu @miller7

occupyhealthcare.net

Lynda.frost@austin.utexas.edu karen@desertvistaconsulting.combhboober@mehaf.org

Contact Information