1 Jeanne-Marie Guise, MD, MPH - PowerPoint Presentation

Slide1

1

Jeanne-Marie Guise, MD, MPH

Professor Oregon Health & Science University

Stakeholder Engagement in the

Evidence-based Center (EPC) Practice ProgramSlide2

Objectives

Describe ways in which the EPC program has engaged stakeholders to improve relevance and applicability of its productsPresent methods used for stakeholder engagement

Identify challenges in stakeholder engagement and strategies employed by the EPC programSlide3

Evidence-based Practice Centers (EPCs)

11 centers throughout the United States and Canada

Systematically and critically appraise existing research and synthesize knowledgeProduce comparative effectiveness and effectiveness reviewsScientific Resource Center (SRC)Responsible for scientific methodological work of reviews and other research projects for EPCs and DEcIDES

EPCs & SRCSlide4

EPCs & SRC

Blue Cross and Blue Shield Association, Technology Evaluation Center,

Chicago, ILBrown University, Providence Rhode IslandECRI Institute, Plymouth Meeting, PAJohns Hopkins University, Baltimore, MDKaiser Foundation Research Institute, Oakland, CAOregon Health & Science University, Portland, ORRTI International – University of North Carolina at Chapel Hill, Chapel Hill, NCRAND Corporation , Santa Monica, CA

University of Alberta, Edmonton,

Edmonton, Alberta

University of Minnesota EPC,

Minneapolis, MNVanderbilt University Medical Center, Nashville, TN

Scientific Resource Center (SRC) for EPCs and

DEcIDES

– Portland VA Research Foundation,

Portland ORSlide5

Opportunities for Stakeholder InputSlide6

EHC Report on Engaging Stakeholders for Development and Prioritization of Future Research Needs

O’Haire C,

McPheeters M, Nakamoto E, LaBrant L, Most C , Lee K, Graham E, Cottrell E, Guise J-M. Engaging Stakeholders To Identify and Prioritize Future Research Needs. Rockville (MD): Agency for Healthcare Research and Quality (US); 2011 Jun. (Methods Future Research Needs Reports, No. 4.) Available from: http://www.ncbi.nlm.nih.gov/books/NBK62565/Slide7

Literature Synthesis

Phase III:

EPC interviews(Vanderbilt EPC)Recruitment and Interview Guide Development

Analysis of Interviews

Key Informant Interviews

Phase II: Key Informant Interviews

(Oregon EPC)

Phase I:

Literature Summary

(Oregon EPC)

Literature Review

Literature Search

Cumulative Synthesis

EPC Protocol Review

Invitations and EPC Discussion Guide

EPC Discussions

Analysis of DiscussionsSlide8

Stakeholder Identification & Recruitment

Identify pertinent stakeholder groups (intentional based upon stakeholder groupings)

Ensure a balance of stakeholder perspectives (i.e. consumer, clinician, researcher, research funder, insurer/payer, manufacturer)The results of EPC FRN pilot projects suggest that engaging 6-10 stakeholders is appropriateOMB restricts the number of participants in surveys to no more than 9 non-federal employees

Research Needs DevelopmentSlide9

Stakeholders

Stakeholder Group

DescriptionConsumer (public, patient, and caregiver)

An individual or advocacy group representing individuals who have a health condition, use health care services and/or who are members of the community

Clinician

(Health professional)

Health care or public health provider (e.g., academic, rural/frontier, and community) and medical and/or public health organizations

Social Service Organizations

An individual or organization that advances human welfare or social work (e.g. community or condition service providers, school-based programs, justice system)

Policymaker

An individual or organization who is involved in health care policy (e.g. local, state, provincial and Federal legislators and staff)

Medical organizations

Governmental organizations (e.g. VA, AHRQ, etc.)

Researcher

An individual who conducts and/or facilitates research activities in:

Basic, translational, clinical sciences; research methodology, public health or health services, and systematic reviews

Research Funder

A public or private organization that funds research (e.g. National Institute of Health, Susan G.

Komen

Foundation, and American Cancer Society)

Insurer/Payer

An organization or agency that pays for health-related goods and services (e.g. Blue Cross Blue Shield, Medicaid, and Medicare) or a business group that pays for health insurance (e.g. employers and government)

Manufacturer

A business group that produces health-related items (e.g. pharmaceuticals and medical devices)Slide10

Methods of Stakeholder Engagement

Engagement

Most common: Focus Groups, Symposia/ConferencesPrioritizationMost common: Priority Questionnaire, Consensus Engagement & Prioritization SimultaneouslyPriority SurveyDelphiSlide11

1:1 Meetings

1:1 Meetings

(phone)Focus GroupsGroup

Calls

Symposia

Consensus or Voting

Survey

Delphi

Short Timeframe

•

+

–

+

–

–

+

–

↓ Budget

–

+

•

+

–

–

+

–

Geographic Barriers

•

+

–

+

–

–

+

+

Differing Perspectives

+

+

–

–

+

–

+

+

Multiple

Stakeholder Groups

+

+

–

–

+

–

++↓Research Experience+++–+––•Complex Program+++–+–•–

+

GOOD

–

FAIR

•

POORSlide12

EHC Program Research Reviews

Topics

Evidence Review

Research

D&ISlide13

Topic Generation Approaches

Nominal Group Process

In-person

Idea generating

Intended to promote group participation

Our experience:

In-person brainstorming with flip chart used to document topics

Each state had 8 stickers to vote for priority topics:

#1 (2 red dots)

#2 (2 green dots)

#3 (2 blue dots), and

#4 (2 yellow dots

)

Weighting

system applied to priorities:

#1 - 4 points for each vote

#2 - 3 points

#3 – 2 points

#4 – 1 pointSlide14

Topic Generation Approaches

Medicaid Medical Directors

Analytic Process:

Weighting system

:

#1 - 4 points

#2 - 3 points#3 – 2 points#4 – 1 pointSlide15

Prevention in Women’s Health

Modified Process:

Local Key InformantsThree web meetings:May 11 – Framework & Initial Topics (Extranet & email topic generation)June 8 – Topic Generation (Evidence Scan) 13 – Final Prioritization

Prioritization

Survey – survey monkey

Web whiteboard during 3

rd

meeting

Topic Generation ApproachesSlide16

Stakeholders Engaged in:

Development of CER key questions Refinement of topic

Review of DraftMethods used:Group or individual callsWeb-based forums with survey type format to provide responses

Scoping CERSlide17

Preparation of Stakeholders is critical:

Assume that all stakeholders need orientation to the CER process

Very important for them to know where they fit in and what to expect from their feedbackFamiliarity with research methods likely to be very variableConflicts of interest are to be expected with stakeholder groups, but must be disclosed.

Scoping CERSlide18

EHC Program Research Reviews

Stakeholder Preparatory MaterialsSlide19

Where We Are

in the Comparative Effectiveness Review Process

Name of Panel

EPC Role &

What to Expect

Describe

what has occurred to date

Describe

Intent of panel

Numbers and types of individuals

Numbers of times and ways they may be engaged

List ways TEP is especially helpful to process

The role of the Evidence-based Practice Center (EPC) is to

…….

List Key Principles

Reviews commissioned under the Effective Healthcare Program (EHC) are posted publically at different stages of the review process, including at the stage of proposed Key Questions and the draft report stage

.

Stakeholder Preparatory MaterialsSlide20

Comparative Effectiveness Reviews (CERs) critically appraise and synthesize bodies of literature and identify areas where research is lacking or insufficient in either quantity or quality

There are limitations in funds available to support researchTo fully actualize their potential, CERs should be utilized to inform and guide future researchFuture Research Needs (FRN) documents are designed to facilitate this process by engaging stakeholders to prioritize future research topics

Future Research Needs Development

PurposeSlide21

Future Research Needs DevelopmentSlide22

Combined Methods:

Semi-structured interviews to identify topicsModified Delphi process for prioritization

Semi-structured interviewsTelephoneConference callsWebinarsDelphi#1 Survey Topics organized by PICOs Likert Scale – 6-point scale of no, low, highest priority#2 Survey Likert Ranking Indicate top 3 in order

Research Needs DevelopmentSlide23

Research Needs Development

Priority

Final Rank

Weighted

Score

a

Studies to test clinical, institutional, or policy interventions to increase access to “safe” TOL

1

72

Research on barriers to providing safe TOL, including factors that limit the ability of hospitals to meet the “immediately available” requirement

2

51

Studies comparing outcomes for mother and infant in settings where physicians are “immediately available” vs. settings where physicians are “readily available”

3

46

Studies to understand best practice models based on institutions that are currently offering safe TOL

4

41

Development of standardized measures for short-and long-term maternal and infant outcomes

5

41

Surveillance to determine long-term clinical outcomes of TOL vs. ERCD

6

38

Research on how patients understand risk, how they respond to different ways of framing risk, and how best to communicate risks of TOL vs. ERCD

7

37

Clinical and policy relevant studies to address the threat of legal liability on practice patterns regarding TOL vs. ERCD

8

34

Development/utilization of a reliable model or tool to predict the probability of successful VBAC for individual women /tool to predict probability of successful VBAC

9

32

Studies to refine, validate, and implement informed consent templates that are informative, reliable, and able to be well documented

10

26Slide24

Stakeholder

Involvement: Across

Research Phases

Research Continuum

Topic

Generation

Study

Design

Implementation

Analysis/

Interpretation

Research Prioritization

Dissemination

Clinicians

Consumers

Funders

Insurers

Researchers

Policymakers

Clinicians

Consumers

Funders

Insurers

Manufacturers

Researchers

Payers

Policymakers

Clinicians

Manufacturers

Researchers

Policymakers

Clinicians

Consumers

Funders

Insurers

Manufacturers

Researchers

Payers

Policymakers

Clinicians

Consumers

Researchers

Policymakers

Clinicians

Researchers

EPC Stakeholder Engagement Expanded Scope of Future Research Needs Beyond CER

Research Needs DevelopmentSlide25

Summary

Several options for methods to use when engaging stakeholders

Method depends on several factors includingIntent (identifying topics, prioritization)ResourcesNumber and variety of stakeholdersWhat is already known in topicExperienceImportant to outline approach and intent a prioriRelationships with stakeholders are importantContact potential stakeholders through multiple venuesConsistent communication and follow-up in the form of emails or phone calls is important to ensuring a balanced selection of stakeholders and a high response ratePreparation of stakeholders is critical to define expectations and allow engagement