Resident Evaluation of a Web-based Integrative Medicine Curriculum: The Function of Formative Evalu

Resident Evaluation of a Web-based Integrative Medicine Curriculum: The Function of Formative Evaluation

Ben Kligler, M.D., Patricia Lebensohn, M.D., Sally Dodds, Ph.D., Raymond Teets, M.D., & Victoria Maizes, M.D.

Presentation Objectives

The objectives of this presentation are to:

Describe the function of formative evaluation for curriculum development.Review the results from learners’ evaluation of the Integrative Medicine in Residency curriculum.

Describe the procedures used in applying feedback from learners’ evaluations to guide curriculum changes and revisions.

Integrative Medicine in Residency (IMR) is…

Competency-based

, online, 200-hour, curriculum.

In-depth training in

Integrative Medicine

.

Incorporated

through all 3 years of Family Medicine residency.

Piloted at 8 residencies nationwide.Seamless, online evaluation of the curriculum and the residents.Responds to ACGME competency requirements.Evaluation developed simultaneously with the curriculum.

University of Arizona

University of Texas Medical Branch

Hennepin County

Carolinas Medical Center

Beth Israel

Maine-Dartmouth

Maine Medical Center

University of Connecticut

IMR Program Locations

Tools in Integrative Medicine

Prevention and Wellness

Women’s Health

Chronic Illness

Pediatrics

Acute Care

IMR Curriculum Units

Introduction to Integrative Medicine

Prevention and Wellness:

U.S. Preventive Health Services

Nutrition and Diet

Supplements for Prevention

Physical Activity

Sleep

Stress and Mind-Body Medicine

Spirituality

Clinical Integration

Tools in Integrative Medicine:

Integrative Medicine Intake and Care Plan

Botanicals

Mind-Body Medicine

Manual Medicine

Introduction to Energy Medicine and Whole Systems

Practice Management

Motivational Interviewing for Behavioral Change

Acute Care:

Acute Back Pain,

Urinary Tract Infection,

Gastroenteritis,

Otitis

Media,

Vaginitis

,

Atypical Chest Pain,

Upper Respiratory Infection

Pediatric Topics:

ADD/ADHD

Chronic Pain Syndrome

Asthma and Allergies

Women’s Health Topics:

PMS/PMDD

Dysmenorrhea

Menopause

Fibromyalgia

Osteoporosis

Depression

Eating Disorders

Pregnancy and Lactation

Chronic Illness:

Cardiovascular Disease

Type II Diabetes

Osteoarthritis

Rheumatoid Arthritis

Obesity Irritable Bowel SyndromeChronic Back PainSpecial Topics:HIV/AIDS Cancer SurvivorshipEnvironmental Medicine

Units and Courses

IMR: Educational Methods

Needs assessment

informed curriculum design.Web-based

curriculum written and edited by Integrative Medicine educators.

Competencies aligned with the

ACGME

Outcomes Project.

Flexible modular format

to meet the needs of residency schedules.Case-based, interactive learning and streaming video.Experiential exercises and process-oriented group activities at the residency sites.A community of learners

through online dialogues with faculty and peers.

Emphasis on teaching and

promoting physician well-being and self-care

.

IMR User Interface

Online portfolios

Flexible modular format

Resources and Links

Direct Observation Checklists

Reflections

“When the cook tastes the soup, that’s formative; when the guests taste the soup, that’s summative.”

- Robert Stakes

What is Formative Evaluation?

Formative Evaluation…

Helps form and strengthen programs

by:Examining the need for them,

Their delivery or technology

The quality of their implementation

Assessing their organizational contexts, procedures, and resources

(Scriven, 1991).

Methods emphasize data collection and analysis

prior to completion.

Why Use Formative Evaluation in the IMR?

Medical knowledge in Integrative Medicine is dynamic, evolving rapidly; revisions are largely driven by new information.

Distributed, web-based curricula requires feedback from learners to be successful.

In the IMR, formative methods include:

Preliminary needs assessment

Feedback from resident evaluation of courses

1. IMR Needs Assessment

(Benn, Maizes, Guerrera,

Sierpina, Cook, & Lebensohn, 2009)

Methods

222 faculty and residents from 8 family medicine programs (60.2% response rate).

Online survey w/ structured and open-ended questions.

Results

Preferred IM be woven throughout all curriculum areas (67%).

Top topics: Nutrition, Supplements, and Physician Wellness.Top curricular areas for IM enhancement: Chronic illness, behavioral health, and outpatient medicineViewed IM central to family medicine training, patient care, and the field of family medicine (84%). Top challenges: Limitations in time, resources, and acceptance.

2. IMR Course Evaluations

Measures assess learners’ ratings of the course in:Meeting course objectives

Clinical utility of the courseTime needed to complete the courseFunctionality of the online technology

Analyzed when 50% of pilot residents complete a course.

Ratings < 8o% and open-ended comments targeted for review.

Review of 01 pilot group suggested Likert-type response categories be changed from 4 to 5 points to increase precision and variability.

Resident Demographics

2011 (n = 67)

2012 (n = 64)Controls (n = 31)

Sex

Female

Male

64%

36%

64%36%45%55%Mean Age range32 (27 – 43)32(24 – 56)30(26 – 38)

Marital Status

Married

Live

w/ partner

Single

Unk

.

45%

13%

27%

15%

27%

6%34%33%

58%10%32%0%

Graduate Status US MD

US DO FMG Unk.

51%18%27%

4%41%26%

28% 5%55%

16%16%13%

Did Course Meet Learning Objectives?

Year 01 Content

*Currently, data are available for 4 courses for the 2012 class.

Clinical Utility of the Course?

Year 01 Content

*Currently, indicators are available for 4 courses for the 2012 class.

How Smooth was the Online Technology?

Year 01 Content

*Currently, indicators are available for 4 courses for the 2012 class.

Meet Learning Objectives?

Clinical Utility?

Year 02 Course Evaluations

Courses currently at 50% threshold

Year 02 Course Evaluations

Courses currently at 50% threshold

Smooth Online technology?

Length of Time Spent

Year 01 &Year 02 Content

Class

Course

< 1

hr

1-2 hr

2-3 hr

3-4 hr4-5 hr

5-6 hr

2011

Introduction

34%

44%

13%

5%

4%

2012

34%

36%

26%

4%

2011

Supplements

32%

35%

12%

12%

5%

3%

2012

50%

36%

9%

5%

2011

Nutrition

& Diet

13%

25%

31%

16%

6%

7%

2012

21%

42%

29%

4%

4%2011 Physical Activity61%38%2%201266%28%7%2011 Stress & Mind Body 34%

43%

17%

3%

2%

2011

Spirituality

72%

26%

2%

2011

Sleep

& Health

94%

6%

2011

Preventive

Services

59%

32%

7%

2011

Clinical Integ

ration

17%

38%

21%

8%

9%

8%

2011

Allergy/Asthma

11%

43%

27%

11%

3%

5%

2011

ADHD

48%

48%

4%

2011

Pediatric

Pain

55%

27%

14%

4%

2011

PMD/PMDD

53%

40%

7%

Year 1

Year2

What Residents Say…

I enjoyed the interactive nature of the modules.

The tools are amazing!

Did an excellent job of relating spirituality to health care.

I think sleep is one thing most neglected in health care topics. This course has rightly identified it.

The ability to self reflect as well as think about our patients.

I loved the video lecture – very engaging and interesting.

This is a very comprehensive topic.

Great links and resources.The seven cases. Great way to pull all of these courses together!

The buttons aren’t working.

There was a lot of data in this module [Nutrition and Diet], but not enough interactive learning.

Too lengthy and too in-depth.

Some links were not available.

Too many resources were given and I did not have enough time to thoroughly review them.

Too much reading, too many links, and too many videos.

Discussion

Acknowledgements

University of Arizona

Pilot

Sites

Emily Sherbrooke

Victor Sierpina, M.D.

Paula

Cook

Selma Sroka, M.D.

Rhonda Hallquist

Mary Guerrera, M.D.

Tieraona Low Dog, M.D.

Dael

Waxman, M.D.

Craig Schneider,

M.D.

John Woytowicz, M.D.