Ben Kligler, M.D., Patricia Lebensohn, M.D., Sally Dodds, Ph.D., Raymond Teets, M.D., & Victoria Maizes, M.D.. Presentation Objectives. The objectives of this presentation are to:. Describe the function of formative evaluation for curriculum development.. ID: 689871
DownloadNote - The PPT/PDF document "Resident Evaluation of a Web-based Integ..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Presentations text content in Resident Evaluation of a Web-based Integrative Medicine Curriculum: The Function of Formative Evalu
Slide1
Resident Evaluation of a Web-based Integrative Medicine Curriculum: The Function of Formative Evaluation
Ben Kligler, M.D., Patricia Lebensohn, M.D., Sally Dodds, Ph.D., Raymond Teets, M.D., & Victoria Maizes, M.D.
Slide2
Presentation Objectives
The objectives of this presentation are to:
Describe the function of formative evaluation for curriculum development.Review the results from learners’ evaluation of the Integrative Medicine in Residency curriculum.
Describe the procedures used in applying feedback from learners’ evaluations to guide curriculum changes and revisions.
Slide3
Integrative Medicine in Residency (IMR) is…
Competency-based
, online, 200-hour, curriculum.
In-depth training in
Integrative Medicine
.
Incorporated
through all 3 years of Family Medicine residency.
Piloted at 8 residencies nationwide.Seamless, online evaluation of the curriculum and the residents.Responds to ACGME competency requirements.Evaluation developed simultaneously with the curriculum.
Slide4
University of Arizona
University of Texas Medical Branch
Hennepin County
Carolinas Medical Center
Beth Israel
Maine-Dartmouth
Maine Medical Center
University of Connecticut
IMR Program Locations
Slide5
Tools in Integrative Medicine
Prevention and Wellness
Women’s Health
Chronic Illness
Pediatrics
Acute Care
IMR Curriculum Units
Slide6
Introduction to Integrative Medicine
Prevention and Wellness:
U.S. Preventive Health Services
Nutrition and Diet
Supplements for Prevention
Physical Activity
Sleep
Stress and Mind-Body Medicine
Spirituality
Clinical Integration
Tools in Integrative Medicine:
Integrative Medicine Intake and Care Plan
Botanicals
Mind-Body Medicine
Manual Medicine
Introduction to Energy Medicine and Whole Systems
Practice Management
Motivational Interviewing for Behavioral Change
Acute Care:
Acute Back Pain,
Urinary Tract Infection,
Gastroenteritis,
Otitis
Media,
Vaginitis
,
Atypical Chest Pain,
Upper Respiratory Infection
Pediatric Topics:
ADD/ADHD
Chronic Pain Syndrome
Asthma and Allergies
Women’s Health Topics:
PMS/PMDD
Dysmenorrhea
Menopause
Fibromyalgia
Osteoporosis
Depression
Eating Disorders
Pregnancy and Lactation
Chronic Illness:
Cardiovascular Disease
Type II Diabetes
Osteoarthritis
Rheumatoid Arthritis
Obesity Irritable Bowel SyndromeChronic Back PainSpecial Topics:HIV/AIDS Cancer SurvivorshipEnvironmental Medicine
Units and Courses
Slide7
IMR: Educational Methods
Needs assessment
informed curriculum design.Web-based
curriculum written and edited by Integrative Medicine educators.
Competencies aligned with the
ACGME
Outcomes Project.
Flexible modular format
to meet the needs of residency schedules.Case-based, interactive learning and streaming video.Experiential exercises and process-oriented group activities at the residency sites.A community of learners
through online dialogues with faculty and peers.
Emphasis on teaching and
promoting physician well-being and self-care
.
Slide8
IMR User Interface
Online portfolios
Flexible modular format
Resources and Links
Direct Observation Checklists
Reflections
Slide9
“When the cook tastes the soup, that’s formative; when the guests taste the soup, that’s summative.”
- Robert Stakes
What is Formative Evaluation?
Slide10
Formative Evaluation…
Helps form and strengthen programs
by:Examining the need for them,
Their delivery or technology
The quality of their implementation
Assessing their organizational contexts, procedures, and resources
(Scriven, 1991).
Methods emphasize data collection and analysis
prior to completion.
Slide11
Why Use Formative Evaluation in the IMR?
Medical knowledge in Integrative Medicine is dynamic, evolving rapidly; revisions are largely driven by new information.
Distributed, web-based curricula requires feedback from learners to be successful.
In the IMR, formative methods include:
Preliminary needs assessment
Feedback from resident evaluation of courses
Slide12
1. IMR Needs Assessment
(Benn, Maizes, Guerrera,
Sierpina, Cook, & Lebensohn, 2009)
Methods
222 faculty and residents from 8 family medicine programs (60.2% response rate).
Online survey w/ structured and open-ended questions.
Results
Preferred IM be woven throughout all curriculum areas (67%).
Top topics: Nutrition, Supplements, and Physician Wellness.Top curricular areas for IM enhancement: Chronic illness, behavioral health, and outpatient medicineViewed IM central to family medicine training, patient care, and the field of family medicine (84%). Top challenges: Limitations in time, resources, and acceptance.
Slide13
2. IMR Course Evaluations
Measures assess learners’ ratings of the course in:Meeting course objectives
Clinical utility of the courseTime needed to complete the courseFunctionality of the online technology
Analyzed when 50% of pilot residents complete a course.
Ratings < 8o% and open-ended comments targeted for review.
Review of 01 pilot group suggested Likert-type response categories be changed from 4 to 5 points to increase precision and variability.
Slide14
Resident Demographics
2011 (n = 67)
2012 (n = 64)Controls (n = 31)
Sex
Female
Male
64%
36%
64%36%45%55%Mean Age range32 (27 – 43)32(24 – 56)30(26 – 38)
Marital Status
Married
Live
w/ partner
Single
Unk
.
45%
13%
27%
15%
27%
6%34%33%
58%10%32%0%
Graduate Status US MD
US DO FMG Unk.
51%18%27%
4%41%26%
28% 5%55%
16%16%13%
Slide15
Did Course Meet Learning Objectives?
Year 01 Content
*Currently, data are available for 4 courses for the 2012 class.
Slide16
Clinical Utility of the Course?
Year 01 Content
*Currently, indicators are available for 4 courses for the 2012 class.
Slide17
How Smooth was the Online Technology?
Year 01 Content
*Currently, indicators are available for 4 courses for the 2012 class.
Slide18
Meet Learning Objectives?
Clinical Utility?
Year 02 Course Evaluations
Courses currently at 50% threshold
Slide19
Year 02 Course Evaluations
Courses currently at 50% threshold
Smooth Online technology?
Slide20
Length of Time Spent
Year 01 &Year 02 Content
Class
Course
< 1
hr
1-2 hr
2-3 hr
3-4 hr4-5 hr
5-6 hr
2011
Introduction
34%
44%
13%
5%
4%
2012
34%
36%
26%
4%
2011
Supplements
32%
35%
12%
12%
5%
3%
2012
50%
36%
9%
5%
2011
Nutrition
& Diet
13%
25%
31%
16%
6%
7%
2012
21%
42%
29%
4%
4%2011 Physical Activity61%38%2%201266%28%7%2011 Stress & Mind Body 34%
43%
17%
3%
2%
2011
Spirituality
72%
26%
2%
2011
Sleep
& Health
94%
6%
2011
Preventive
Services
59%
32%
7%
2011
Clinical Integ
ration
17%
38%
21%
8%
9%
8%
2011
Allergy/Asthma
11%
43%
27%
11%
3%
5%
2011
ADHD
48%
48%
4%
2011
Pediatric
Pain
55%
27%
14%
4%
2011
PMD/PMDD
53%
40%
7%
Year 1
Year2
Slide21
What Residents Say…
I enjoyed the interactive nature of the modules.
The tools are amazing!
Did an excellent job of relating spirituality to health care.
I think sleep is one thing most neglected in health care topics. This course has rightly identified it.
The ability to self reflect as well as think about our patients.
I loved the video lecture – very engaging and interesting.
This is a very comprehensive topic.
Great links and resources.The seven cases. Great way to pull all of these courses together!
The buttons aren’t working.
There was a lot of data in this module [Nutrition and Diet], but not enough interactive learning.
Too lengthy and too in-depth.
Some links were not available.
Too many resources were given and I did not have enough time to thoroughly review them.
Too much reading, too many links, and too many videos.
Download this presentation
DownloadNote - The PPT/PDF document "Resident Evaluation of a Web-based Integ..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Presentations text content in Resident Evaluation of a Web-based Integrative Medicine Curriculum: The Function of Formative Evalu
Resident Evaluation of a Web-based Integrative Medicine Curriculum: The Function of Formative Evaluation
Ben Kligler, M.D., Patricia Lebensohn, M.D., Sally Dodds, Ph.D., Raymond Teets, M.D., & Victoria Maizes, M.D.
Slide2Presentation Objectives
The objectives of this presentation are to:
Describe the function of formative evaluation for curriculum development.Review the results from learners’ evaluation of the Integrative Medicine in Residency curriculum.
Describe the procedures used in applying feedback from learners’ evaluations to guide curriculum changes and revisions.
Slide3Integrative Medicine in Residency (IMR) is…
Competency-based
, online, 200-hour, curriculum.
In-depth training in
Integrative Medicine
.
Incorporated
through all 3 years of Family Medicine residency.
Piloted at 8 residencies nationwide.Seamless, online evaluation of the curriculum and the residents.Responds to ACGME competency requirements.Evaluation developed simultaneously with the curriculum.
Slide4University of Arizona
University of Texas Medical Branch
Hennepin County
Carolinas Medical Center
Beth Israel
Maine-Dartmouth
Maine Medical Center
University of Connecticut
IMR Program Locations
Slide5Tools in Integrative Medicine
Prevention and Wellness
Women’s Health
Chronic Illness
Pediatrics
Acute Care
IMR Curriculum Units
Slide6Introduction to Integrative Medicine
Prevention and Wellness:
U.S. Preventive Health Services
Nutrition and Diet
Supplements for Prevention
Physical Activity
Sleep
Stress and Mind-Body Medicine
Spirituality
Clinical Integration
Tools in Integrative Medicine:
Integrative Medicine Intake and Care Plan
Botanicals
Mind-Body Medicine
Manual Medicine
Introduction to Energy Medicine and Whole Systems
Practice Management
Motivational Interviewing for Behavioral Change
Acute Care:
Acute Back Pain,
Urinary Tract Infection,
Gastroenteritis,
Otitis
Media,
Vaginitis
,
Atypical Chest Pain,
Upper Respiratory Infection
Pediatric Topics:
ADD/ADHD
Chronic Pain Syndrome
Asthma and Allergies
Women’s Health Topics:
PMS/PMDD
Dysmenorrhea
Menopause
Fibromyalgia
Osteoporosis
Depression
Eating Disorders
Pregnancy and Lactation
Chronic Illness:
Cardiovascular Disease
Type II Diabetes
Osteoarthritis
Rheumatoid Arthritis
Obesity Irritable Bowel SyndromeChronic Back PainSpecial Topics:HIV/AIDS Cancer SurvivorshipEnvironmental Medicine
Units and Courses
Slide7IMR: Educational Methods
Needs assessment
informed curriculum design.Web-based
curriculum written and edited by Integrative Medicine educators.
Competencies aligned with the
ACGME
Outcomes Project.
Flexible modular format
to meet the needs of residency schedules.Case-based, interactive learning and streaming video.Experiential exercises and process-oriented group activities at the residency sites.A community of learners
through online dialogues with faculty and peers.
Emphasis on teaching and
promoting physician well-being and self-care
.
Slide8IMR User Interface
Online portfolios
Flexible modular format
Resources and Links
Direct Observation Checklists
Reflections
Slide9“When the cook tastes the soup, that’s formative; when the guests taste the soup, that’s summative.”
- Robert Stakes
What is Formative Evaluation?
Slide10Formative Evaluation…
Helps form and strengthen programs
by:Examining the need for them,
Their delivery or technology
The quality of their implementation
Assessing their organizational contexts, procedures, and resources
(Scriven, 1991).
Methods emphasize data collection and analysis
prior to completion.
Slide11Why Use Formative Evaluation in the IMR?
Medical knowledge in Integrative Medicine is dynamic, evolving rapidly; revisions are largely driven by new information.
Distributed, web-based curricula requires feedback from learners to be successful.
In the IMR, formative methods include:
Preliminary needs assessment
Feedback from resident evaluation of courses
Slide121. IMR Needs Assessment
(Benn, Maizes, Guerrera,
Sierpina, Cook, & Lebensohn, 2009)
Methods
222 faculty and residents from 8 family medicine programs (60.2% response rate).
Online survey w/ structured and open-ended questions.
Results
Preferred IM be woven throughout all curriculum areas (67%).
Top topics: Nutrition, Supplements, and Physician Wellness.Top curricular areas for IM enhancement: Chronic illness, behavioral health, and outpatient medicineViewed IM central to family medicine training, patient care, and the field of family medicine (84%). Top challenges: Limitations in time, resources, and acceptance.
Slide132. IMR Course Evaluations
Measures assess learners’ ratings of the course in:Meeting course objectives
Clinical utility of the courseTime needed to complete the courseFunctionality of the online technology
Analyzed when 50% of pilot residents complete a course.
Ratings < 8o% and open-ended comments targeted for review.
Review of 01 pilot group suggested Likert-type response categories be changed from 4 to 5 points to increase precision and variability.
Slide14Resident Demographics
2011 (n = 67)
2012 (n = 64)Controls (n = 31)
Sex
Female
Male
64%
36%
64%36%45%55%Mean Age range32 (27 – 43)32(24 – 56)30(26 – 38)
Marital Status
Married
Live
w/ partner
Single
Unk
.
45%
13%
27%
15%
27%
6%34%33%
58%10%32%0%
Graduate Status US MD
US DO FMG Unk.
51%18%27%
4%41%26%
28% 5%55%
16%16%13%
Slide15Did Course Meet Learning Objectives?
Year 01 Content
*Currently, data are available for 4 courses for the 2012 class.
Slide16Clinical Utility of the Course?
Year 01 Content
*Currently, indicators are available for 4 courses for the 2012 class.
Slide17How Smooth was the Online Technology?
Year 01 Content
*Currently, indicators are available for 4 courses for the 2012 class.
Slide18Meet Learning Objectives?
Clinical Utility?
Year 02 Course Evaluations
Courses currently at 50% threshold
Slide19Year 02 Course Evaluations
Courses currently at 50% threshold
Smooth Online technology?
Slide20Length of Time Spent
Year 01 &Year 02 Content
Class
Course
< 1
hr
1-2 hr
2-3 hr
3-4 hr4-5 hr
5-6 hr
2011
Introduction
34%
44%
13%
5%
4%
2012
34%
36%
26%
4%
2011
Supplements
32%
35%
12%
12%
5%
3%
2012
50%
36%
9%
5%
2011
Nutrition
& Diet
13%
25%
31%
16%
6%
7%
2012
21%
42%
29%
4%
4%2011 Physical Activity61%38%2%201266%28%7%2011 Stress & Mind Body 34%
43%
17%
3%
2%
2011
Spirituality
72%
26%
2%
2011
Sleep
& Health
94%
6%
2011
Preventive
Services
59%
32%
7%
2011
Clinical Integ
ration
17%
38%
21%
8%
9%
8%
2011
Allergy/Asthma
11%
43%
27%
11%
3%
5%
2011
ADHD
48%
48%
4%
2011
Pediatric
Pain
55%
27%
14%
4%
2011
PMD/PMDD
53%
40%
7%
Year 1
Year2
Slide21What Residents Say…
I enjoyed the interactive nature of the modules.
The tools are amazing!
Did an excellent job of relating spirituality to health care.
I think sleep is one thing most neglected in health care topics. This course has rightly identified it.
The ability to self reflect as well as think about our patients.
I loved the video lecture – very engaging and interesting.
This is a very comprehensive topic.
Great links and resources.The seven cases. Great way to pull all of these courses together!
The buttons aren’t working.
There was a lot of data in this module [Nutrition and Diet], but not enough interactive learning.
Too lengthy and too in-depth.
Some links were not available.
Too many resources were given and I did not have enough time to thoroughly review them.
Too much reading, too many links, and too many videos.
Slide22Discussion
Acknowledgements
University of Arizona
Pilot
Sites
Emily Sherbrooke
Victor Sierpina, M.D.
Paula
Cook
Selma Sroka, M.D.
Rhonda Hallquist
Mary Guerrera, M.D.
Tieraona Low Dog, M.D.
Dael
Waxman, M.D.
Craig Schneider,
M.D.
John Woytowicz, M.D.
Slide23