San Antonio May 7 2022 THE AMERICAN BOARD OF SURGERY wwwabsurgeryorg Outline WelcomeJohn Mellinger Why EPAsDan Dent What have we learned from the pilotJake Greenberg What does the scholarship to date sayGeorge ID: 1047993
Download Presentation The PPT/PDF document "All Things EPA: An SEW Update" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
1. All Things EPA: An SEW UpdateSan AntonioMay 7, 2022THE AMERICAN BOARD OF SURGERY | www.absurgery.org
2. OutlineWelcome-John MellingerWhy EPAs?—Dan DentWhat have we learned from the pilot?—Jake GreenbergWhat does the scholarship to date say?—George SarosiIntegrating it all; SCORE and beyond—Amit JoshiWrap up and what’s coming—John MellingerTHE AMERICAN BOARD OF SURGERY | www.absurgery.org
3. Why EPAs?APDS 2022Daniel L. Dent, MD, FACSProfessor, Departments of Medical Education and SurgeryVice Chair for EducationChair, ABS Certifying Exam CommitteeChair-Elect, General Surgery Board, ABS
4. Financial DisclosureDaniel Dent, MD, has no relevant financial relationship with commercial interests to disclose.
5. Why EPAs?1) Until EVERY graduate from EVERY program is someone we would trust to care for our family, we are coming up short2) Competency-Based Education can help us achieve #1 Entrustable Professional Activities are units of Competency-Based Education3) Faculty will like EPAs better than Competencies or Milestones
6. Surgery Residency Graduates Are Coming Up ShortMattar, et. Al., Ann Surg 2013-21% of fellows arrive unprepared for OR-30% can’t do Lap Chole-38% don’t demonstrate patient ownership-66% can’t be left in OR 30 minutes
7. Counting Numbers Isn’t Enough95% of Trainees Reach Independence50% of Trainees Reach IndependenceDe Siqueira, Gough BJS (2016)
8. Results: Intracorporeal Suturing
9. Current Efforts in Competency Based Resident EducationABMS and ACGME are collaborating to move to CBREPilot trials to inform CBRE developmentThe NetherlandsOrthopaedic Surgery in CanadaGeneral Surgery in CanadaPediatrics pilot in the US
10. Why Entrustable Professional Activities (EPAs)?EPAs are the units of CBREEPAs define a specialtyEPAs are NOT defined by a procedure – but include procedural competencyFaculty like EPAs MUCH BETTER than Competencies
11. Why Will Faculty Like EPAs more than Competencies?Faculty will be asked if they “TRUST” the resident to do what physicians in that specialty are expected to do?Manage Right Lower Quadrant Pain vs ?Problem Based Learning and Improvement ?Manage Biliary Stone Disease vs ?Systems Based Practice?Provide Consultation vs ?Interpersonal and Communication Skills
12. What conversation do you hear/have as you walk down the hallway?“I don’t think Resident Steve is meeting expectations in Problem Based Learning and Improvement”? OR“Resident Shaina did a great job last night with that patient with Right Lower Quadrant Pain”?
13. Why EPAs?1) Until EVERY graduate from EVERY program is someone we would trust to care for our family, we are coming up short2) Competency-Based Education can help us achieve #1 Entrustable Professional Activities are units of Competency-Based Education3) Faculty will like EPAs better than Competencies or Milestones
14. ?
15. What Have We Learned Thus Far?Jacob A. Greenberg, MD, EdMProfessor of SurgeryLeon Henri Charbonnier Endowed Chair of Surgical EducationMedical College of Georgia at Augusta University
16. Pilot Data OverviewTotal possible residents1,115 unique resident IDs in file 6,236 ObservationsAverage of 5.6 observations per residentResidents who received at least one observation had a mean of 7.25 observationSource of Faculty ObservationsSurgical Faculty 85%Non-surgical faculty 13.8%Others 1.2%
17. Active residents per cycleCycleNumber of ResidentsNumber (%) of Residents with observationsNumber (%) of Residents with No observations1845265 (31.4%)580 (68.6%)2826355 (43%)471 (57.0%)3432186 (43.1%)246 (56.9%)414054 (38.6%)86 (61.4%)8,143 Surgery residents in US in 2019-20
18. Cumulative Observations by Program
19. Program Level Data
20. Program Level Data
21. Program Level Data OverviewHigh OutliersMiddle TierLow OutliersReason for Low OutlierPattern of PerformancecaImproving momentum, but only did 1 epaccccOnly data from Period 2 Only implemented 1 EPA and only had data from period 2e8Solid Improvement all the way throughf7Never got off the groundNot blank, but never got things goingfcReasonable response throughout with small number of residents2fZero Assessments (consider talking to them)"04"4e4eLost significant steam in period 35aOnly period 1Only intraop evaluations and nothing pre or post8cOnly period 2 Interesting that they had 20 for all pre, intra, and post hernia12Only period 1Trailed off after period 118Success with Gallbladder and complete failure with Consult2121Did Great in 1 and 2 and then fell off cliff. ? Data 33Better Intraop than Outside of OR47Performed way better with gallbladder than with trauma54Only period 1Never got started61Struggled to implement trauma6464Peters out consistentlyLow number per overall number of residents6767Steady performance, but not a ton of observations for program size69Was gaining momentum (also had many pre and post op compared to other programs)82Okay for one, struggled with trauma85Solid increase from 1 to 2. ? data from 3 to 4 missing89Only period 2 and minimal numbers98Zero Assessments (consider talking to them)a6Solid across the board. Lots of Pre and PostadWas gaining momentum. Missing datat from cycle 3 and 4bfSmall program that did very well (especially for consults!)c1Gained momentum, but overall very small numbers
22. Stakeholder Surveys (Andrew Jones)
23. Faculty Surveys
24. Faculty Surveys
25. Faculty Survey ThemesNeed for more faculty developmentGrand RoundsDidactic SessionsOnline ResourcesEPAs are valuable for residentsNeed more EPAs so that all faculty can be involved
26. Resident Surveys
27. Resident Surveys
28. Resident Survey ThemesNeed more education around EPAsDifficult to integrate into daily workflowAdditional administrative burdenIncrease faculty buy-inFelt to be a large barrier to implementationMore EPAs for broader use
29. PD Surveys
30. PD Surveys
31. PD Survey ThemesHit or MissSome thought this was very positive…………others not so muchCan clearly replace certain ABS requirementsCase Observations/Clinical AssessmentsNot currently a replacement for othersMilestones/EOR EvaluationsFocusing on resident autonomy could ease a transition to practice and build greater confidence
32. Semi-Structured InterviewsConducted semi-structured interviews with 8 Program DirectorsTranscribed and Coded for Thematic AnalysisThemesNeed for robust faculty/resident development toolsThe assessment solution needs to be EASY!
33. Resident Focus Groups
34. Best Practices Working GroupFacultyMackenzie Cook (OHSU) Prasad Poola (SIU)Nicole Garcia (ECU)Rabih Salloum (Rochester)Jason Johnson (Saint Joseph Hospital)Residency CoordinatorBrianne Nickel (IU)ResidentsErin White (Yale)Christina Theodorou (UC Davis)Kyle Cassling (Vanderbilt)Amy Holmstrom (Northwestern)Rebecca Williams (New Mexico)
35. Thank YouJacob A. Greenberg, MD, EdMjagreenberg@augusta.edu@Georgiahernia
36. What Does the Scholarship Say:Published Work From the ABS EPA PilotGeorgE A. Sarosi Jr. MD.Program DirectorUniversity of Florida
37. Descriptions of the ProjectEntrustable Professional Activities in General Surgery: Development and Implementation.Journal of Surgical Education 76(5) 2019 1174-1186A Phased Approach: The General Surgery Experience Adopting Entrustable Professional Activities in the United StatesAcademic Medicine 97(7S) 2021 s9-s13THE AMERICAN BOARD OF SURGERY | www.absurgery.org
38. Relationship Between EPAs and ACGME MilestonesAssociation Between Entrustable Professional Activities and Milestones Evaluations: Real-time Assessments Correlate With Semiannual Reviews Journal of Surgical Education 77(6) 2020 e220-228Concurrent Validity Evidence for Entrustable Professional Activities in General Surgery Residents Journal of the American College of Surgeons 234 (5) 2022 938-46THE AMERICAN BOARD OF SURGERY | www.absurgery.org
39. Implementation of EPAs in a Training programImplementation of Entrustable Professional Activities into a General Surgery Residency Journal of Surgical Education 77(4) 2020 739-48THE AMERICAN BOARD OF SURGERY | www.absurgery.org
40. Differences in Resident and Faculty AssessmentsEntrustable Professional Activities in General Surgery: Trends in Resident Self-Assessment Journal of Surgical Education 77 (6) 2020 1562-67Gender Differences in Entrustable Professional Activity Evaluations of General Surgery Residents Annals of Surgery 275 (2) 222-29THE AMERICAN BOARD OF SURGERY | www.absurgery.org
41. Concordance Between Comments and Anchoring BehaviorsNatural language processing and entrustable professional activity text feedback in surgery: A machine learning model of resident autonomy American Journal of Surgery 221 (2) 2021 369-75THE AMERICAN BOARD OF SURGERY | www.absurgery.org
42. Work in ProgressBest practice lessons learned Companion Papers based on survey data and structured interviews from pilot programsResident perspectiveFaculty and PD perspectiveEntrustment concordance between phases of care Primary report of pilot dataDid EPA assessment change milestone trajectory?How many assessments are required for entrustment?THE AMERICAN BOARD OF SURGERY | www.absurgery.org
43. Integrating It AllAPDS, San AntonioMay 2022Amit R.T. Joshi, MD, FACSProfessor of SurgeryAssociate Dean for GMECooper University Health CareCooper Medical School of Rowan University43
44. Disclosures44
45. ProgressHalsted (1900)AutocraticPyramidalChurchill (1930)Training by “Master Surgeons”RectangularCameron. Our Surgical Heritage. Ann Surg 1997Moore. Edward Delos Churchill. Ann Surg 1973Grillo. Edward D. Churchill and the “rectangular surgical residency.” Surgery 200445
46. ProgressFlexner (1910)Assessed the quality of education in medical schools. 12 of 168 medical schools closed, and 28 more were affected46
47. American College of SurgeonsFounded 1913First accreditors of hospitalsEducationQualityAdvocacy47
48. ABSFounded 1937Non-governmental & non-profitCertification of surgeonsSCOREEPAs48
49. ACGMEFounded 1981Standards for Training ProgramsCompetency-Based EducationConduit for CMS $49
50. Competencies50
51. Milestones51
52. Much Work to be Done52Medical SchoolUMETraining ProgramsGMEIndependentPractice
53. UME-GME Continuum53
54. Curriculum / AssessmentUME & GME educators should define a common framework and set of outcomes to apply to learners across the UME-GME transitionUME must commit to using robust and valid assessment tools and strategies (and develop their faculty)Lovell, Mejicano. The UME-GME Review Committee Final Report. Coalition for Physician Accountability Planning Committee. 2021. 54
55. What success looks likeTrainees/learnersunderstand the target to which they are trainingreceive useful & frequent formative feedbackFacultyeffectively teach, guide, rate, and provide feedbackare compensated for their education effortsPatients who receive higher quality care by our graduatesWithin training environments . . . and beyond55
56. EPA DatabaseMobileAppWeb-based InterfaceSummative Entrustment DecisionsABSSecure Backdoor Access to Hashed Data for Quality and ResearchPDs / PAsSpreadsheet Micro-assessment entry form3rd Party ServerCCCData AgreementAny app that can generate a spreadsheet ManualEntryDe-identification ProcessAutomatictransmissionDrag & Drop via webpageMilestones MapsSummative ReportsDashboardsEPAs56
57. Other Initiatives57
58. What success looks like with EPAsGMEBreak down the wall between UME/GMECollaborate across organizations58
59. The Educational WebTraining ProgramsMedical SchoolsIndependentPractice/OutcomesEPAs59
60. What success looks like60
61. 61
62. Wrap up and What’s ComingJohn D. MellingerVice President, ABS
63. Disclosure: John MellingerNothing to discloseEmployee of ABSTHE AMERICAN BOARD OF SURGERY | www.absurgery.org
64. What Will the ABS Provide?Mobile app platform for efficient data capture of all 19 EPAs including anchors for entrustment levels for each and dictation functionUltimate potential to integrate with EQiP and ACGME quality initiativesSecure strategy for data protectionMeaningful dashboard reports for multiple stakeholdersResidentsFacultyPD’sResidency AdministratorsCCC’sTHE AMERICAN BOARD OF SURGERY | www.absurgery.org
65. Full EPA Menu for General SurgeryE&M of a patient with RLQ pain/AppendicitisE&M of a patient with gallbladder diseaseE&M of a patient with inguinal hernia (groin)Evaluation and initial management of a trauma patientProvide general surgery consultationE&M of a patient with abdominal wall herniaE&M of a patient with the acute abdomenE&M of a patient with benign anorectal diseaseE&M of a patient with small bowel obstructionE&M of a patient with thyroid and parathyroid diseaseE&M of a patient with cutaneous and subcutaneous neoplasmsE&M of a patient with benign or malignant breast diseaseE&M of a patient with benign or malignant colon diseaseE&M of a patient with severe acute or necrotizing pancreatitisPerioperative care of the critically ill surgery patient (Inc Sepsis and Hemorrhage)E&M of a patient needing dialysis accessE&M of a patient with soft tissue infection (Inc NSTI)Flexible GI Endoscopy Perioperative care of the patient with significant comorbid disease
66. THE AMERICAN BOARD OF SURGERY | www.absurgery.org Note language that is provided, formative, and specific—automated documentation for learner, PD
67. THE AMERICAN BOARD OF SURGERY | www.absurgery.org
68. THE AMERICAN BOARD OF SURGERY | www.absurgery.org
69. THE AMERICAN BOARD OF SURGERY | www.absurgery.org
70. What Will the ABS Provide?Faculty, resident, and program leadership development materialsSupport and help desk functions to support implementationResearch to assess value of program or lack thereofConcerted effort to ‘add by subtracting’, i.e., dovetail with other key initiatives to make program time efficient for programsCCC feedCase log feedElimination of end of rotation and other less meaningful forms for assessment and feedbackMeans of addressing program requirements including populating WebADS Annual Report with meaningful initiativesTHE AMERICAN BOARD OF SURGERY | www.absurgery.org
71. What Will the ABS Expect From Programs?Summative report of level of entrustment for all EPAs at time of application to sit for Qualifying Exam for class commencing July 2023Engagement with project using materials ABS will provide for faculty and resident development and associated best practice strategiesIncorporating EPAs into the broader strategy of CBRETHE AMERICAN BOARD OF SURGERY | www.absurgery.org
72. Help us form a new habit…Feedback that is:TimelyFormativeDirect observation basedProximateRegularRelevant to practiceAutonomy-focusedTHE AMERICAN BOARD OF SURGERY | www.absurgery.org
73. …and a movement“By thinking globally, I can analyze all phenomena, but when it comes to acting, it can only be local and at a grassroots level if it is to be honest, realistic, and authentic.”—Jacques Ellul THE AMERICAN BOARD OF SURGERY | www.absurgery.org
74. EPA Program ChampionsThe feedback that we receive from EPA participants is invaluable. Local champions who serve as promoters and exemplars for the project will be critical to its success. That's where you come in!THE AMERICAN BOARD OF SURGERY | www.absurgery.org We covet resident, program director and faculty volunteers from each general surgery residency program to become an "EPA Program Champion“—a local leader and liaison. Scan this QR code with your smartphone’s camera to learn more and sign up!
75. Thank you, and questions?jmellinger@absurgery.org