Criteria 11, 12 and 13 David Pieper (dpieper@med.wayne.edu) - PowerPoint Presentation

1. Criteria 11, 12 and 13David Pieper (dpieper@med.wayne.edu)with help from Brooke Taylor and Rebecca Daniel

2. ACCMEAccreditation Requirements and Descriptions Criterion 11: The provider analyzes changes in learners (competence, performance, or patient outcomes) achieved as a result of the overall program's activities/educational interventions.Criterion 12The provider gathers data or information and conducts a program based analysis on the degree to which the CME mission of the provider has been met through the conduct of CME activities/educational interventions.Criterion 13The provider identifies, plans and implements the needed or desired changes in the overall program (eg, planners, teachers, infrastructure, methods, resources, facilities, interventions) that are required to improve on ability to meet the CME mission.

3. Criterion 11The provider analyzes changes in learners (competence, performance, or patient outcomes) achieved as a result of the overall program's activities/educational interventions.

4. Criterion 11: Example of CitationThe evidence presented in the self-study report and in the activities reviewed demonstrated that the provider evaluated learner satisfaction and whether learning objectives were met. The provider did not present data related to, or an analysis of, changes in physician competence, performance, and/or patient outcomes

5. Majority of WSU Activities Assess Competence

6. Example of Evaluation of “Competence”Based on what you learned in this session, will you make changes in your practice? Yes or NoIf yes, please describe the changes you intend to makeWhat barriers to change do you anticipate?What strategies or mechanisms will you apply to overcome the barriers?

7. Example of ResponsesWSU/DMC Antithrombosis Conference BASED ON WHAT YOU LEARNED TODAY, WILL YOU MAKE CHANGES IN YOUR PRACTICE? YES: 101 NO (will keep the same): 21 NO (Retired): 2If Yes, please describe the changes you intend to make:will encourage our cardiologists to prescribe anticoagulation meds in stable non stented CAD patients no ASA – without assessment of patientwill look at CHADS2 VASC2 score along with CHADSestablish TSOAC monitoring procedureduration of therapy of anticoagulants in first event DVT/PE => more than 6 months is beneficialmaking recommendations for candidates for anticoagulation therapyI intend to implant formed INR goals on referring physicians to eliminate unwarranted goalsCHADS2VASC score initiatedevaluate bridging protocol for warfarin interruptions for patients with old (> 612 months) PE/DVT and factor check Leiden (didn’t realize this doesn’t increase risk of recurrent VTE)more confident to defend the role of warfarin over NOAC. I have more understanding of that datarecommend wider use of NOACs for VTE

8. What barriers to change do you anticipate?insurance prior authorizations and cost of new medsnone – DMC will implement changesnonetime away from clinic to discussinsurance coverage, or lack ofpatient preference to DOT and MD preference of DOTphysician resistant/threat to autonomyphysician pushbackphysician resistance to new agentseducation; physician buyintimecost of new anticoagulants (No GEQ)time and resourcesbilling for serviceExample of ResponsesWSU/DMC Antithrombosis Conference

9. What strategies or mechanisms will you apply to overcome these barriers?follow policies – DMCspeak to leadershipassistance programs (?)describe evidence support longer treatment of AC in first event of DVT/PEchange referrals, enforce new rulespersistencedelegate to resident assignmentsrefer programs to help patients pay for meds – patient assistance programsuse of EMR, resource allocation/shift, costavoidance justification, costbenefit justificationpresent primary literature discussed/presented todaychange note template in chartsmall sections at a time (of learning information)back up change in studies from programExample of ResponsesWSU/DMC Antithrombosis Conference

10. Case Conferences / Tumor BoardsExample from WSU/DMC Tumor Board Nodule SizeStatus or CommentsRecommendationsCase #1Nodular density 18mm RML & scarring; subcm LUL subpleural noduleNew PET/CT done 1/30 for review.In the board's opinion these findings are: Consistant with malignancy. At this time it is recommended that patient is referred to CT surgery. Case #2Mediastinal lymphadenopathy with subcarinal 15 mm & R hilar 19 mmHx NSCLC & melanoma. PET/CT done 2/4 for review.In the board's opinion these findings are: Inconsistant with malignancy. At this time it is recommended that the patient continue surveillance per oncology.Case #310mm spiculated nodulePET/CT done 2/3 to review. PFT's scheduled for Pt at NH.In the board's opinion these findings are: highly suspicious for malignancy. The board agrees with primary pulmonary team plan to refer patient to thoracic surgery and obtain PFTs.Case #4RUL mass 24x21mm & nodule 17x13mm; LUL nodule 10x7mm; RLL infiltrateHx of Renal Cell CA. New CT from 1/27 to review.In the board's opinion these findings are: Consistant with malignancy. The board recommends: the patient should be referred to CT surgery.No Patient Identifiers

11. Example from DMC VTE Prophylaxis Performance Improvement Project

12. Other Evaluation MethodsSurvey Monkey3 Month follow-upAudience ResponseSimulationMock scenariosHands-on performance

13. C11 Example from ACCMEFocus Group Reports (for sessions held March 2015June 2015) Facilitator: Dr. Dre7 participants from conference sessions (including 3 attendings, 2 nurse leaders, 1 resident, and 1 genetics counselor)All representatives shared new strategies for managing early diagnosis and consultation regarding surgeryIdentified systembased obstacles for implementing some strategies (e.g. scheduling delays for genetic counselors, poor communication between clinical and social work staff).Shared that it would be helpful to share this feedback with the group as a whole. (for sessions held July 2015September 2015) Facilitator: Ms. Black8 participants (2 surgeons, 1 medical records, chief quality officer, 1 attending, 2 nurse practitioners, and 1 medical scribe)Sessions on medical informatics were interesting, but didn’t provide actionable next steps. Participants didn’t know how to apply the information.Suggestions to include a tool (maybe a checklist) for future sessions(end of year) Facilitator: Dr. Long9 participants (2 surgical staff, 1 pathology attending, 1 path student, 1 case manager, quality manager, 1 pharmacist, and 2 oncology residents)“The coffee is terrible. Can we go back to having Starbucks?!”Participants said it was helpful to have the team participate this past quarter – had great discussions how to implement changes to our shared processes.Participants appreciated seeing their data “Great to see how survivor data is improving.” – and suggest that we should bring more data into conferences to provide feedback on how we’re improving.

14. Answer 

15. Beaumont HealthCriterion 11 – Evaluate Changes in CompetenceImmediate post-conference survey:Based on information received at this CME activity, what primary action will you take to change your practice?Seek additional informationImplement new information/skill(s) into my practiceDo nothing, current practice reflects activity recommendationsDo nothing, system barriers prevent me from changing my practiceOther, specifyGive an example of at least 1 action you plan to take to change your practice:

16. Beaumont HealthExample of Responses

17. Beaumont Health Criterion 11 – Evaluate Changes in Performance6 Week follow-up survey:

18. Beaumont Health Example of Reponses

19. Beaumont HealthCriterion 11 – Evaluate Changes in PerformanceHave you noticed an improvement or change in your patient outcomes since implementing this change in practice?Yes – please describeNoHave not tracked

20. Beaumont HealthExample of Reponses

21. Beamont Health Examples for C11Competence – what will you do?Performance – what did you do?

22. Beaumont Health Criterion 11: Activity Analysis of Data

23. Beaumont Health Criterion 11: Activity Analysis of DataOf the learners who successfully implemented a change in practice, 50% indicated they had noticed an improvement in their patient outcomes.“Patients happier” “Shortened wait times”“Better outcomes with bladder control”“Improved headache control in selected patients”

24. C11 Example from SJMAA Dermatology RSSAreas of assessmentYes NoCME activity met all objectives100Material was relevant to the learner’s practice100Learner intends on utilizing content of activity to make a practice change982Presentation was free of commercial bias100Learner feels this was a valuable activity and would attend again100Ways in which this activity changed competence, performance or patient outcomesIncreased knowledge base to take care of cong nevi, eruptive KA, Improved ability to utilize broad approach and perspective in management, awareness of morphine metabolism to enable better dosing, improve outcomes in NICU, facilitate early diagnosis of histiocytosis, helped improve my decision making with melanoma management.Main discussion points during the presentationTopical dermatology, excellent review Comments or suggestions for this educational activityMade me smart!

25. Evaluation Challenges Low percent of evaluations returnedMeaningless responses (“All 4 syndrome”)Information not summarized or utilizedSpeakers need individual ratings for P&T

26. Criterion 12Based on data and information gathered, provide your program based analysis on the degree to which the expected results component of your CME mission has been met through the conduct of your CME activities/educational interventions.

27. WSUReached Target Audience (Attendance Data)

28. WSU: Produced the types of activities stated in the mission:

29. DMC: Expected Results

30. (WSU)

31. Survey Responses: What do you like about the CME program at WSU/DMC?AccessibilityTracking my CMEReadily availableVariety of activitiesEasy, convenientVariety of speakers at Grand Rounds. Easily available, Practical.Close by, incorporated in clinical scheduleWell organizedAbility to present and discuss new informationIt is collegialVery activeAvailable and high qualityCell phone sign inBroad coverage

32. Survey Responses: What would you like to see improved with regard to the CME program at WSU/DMC?Website listing upcoming CME activities, including those coming in months ahead, so that I can plan my Schedule a few months in advanceMore use of teleconferencing to allow physicians that aren't in proximity to participateMore timely documentation of participationImproved access to my updated CME hoursOnline / SMART phone effortOnline versions for credit if unable to attend in personInterdepartmental CME coursesEasier access, web based learning without blackboard hassels

33. Continuity for current molecular diagnosticsContinued medical knowledge improvementWork life balanceSafetyCommunity Resources in Wayne County (Free / Low Cost Clinics; etc.)Use of rating scales to measure changePatient schedulingPractice organizationAutomatic access to current practice guidelines and updates preparing for ICD 10 implementationWSU Survey Responses: What are the areas of your practice that are most in need of improvement?

34. Beaumont Health Criterion 12: Program-based AnalysisTypes of Activities – what formats did you certify? Hours of Instruction – did you offer more than just AMA credit?Target Audience – who attended your activities? Internal staff? External?Learners – who were your learners? Physicians? Others?Content Areas – did you offer content in all areas you hoped?Are you satisfied with the results? How did this year compare to last year?

35. Beaumont Health Criterion 12 201020112012201320142015Seminars5293 263151 122138Regularly Scheduled Series96104125132147175Web-based Enduring32041868794Web-based Live042148Learning from Teaching000011Total Activities151221431370361416Number of Activities

36. Beaumont Health Criterion 12: Program-based Analysis – Expected OutcomesLearners are expected to gain competence in their approach to clinical problem solving, apply newly acquired strategies in their practice, and change their performance in order to provide evidence-based and patient-centered care.

37. C13 The provider identifies, plans and implements the needed or desired changes in the overall program (eg, planners, teachers, infrastructure, methods, resources, facilities, interventions) that are required to improve on ability to meet the CME missionAreas for ImprovementPlan(s) for ImplementationPlans for measuring impact of improvements on WSU’s ability to meet its CME missionSteps Taken and Changes Implemented    Need to integrate CME with quality improvement and GMEJoined AAMC AE4Q. Implemented large QI/PICME projects with QI and GME.Improved patient careCMS Core measures for VTE prophylaxis showed marked improvementNeed for online CME activitiesDeveloped new template for web based CME using CME TrackerA number of CME activities are now available on web using the new web portal infrastructure (CME Tracker)New web template established and a variety of activities now available on web for CMENeed to increase number of RSS with QI related topicPlan more RSS based on QI related topicsIncrease number of RSS based on QI related topicsOrganized Grand Rounds for Internal Medicine and Surgery on QI topicsNeed to improve efficiency of attendance data entry for RSSImplemented Cell Phone call in attendance systemGradually increasing number of RSS on the new call in attendance systemAbout 30% of RSS now on the call in systemNeed to give WSU Faculty ability to receive credit for Learning from TeachingDeveloped policy and forms for Learning from TeachingPolicies and forms developed and implementedNeed to revise the existing policy and forms to make the expectations easier to understandNeed to improve compliance of presentations with regard to trade namesContinue to review presentations and work with faculty to get them to understand policySystem is working but is cumbersome and requires a lot of diligenceWorking hard to make sure speakers do not use trade names in their presentationsNeed to integrate MOC Part IV projects into QI type CME programsConsidering ABMS Portfolio sponsorshipWould move WSU SOM more into demonstrable patient care quality improvementWe have participated in several quality improvement projects that were accredited for CME and were also awarded MOC Part IV credits(WSU)

38. ACCME Example C13In its self-study report, County Medical Hospital has described areas where it identified, planned and implemented needed or desired changes to its CME program. These included, “incorporating an audience response system,” “getting a better database to store and analyze participant data and records,”and “making changes in the evaluation forms to collect more information from learners about future topics.”

39. AnswerHas the provider described improvements to the overall program that will help their ability to meet their CME mission per the expectation of Criterion 13?  

40. ACCME Example C13The Accredited Provider conducts an overall program review and finds that they’ve “met their mission”. During the conduct of that overall program review, however, they have identified organizational challenges with regard to managing the planning and evaluation of CME activities. They describe implementation of a new CME tracking database as a solution to overcome these obstacles.  Has the provider described improvements to the overall program that will help their ability to meet the CME mission to the expectation of Criterion 13?

41. Answer

42. ACCME Example C12 and C13The accredited provider’s self-study narrative includes a description that it achieved some of the expected results (i.e., changing performance) in its mission statement, but has fallen short of goals to change patient outcomes. As a result, the provider described implementing strategic planning with its Executive Committee to identify ways to better align education with quality and safety for better patient outcomes. The provider reported at the interview that early results are showing progress in addressing some patient care issues like falls and poor HCAHPS (patient satisfaction) scores.Please indicate your findings for the provider’s performance for Criterion 12 and Criterion 13: Compliance with both C12 and C13Compliance with C12, but not C13Compliance with C13, but not C12Noncompliance for both C12 and C13

43. Answer