The Use of a Surgery Specific Written Examination in the Selection Process of Surgical - PowerPoint Presentation

The Use of a Surgery Specific Written Examination in the Selection Process of Surgical Residents Daniel Farkas, Kamal Nagpal, Ernesto Curras,Ajay Shah, John CosgroveDepartment of SurgeryBronx-Lebanon Hospital CenterAlbert Einstein College of Medicine

BackgroundChoosing residents for a surgical residency program is difficult and time consuming Programs are looking for many characteristics, and among them is the ability to acquire surgical knowledgePrograms are required by the RRC to have 65% of their residents pass the ABS exams on the first tryThese pass rates are available online to prospective candidates

BackgroundMany programs use a candidates USMLE scores to help identify their ability to gain surgical knowledge Some studies have found a correlation between USMLE scores and ABS exams passing scoresSome conflicting evidence as to whether step 1 or step 2 is the better testIn addition, the USMLE is broad based and not surgery specific

Study Purpose The purpose of this study was to look at a different method of assessing a candidate’s ability to gain surgical knowledgeWould using a surgery specific written exam (SSWE) be a better assessment of this ability as compared with the USMLE?

SSWE 50 question multiple choice SSWE was createdQuestions made by a faculty member, using standard surgical textbooksResidency candidates were given this exam on their interview dayEach exam scored out of 100% and added to candidates folders

Study Design Retrospective study of residents in our program between 2004 and 2012Academic folders were reviewed and pre-residency information was collectedUSMLE part 1 score (USMLE-1)USMLE part 2 score (USMLE-2)SSWE scoreIntra-residency data collectedABSITE score in PGY 1 through PGY 5 (ABSITE-1 to ABSITE-5)

Study Design Residents with all three pre-residency scores were included in the studyCorrelation coefficients were calculated between the pre-residency exam scores and the intra-residency exam scoresP values of less than 0.05 were considered significant

Results 53 residents had partial information43 residents had all three pre-residency scores and were included in the study38 had an ABSITE-1 score24 had an ABSITE-2 score18 had an ABSITE-3 score17 had an ABSITE-4 score14 had an ABSITE-5 score

  ABSITE-1ABSITE-2 ABSITE-3ABSITE-4ABSITE-5  n = 38 n = 24 n = 18 n = 17 n = 14 USMLE-1 r = 0.327 p = 0.045* r = 0.321 p = 0.126 r = 0.346 p = 0.160 r = 0.165 p = 0.527 r = -0.150 p = 0.610 USMLE-2 r = 0.314p = 0.055r = 0.187p = 0.381r = 0.415p = 0.086r = 0.245p = 0.343r = 0.044p = 0.880SSWEr = 0.656p < 0.001*r = 0.275p = 0.194r = -0.179p = 0.476r = -0.399p = 0.113r = -0.060p = 0.838 * Statistically significant

  ABSITE-1ABSITE-2 ABSITE-3ABSITE-4ABSITE-5 ABSITE-1  n = 21 r = 0.573 p = 0.007* n = 15 r = 0.850 p < 0.001* n = 12 r = 0.593 p = 0.042* n = 9 r = 0.870 p = 0.02* ABSITE-2     n = 17 r = 0.906p < 0.001*n = 14r = 0.904p < 0.001*n = 11r = 0.765p = 0.006*ABSITE-3   n = 15r = 0.824p < 0.001*n = 12r = 0.682p = 0.015*ABSITE-4    n = 14 r = 0.406 p = 0.150 * Statistically significant

Results - SummaryUSMLE may correlate with ABSITE-1 scores, but a SSWE has a much stronger correlation with ABSITE-1 than either step 1 or 2 ABSITE-1 scores correlate significantly with ABSITE scores in future years

Discussion Identifying residents that will do well on their board exams is critical to the success of a surgical residency programPart of a resident’s academic achievement is due to the educational activities within the programHowever, studies have shown that residents with poor USMLE scores have higher chances of failing the boards.1,2 Shellito JL, Osland JS, Helmer SD, Chang FC. American Board of Surgery examinations: can we identify surgery residency applicants and residents who will pass the examinations on the first attempt? Am J Surg . 2010;199:216-22. de Virgilio C, Yaghoubian A, Kaji A, et al. Predicting performance on the American Board of Surgery qualifying and certifying examinations: a multi-institutional study. Arch Surg . 2010;145:852-6.

DiscussionCorneille et al 1 showed that a targeted surgery exam during residency was able to predict which residents would do well on the ABSITEOur study showed that by giving this exam to residency candidates, we could predict which candidates were more likely to do well on the ABSITE Corneille MG, Willis R, Stewart RM, Dent DL. Performance on brief practice examination identifies residents at risk for poor ABSITE and ABS qualifying examination performance. J Surg Educ . 2011;68:246-9.

LimitationsCorrelation only significant with ABSITE-1 Possible with a bigger sample size this would changeABSITE-1 correlated with ABSITE scores in later yearsABSITE scores have been shown to correlate with board passing rates1SSWE scores were absolute whereas other exams were percentile scoresMakes it harder to compare between years de Virgilio C, Yaghoubian A, Kaji A, et al. Predicting performance on the American Board of Surgery qualifying and certifying examinations: a multi-institutional study. Arch Surg . 2010;145:852-6.

LimitationsSSWE only useful for surgical knowledge 5 other competenciesMany other important skills in being a successful surgical resident and a successful surgeonNot practical for every programMany candidates are wooed by other programs on their interview dayGood potential candidates could be turned away by getting an extra exam

Conclusions Notwithstanding the limitations, the SSWE is a very strong predictor of who will do well on ABSITE-1Higher ABSITE-1 scores is correlated with higher ABSITE scores in later yearsHigher ABSITE scores can predict who will pass their board exams on the first chance

Final ConclusionResidency programs looking for residents who will pass their boards on the first attempt are more likely to find these residents using a SSWE than by using the USMLE

Acknowledgements Kamal NagpalErnesto CurrasAjay ShahJohn CosgroveSaundra KingNancy Rivera