Jonathan Fryer Noreen Corcoran Brian George Ed Wang Deb DaRosa Northwestern University Feinberg School of Medicine Chicago Illinois Resident Selection Process 1 How effective is the ranking process in selecting residents who will perform well ID: 602545
Download Presentation The PPT/PDF document "Does resident recruitment ranking predic..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Does resident recruitment ranking predict subsequent performance?
Jonathan Fryer, Noreen Corcoran,
Brian George, Ed Wang, Deb DaRosa
Northwestern
University Feinberg School of Medicine,
Chicago
IllinoisSlide2
Resident Selection ProcessSlide3
1) How effective is the ranking process in selecting residents who will perform well?
2) How
effective is the ranking process in predicting subsequent resident performances?
3) How
effective are “adjustments” made to the preliminary rank list in improving the selection of residents who will perform well?
Questions?Slide4
General surgery categorical resident recruitment between
2002-2011
inclusive (n=46).
4 categorical residents
2002-2004.5 categorical residents 2005-2011.Residents who dropped out (n=1) or who were recruited after the PGY1 year (n=2) were excluded from the analyses.
MethodsSlide5
We compared how successful candidates were ranked during recruitment with their subsequent performance in our program.
MethodsSlide6
1) USMLE˄
Scores
alone
2)
Unadjusted Ranking Score (URS˄): based
on sum
of 3 assessment
scores
Academic Profile
(Coordinators)
Medical school rank, USMLE Step 1, Class Rank, Honors in SurgeryProgram Director Review (PDs)Research experience, extracurricular/community involvement, LORs, Personal Statement, Dean’s Letter Faculty Interview score (Faculty)Averaged for 2 independent faculty interview scores.
Recruitment Ranking Parameters
˄
Higher score is betterSlide7
3) Final Adjusted
Ranking
(
FAR˅
)Modification of the preliminary rank list generated by the URS Based on additional insights about specific candidates provided by the resident selection committee and/or leadership
Endorsements from trusted colleagues
Negative interactions with staff
Concerns raised by residents, coordinators
Other
?
Recruitment Ranking Parameters
˅ Lower is betterSlide8
Resident Selection Process
NUFSM
*USMLE includedSlide9
1)
ABSITE˄
percentile alone
2)
Resident Evaluation Grade (REG˄)Semiannual evaluation scores (Letter grade: A-F)Group discussion and grade assignment based on: Clinical Evaluations (360°): faculty, peers, med students, nurses, patients, etc.
Compliance: evaluations, case log, duty hour log, conference attendance, etc.
ABSITE, Mock Oral, PAME scores.
Resident Performance Measures
˄
higher is betterSlide10
3) Independent Faculty Rating/Ranking (IFRR˅)
Confidential survey with faculty independently rating all residents using a 7-point
Likert
scale and ranking resident within their PGY1 recruitment cohorts.* not part of standard resident evaluation at
NU
Resident Performance Measures
˅
lower is betterSlide11
Full IRB approval was obtained.All resident ranking and performance data was de-identified after collection and aggregated to protect resident confidentiality.
MethodsSlide12
Semiannual Resident Evaluation Grades (A-F) were converted to numerical values (5-0, respectively) and averaged for analyses.
Data from Individual Faculty Rating/Ranking surveys were averaged for individual residents.
MethodsSlide13
Statistical Analyses performed using SAS 9.2
software (Cary
, NC
)
.Associations between ranking and performance parameters were analyzed using Spearman’s correlation coefficient.Comparison of ranking
parameters between
poor and satisfactory performance used
student
t-test.
Differences in
performance based on ranking range were compared using F-test. MethodsSlide14
Poor Resident
Performance
Parameter
# (%) comments
Drop
outs
1 of
46 (2.2%)
excellent evaluations
ABS QE/CE fail0 of 13 (0%)
REG < C *
1 of
46 (2.2%)
Probation
IFFR > 4.0
6
of
46 (13.0%)
>4.0
= below average
ABSITE
< 35%
*
12 of
46 (26.1%)
Results
Overall resident performance
* Occurring at any time during residency trainingSlide15
Recruitment
Ranking
USMLE˄
URS˄
FAR˅
Resident Performance
R*
p
R*
p
R*
p
ABSITE (%tile
)˄
0.61
<0.0001
0.06
0.6952
0.09
0.5891
PGY1
REG˄
0.12
<0.4087
0.40
0.0058
0.17
0.2597
Overall
Grade˄
0.16
<0.2783
0.34
0.02190.160.2783Independent Faculty Rating˅0.22<0.14090.020.9020-0.120.4245
Recruitment Ranking vs Performance
* Spearman correlation coefficient
˄ higher is better
˅ lower is betterSlide16
USMLE˄
URS˄
FAR˅
Poor
Resident P
erformance
Criteria
Mean
SD
Mean
SD
Mean
SD
*REG
˄
<
C (n=1)
239.00
-
111.00
-
27.00
-
≥ C (n=46)
238.42
15.27
110.44
6.00
18.48
11.83p value0.80080.6041<0.0001IFRR ˅ ≥ 4.0 (n=6)232.5017.44111.335.20
13.149.12
< 4.0 (n=41)
239.33
14.76110.336.0819.5212.00p value0.30730.70230.2199*ABSITE ˄ % < 35 (n=12)228.9217.70110.255.9319.2513.59 ≥ 35 (n=35)243.4813.05110.326.3717.9511.95p value0.00570.97520.7625
Predicting Poor Resident PerformancesPredictors of Poor Resident Performances
* Occurring at any time during the residencySlide17
USMLE˄
URS˄
FAR˅
Poor
Resident P
erformance
Criteria
Mean
SD
Mean
SD
Mean
SD
*REG
˄
<
C (n=1)
239.00
-
111.00
-
27.00
-
≥
C (n=46)
238.42
15.27
110.44
6.00
18.48
11.83
p value
0.8008
0.6041
<0.0001IFRR ˅ ≥ 4.0 (n=6)232.5017.44111.335.2013.149.12 < 4.0 (n=41)239.3314.76
110.336.0819.5212.00p value0.30730.7023
0.2199
*ABSITE ˄ % < 35 (n=12)228.9217.70110.255.9319.2513.59 ≥ 35 (n=35)243.4813.05110.326.3717.9511.95p value0.00570.97520.7625
Predicting Poor Resident PerformancesPredictors of Poor Resident Performances
* Occurring at any time during the residencySlide18
Performance criteria
ABSITE
(%
tile)
PGY1
REG
Overall REG
IFRR
Rank
range
( % tile
)˅
Mean
SD
Mean
SD
Mean
SD
Mean
SD
<15 %
(n=14)
61.69
28.82
4.57
0.55
4.50
0.44
3.10
0.78
15 -30%
(n=17)
65.57
21.774.290.794.260.643.160.80>30% (n=22)66.4323.734.000.764.07
0.563.220.68
P value (F-test)
0.8831
0.11060.13380.9158Rank range vs. performance(URS)Slide19
Performance criteria
ABSITE˄
PGY1
REG˄
Overall
REG˄
IFRR˅
Rank Range
(% tile
)˅
Mean
SD
Mean
SD
Mean
SD
Mean
SD
<15%
(n=19)
60.84
26.40
4.63
0.62
4.42
0.50
3.13
0.83
15-30%
(n=9)
66.17
22.26
3.81
0.663.920.613.520.65>30% (n=22)70.0028.814.36
0.714.540.38
2.700.43P value
(F-test)0.64750.00250.00510.0156Rank range vs. performance(FAR)Slide20
Performance criteria
ABSITE˄
PGY1
REG˄
Overall
REG˄
IFRR˅
Rank Range
(% tile
)˅
Mean
SD
Mean
SD
Mean
SD
Mean
SD
<15%
(n=19)
60.84
26.40
4.63
0.62
4.42
0.50
3.13
0.83
15-30%
(n=9)
66.17
22.26
3.81
0.66
3.92
0.61
3.52
0.65>30% (n=22)70.0028.814.360.714.540.382.700.43
P value (F-test)0.64750.00250.0051
0.0156
Rank range vs. performance(FAR)Slide21
Single center studyURS confounded by USMLE score
REG confounded by ABSITE
No formal faculty orientation for IFRR survey
Study LimitationsSlide22
USMLE scores were predictive of subsequent ABSITE performance only.Unadjusted Ranking Scores (URS) were predictive of subsequent performance based on resident evaluation grades (REGs), while Final Adjusted Rankings (FAR) were not.
SummarySlide23
Our resident selection process has generally been successful in providing us with residents who perform well.
Our
unadjusted ranking score
appears to be a better predictor of subsequent resident performance than our final adjusted ranking… ….therefore caution should be exercised when considering adjustments
to the preliminary
rank
list,
as
they
may not engender selection of better performing residents.Effectively defining a reliable rank list “cutoff”, beyond which performance will predictably decrease, may not be possible in our system.ConclusionsSlide24Slide25Slide26Slide27
Drop outs: 1 (after PGY1 despite excellent performance)
ABS exams first try pass rate (n=13):
100%
Ever with REG <C 100% (i.e. probation):
1IFR > 4.0 (i.e. below average): 6Ever with ABSITE scores < 35%tile ever: 12
Results
Overall resident performance Slide28
Range
range
vs. performanceSlide29
USMLE
URS
FAR
Poor
Resident P
erformance
Criteria
Mean
SD
Mean
SDMean
SD
REG < C (n=1)
239.00
-
111.00
-
27.96
-
≥ C (n=46)
238.42
15.27
110.44
6.00
18.48
11.83
p value
0.8008
0.6041
<
0.0001
IFR ≥ 4.0
232.5017.44111.335.2013.149.12 < 4.0239.3314.76110.33
6.0819.5212.00
p value0.3073
0.7023
0.2199ABSITE % < 35228.9217.70110.255.9319.2513.59 ≥ 35243.4813.05110.326.3717.9511.95p value0.00570.97520.7625Predicting Poor Resident PerformancesPredictors of Poor Resident PerformanceSlide30
Absolute ranking correlation with resident performance:Absite
Semiannual evaluation grade
Faculty survey rating
Results
Absolute RankingSlide31
Absolute ranking (AR): Ranking among entire candidate group (n= 60-80).
Relative ranking (RR):
Ranking among cohort of successful PGY1 applicants (n=4 or 5).
Recruitment RankingSlide32
Within resident cohorts FAR did not correlate significantly with subsequent:
ABSITE scores (r=0.22; p=0.1760)
Semi-annual evaluation scores (r=0.20; p=0.1987)
Faculty survey cohort rankings (r=0.23;0.1175)
Conversely, USMLE scores exhibited a significantly positive correlation with subsequent:ABSITE scores (r=0.46; p=0.0022), Semi-annual evaluation scores (r=0.41; p=0.0163)Faculty cohort rankings (r=0.35; p=0.163)
Results (ABSTRACT)
Relative RankingSlide33
Resident recruitment involves a formal evaluation of candidates where a variety of objective and subjective criteria are used to rank candidates from best to worst.
Preliminary rank lists are often subsequently “adjusted” based on additional
insights
about the candidates.
IntroductionSlide34
Resident Selection Process
NUFSM