/
Does resident recruitment ranking predict subsequent perfor Does resident recruitment ranking predict subsequent perfor

Does resident recruitment ranking predict subsequent perfor - PowerPoint Presentation

sherrill-nordquist
sherrill-nordquist . @sherrill-nordquist
Follow
381 views
Uploaded On 2017-11-04

Does resident recruitment ranking predict subsequent perfor - PPT Presentation

Jonathan Fryer Noreen Corcoran Brian George Ed Wang Deb DaRosa Northwestern University Feinberg School of Medicine Chicago Illinois Resident Selection Process 1 How effective is the ranking process in selecting residents who will perform well ID: 602545

performance resident scores ranking resident performance ranking scores rank faculty residents poor recruitment range evaluation value0 subsequent absite selection

Share:

Link:

Embed:

Download Presentation from below link

Download Presentation The PPT/PDF document "Does resident recruitment ranking predic..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.


Presentation Transcript

Slide1

Does resident recruitment ranking predict subsequent performance?

Jonathan Fryer, Noreen Corcoran,

Brian George, Ed Wang, Deb DaRosa

Northwestern

University Feinberg School of Medicine,

Chicago

IllinoisSlide2

Resident Selection ProcessSlide3

1) How effective is the ranking process in selecting residents who will perform well?

2) How

effective is the ranking process in predicting subsequent resident performances?

3) How

effective are “adjustments” made to the preliminary rank list in improving the selection of residents who will perform well?

Questions?Slide4

General surgery categorical resident recruitment between

2002-2011

inclusive (n=46).

4 categorical residents

2002-2004.5 categorical residents 2005-2011.Residents who dropped out (n=1) or who were recruited after the PGY1 year (n=2) were excluded from the analyses.

MethodsSlide5

We compared how successful candidates were ranked during recruitment with their subsequent performance in our program.

MethodsSlide6

1) USMLE˄

Scores

alone

2)

Unadjusted Ranking Score (URS˄): based

on sum

of 3 assessment

scores

Academic Profile

(Coordinators)

Medical school rank, USMLE Step 1, Class Rank, Honors in SurgeryProgram Director Review (PDs)Research experience, extracurricular/community involvement, LORs, Personal Statement, Dean’s Letter Faculty Interview score (Faculty)Averaged for 2 independent faculty interview scores.

Recruitment Ranking Parameters

˄

Higher score is betterSlide7

3) Final Adjusted

Ranking

(

FAR˅

)Modification of the preliminary rank list generated by the URS Based on additional insights about specific candidates provided by the resident selection committee and/or leadership

Endorsements from trusted colleagues

Negative interactions with staff

Concerns raised by residents, coordinators

Other

?

Recruitment Ranking Parameters

˅ Lower is betterSlide8

Resident Selection Process

NUFSM

*USMLE includedSlide9

1)

ABSITE˄

percentile alone

2)

Resident Evaluation Grade (REG˄)Semiannual evaluation scores (Letter grade: A-F)Group discussion and grade assignment based on: Clinical Evaluations (360°): faculty, peers, med students, nurses, patients, etc.

Compliance: evaluations, case log, duty hour log, conference attendance, etc.

ABSITE, Mock Oral, PAME scores.

Resident Performance Measures

˄

higher is betterSlide10

3) Independent Faculty Rating/Ranking (IFRR˅)

Confidential survey with faculty independently rating all residents using a 7-point

Likert

scale and ranking resident within their PGY1 recruitment cohorts.* not part of standard resident evaluation at

NU

Resident Performance Measures

˅

lower is betterSlide11

Full IRB approval was obtained.All resident ranking and performance data was de-identified after collection and aggregated to protect resident confidentiality.

MethodsSlide12

Semiannual Resident Evaluation Grades (A-F) were converted to numerical values (5-0, respectively) and averaged for analyses.

Data from Individual Faculty Rating/Ranking surveys were averaged for individual residents.

MethodsSlide13

Statistical Analyses performed using SAS 9.2

software (Cary

, NC

)

.Associations between ranking and performance parameters were analyzed using Spearman’s correlation coefficient.Comparison of ranking

parameters between

poor and satisfactory performance used

student

t-test.

Differences in

performance based on ranking range were compared using F-test. MethodsSlide14

Poor Resident

Performance

Parameter

# (%) comments

Drop

outs

1 of

46 (2.2%)

excellent evaluations

ABS QE/CE fail0 of 13 (0%)

REG < C *

1 of

46 (2.2%)

Probation

IFFR > 4.0

6

of

46 (13.0%)

>4.0

= below average

ABSITE

< 35%

*

12 of

46 (26.1%)

Results

Overall resident performance

* Occurring at any time during residency trainingSlide15

Recruitment

Ranking

USMLE˄

URS˄

FAR˅

Resident Performance

R*

p

R*

p

R*

p

ABSITE (%tile

0.61

<0.0001

0.06

0.6952

0.09

0.5891

PGY1

REG˄

0.12

<0.4087

0.40

0.0058

0.17

0.2597

Overall

Grade˄

0.16

<0.2783

0.34

0.02190.160.2783Independent Faculty Rating˅0.22<0.14090.020.9020-0.120.4245

Recruitment Ranking vs Performance

* Spearman correlation coefficient

˄ higher is better

˅ lower is betterSlide16

USMLE˄

URS˄

FAR˅

Poor

Resident P

erformance

Criteria

Mean

SD

Mean

SD

Mean

SD

*REG

˄

<

C (n=1)

239.00

-

111.00

-

27.00

-

≥ C (n=46)

238.42

15.27

110.44

6.00

18.48

11.83p value0.80080.6041<0.0001IFRR ˅ ≥ 4.0 (n=6)232.5017.44111.335.20

13.149.12

< 4.0 (n=41)

239.33

14.76110.336.0819.5212.00p value0.30730.70230.2199*ABSITE ˄ % < 35 (n=12)228.9217.70110.255.9319.2513.59 ≥ 35 (n=35)243.4813.05110.326.3717.9511.95p value0.00570.97520.7625

Predicting Poor Resident PerformancesPredictors of Poor Resident Performances

* Occurring at any time during the residencySlide17

USMLE˄

URS˄

FAR˅

Poor

Resident P

erformance

Criteria

Mean

SD

Mean

SD

Mean

SD

*REG

˄

<

C (n=1)

239.00

-

111.00

-

27.00

-

C (n=46)

238.42

15.27

110.44

6.00

18.48

11.83

p value

0.8008

0.6041

<0.0001IFRR ˅ ≥ 4.0 (n=6)232.5017.44111.335.2013.149.12 < 4.0 (n=41)239.3314.76

110.336.0819.5212.00p value0.30730.7023

0.2199

*ABSITE ˄ % < 35 (n=12)228.9217.70110.255.9319.2513.59 ≥ 35 (n=35)243.4813.05110.326.3717.9511.95p value0.00570.97520.7625

Predicting Poor Resident PerformancesPredictors of Poor Resident Performances

* Occurring at any time during the residencySlide18

Performance criteria

ABSITE

(%

tile)

PGY1

REG

Overall REG

IFRR

Rank

range

( % tile

Mean

SD

Mean

SD

Mean

SD

Mean

SD

<15 %

(n=14)

61.69

28.82

4.57

0.55

4.50

0.44

3.10

0.78

15 -30%

(n=17)

65.57

21.774.290.794.260.643.160.80>30% (n=22)66.4323.734.000.764.07

0.563.220.68

P value (F-test)

0.8831

0.11060.13380.9158Rank range vs. performance(URS)Slide19

Performance criteria

ABSITE˄

PGY1

REG˄

Overall

REG˄

IFRR˅

Rank Range

(% tile

Mean

SD

Mean

SD

Mean

SD

Mean

SD

<15%

(n=19)

60.84

26.40

4.63

0.62

4.42

0.50

3.13

0.83

15-30%

(n=9)

66.17

22.26

3.81

0.663.920.613.520.65>30% (n=22)70.0028.814.36

0.714.540.38

2.700.43P value

(F-test)0.64750.00250.00510.0156Rank range vs. performance(FAR)Slide20

Performance criteria

ABSITE˄

PGY1

REG˄

Overall

REG˄

IFRR˅

Rank Range

(% tile

Mean

SD

Mean

SD

Mean

SD

Mean

SD

<15%

(n=19)

60.84

26.40

4.63

0.62

4.42

0.50

3.13

0.83

15-30%

(n=9)

66.17

22.26

3.81

0.66

3.92

0.61

3.52

0.65>30% (n=22)70.0028.814.360.714.540.382.700.43

P value (F-test)0.64750.00250.0051

0.0156

Rank range vs. performance(FAR)Slide21

Single center studyURS confounded by USMLE score

REG confounded by ABSITE

No formal faculty orientation for IFRR survey

Study LimitationsSlide22

USMLE scores were predictive of subsequent ABSITE performance only.Unadjusted Ranking Scores (URS) were predictive of subsequent performance based on resident evaluation grades (REGs), while Final Adjusted Rankings (FAR) were not.

SummarySlide23

Our resident selection process has generally been successful in providing us with residents who perform well.

Our

unadjusted ranking score

appears to be a better predictor of subsequent resident performance than our final adjusted ranking… ….therefore caution should be exercised when considering adjustments

to the preliminary

rank

list,

as

they

may not engender selection of better performing residents.Effectively defining a reliable rank list “cutoff”, beyond which performance will predictably decrease, may not be possible in our system.ConclusionsSlide24
Slide25
Slide26
Slide27

Drop outs: 1 (after PGY1 despite excellent performance)

ABS exams first try pass rate (n=13):

100%

Ever with REG <C 100% (i.e. probation):

1IFR > 4.0 (i.e. below average): 6Ever with ABSITE scores < 35%tile ever: 12

Results

Overall resident performance Slide28

Range

range

vs. performanceSlide29

USMLE

URS

FAR

Poor

Resident P

erformance

Criteria

Mean

SD

Mean

SDMean

SD

REG < C (n=1)

239.00

-

111.00

-

27.96

-

≥ C (n=46)

238.42

15.27

110.44

6.00

18.48

11.83

p value

0.8008

0.6041

<

0.0001

IFR ≥ 4.0

232.5017.44111.335.2013.149.12 < 4.0239.3314.76110.33

6.0819.5212.00

p value0.3073

0.7023

0.2199ABSITE % < 35228.9217.70110.255.9319.2513.59 ≥ 35243.4813.05110.326.3717.9511.95p value0.00570.97520.7625Predicting Poor Resident PerformancesPredictors of Poor Resident PerformanceSlide30

Absolute ranking correlation with resident performance:Absite

Semiannual evaluation grade

Faculty survey rating

Results

Absolute RankingSlide31

Absolute ranking (AR): Ranking among entire candidate group (n= 60-80).

Relative ranking (RR):

Ranking among cohort of successful PGY1 applicants (n=4 or 5).

Recruitment RankingSlide32

Within resident cohorts FAR did not correlate significantly with subsequent:

ABSITE scores (r=0.22; p=0.1760)

Semi-annual evaluation scores (r=0.20; p=0.1987)

Faculty survey cohort rankings (r=0.23;0.1175)

Conversely, USMLE scores exhibited a significantly positive correlation with subsequent:ABSITE scores (r=0.46; p=0.0022), Semi-annual evaluation scores (r=0.41; p=0.0163)Faculty cohort rankings (r=0.35; p=0.163)

Results (ABSTRACT)

Relative RankingSlide33

Resident recruitment involves a formal evaluation of candidates where a variety of objective and subjective criteria are used to rank candidates from best to worst.

Preliminary rank lists are often subsequently “adjusted” based on additional

insights

about the candidates.

IntroductionSlide34

Resident Selection Process

NUFSM