Behrang Amini How good a radiologist am I Do I talk good Do my words mean anything Words Word Percentage Is 100 Is not 0 Words Word Percentage Is 100 Possibly 199 ID: 914266
Download Presentation The PPT/PDF document "Confidence Calibration in Radiology" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Confidence Calibration in Radiology
Behrang Amini
Slide2How good a radiologist am I?
Slide3Do I talk good?
Slide4Slide5Do my words mean anything?
Slide6Words
%
Word
Percentage
Is
100%
Is not
0%
Slide7Words
%
Word
Percentage
Is
100%
Possibly
1-99%
May
1-99%
Can’t exclude
1-99%
Is not
0%
Slide8How well-calibrated am I?
Slide9Calibration?
Slide10Slide110
0.2
0.4
0.6
0.8
1
0
1
Predicted chance of event
Out-come
Slide120
0.2
0.4
0.6
0.8
1
0
33
5
5
4
1
0
1
0
2
3551
Predicted chance of event
Out-come
Slide130
0.2
0.4
0.6
0.8
1
0
33
5
5
4
1
0
1
0
2
3551
Sum337
8
9
6
1
Predicted chance of event
Out-come
Slide140
0.2
0.4
0.6
0.8
1
0
33
5
5
4
1
0
1
0
2
3551
Sum3378
9
6
1
Predicted chance of event
Out-come
total = 33 + 7 + 8 + 9 + 6 + 1
total = 64
positives = 0 + 2 + 3 + 5 + 5 + 1
positives = 16
prevalence = 16/64
prevalence = 0.25
Slide150
0.2
0.4
0.6
0.8
1
0
33
5
5
4
1
0
1
0
2
3551
Predicted chance of event
Out-come
Slide160
0.2
0.4
0.6
0.8
1
0
33
5
5
4
1
0
1
0
2
3551
0/33
2/7
3/8
5/9
6/5
1/1
0.29
0.38
0.56
0.83
1
Out-come
Predicted chance of event
Slide170
0.2
0.4
0.6
0.8
1
0
33
5
5
4
1
0
1
0
2
3551
0/33
2/7
3/8
5/9
6/5
1/1
0
0.29
0.38
0.56
0.83
1
Out-come
Predicted chance of event
Slide180
0.2
0.4
0.6
0.8
1
0
33
5
5
4
1
0
1
0
2
3551
0/33
2/7
3/8
5/9
6/5
1/1
0
0.29
0.38
0.56
0.83
1
Out-come
Predicted chance of event
Observed probability of event
Slide190
0.2
0.4
0.6
0.8
1
0
33
5
5
4
1
0
1
0
2
3551
0/33
2/7
3/8
5/9
6/5
1/1
0
0.29
0.38
0.56
0.83
1
Out-come
Predicted chance of event
Observed probability of event
Slide200
0.2
0.4
0.6
0.8
1
0
33
5
5
4
1
0
1
0
2
3551
0/33
2/7
3/8
5/9
6/5
1/1
0
0.29
0.38
0.56
0.83
1
Out-come
Predicted chance of event
Observed probability of event
Slide210
0.2
0.4
0.6
0.8
1
0
33
5
5
4
1
0
1
0
2
3551
0/33
2/7
3/8
5/9
6/5
1/1
0
0.29
0.38
0.56
0.83
1
Out-come
Predicted chance of event
Observed probability of event
Slide220
0.2
0.4
0.6
0.8
1
0
33
5
5
4
1
0
1
0
2
3551
0/33
2/7
3/8
5/9
6/5
1/1
0
0.29
0.38
0.56
0.83
1
Out-come
Predicted chance of event
Observed probability of event
Slide230
0.2
0.4
0.6
0.8
1
0
33
5
5
4
1
0
1
0
2
3551
0/33
2/7
3/8
5/9
6/5
1/1
0
0.29
0.38
0.56
0.83
1
Out-come
Predicted chance of event
Observed probability of event
Slide24Slide25http://
CheckMyAi.com
/
Slide26How to interpret curves?
Slide27Line of optimal calibration
Slide28Line of no resolution
0.25 (Prev.)
Slide29Lines of optimal resolution
0.25 (Prev.)
Slide30Lines of optimal accuracy
0.5
Slide31Lines of zero skill
0.5
Slide32Under-confident
Slide33Over-confident
Slide34No resolution
Slide35Optimal accuracy
Slide36How do we get there?
Slide37Slide380%
10%
30%
50%
70%
90%
100%
Slide390%
10%
30%
50%
70%
90%
100%
0%
10%
30%
50%
70%
90%
100%
0%
Slide400%
10%
30%
50%
70%
90%
100%
10%
Slide410%
10%
30%
50%
70%
90%
100%
30%
Slide420%
10%
30%
50%
70%
90%
100%
50%
Slide430%
10%
30%
50%
70%
90%
100%
70%
Slide440%
10%
30%
50%
70%
90%
100%
90%
Slide450%
10%
30%
50%
70%
90%
100%
100%
Slide461 year later
41 patients with > 6 months of follow-up
Slide47How well-calibrated am I?
Slide48Slide49“Findings concerning for locally recurrent disease”
“When I’m 80% sure there’s recurrence, I over-call recurrence”
Slide50How easy is this?
Slide510%
10%
30%
50%
70%
90%
100%
70%
Slide52Slide53Slide54bamini@mdanderson.org