semiautomated adjudication of vital sign alerts in Step Down Units Society of Critical Care Medicine Annual Congress January 2016 Madalina Fiterau mfiteraucscmuedu Donghan Wang ID: 536690
Download Presentation The PPT/PDF document "Using expert review to calibrate" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Using expert review to calibrate semi-automated adjudication of vital sign alerts in Step Down Units
Society of Critical Care Medicine Annual CongressJanuary 2016
Madalina Fiterau (mfiterau@cs.cmu.edu)Donghan Wang (donghanw@cs.cmu.edu)Artur Dubrawski (awd@cs.cmu.edu)
Gilles Clermont
(
cler@pitt.edu
)
Marilyn
Hravnak
(
mhra@pitt.edu
)
Michael
R. Pinsky
(
pinsky@pitt.edu
)Slide2
DisclosuresFunded by awards:National Science Foundation awards 0911032 and 1320347National
Institutes of Healthaward R01NR013912No commercial conflicts of interestSlide3
BackgroundPatients are monitored using non-invasive vital sign (VS) data
Alerts issued when a VS exceeds predefined thresholdsMany alerts are artifacts, due to threshold-based issuanceArtifacts cause alarm fatigue
Machine Learning has proven useful in classifying clinical dataTraining data requires laborious expert annotationSlide4
ObjectiveReduce expert annotation effort through semi-automatic adjudication of VS alerts as
real or artifacts, while maintaining high accuracy.Slide5
Heart Rate<40 or >140Respiratory Rate
<8 or >36Systolic Blood Pressure<80 or >200Diastolic Blood Pressure>110SPO2<85%
w
indow preceding alert
a
lert duration
Features computed from time series include common statistics of each VS: m
ean,
stdev
, min, max,
range of values, duty
cycle ...
Alerts
some are
artifacts
, not
true alerts
Data DescriptionSlide6
Artifact adjudication modelsSPO2 model trained on 91 committee-labeled events
RR model trained on 194 committee-labeled events
Alert events
Events
l
abeled by committee
S
elected for expert review
Review based on
Informative Projections
Adjudication
Model
Expert Review SystemSlide7
Extract
simple projections of data in which alerts appear as either convincingly correct or easily dismissible
Informative ProjectionsSlide8
Artifact adjudication modelsSPO2 model trained on 91 committee-labeled events
RR model trained on 194 committee-labeled events
Alert events
Events
l
abeled by committee
S
elected for expert review
Review based on
Informative Projections
Chart-based review
Automatic Adjudication
Adjudication
Model
Calibration
Expert Review SystemSlide9
Maximum
Respiratory Rate
Median Respiratory Rate
Artifacts
Real alerts
New alert
New alert
can be
confidently adjudicated
with the informative projection.
Review based on projectionsSlide10
Chart-based reviewSlide11
32 alerts are confidently adjudicated.
17 alerts are
ambiguous.31 alerts are confidently adjudicated.
Expert
r
eview based on Informative Projection
(80 alerts)
Experts agree with each other regarding label.
(32 alerts)
Experts disagree
(48 alerts)
Chart review
Experts agree
(31 alerts)
Experts disagree
(17 alerts)
Study ResultsSlide12
Semi-automated adjudication
reduces error
Number of alerts
Correct classification
from projections
Correct classification from chart
Incorrect classification
Alerts marked ambiguous
Adjudication errorSlide13
Half of alerts that can be classified are
handled automatically
3 ways
ML reduces expert labeling effort
Use of
ML models
for semi-automatic adjudication
Active
sample selection
for expert review
Threshold adjustment maximizes
confident adjudication
1/5 of alerts
could not be classified
by system or reviewers
Semi-automated adjudication model filters
out
artifactual
alerts, helping to reduce alarm fatigueConclusions