/
Using expert review to calibrate Using expert review to calibrate

Using expert review to calibrate - PowerPoint Presentation

luanne-stotts
luanne-stotts . @luanne-stotts
Follow
386 views
Uploaded On 2017-04-12

Using expert review to calibrate - PPT Presentation

semiautomated adjudication of vital sign alerts in Step Down Units Society of Critical Care Medicine Annual Congress January 2016 Madalina Fiterau mfiteraucscmuedu Donghan Wang ID: 536690

adjudication alerts expert review alerts adjudication review expert based model events projections committee informative artifacts data alert chart rate trained semi labeled

Share:

Link:

Embed:

Download Presentation from below link

Download Presentation The PPT/PDF document "Using expert review to calibrate" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.


Presentation Transcript

Slide1

Using expert review to calibrate semi-automated adjudication of vital sign alerts in Step Down Units

Society of Critical Care Medicine Annual CongressJanuary 2016

Madalina Fiterau (mfiterau@cs.cmu.edu)Donghan Wang (donghanw@cs.cmu.edu)Artur Dubrawski (awd@cs.cmu.edu)

Gilles Clermont

(

cler@pitt.edu

)

Marilyn

Hravnak

(

mhra@pitt.edu

)

Michael

R. Pinsky

(

pinsky@pitt.edu

)Slide2

DisclosuresFunded by awards:National Science Foundation awards 0911032 and 1320347National

Institutes of Healthaward R01NR013912No commercial conflicts of interestSlide3

BackgroundPatients are monitored using non-invasive vital sign (VS) data

Alerts issued when a VS exceeds predefined thresholdsMany alerts are artifacts, due to threshold-based issuanceArtifacts cause alarm fatigue

Machine Learning has proven useful in classifying clinical dataTraining data requires laborious expert annotationSlide4

ObjectiveReduce expert annotation effort through semi-automatic adjudication of VS alerts as

real or artifacts, while maintaining high accuracy.Slide5

Heart Rate<40 or >140Respiratory Rate

<8 or >36Systolic Blood Pressure<80 or >200Diastolic Blood Pressure>110SPO2<85%

w

indow preceding alert

a

lert duration

Features computed from time series include common statistics of each VS: m

ean,

stdev

, min, max,

range of values, duty

cycle ...

Alerts

some are

artifacts

, not

true alerts

Data DescriptionSlide6

Artifact adjudication modelsSPO2 model trained on 91 committee-labeled events

RR model trained on 194 committee-labeled events

Alert events

Events

l

abeled by committee

S

elected for expert review

Review based on

Informative Projections

Adjudication

Model

Expert Review SystemSlide7

Extract

simple projections of data in which alerts appear as either convincingly correct or easily dismissible

Informative ProjectionsSlide8

Artifact adjudication modelsSPO2 model trained on 91 committee-labeled events

RR model trained on 194 committee-labeled events

Alert events

Events

l

abeled by committee

S

elected for expert review

Review based on

Informative Projections

Chart-based review

Automatic Adjudication

Adjudication

Model

Calibration

Expert Review SystemSlide9

Maximum

Respiratory Rate

Median Respiratory Rate

Artifacts

Real alerts

New alert

New alert

can be

confidently adjudicated

with the informative projection.

Review based on projectionsSlide10

Chart-based reviewSlide11

32 alerts are confidently adjudicated.

17 alerts are

ambiguous.31 alerts are confidently adjudicated.

Expert

r

eview based on Informative Projection

(80 alerts)

Experts agree with each other regarding label.

(32 alerts)

Experts disagree

(48 alerts)

Chart review

Experts agree

(31 alerts)

Experts disagree

(17 alerts)

Study ResultsSlide12

Semi-automated adjudication

reduces error

Number of alerts

Correct classification

from projections

Correct classification from chart

Incorrect classification

Alerts marked ambiguous

Adjudication errorSlide13

Half of alerts that can be classified are

handled automatically

3 ways

ML reduces expert labeling effort

Use of

ML models

for semi-automatic adjudication

Active

sample selection

for expert review

Threshold adjustment maximizes

confident adjudication

1/5 of alerts

could not be classified

by system or reviewers

Semi-automated adjudication model filters

out

artifactual

alerts, helping to reduce alarm fatigueConclusions