Systematically Fair Modeling of Perioperative Cognitive Outcomes: - PowerPoint Presentation

Slide1

Systematically Fair Modeling of Perioperative Cognitive Outcomes:

A Learning Health Systems Approach

Patrick Tighe MD MS

Associate Professor

Donn M. Dennis M.D. Professor in Anesthetic Innovation

University of Florida Term Professor

Depts. of Anesthesiology, Orthopedics,

& Information Systems/Operations Management

University of Florida

ptighe@anest.ufl.edu

Specific Aims

Review Public Health Impact of Perioperative Neurocognitive ChangesPreliminary Data on Disparities and Fairness in Outcome ModelsLHS Aim 1: Add novel virtual mobile (tablet) based executive, episodic memory, and language platforms to the PeCAN Assessment and Data Model in a pilot set of participants (n=157) enrolled in an ongoing R01. Aim 2: Work within UFHealth to develop a “delirium supplement” for data integration into UFHealth Integrated Data Repository (IDR) and then plan piloting to the

OneFlorida

DataTrust

.

LHS Aim 3: Develop an “

MLOps

” Platform for ethical implementation of ML-based cognitive risk assessments.

Motivation

54% of all surgical procedures are performed on older adults, and up to 20% of those who present for elective surgery show early signs of cognitive impairmentACS recommends preoperative cognitive screening in older adultsPerioperative neurocognitive disorders (e.g. delirium) are the most common complication in older adults following surgery, with an incidence ranging from between 10% and 65%.

Independently associated with increased cost of postoperative recovery of over $17,000 dollars in the first year after surgery, 2x 1-year mortality

Existing cognitive assessments either

shallow

or

useful

dCDT

generates over 100k features per patient….now have 16k patients!

Require deep learning approaches for comprehensive analyses

Routine

Preoperative Care

Trained Preop RN

Screens Image

Form or drawing

into EHR

Raw Output

Example

(

from R01

AG055337)

Screening & Preoperative Evaluation

TM

Cognitive Data Abstraction Layers

Time Series (~100k)

[

x,y,pressure,time

] @ 80Hz

Structured Features (~4k)

Hand angle, digit placement, etc.

Aggregate Features (~100)

Time to Complete, etc.

Screening Score

Example clock drawing

Machine Learning

Statistical inference

Clinical Interpretation

Trained observation, contextualization

R01 AG055337

(….after UMAP)

Human attributes clock landmarks

Deep Learning model attributes pixels

Semi-supervised learning using BYOL for nominal scoring of CDT

Setup –

Datasets – We created 2 datasets namely train and test. The train set consists of 5000 images of which 457 are “Good” and 4543 are “Bad” clocks. The test set consists of 1342 images of which 110 are “Good” and 1214 are “Bad” clocks.

BYOL is trained using unlabeled dataset of 13480 clocks.

Encoder network (Resnet architecture) of BYOL, is extracted and appended to a single-layer linear neural net having 2 output nodes (corresponding to “Good” and “Bad” clock classes).

L

ast 3 layers are fine-tuned for 10 epochs using labelled dataset.

Classifier is weighted to remove class-imbalance problem.

Semi-supervised learning using BYOL for predicting patient

Education level

Edu categories

1 = No Schooling Completed

2 = Nursery School to 8th Grade

3 = Some High School, No Diploma

4 = High School Graduate

5 = GED

6 = Some College Credit, No Degree

7 = Trade/Technical/Vocational Training

8 = Associate's Degree

9 = Bachelor's Degree

10 = Master's Degree

11 = Professional Degree

12 = Doctorate Degree

13 = Other

Observation – Class 4 (High school graduate), Class 8 (Associate’s degree) and Class 9 (Bachelor’s degree) shows good performance.

Community-Level Disparities in Older Adults for Elective Surgeries: A Cognitive Perspective

MSRP Project 2020

Precious

Ichite

MS2

Erin

Locey

MS2

M. Juliana Peña MS2

Getis

-Ord Gi*

Hot-Spot AnalysisCold SpotsHot Spots

Aim 1: LINUS Health Integration

≧ 65 presenting for elective surgeryBaseline (preop) neuropsychological measures, dCDT, and Mobile Cognitive Platform Metrics from LHFollow-up Timepoints: 6-week3-month

Aim 1 Process: Longitudinal Next-Generation Cognitive Assessment Pathway

Mobile Platform Task

Description

Simple Reaction Time

1

Basic reaction time metric based on foundational work by Stuss and colleagues[13]. Reaction time changes are commonly observed perioperatively via dCDT. This test will be compared to baseline dCDT metrics for validation and assessed for pre to postoperative change.

Procedural Reaction Time

1

A number (1, 2, 3, or 4) appears on the screen, and the test taker must indicate which number was displayed by tapping either the “1 or 2” button or the “3 or 4” button.

Go/No-Go

1

A building with six windows is displayed, and either a “friend” (green) or “foe” (gray) alien will appear in a window. The test taker must tap the “BLAST” button only when a foe stimuli appears.

Story Recall

Novel story paragraphs with 11 alternative versions created by John Newcomer; validated via Price lab and others[14-16] ; in process of incorporating with LINUS Health.

Category and Letter F

Fluency to animal category and letter F category in one minute duration. In process of incorporating with Linus Health

Picture Description

Participants describe a picture of a complex scene in their own words.

Aim 3: MLOps

Implementation of Cognitive Risk Assessments

IDR

Precede

CatBoost

Gradient-boosting decision tree library

GPU Accelerated

Missing + Categorical Variables

Fairness Evaluations

FairML

Themis

SHAP

MLOps

Pipeline

Kubernetes

Docker

Moving AI Forward @

UFHealth? An Implementation Perspective with DLOps

Gather

data

Preprocess

DL Model Development

Featurize

Train

Evaluate

UF Data

Repositories &

Registries

Model Registries

Release Model

Package

Validate

Profile

Approve

Deploy

DLOps

?

Implementation & Application Science

Evidence?

Outcomes?

Impact?

Interpret?

Drift?

Adapted from: https://

gigaom.com

/report/delivering-on-the-vision-of-

mlops

/

Analyze?

Simulate

New Infrastructure

Current State

Thank you!!!

ptighe@anest.ufl.edu@ptighe

Clock Drawing Test

The test demands verbal understanding, memory, spatially coded knowledge and fine motor skills [1]. In the “command” setup a subject is asked to draw a clock from his/her memory. In the “copy” setup the subject is asked to copy an already drawn clock. Scoring a test is not straightforward. There are many protocols for scoring: ranging from a nominal Right/Wrong to detailed 22-31 points scoring systems [2].

Copy Command

[1] Freedman MI, Leach L, Kaplan E,

Winocur

G, Shulman KJ, Delis DC eds. Clock Drawing. Oxford: Oxford University Press, 1994.

[2]

Agrell

B,

Dehlin

O. The clock-drawing test. Age and Ageing 1998; 27: 399-403

Aim 2: Establishing a “Delirium Supplement” for extra-ICU Surgical Patients

Aim 2: Staged Approach to UFHealth IDR/

OneFlorida DataTrust Perioperative Delirium Supplement

Wave 1

Wave 2

Wave 3

Wave 4

Wave 5

CAM ICU/PACU

Anticholinergics

Pain intensity

CPT Code

Intraoperative Vitals

Floor-based delirium assessment

Benzodiazepines

Pain assessment context (static/dynamic)

Primary ICD10 Code

Intraoperative Medications

Geriatric Consult

Antipsychotics

Preoperative Pain Intensity

Day of Week of Surgery

Intraoperative Fluids

Discharge Disposition

Opioids

Preoperative Opioid Use

Time of Day of Surgery

Perioperative Transfusions

Preoperative cognitive disorder

Muscle Relaxants

History of Chronic Pain

Postoperative Destination (PACU, IMC, ICU)

EBL

History of neurodegenerative disease

Gabapentinoids

History of Mood Disorder

Ambulatory, POA, Inpatient, H2HTx, ED

Type of Anesthesia

History of Stroke

Elective/Emergent

Duration

Aim 3: MLOps

Implementation of Cognitive Risk Assessments

VAS implies that…

0 is Optimal?

Less isn’t More!

Consequences of the Pareto Barrier

POD1 Mean Pain Intensity

POD1 Opioid OME (mg)

Clinical Outcome Assessments

An assessment that describes or reflects how an individual feels, functions or survives.

Adapted from: E. Papadopoulos, MD, MPH

Acting Director Division of Clinical Outcome Assessment

Office of New Drugs, CDER, FDA