Risk-based Quality Management in Clinical Data Management - PowerPoint Presentation

Slide1

Slide2

Risk-based Quality Management in Clinical Data Management

Informal CDM Leadership Network

in collaboration with

Medidata

Represented by

Peter Stokman (Bayer) & Lisa Ensign (Medidata)

Slide3

RBQM in CDM

Informal CDM Leadership Network

Currently 20 companies; 44 participants

Participating companies:

Slide4

RBQM in CDM

Background

Methodology

Preliminary Results

Preliminary Conclusions

Slide5

Currently 20 companies; 44 participants

Subgroups – trying to create common understanding, consensus, basic approaches

RBQM in Clinical Data Management

CRO Oversight

Audit Trail Review

GDPR

CDM Network Europe

Slide6

Guidances to encourage better results by using smarter approaches

ICH Q9 Quality Risk Management (January 2006)

‘…

provides principles (…) for QRM that can be applied.’

ISO 31000: Risk Management Principles Overview 2009

FDA Guidance: Oversight of Clinical Investigations — A Risk-Based Approach to Monitoring (Aug 2013)

Transcelerate: Position Paper: Risk-Based Monitoring Methodology (May 2013; + Updates)

EMA Reflection Paper on risk based quality management in clinical trials (November 2013)

ICH E6 (R2) Addendum (June 2017)

ICH E8 (R1) Designing Quality in Clinical Trials (May 2019)

RBQM in Clinical Development

Slide7

RBQM in Clinical Development

EMA Reflection Paper on risk based quality management in clinical trials:

‘lack of proportionality (one size fits all)

in the implementation of quality control activities (e.g. monitoring etc.) often related to a lack of understanding of the impact of variability in trial conduct and measurement or data collection on the study results and their reliability;’

Slide8

Heart of ICH E6 R2: focus on efficient

and

effective

, identifying quality standards (Quality Tolerance Limits)

Movement towards understanding what each aspect of the quality cycle (Monitoring

(central, onsite, offsite, statistical), Data Management) will – and will not – add to qualityAs a consequence: quantify the effectiveness of each part & evaluate whether it should be used - even on critical data.

RBQM in Clinical Development

Slide9

So what can we do?

Work harder!

RBQM in Clinical Development

Evaluating Source Data Verification as a Quality Control Measure in Clinical Trials

Nicole Sheetz, Therapeutic Innovation & Regulatory Science, 2014, Vol. 48(6) 671-680

3.7%

1.1%

In Summary, all TAs combined:

Queries

(N)

Q from SDV (N)

Qs SDV/

Qs All (%)

Qs

SDV

Critical

/ Qs All (%)

349,081

30,680

7.8 (0.5-22.1)

2.3 (0.1-6.6)

1.2%

1.4%

Slide10

RBQM in Clinical Development

Focus where it matters most

 Identify where it matters most

For

Monitoring

:

Source Data Verification - Source Data Review

Trial sites with characteristics correlated with poor performance or noncompliance

Try to limit time wasters: SDV

For

Clinical Data Management – particularly Data Cleaning Data Changes

Critical Data

Time wasters: queries in non-critical domains that rarely lead to data changes

28-71 $ ($50)

https://www.linkedin.com/pulse/clinical-data-quality-query-resolution-costs-steven-law/

Slide11

RBQM in Clinical Data Management

Seven companies teamed up with Medidata to answer the following questions:

Which clinical domains are queried most

What percentage of these queries are actually leading to data changes

In which data fields

How is requerying related to data changes (future work)

Slide12

Study Methods

Select recently completed Phase III studies

representing a variety of sponsors and TAs

primary endpoint within last 5 years

Extract query data and standardize using Medidata’s FFC ML modeling approach with supervised review

Analyze data

Primary: by TA, Form Domain, Field, Direct/Indirect Change,

Requeried

Secondary: Query Initiator, Time to Query Resolution

No sponsor-identifiable data will be disclosed. No patient-level data utilized.

Slide13

PULM/RESP

Results – Study Summary

Range: 200 – 4200 subjects

Median: 600

Range: 2 – 54 mos

Median: 29.3

Slide14

Query File Extraction

Slide15

2,055,808 Queries

Query Summary

Automatic

Slide16

Who is Initiating Manual Queries?

e.g., Includes queries by CRA & Site Monitor

30%

70%

N = 611,265

e.g., Includes queries by DM, Data Monitor, Safety MM, Medical Monitor, ME

Slide17

Query Standardization

Use ML to Standardize Data across Multiple Trials

Standardized Forms & Fields

Model Applied to RBQM Queries

Slide18

Query Standardization Form Domains

2,055,808 Queries standardized to

27

standardized form domains

196

standardized field names

Add Example

Datapage Name

Dataset Name

Variable Name

Control Type

Field Pretext

AI

Predicted For

m

Domain

AI

Predicted Field

death complementary information

dd_pv_all

ddseccs

LongText

Secondary Causes of Death

S

urvival Status/Death Details

DEATH_CAUSE

death

dd_onc

ddorres

RadioButton (Vertical)

Cause of death

SSDD

DEATH_CAUSE

death

dt

dtcause

LongText

Primary cause of death

SSDD

DEATH_CAUSE

death complementary information

dd_pv_all

ddprcs

LongText

Primary Cause of Death

SSDD

DEATH_CAUSE

death data

deth

decu

DropDownList

Primary cause of death

SSDD

DEATH_CAUSE

death form

death

dthreas

LongText

Primary cause of death

SSDD

DEATH_CAUSE

death form

death

deathreas

Text

Cause of death

SSDD

DEATH_CAUSE

death report form review

dd

prcdth

DropDownList

Cause of Death (indicate most probable):

SSDDDEATH_CAUSE

Slide19

Results – Manual Query Counts by

Standardized Domain

More

queries in domains that are collected throughout the study:

CM, LB & AE

Slide20

Results – Manual Queries by Data Change

Data Change

“Direct”: change in queried field

“Indirect”: change in other field (based on verbiage used)

70% of manual queries

DID

result in a data change

58% “Direct” change (Answer Data ≠ Query Data)

12% “Indirect” change (Answer Data = Query Data)

Slide21

Field-Specific Result Examples

Mean 70% Changed

Slide22

Results – Example Fields with

Lower

Query Efficiency

AE Serious

DS Death Date, IC Date, Treatment Completion/Discontinuation Date

EX Treatment Name, Dose Frequency, Dose Units

CM Regimen Number, Start Time, Discontinuation Reason, Response/Result

LB Name

MH Body System

SV Visit Date

VS Systolic BP

Consider Query Efficiency

AND

overall / TA-specific

Criticality

Slide23

Results – Example Fields with

Lower

Query Efficiency

AE Serious

DS Death Date, IC Date, Treatment Completion/Discontinuation Date

EX Treatment Name, Dose Frequency, Dose Units

CM Regimen Number, Start Time, Discontinuation Reason, Response/Result

LB Name

MH Body System

SV Visit Date

VS Systolic BP

Consider Query Efficiency

AND

overall / TA-specific

Criticality

Domain Criticality

(to be determined by study team)

High

Medium

Low

Slide24

Comparison to Previous Work

Prior Work

Transcelerate RBM Position Paper

RBQM Analysis

SDV =

8.9%

of total queries

SDV Queries =

7.8%

of All Queries

Sheetz et al Paper

Focus on

all

Manual queries

1% eCRF data corrected by manual queries

No data removed

Included changes where missing data updated to a populated value

Included “indirect” changes

Focus on SDV process

1.1% eCRF

data

*

corrected by SDV

*

Removed

Batch-loaded data

Non SDV data

Excluded changes where missing data updated to a populated value

Did not include “indirect” changes

Slide25

Preliminary Conclusions

Manual queries only lead to a correction in



1% all data

Yet: 70% of manual queries are effective - i.e.: lead to a correction of the data

Domains and fields can be identified where manual queries lead to relatively

low & relatively

high percentages of data changes – impact on quality not clear

By using this data as historical estimates of data cleaning efficacy, and by combining them with data criticality assessment, these findings can be used to help focus on where it matters mostBy looking at the efficacy of queries on an individual trial basis, the efficiency of the query process can be improved incrementally in the Plan/Do/Study/Act quality circle

Slide26

Future Analyses

Evaluation of re-query efficacy

Further dataset mining

Comparison to ML data quality detection algorithms

Use as historical data to help specify TA specific tolerance limits

Publish the results in more detail:

SCDM’s Data Basics, DIA’s TIRS, ACT, Perspectives in Clinical Research,

International Journal of Big Data Management2020 Annual DIA Conference – abstract submitted

2020 Japan DIA Clinical Data Management Workshop - Invited

Medidata NEXT

Slide27

Thank you!