Pragmatic Clinical Trials: Challenges with Clinical Event Ascertainment using - PowerPoint Presentation

1. Pragmatic Clinical Trials: Challenges with Clinical Event Ascertainment using RWDFrank W. Rockhold, PhDProfessor of Biostatistics and BioinformaticsDuke University Medical CenterDuke/Stanford CEC SummitSeptember 26-27, 2018, Chicago, Il

2. Disclosure Statement – Frank W Rockhold, PhDResearch Funding: NIH, PCORI, Duke Clinical Research Institute, Astra Zeneca, ReNeuron, Luitpold, Alzheimer's Drug Discovery Foundation, Janssen, BMSConsulting and Honoraria: https://dcri.org/about-us/conflict-of-interest/Boards: Frontier Science Board of Directors, DataVant Scientific Advisory Board, European Medicines Agency Technical Advisory GroupEquity Interest: GlaxoSmithKline, DataVant

3. Topics to DiscussPragmatic Clinical Trials (PCTs) answer questions that are about real world effectiveness. It can be a supplement to and not necessarily a replacement for “classic” (explanatory/confirmatory) RCT’s. Depends on the question and inference desiredFor this discussion RWD is EMR/Claims data, i.e. data that are a result of a patient health encounter“Investigator” vs GP: Has an impact on recruitment and event ascertainment.Is it really less expensive? In the near term the focus should be on ”how” and not “how much”.Data management vs healthcare informatics- resource and cost shifting. Lessons learned should be of value to all clinical trial conduct.The approach can be useful for safety studies but there needs to be agreement on potential tradeoffs in event ascertainment.

4. Some questions to considerWhat loss in ascertainment sensitivity are we willing to live with in a study with RWD study? For safety studies of uncommon or serious events are PCT’s using RWD sources an adequate alternative to RCT’s? OBS study is not likely to fully discharge a safety risk.Do not abandon the concept of randomization too quicklyGiven practicality considerations what should be our threshold for saying an RWD study is the “only way” even in situations where it is possible to conduct an RCT?? And what is the tradeoff in scientific credibility vs practicality?

5. PRECIS-2 (Loudon, K., Treweek, S., Sullivan, F., Donnan, P., Thorpe, K.E., and Zwarenstein, M., The PRECIS-2 tool: designing trials that are fit for purpose. BMJ, 2015. 350: p. h2147 )

6. Intentionally homogeneous to maximise treatment effectRandomisation and blindingClinical measures, intermediate endpoints, composite endpoints, clinical outcomesProtocol defines the level and timing of testing. Physicians blinded to dataFixed standard of care or placeboConducted only by investigators with proven track recordVisit schedule and treatment pathway defined in the protocol Patients wishing to change treatment must withdraw from the studyCompliance is monitored closely Close monitoring of adherence Intent to treat, per-protocol and completers“Classic” RCTDesigning a randomised pragmatic clinical trialWays that Randomised Controlled Trials (RCTs) and PCTs can differHeterogeneous - representative of normal treatment populationRandomisation and rarely blindingClinical outcomes, PROs, QoL, resource useMeasured according to standard practiceStandard clinical practiceEmployment of a variety of practitioners with differing expertise and experience Visits at the discretion of physician and patient.Standard clinical practice – switching therapy according to patient needsPassive monitoring of patient compliancePassive monitoring of practitioner adherenceAll patients included P”R”CT

7. ADAPTABLE*, the Aspirin Study – A Patient-Centered Trial*theaspirinstudy.org

8. PCORnet seeks to improve the nation’s capacity to conduct clinical research by creating a large, highly representative, national patient-centered network that supports more efficient clinical trials and observational studies.

9. Study DesignPatients with known SCVD(ie MI, OR cath ≥75% stenosis of ≥1 epicardial vessel or PCI/CA BG)AND ≥ 1 Enrichment FactorPts. contacted with trial information and link to eConsent; Treatment assignment provided directly to patientIdentified through EMR (computable phenotype) by CDRNs (PPRN pts. already part of a CDRN are eligible)ASA 81 mg QDASA 325 mg QDElectronic F/U Q 3-6 months;Supplemented with EMR/CDM/claims dataDuration: Enrollment over 24 months;Maximum f/u of 30 monthsPrimary Endpoint: Composite of all-cause mortality, nonfatal MI, nonfatal strokePrimary Safety Endpoint: Major bleeding complicationsExclusion CriteriaAge < 18 yrsASA allergy or contraindication (including pregnancy or nursing)Significant GI bleed within past 12 monthsSignificant bleeding disorderRequires warfarin or NOAC or Ticagrelor*Enrichment factorsAge > 65 yearsCreatinine > 1.5Diabetes (Type 1 or 2)3-vessel coronary artery diseaseCerebrovascular disease and/or peripheral artery diseaseEF <50% by echo, cath, nuclear studyCurrent smoker

10. ADAPTABLE – How Pragmatic is it?

11. Centralized Enrollment & Follow-Up391230DCRI Call CenterPatients who miss 2 contactsPatient Reported OutcomesMedication useHealth outcomesBaseline DataADAPTABLEPatient615CMS and private health plans FOLLOW-UP Longitudinal health outcomesPatient PortalInfo/eConsent/RandomizationPatient Reported OutcomesMedication useHealth outcomesRandomized to Q3 or Q6PCORNet Coordinating Center FOLLOW-UP Via Common Data Model Longitudinal health outcomes

12. Information FlowMytrus Patient PortalEMRMedicare ClaimsNational Death IndexPrivate Health Plan DataPatientPCORnetSupplemental LinkagesADAPTABLE Study DatabaseEach data source arrives at the coordinating center via a different mechanismAll will contribute to eventual study databaseAlgorithm based decisions for discrepant data/event ascertainment

13. Information AsymmetryPtPEMRCMSNDIHPPatient #1PtPCMSNDIHPPatient #2PtPEMRCMSNDIHPPatient #3PtPCMSNDIHPPatient #4Different participants with different sources of data contributing to endpoint ascertainmentVary by site, Medicare & health plan coverage are not uniformVary by site or patient if fields are inaccurate or missing EMREMR

14. Information LatencyWg Implication: complete data for the study will become available 1 year after the last patient last follow upData Source Availability Min – Max Delay Participant Self-reported data Instant NoneElectronic Records EMR from PCORnet DataMart Quarterly3 – 6 months Medicare Claims DataAnnual 1 – 12 months National Death IndexAnnual 1 – 12 months Private Health Plan DataEnd of study?

15. Handling Disagreement Across Different Data SourcesPatient reported hospitalizations that are not observed in EMR data will be queried via:Medicare fee-for-service claimsLarge national health plans(FDA’s Mini-Sentinel initiative)DCRI Call Center

16. Missing Follow-upDCRI Call CenterPatient FinderSocial Security Death IndexContact with the site

17. Endpoint Ascertainment and Validation Are we capturing information completely and accurately?A multi-faceted approach to capture outcomesEndpoint validation and Endpoint reconciliation of MACE and Major Bleeding eventsCan endpoints identified in EMR comparable to clinical endpoints confirmed by traditional adjudication process?Events of interest: MI, stroke, major bleedingAssess agreement: True Positive, False Positive, PPVProvide insights on accuracy of coding algorithms and the data curation processCould a machine algorithm help speed confirmation of events?Absence of events from EMR is not usually verified due to low event ratesCould machine algorithms be used to look for potential false negatives?

18. Approach to Endpoint AscertainmentRoutine queries of the PCORnet CDM to capture and classify endpoints using validated coding algorithmsHospitalizations will be identified via standardized, validated coding algorithms developed centrally and applied to the CDMADAPTABLE patient portal will ask about possible endpoint events (hospitalizations for myocardial infarction, stroke, or major bleeding) during participant contacts (every 3-6 months) Such information will provide additional confirmation for the CDM-generated hospitalization data that meet criteria for the endpoints specified Death ascertainment via Social Security Administration (Medicare beneficiaries)18

19. Validation Plan – Background (courtesy of Dr. Schuyler Jones) Formal clinical endpoint classification (CEC) cannot be performed within the budget and will need to be modified in order to ensure efficiency and lower costs Concerns exist that event ascertainment via coding algorithms of administrative claims data needs to demonstrate acceptable levels of validity and be accepted by physicians and patients alike Conflicting data exist regarding the agreement between administrative claims data and clinical study adjudication, thus necessitating the need for this supplemental, validation plan19

20. Validation Plan – Expected Number of Endpoints to ReviewRandom sample of approximately 20% of the MI, stroke, and major bleeding endpoints sampled proportionally from enrolling health systems within participating CDRNsselect the first 10 MI events, 10 stroke events, and 5 bleeding events that occur within each CDRN (total=25 events per CDRN). We will randomly select approximately 175 non-fatal MI and stroke endpoints and 50 major bleeding endpoints Approximately 225 overall endpoints to review during the course of the study.20

21. Validation Plan – Adjudication of EventsAdjudicationEach case randomly selected will be reviewed in a blinded manner by a disease-specific, expert adjudicator a single cardiologist will review MI and major bleeding endpointsa single neurologist will review stroke endpoints Standard endpoint definitions for MI, stroke, and major bleeding will be used to adjudicate these endpoints using standardized adjudication forms21

22. Validation Plan – Analysis Formal statistical analyses will be performed to measure: percentages of true positive results and false positive results agreement/concordance rates for each endpoint that will be validated (non-fatal stroke, non-fatal MI, and major bleeding)These results (if different) will not change the primary analysis of ADAPTABLE which will count those events identified via coding algorithms as described in the protocol22

23. Data and Safety Monitoring Source documentation adjudicationConcordanceanalysesRole of a DMC in a pragmatic trialStandard role Additional focus: feasibility, protocol adherence, data validityInvolvement of patient representatives If and how could the DMC make critical decisions with fragmented information during the study? Interim analyses: differentiate a signal from noise with limited access to EMR outcome dataDifferential data lag timesBenefit to risk analysisCould a machine algorithm help even with partial data?Is the DMC protected?Indemnification

24. Summary: ADAPTABLEADAPTABLE is the largest pragmatic trial in the world to evaluate aspirin dose with expected 15,000 participantsAttempts to mimic the real world patient experience of a patient with heart diseaseCollects data through different sources and employs a multi-faceted approach to capture outcomes will teach us a great deal about the value and issues with this approach in the future.Maintained scientific rigorrandomisedactive controlrobust primary endpointADAPTABLE will tell us a great deal about the utility of the approach to perform “mega trials” in a very different way.

25. Figure 1.DCRI CEC ProcessMachine Learning for Clinical Events Classification and Safety Surveillance: An Innovation Project from the DCRI ICE-SS groupThe mission of the Duke Clinical Events Classification GroupTo provide high quality adjudicated endpoint data with scientific rigor, efficiency and innovation by coordinating and conducting Systematic, Comprehensive, Unbiased, Blinded, and Independent clinical events adjudication.Figure 3. Classification Results on Test Set (using trigger summary)Figure 2. Euclid Trial Summary Statistics11,082 eventsAverage trigger summary: 46 words (min: 1, max: 1141)Average dossier: 1317 words (min: 1, max: 24984)80/20/20% partition for training, validation and test, respectively.MIBLEEDINGALISTROKE/TIADEATH2088396630196921317YesNoYesNoYesNoStroke Yes TIA YesNoNA853123530449622932726317107214NALopes RD, et al. Not published.

26. Endpoints in Studies (Trial) with RWD PCTs answer questions that are more real world effectiveness. Should be viewed as a supplement to and not a replacement for “classic” RCT’s.“Investigator” vs GP: Has an impact on recruitment and event ascertainment.Is it really less expensive? Time will tell and in short term the focus should be on ”how” and not “how much”. Cost per information unit is unknown.Data management vs healthcare informatics- resource and cost shifting. Lessons learned should be of value to all clinical trial conduct.Machine algorithms could likely help with EMR endpoint PPV.The approach can be useful for safety studies but there needs to be agreement on tradeoffs in event ascertainment and effort involved in data acquisitionEven with tradeoffs of potential data gaps these algorithms could be useful in increasing the probability of detecting an event

27. RecommendationsUnderstand the data- don’t try to make RWD look like an RCT CRFWhat you think is “missing” may not be- it just was not needed for patient careDo studies using RWD because of the clinical utility and scientific value and not driven solely by speed and costDoing a pragmatic trial is not a euphemism for “sloppy” or “easy to conduct”.These trials have a particular place in the spectrum of clinical research methods that continues to evolveLeverage new technologies to help speed and standardize endpoint “precision” in RWD trials and we are going to be limited by the data.Transparency and validity in RWD studies are real issues that need to evolve.This applies both to the data and the algorithms used.

28. Questions and discussion

29. Validation Plan – Source DocumentsSource Document CollectionDCRI Coordinating Center will collect source documents via electronic means, fax, and/or mail for all specified endpointshospital discharge summaries, brain imaging reports (e.g. CT scans), laboratory values (e.g. hemoglobin levels for bleeding endpoints, cardiac marker values for MI endpoints), and electrocardiograms (for MI endpoints)These source documents will be collated and private health information and treatment assignment (i.e. aspirin dosing) information will be redacted 29

30. Validation Plan – Endpoints Plan to review hospitalizations for (1) myocardial infarction (MI), (2) stroke and (3) major bleedingDeath will not be reviewed since studies have consistently shown that the standard methods to ascertain vital status (i.e. all-cause death) that will be used in ADAPTABLE are valid and accurateThe occurrence of percutaneous coronary intervention (PCI) and coronary artery bypass grafting (CABG) surgery will also not be reviewed since prior studies have shown a high agreement rate for the confirmation of coronary revascularization procedures (κ = 0.88–0.91)30

31. 10/2016ClinicalTrials.gov: NCT02697916