R eal-world evidence Value, methodological strengths and weaknesses - PowerPoint Presentation

Slide1

Real-world evidence

Value, methodological strengths and weaknesses

What is RWE?

RWE, real-world evidence

.

1. Food and Drug Administration. RWE. https://www.fda.gov/science-research/science-and-research-special-topics/real-world-evidence (Accessed July 2019).

RWE is a term for clinical evidence relating to the usage and potential benefits or risks of medical products derived from analyses of real-world data. It can be generated via a range of different study designs, including:1Pragmatic trialsObservational studies (prospective or retrospective)

Real-world

data

(e.g. electronic medical records, insurance claims data)

Analyzed in real-world

studies

(e.g. retrospective claims review)

Real-world evidence

Context

Growing recognition of RWE within medicine

*Search algorithm: “real-world” OR “real world” OR “real-life” Or “real life”.

RWE, real-world evidence.

1. US National Library of Medicine National Institutes of Health. PubMed. https://www.ncbi.nlm.nih.gov/pubmed/ (Accessed April 2019).

ContextThere has been increasing recognition of the potential value of RWE in recent years as reflected by its increasing presence within the medical literature1Year

RWE complements RCT evidence

RWE can:

Confirm whether RCT results are observed in everyday clinical practice

1 Provide complementary insights from more varied settings1Help to translate RCT findings into evidence that can guide routine clinical practice1*e.g. different age, race, comorbidities, co-medications, adherence.

RCT, randomized controlled trial; RWE, real-world evidence.1. Nallamothu BK, et al. Circulation 2008;118(12):1294–303.

RCTs: required

for

regulatory

approval

Tightly controlled patient population

Real-world

studies

enable evaluation of:

Different

settings

Different

outcomes

Different treatments

Different populations*

Context

Real-world data source example: claims data

Healthcare service

(hospital, clinic, pharmacy)

Physical examination

Laboratory procedures

Pharmacy prescriptions

Insurance claim

Data abstraction CodingAdjudication Health databasePrivacy protectionIntegrationStandardization Codes: disease, procedures, drugs

Healthcare/pharmacy encounters RWE dataData

Real-world data source example: EHRs

Physician office records

Medical history

Physical examinationDiagnosisLaboratory

resultsPrescription informationHospital admissionsSpecialist referrals

Patient enrolled in database

Hospital visits (typically outpatient)

EHRs: electronic health records.

RWE dataData

Examples of real-world data sources and the oncology-related questions they can address

Research question

Insurance

claimsPragmatic trialCancer registriesChart reviewElectronic health records*Incidence, prevalence

Patient demographics

Cancer diagnosis

Clinical characteristicsCancer staging

Survival, mortalityBiomarkers, mutations

Laboratory resultsProcedures: surgeries, resections, radiation

Disease progressionTreatment duration

Treatment sequencingNon-oncological comorbidities

Safety events

*

Cancer-specific (i.e. not linked to wider health

records).

This slide

was developed

based

upon

Boehringer Ingelheim's experience of real-world data collection and

analysis.

Generally:

Evaluable

;

Unevaluable

;

Feasible to

infer,

or subject to data

collection

.

RWE data

Data

AE, adverse events; RCT, randomized

controlled

trial.

In addition to the two cited references, this slide was developed based upon Boehringer Ingelheim's experience of real-world data limitations.1. Southworth

MR, et al. N Engl J Med 2013;368(14):1272; 2. Larsen TB, et al. J Am Coll Cardiol 2013;61(22):2264–73. Real-world data limitationsReal-world data sourceExamples

of potential limitations (not exhaustive)

GeneralMay include off-label use Limited comorbidities/dosing information Other unmeasured confounding factors and/or methodological limitations (selection and assessment bias)

Safety/AE reporting databasesEvent reporting is not always mandatory (i.e. true number of events not accurately reflected)New drugs more likely to have AEs reported (‘Weber effect’)1,2Influenced by whether an AE is described in a drug label, publicity about the event/safety concern, class action lawsuits

Claims databasesHealth claims codes not well defined, may be inaccurate and without source data verificationOutcome definitions not always clearMissing information on comparator treatment can affect outcome interpretation

Observational data from cohort studiesInconsistent definitions of diagnoses and coding practice (e.g. grading of disease severity) Seldom

multi-national RegistriesOften lack comparator arm

Case reportsSmall sample size, limited generalizability No comparator Endpoints seldom clearly defined

Publication

bias results in greater editorial weighting towards

difficult cases

and/or

extreme outcomes (positive

or negative)

RWE data

Data

RCT, randomized

controlled

trial.

1. Khozin S, et al. J Natl Cancer Inst 2017;109(11); 2. Roche N, et al. Ann Am Thorac

Soc 2014;11(Suppl. 2):S99–S104; 3. Nallamothu BK, et al. Circulation 2008;118(12):1294–303. RCTs vs real-world studies: characteristicsRCT1–3

Real-world studies

1–3 PurposeEvaluate the efficacy and safety of therapies

Post-marketing surveillance, cost-effectiveness evaluation, comparative effectiveness, service implementation, center audit, hypothesis generationPopulation characteristicsPredefined, HIGHLY SELECTED population

(stringent inclusion/exclusion criteria to assess causality and to minimize potential confounding)More DIVERSE, ROUTINE CARE patients, including those with characteristics that would prohibit their RCT eligibility (e.g. comorbidities, advanced disease stage)Intervention or treatmentRandomized treatment allocation; c

learly defined, controlled and monitored regimensNon-randomized regimen; physician judgement as informed by label and other considerationsComparator (typically)Placebo or current gold standard

Routinely prescribed treatment alternativesOutcomeEfficacy, safety

Effectiveness, side-effect profile, patient and clinical behaviors, natural history of diseaseTime/observation period Usually short termVariable with potential for long-term evaluation

Treatmen

t adherence

Usually high

(monitored

and reinforced)

Variable, reflecting routine usage

Methods

*The approval of an extended indication of palbociclib (

for the treatment of men with HR+, HER2- metastatic breast cancer) was based predominantly on RWE.

4

**Through patient selection and optimized standards of care.HCP, healthcare professional; RCT, randomized controlled

trial; RWE, real-world evidence.1. Khozin S, et al. J Natl Cancer Inst 2017;109(11); 2. Roche N, et al. Ann Am Thorac Soc 2014;11(Suppl. 2):S99–S104; 3. Nallamothu BK, et al. Circulation 2008;118(12):1294–303; 4. Wong GWK, et al. Ann Am Thorac Soc 2014;11(Suppl. 2):S85–91; 4. Pfizer. Press Release. https://www.pfizer.com/news/press-release/press-release-detail/u_s_fda_approves_ibrance_palbociclib_for_the_treatment_of_men_with_hr_her2_metastatic_breast_cancer (Accessed July 2019). RCTs vs real-world studies: characteristics1–4

RCT

Real-world studiesAccepted by regulatory authorities

YesNot routinely*Internal validityHigh

LowGeneralizability/external validityLowHighPotential for confounding

LowHighMaximizes potential treatment effect**Yes

NoPotential to:Detect rare events

LowHigherEvaluate long-term outcomesLow

High

Evaluate

aspects of HCP/patient behaviors

Low

High

Evaluate aspects

of healthcare delivery

Low

High

Methods

RWE/RCT terminology differences

This slide was developed based upon Boehringer Ingelheim's

experience of terminology used in real-world studies and randomized clinical trials.

PFS, progression-free survival; RCT, randomized

controlled trial; RWE, real-world evidence; TTP, time to progression.Commonly used terminology often differs for real-world studies and RCTs:RCT Real-world studyTrialStudyAllocated treatment

Prescribed treatment

EndpointsOutcomesTreatment arm Treatment group/cohortTrial resultsRWEEfficacy Effectiveness

SafetySide-effect profilePFSTTPMethods

1. Roche N, et al. Lancet Respir

Med

2013;1(10):

e29–30.Describing study designs in real-world terms

MethodsThe framework links various types of studies based on:population characteristics and ecology of care The typical position of common study designs are illustrated, but can be moved in any direction depending on the specifics of its study design

Conceptual framework of therapeutic

research1STUDY DESIGN ECOLOGY OF CAREFreeNarrowBroad

POPULATIONConstrained

RCT, randomized controlled trial.

1. Thorpe KE, et al. J Clin Epidemiol 2009;62(5):464–75.

PRECIS wheel: describing study designs using ten explanatory pragmatic axes

1

Methods

Explanatory studies:

Tests a causal research hypothesis: ‘Does a treatment work under ideal conditions?’

Pragmatic studies:

Helps to choose between care options: ‘Does a treatment work under usual conditions?’

Interpreting RCT findings and RWE

Results

Real-world studies often make use of

proxy endpoints

if specific variables are not routinely availableRCT endpoints and real-world outcomes can differ; methodological review is required before comparing their findings: RCT endpointTypical

RCT definition

Disaggregated componentsChallenges of real-world evaluationPFSTime from randomization* until objective tumor progression

by RECIST or deathDeathPDMonitoring and follow-up varies across healthcare settings, centers and clinicians, which limits

(or prevents) evaluation of RECIST TTPTime from randomization* until clinical progression, or deathDeath

Clinical progressionPDRoutine care methods of assessing and monitoring clinical progression are difficult to standardize for real-world evaluationTTFTime from randomization* to discontinuation for any reason

DeathClinical progressionPDAEs**

Pragmatic routine care management approaches may result in treatment through progression resulting in “time on treatment” being used as a proxy for TTF*Time from study enrollment for single-arm trials; **TTF may discriminate between discontinuations due to AEs and discontinuations for other

reasons.

This slide was developed based upon Boehringer Ingelheim's experience analyzing data from randomized controlled trials and real-world studies.AE, adverse event; PD, progressive disease;

PFS, progression-free survival; RCT, randomized controlled trial; RECIST, Response Evaluation Criteria in Solid Tumors; RWE, real-world evidence; TTF, time to treatment failure; TTP, time to progression.

Endpoints – PFS, TTP, TTF: RCT scenario 1

Conceptual scenario

:

The effect of treatment beyond progression results in TTP being longer than mPFS, and outweighs the effect of treatment discontinuation for other reasons

PDPDPDDeathPD

Results

PDPD

PDDeathPDClinical PD

Clinical PDClinical PDPDPDPDDeath

PDClinical PDClinical PDClinical PD

AEAEAEAE

AE, adverse events; mPFS, median progression-free survival; RCT, randomized controlled trial; TTF, time to treatment failure; TTP, time to progression.

Conceptual scenario

:

The effect of treatment discontinuation for other reasons has more effect than prolongation of

mPFS by

treating beyond RECIST progression

Endpoints – PFS, TTP, TTF: RCT scenario

2AE, adverse events; mPFS, median progression-free survival; RCT, randomized controlled trial; RECIST, Response Evaluation Criteria in Solid Tumors; TTF, time to treatment failure; TTP, time to progression.PD

PDPDDeathPD

ResultsPDPDPDDeath

PDClinical PDClinical PDClinical PD

PDPDPDDeathPD

Clinical PDClinical PDClinical PDAE

AE

AE

AE

AE

AE

Outcomes – TTP, ToT (TTF/TTD): Real-world scenario

Conceptual scenario

:

Routine care patients treated as long as clinician deems beneficial. A greater number of events (e.g. PD, withdrawal, discontinuation due to tolerability) contribute to ToT (TTF) analyses, making it shorter than TTP

AE, adverse events; PD, progressive disease; RECIST, Response Evaluation Criteria in Solid Tumors; ToT, time on treatment; TTF, time to treatment failure; TTP, time to progression.

Progression-free survival

mPFS: 21.5 monthsContributing events:

PDPDPDDeath

PD

Time to progressionTTP: 23.7 monthsContributing events:

Time on treatment / time to treatment failureToT/ TTF: 20.2 monthsContributing events:

PDPDPDDeathPD

Clinical

PD

Clinical

PD

Clinical

PD

PD

PD

PD

Death

PD

Clinical

PD

Clinical

PD

Clinical

PD

AE

AE

AE

AE

AE

AE

Not evaluable: in routine care, patients do not stop according to RECIST and do not have a standardi

z

ed imaging schedule

Results

Interpreting RWE: comparing vs RCT results

Scenario

Interpretation

and considerations

RCT efficacy

shown, andreal-world effectiveness shown

RWE validates RCT findings in routine care patients and/or settingsIncreases confidence in the finding, e.g. real-world studies of adjuvant chemotherapy have shown OS results comparable with those reported in several pivotal RCTs1

RCT efficacy shown, but real-world effectiveness not shown

Result raises questions about the extent to which a novel therapy can offer benefit in the real world:Does treatment only work in selected populations?

Should (particularly toxic) treatments be prescribed in older and/or sicker patients than those who were eligible for the RCT?RCT efficacy marginal, butreal-world effectiveness shown

May suggest RCT was underpowered

and real-world studies

can provide further confirmatory evidence

RCT efficacy

not shown

, and

real-world effectiveness

shown

X

The demonstration of real-world effectiveness in the absence of any RCT efficacy signal requires

close statistical scrutiny to assess potential sources of bias

*Through

patient selection and

optimized

standards of

care.

OS,

overall

survival;

RCT

,

randomized

controlled

trial; RWE, real-world evidence.1. Karim S, Booth CM. J Clin Oncol 2019:37(13):1047–50.

Results

Side-by-side comparison: real-world effectiveness vs RCT efficacy

Objective:

Measure the relationship between RCT efficacy and real-world effectiveness for oncology treatments, in:

Selected real-world patients who meet routine RCT eligibility criteria (

baseline cohort) All real-world patients with evaluable records (full cohort)Methods:RCT data: abstracted from 21 Phase III oncology RCTs reporting OS, PFS or TTP Real-world data: EHRs – SEERs Medicare dataPrimary outcome: real-world OS, estimated as MHRCox proportional hazard regression model used to calibrate difference between real-world MHR and:RCT MHRRCT PPS/TTP SHR

Predicting

Real-World Effectiveness of Cancer Therapies Using Overall Survival and Progression-Free Survival from Clinical Trials: Empirical Evidence for the ASCO Value Framework1

Author conclusions: Baseline cohort: in real-world patients eligible for RCTs, real-world OS benefits were:Similar to those observed in RCTs based on OS endpoints

16% less than RCTs based on surrogate endpoints (PFS, TTP)Full cohort: in non-selected real-world populations, treatment benefit was predicted to be even less than in the baseline, RCT-eligible real-world cohortResults

*Not reported in Ladawalla et al; included for illustrative purposes to exemplify the direction of the observed differences. EHR, electronic health record; HR, hazard ratio; MHR, mortality hazard ratio; OS, overall survival; PFS, progression-free survival; RCT, randomized controlled trial; SEERS, Surveillance and Epidemiology End Results;

SHR, surrogate hazard ratio; TTP, time to progression.1. Lakdawalla DN, et al. Value in Health 2017;20(7):866–75.

Results:

Pragmatic real-world endpoints in NSCLC studies: considerations

Methods:

18 metastatic NSCLC RCTs:

Initiated after 2007

Submitted to the FDA Involved 8,947 patientsCompared TTD to PFS and OSTTD: defined as date of randomization to date of discontinuation or death Results: Overall: TTD was more closely associated with PFS (r=0.87, 95% CI 0.86–0.87) than with OS (r=0.68, 95% CI 0.67–0.69)Oncogene-targeted subgroups: mTTDs exceeded mPFS:EGFR+: 13.4 months vs 11.4 months, respectively ALK+

: 14.1 months vs 11.3 months, respectively

Analysis of time-to-treatment discontinuation of targeted therapy, immunotherapy, and chemotherapy in clinical trials of patients with NSCLC1

Author conclusions: mTTD exceeds mPFS for oncogene-targeted subgroups due to treatment beyond RECIST progression. The ability to treat beyond progression may reflect real-world treatment tolerability and the need to continually

suppress driver mutations despite emergence of alternative resistance mutationsWith ICI therapy, mPFS (4.2 months) was slightly longer than mTTD (3.5 months), with cases of both early and late TTD. This may reflect some: cases of early termination due to immune-mediated AEs; some instances of durable treatment benefit, and others of treatment beyond conventional progressionFurther research is needed to validate TTD as a measure for pragmatic RCTsSelected KM analysis of TTD and PFS: (A) EGFR TKI vs doublet ChT with maintenance EGFR mutant population; (B) EGFR TKI vs doublet ChT (no maintenance) in EGFR mutant population; (C) EGFR TKI vs EGFR TKI in EGFR mutant population; (D) ICI vs monotherapy ChT

Results (cont’d):

PFS ChT double + maint.PFS EGFR TKI

TTD ChT double + maint.TTD EGFR TKI

PFS ChT (no maint.)

PFS EGFR TKI

TTD ChT

(no

maint.)

TTD EGFR TKI

PFS EGFR TKI Arm 1

PFS EGFR TKI Arm

2

TTD EGFT TKI Arm

1

TTD EGFT TKI Arm 2

PFS ChT monotherapy

PFS ICI

TTD Cht monotherapy

TTD ICI

Results

AE

, adverse

event;

ALK

,

anaplastic lymphoma

kinase;

ChT,

chemotherapy;

EGFR

, epidermal growth factor receptor; FDA, Food and Drug Administration; ICI, immune checkpoint inhibitor; maint., maintenance; mPFS, median PFS; mTTD, median TTD; NSCLC, non-small cell lung cancer; OS, overall survival; PFS, progression-free survival; RCT, randomized controlled trial; RECIST, Response Evaluation Criteria in Solid Tumors; TKI, tyrosine kinase inhibitor;

TTD, time to discontinuation.1. Blumenthal GM, et al. Ann Oncol 2019;30(5):830–8.

RCT,

randomized

controlled

trial; RWE, real-world evidence.1. Wong GWK, et al. Ann Am Thorac Soc 2014;11(Suppl. 2):S85–91; 2. Nallamothu BK, et al. Circulation 2008;118(12):1294–303; 3. Roche N, et al. Ann Am Thorac Soc 2014;11(Suppl. 2):S99–S104

; 4. Karim S, Booth CM. J Clin Oncol 2019:37(13):1047–50; 5. Lakdawalla DN, et al. Value Health 2017;20(7):866–75. RWE summaryBackground: Real-world studies complement RCTs – they provide evidence from everyday patient populations managed in routine care settings, including those with characteristics that would preclude their participation in RCTs1–3 Real-world data sources: There is a wide range of real-world data sources available; each has specific strengths and limitationsReal-world data source examples include: electronic medical record databases (primary care ± secondary care ± pharmacy ± other linked data); insurance claims databases; patient access schemes; registries

Real-world studies vs RCTs:

1,3Conducted for different purposes, e.g. drug registration (RCTs) vs long-term surveillance (real-world studies) Differ according to how much their design reflects (i) real-world patients and (ii) real-world clinical management Different strengths and weaknesses, e.g. real-world studies offer high external validity, but low internal validityReal-world vs RCT findings:Often evaluate different outcomes; real-world studies make use of proxy measures where specific variables are unavailableRCTs may over-estimate treatment benefits through their selection of ‘ideal’ populations4

Real-world studies and RCTs provide complementary evidence; a judicious approach combining evidence from both can provide a more complete view of the evidence base1–5