Answers to Frequently Asked Questions Brenda Crowe Research Advisor Eli Lilly and Company With special thanks to Jesse Berlin Midwest Biopharmaceutical statistics workshop May 21 2013 Disclaimer ID: 726528
Download Presentation The PPT/PDF document "Meta-Analysis of Clinical Data for Regul..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Meta-Analysis of Clinical Data for Regulated Biopharmaceutical Products: Answers to Frequently Asked Questions
Brenda Crowe, Research Advisor, Eli Lilly and CompanyWith special thanks to Jesse Berlin
Midwest Biopharmaceutical statistics workshop
May 21, 2013Slide2
DisclaimerThe views expressed herein represent those of the presenter and do not necessarily represent the views or practices of the presenter’s employer or any other partyMBSW May 21, 2013
2Slide3
Acknowledgements
Jesse Berlin
Amy Xia
Juergen
Kuebler
Carol Koro
Ed Whalen
MBSW May 21, 2013
3Slide4
AgendaBackgroundThe 6 questionsWhat studies should be pooled/combined?
Method of ascertainment?Individual patient data (vs. aggregate patient data)?Multiple looks and/or multiple endpoints?Heterogeneity of design and results?
Fixed-effect models or random-effects models?
Concluding remarks
MBSW May 21, 2013
4Slide5
BackgroundDuring drug development, sponsors need to recognize safety signals early and adjust the development program accordinglyCrowe et al. (SPERT): overview of the framework and planning of MA in drug development but did not provide details regarding practical issues arising during implementation.
Focus here on common analytical topics (6 questions)Emphasis on situations that arise in drug development, mostly premarketingSPERT = Safety Planning, evaluation and Reporting Team
MBSW May 21, 2013
5Slide6
A little vocabulary (in today’s context)POOL (noun): a grouping of studies used to address a specific research questionSwimming in data (avoid drowning) MBSW May 21, 2013
6Slide7
Q1: What studies should be pooled in the meta-analysis?MBSW May 21, 20137Slide8
Existing GuidanceFDA guidance on premarketing risk assessment
MBSW May 21, 20138Slide9
Existing GuidanceInternational Conference on Harmonization (ICH) M4E
MBSW May 21, 20139Slide10
Existing GuidanceCouncil for International Organizations of Medical Sciences VI (CIOMS VI) report MBSW May 21, 2013
10Slide11
What to pool?Decisions on what to combine depend on the specific questions to be answered (duh)Often there are several questions and these might require different subsets of studies or subjectsMBSW May 21, 2013
11Slide12
Pools may be based onType of control: placebo vs. activeDose route or regimenConcomitant (background) therapyMethods of eliciting adverse events (e.g., active vs. passive).Disease stateDuration of treatment (and follow-up?)
Subgroups of patients based on age groups, geographies, ethnicity groups, or severity of disease, etc.MBSW May 21, 201312Slide13
Table to help pick the right studiesMBSW May 21, 2013
13Slide14
Usually exclude Phase 1 pharmacokinetic and pharmacodynamic studies (because short duration, healthy subjects or patients with incurable end-stage disease).Studies that cannot / will not provide individual patient level data if required for analysis.Considerations for inclusion in a poolMBSW May 21, 2013
14Slide15
It is generally most appropriate to combine data from studies that are similar.Strong similarity is not required for pooling, if the effects of treatment don’t depend on the trial characteristics being considered.Considerations for inclusion in a poolMBSW May 21, 2013
15Slide16
Suppose some studies (or arms) were conducted at a higher dose than the sponsor is proposing for the marketing label. Would you exclude those arms from the analysis?Yes, if the goal for those analyses is to characterize adverse events from proposed indications at the proposed doses.However, one might choose to combine the high-dose studies or arms in a different pool to help assess what could happen in an overdose situation.For example . . .
MBSW May 21, 201316Slide17
Studies (or arms) at a higher or lower dose than proposed for marketing?In general, exclude dose arms that are lower than the proposed dose for marketing, as these may dilute the effects seen at the higher marketed doseHowever, events may occur in the lower dose studies that should not be ignored
Including low-dose and high-dose studies may help understand the dose-response relationshipMBSW May 21, 201317Slide18
AEs in all those who took the drug?Can analyze ALL who took drug as a single cohort without a comparator group: useful for accounting for all events and estimating event rates for infrequent eventsCan then be compared to external reference population rates
However, external population rates limited by the availability of event rates for a specific subset of the population that is comparable to the trial populationIf the underlying disease increases the risk of a particular event, comparisons with an external reference could be biased against the study drug. Conversely, if enrollment criteria are such that high-risk patients are excluded from trials, the on-study rates could appear to be artificially low.
MBSW May 21, 2013
18Slide19
What if a safety signal was detected in Phase 2 that resulted in a change in ascertainment of an AE in Phase 3 (e.g., an adjudication process, special case report form)?Create a grouping of Phase 3 studies designed for that particular event AdvantagesStudies with consistent ascertainment analyzed togetherExcludes studies that generated the hypothesis being tested
Hypothesis generating studies?MBSW May 21, 201319Slide20
Previous addresses type I error but sacrifices statistical powerdiscards data from what may be studies in a closely monitored population, which may also be at differential risk due to exposure to the compoundAnd it can raise all kinds of red flags (so transparency is key – do the analysis with and without those studies)Hypothesis generating studies (cont.)
MBSW May 21, 201320Slide21
Do not do a crude unstratified analysis that combines studies with a comparator and studies without a comparator.Results can be very misleading. See Lièvre 2002, Chuang-Stein 2010 for further information on dangers of not stratifying.CaveatsMBSW May 21, 2013
21Slide22
Q2: How does the method of ascertainment impact the quality of the meta-analysis?MBSW May 21, 201322Slide23
Ascertainment methodCan affect observed event rates, e.g., actively solicited events will have higher reporting rates than passively collected eventsE.g., for drugs that cross the blood–brain barrier, use prospective tool to assess suicidal ideation and behavior (vs. post hoc adjudication)
MBSW May 21, 201323Slide24
Retrospective adjudicationEven with strict criteria using previously collected data, bias could be introduced by retrospective adjudicationImportant detailed clinical information may be missingIf post hoc
adjudication is necessary, use an external, independent adjudication committee that Is masked to treatment assignment ANDAdjudicates events across the entire development programMBSW May 21, 2013
24Slide25
Q3: What are the advantages of using individual patient data (vs. aggregate summaries)?MBSW May 21, 201325Slide26
Individual or aggregate-level data? For many questions get same answer with IPD as with APDFor analyses that do not require patient-level data, including all relevant studies improves precisionMay also reduce bias that could be introduced by limiting the analysis to those where patient-level data are available
However, there can be advantages to IPDMuch easier to detect interactions between treatment and patient-level characteristic with IPD than with APDMBSW May 21, 201326Slide27
Advantages of patient-level dataAllows mapping all data to a common version of MedDRA (or other) increasing consistency of terminology across trialsGenerally permits creation of common variables across trialsE.g., age categories may have been defined using different category boundariesDifferent threshold hemoglobin values may have been used to define ‘anemia’
MBSW May 21, 201327Slide28
More advantages of IPDAllows specification of a common set of patient-level covariates so subgroup analyses across trials can be performedCan define outcomes based on combinations of variables defining specific events but that may indicate a common mechanism, e.g., a combination of weight loss or appetite reductionMBSW May 21, 2013
28Slide29
And still more advantages of IPDPost hoc analyses of outcomes that require adjudication can sometimes be derived, as in the case of suicide event grading according to Columbia Classification Algorithm of Suicide Assessment (C-CASA criteria)Creation of time-to-event variables (may not be available in publications)Flexibility in defining time periods of interest for analyses, e.g., events occurring during “short-term” follow-up
MBSW May 21, 201329Slide30
Why not always use IPD?Integration required to provide the database is labor intensive, especially if done in retrospectSometimes summary statistics may be the only information available for some studies of interest, e.g., studies of a new therapeutic approach done by an academic group that does not share patient-level data, or the drug of interest may have been included as an active control by another sponsor
MBSW May 21, 201330Slide31
Q4: should we adjust for multiple looks and/or multiple endpoints in the context of meta-analysis?MBSW May 21, 201331Slide32
Q4: Multiple comparisonsComplicated by having multiple looks over time and multiple (and an unknown number of) endpointsSafety Planning, Evaluation, and Reporting Team (SPERT) defined “Tier 1 events” as those for which a prespecified hypothesis has been defined MBSW May 21, 2013
32Slide33
Tier 1 EventsE.g., to rule out an effect of a certain magnitude for assessing a particular risk (a noninferiority test – as for diabetes drugs)Generally, should consider performing formal adjustment for multiple looks for Tier 1 events and for multiple endpoints for other eventsMBSW May 21, 2013
33Slide34
Diabetes drugsNeed to rule out a relative risk of 1.8 (for CV events) for conditional approval, and 1.3 for final approvalConfidence level for that specific outcome may need to be adjusted for multiple looks, which can be considered separately from non-Tier 1 events because it needs to be met for the drug to move forwardAn event of interest: important regardless of the specific side effect profile and Analogous to a primary analysis in the efficacy setting
MBSW May 21, 201334Slide35
Multiplicity is a complicated issue in the safety contextOften have low power, lack of a priori definitions, and extraneous variabilityValue in trying not to miss a safety signal, but remember that initial detection is not the same as proving that a given AE is definitively related to a given drugWorry about reducing false negative findings in drug safety given the known limitations of our tools
MBSW May 21, 201335Slide36
Q5: what is heterogeneity and what are sources of heterogeneity?MBSW May 21, 201336Slide37
Heterogeneity refers to differences among studies and/or study results.Can be classified in 3 ways: clinical, methodological and statistical.MBSW May 21, 201337Slide38
Differences among trials in their Patient selection (e.g., disease conditions under investigation, eligibility criteria, patient characteristics, or geographic differences) Clinical Heterogeneity
MBSW May 21, 2013
38Slide39
Differences among trials in their Interventions (e.g., duration, dosing, nature of the control)Outcomes (e.g., definitions of endpoints, follow-up duration, cut-off points for scales)Clinical Heterogeneity
MBSW May 21, 201339Slide40
Differences in Study design (e.g., the mechanism of randomization).Study conduct (e.g., allocation concealment, blinding, extent and handling of withdrawals and loss to follow up, or analysis methods).Decisions about what constitutes clinical heterogeneity and methodological heterogeneity do not involve any calculation and are based on judgment.Methodological Heterogeneity
MBSW May 21, 201340Slide41
Numerical variability in results, beyond expected by sampling variabilityMay be caused byKnown (or unknown) clinical and methodological differences among trialsChanceStatistical heterogeneity
MBSW May 21, 201341Slide42
Hypothetical exampleMBSW May 21, 201342Slide43
Clinical heterogeneity may not always result in statistical heterogeneity. If there is clinical heterogeneity but little variation in study results, may represent robust, generalizable treatment effects. MBSW May 21, 201343Slide44
Beware of Q(unless you are James Bond)Cochran’s Q is a global test of heterogeneityI2 is a measure of global heterogeneityKEY POINT: They are informative, but rely on neither of these statisticsApparent lack of overall heterogeneity does not rule out a specific source of heterogeneity
Conversely, large studies with clinically small variability can yield spuriously high statistical heterogeneityMBSW May 21, 201344Slide45
Q6: is it sufficient to use fixed-effects models when combining studies or do we need to consider random-effects models?MBSW May 21, 201345Slide46
Fixed-effect vs. random-effectsFixed = common effect across all studiesInference is to the studies at handReasonable to expect (?) when designs and populations are similar across studiesRandom-effects models: true underlying population effects differ from study to study and that the true individual study effects follow a statistical distributionThe analytic goal is then to estimate the overall mean and variance of the distribution of true study effects
MBSW May 21, 201346Slide47
More on FE vs. REIn some situations, it may not be appropriate to produce a single overall treatment-effect estimateGoal should sometimes (often) be to model and understand sources of heterogeneityMBSW May 21, 2013
47Slide48
More points on FE vs. RERisk differences more heterogeneous than odds ratios (OR) or relative risks (RR, a point that is also made in an FDA’s draft guidance for industry on noninferiority trials)Can model on OR scale then convert to RD or RR to help with clinical interpretabilityConstant OR implies effect size must vary for RD, so - must decide whether to estimate the baseline (control) event rate from the external data or from the data included in the actual meta-analysis (implications for variance estimation)
MBSW May 21, 201348Slide49
How to decide on FE or RE? Do you expect a common effect or not?Single indication, similar protocols, same data collection methods, definitions, etc., FE likely to be appropriate. Different populations, etc., use RE but ALSO explore sources of heterogeneityEnough data? Sparse data, few studies, may not permit RE estimationSmall studies may get “up-weighted” with RE: are small study results systematically different?
MBSW May 21, 201349Slide50
Once you go Bayesian, you’ll never go backSpecify a prior probability distributionToday’s posterior becomes tomorrow’s priorFlexibility to deal with heterogeneity through complex modelingAvailable under both FE and RE (use Deviance Information Criterion to decide?) Bayesian inferences are based on the full ‘exact’ posterior distributions (so useful for small numbers of events)
MBSW May 21, 201350Slide51
For more details …MBSW May 21, 2013
51Slide52
Meta-analysis increasingly used to address safety concerns in drug development.Up-front thought allows teams to improve planning and enhance data capture, and enhances transparency and interpretation of the results.Concluding RemarksMBSW May 21, 2013
52Slide53
OK, folks! It’s a wrap!MBSW May 21, 2013
53Slide54
Christy Chuang-Stein, and Mohan Beltangady. Reporting cumulative proportion of subjects with an adverse event based on data from multiple studies. Pharmaceut. Statist. 2010Crowe, Xia, Berlin et al. Recommendations for safety planning, data collection, evaluation and reporting during drug, biologic and vaccine development: a report of the safety planning, evaluation, and reporting team. Clin Trials 2009; 6 430-440Lièvre,
Cucherat and Leizorovicz. Pooling, meta-analysis, and the evaluation of drug safety. Current Controlled Trials in Cardiovascular Medicine 2002Olkin I, Sampson A. Comparison of meta-analysis versus analysis of variance of individual patient data. Biometrics. Mar 1998;54(1):317-322.
References
MBSW May 21, 2013
54