Martijn Schuemie PhD Janssen Research and Development Previous meetings Oct 19 Sequence Symmetry Analysis Nov 2 Distributed regression amp biosignal collection Nov 16 Propensity score lessons learned ID: 1033129
Download Presentation The PPT/PDF document "Confidence interval calibration" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
1. Confidence interval calibrationMartijn Schuemie, PhDJanssen Research and Development
2. Previous meetingsOct 19: Sequence Symmetry AnalysisNov 2: Distributed regression & bio-signal collectionNov 16: Propensity score lessons learnedNov 30: Constructing a drug monitoring system in Korea2
3. DisclaimerThese are preliminary resultsAwaiting new set of negative controls (more power)3
4. A tale of two studiesStudy 1: Mini-Sentinel estimated incidence of GI bleeding in dabigatran vs warfarinCrude comparisonIRR = 1.6 / 3.5 = 0.46NEJM publicationFDA subsequently communicated dabigatran is safe4
5. A tale of two studiesStudy 2: Graham et al. also estimated incidence of GI bleeding in dabigatran vs warfarinPropensity-score adjustmentRestricted to elderly populationHR = 1.28 (1.14–1.44)5
6. Opposite effectsSouthworth: IRR = 0.46Graham: HR = 1.28 (1.14–1.44)6
7. Opposite effectsSouthworth: IRR = 0.46Graham: HR = 1.28 (1.14–1.44)One explanation: different bias in the different studiesCan we quantify the bias?Can we calibrate for the bias?Does calibration make the results comparable?7
8. Step 1: Replicate studiesFaithful replication of all study details except database choice8
9. Step 2: Negative controlsAllergic rhinitisInfection due to EnterobacteriaceaeAnxiety disorderInfluenzaArthritis of spineIngrowing nailArthropathy of pelvisInstability of jointAtelectasisIrritable bowel syndromeBlepharitisMalignant neoplasm of genital structureCellulitisMalignant tumor of breastChronic sinusitisMegaloblastic anemiaChronic ulcer of skinObesityCurvature of spineOsteochondropathyCutis laxaPeripheral vertigoDehydrationPlantar fasciitisDiabetic oculopathyPoisoningDiabetic renal diseaseProlapse of female genital organsDislocation of jointPsychotic disorderDrug dependencePtosis of eyelidDyssomniaSciaticaEffusion of jointSeborrheic dermatitisFibrocystic disease of breastSimple goiterFracture of lower limbSleep apneaGallstoneStreptococcal infectious diseaseGastro-esophageal reflux disease with esophagitisSuperficial mycosisHuman papilloma virus infectionUlcerative colitisHyperplasia of prostateUrge incontinence of urineHypokalemiaUrinary tract infectious disease9
10. Negative control distribution10Southworth replication
11. Negative control distribution11Graham replication
12. Trouble with positive controlsOften very few positive examples for a particular comparisonExact effect size never known with certainty (and depends on population)Doctors also know they’re positive, and will change behavior accordingly12
13. Creating positive controlsStart with negative controls: RR = 1Add simulated outcomes during exposure until desired RR is achievedInjected outcomes should behave like ‘real’ outcomes: preserve confounding structure by injecting outcomes for people at high risk 13
14. Creating positive controls14Patient 1DabigatranPatient 2WarfarinPatient 3DabigatranPatient 4WarfarinDabigatranPatient 6WarfarinIngrowing nailInjected ingrowing nailPatient 5Predictive model of outcome indicates this is a high-risk patientNew RR = 2 (but with same confounding)
15. 15Estimating effects for positive controlsBlack line indicates true hazard ratioSouthworth replication
16. Estimating effects for positive controls16Ingrowing nailTrue RR = 1 Estimated RR = 0.88 (0.77 – 1.00)Southworth replication
17. Estimating effects for positive controls17Ingrowing nail+True RR = 1.5Estimated RR = 1.30 (1.16 – 1.46)Southworth replication
18. Estimating effects for positive controls18Analysis suggests bias remains constant with effect sizeSouthworth replication
19. Southworth replicationEvaluating coverage of the CI1942%38%38%34%CoverageCoverage decreases with true effect sizeMissing the true effect size 62% of the time when the true RR = 2!Coverage of 42% means the true effect size is outside of the 95% confidence interval 58% of the time (when the true RR = 1)
20. Confidence interval calibration20= 1 µσ σ Estimated systematic error distribution= 2 µσ
21. Calibrating a confidence interval21 σ 0.74 (0.66 – 0.85)0.81 (0.52 – 1.26)CICalibrationUncalibratedCalibratedConfidence intervals were too narrow, so made wider to get to nominal coverage
22. 22Confidence interval calibrationUncalibratedCalibrated
23. Confidence interval calibration92%91%94%96%CoverageCalibrated
24. Internal validity: Southworth24
25. Internal validity: Graham25
26. External validity26We could explain some but not all of the difference with study biasHad Mini-Sentinel used CI calibration they could have come to a different conclusion
27. Additional exampleTata et al. studied relationship between SSRIs and upper GI bleeding Case-control: OR = 2.38 (2.08–2.72)SCCS: IRR = 1.71 (1.48–1.98)Reproduce in CPRD (need ISAC approval)27
28. ConclusionConfidence interval calibration accounts for systematic error (in addition to random error)Confidence interval calibration restores coverage of 95% CI to approximately 95%Account for systematic error reduces between-study heterogeneity (but doesn’t eliminate it)28
29. Questions?29
30. Next workgroup meetingEastern hemisphere: January 113pm Hong Kong / Taiwan4pm South Korea5:30pm Adelaide8am Central European time7am UK timeWestern hemisphere: January 4?6pm Central European time12pm New York9am Los Angeles / Stanfordhttp://www.ohdsi.org/web/wiki/doku.php?id=projects:workgroups:est-methods