Seng Chan You What should OHDSI studies look like 2 A study should be like a pipeline A fully automated process from database to paper Performing a study building the pipeline Database ID: 1033045
Download Presentation The PPT/PDF document "Does large-scale propensity score matchi..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
1. Does large-scale propensity score matching really match the hidden baseline characteristics?Seng Chan You
2. What should OHDSI studies look like?2A study should be like a pipelineA fully automated process from database to paper‘Performing a study’ = building the pipelineDatabasePaper
3. Current OHDSI best practices123
4. General principlesPrespecify what you're going to estimate and how: this will avoid hidden multiple testing (publication bias, p-value hacking). Run your analysis only once.Validation of your analysis: you should have evidence that your analysis does what you say it does (showing that statistics that are produced have nominal operating characteristics (e.g. p-value calibration), showing that specific important assumptions are met (e.g. covariate balance), using unit tests to validate pieces of code, etc.)Transparency: others should be able to reproduce your study in every detail using the information you provide.4
5. Best practices (generic)Write a full protocol, and make it public prior to running the studyResearch question + hypotheses to be testedWhich method(s), data, cohort definitions. What is the primary analyses and what are sensitivity analyses?Quality controlAmendments and UpdatesValidate all code used to produce estimates. The purpose of validation is to ensure the code is doing what we require it to do. Possible options are:Unit testingSimulationDouble codingCode reviewInclude negative controls (exposure-outcome pairs where we believe there is no effect)Produce calibrated p-valuesMake all analysis code available as open source so others can easily replicate your study5
6. Best practices (new-user cohort design)Use propensity scores (PS)Build PS model using regularized regression and a large set of candidate covariates (as implemented in the CohortMethod package)Large-scale propensity score matchingUse either variable-ratio matching or stratification on the PSCompute covariate balance after matching for all covariates, and terminate study if a covariate has standardized difference > 0.26
7. My studyComparison between first line dual-combination therapies in hypertension: multi-national collaborative research through OHDSI networkAIMTo compare the mortality risk of combination regimens among patients initiating antihypertensive treatmentAnalysis based on CDM for reproducible research7
8. Method: StatisticsLarge scale propensity score matchingCaliper: 0.15Max Ratio: 1:1Univariate Cox regression with stratificationSensitivity analysisSame analysis on patients with various minimum periods (30, 365,730 days) of continuing the drug regimenAnalytic R code is available for reproducible research:https://github.com/OHDSI/StudyProtocolSandbox/tree/master/HypertensionCombinationMartijn J. Schuemie, Marc A. Suchard and Patrick B. Ryan (2017). CohortMethod: New-user cohort method with large scale propensity and outcome models. R package version 2.4.3.
9. Validation: balance scatter plot (Korea-NHIS)9You et al., ESC Congress[Abstract], 2017CD vs ACAC vs ADCD vs AD
10. Validation: Propensity score distribution before PSM(Korea-NHIS)10You et al., ESC Congress[Abstract], 2017CD vs ACAC vs ADCD vs AD
11. 11You et al., ESC Congress[Abstract], 2017CD vs ACAC vs ADCD vs ADValidation: Propensity score distribution after PSM (Korea-NHIS)
12. Validation: Negative controls (Korea-NHIS)12You et al., ESC Congress[Abstract], 2017
13. Validation: Negative controls (Medicaid & Medicare)13MedicaidMedicare
14. Result: Primary endpoint (All-cause mortality) in Korean NHISP = 0.465P = 0.465P = 0.478Survival probabilityA+CA+DC+DA+CC+DA+D
15. Result: Primary endpoint (All-cause mortality) in Korean NHISResult: All-cause mortality between dual combination treatment group after large scale propensity score matching (Minimum drug period : 180 days) Active drug groupComparator groupNumber of active groupafter matchingNumber of comparator group after matchingHazard ratio95% CIP valueA+CA+D475147511.110.84-1.490.465C+DA+C173917391.030.71-1.330.465C+DA+D238223821.090.85-1.410.478Abbreviations: CI, confidential interval; A, angiotensin-converting enzyme inhibitor/angiotensin receptor blocker; B, β-blocker; C, calcium channel blocker; D, thiazide-diuretics; CV, cardiovascular
16. Result: Primary endpoint (All-cause mortality) in US (medicaid & medicare))16A+CA+DC+DA+CC+DA+DMedicaidMedicare
17. Result: Primary endpoint (All-cause mortality)There is no difference in mortality between dual combination of anti-hypertensive medicationActive drugComparatorDatabaseNumber of active groupafter matchingNumber of comparator group after matchingHazard ratio95% CIP valueA+CA+DKorean NHIS475147511.110.84-1.490.465Medicare34329343290.980.84-1.140.784Medicaid400640060.910.64-1.290.594C+DA+CKorean NHIS173917391.030.71-1.330.465Medicare546554651.000.69-1.441.000Medicaid100510050.790.35-1.730.555C+DA+DKorean NHIS238223821.090.85-1.410.478Medicare663966391.380.98-1.950.070Medicaid119111910.690.29-1.600.356
18. Expected Question from Reviewer 1How are you convinced that important variables are adequately matched?18
19. Previous studies using High-Dimensional Propensity Score MatchingHuybrechts et al., NEJM, 2014
20. Previous studies using High-Dimensional Propensity Score MatchingThey demonstrated that there is no association between antidepressant use and cardiac malformationBy 5 levels of analysis-Unadjusted analysis-Depression-restricted analysis-PS stratification-HDPS (data are not shown)-Sensitivity analysis for stratified race and age groupHuybrechts et al., NEJM, 2014
21. Previous studies using High-Dimensional Propensity Score MatchingHuybrechts et al., NEJM, 2014
22. Previous studies using High-Dimensional Propensity Score MatchingZhou et al., Epidemiology, 2017
23. Previous studies using High-Dimensional Propensity Score MatchingZhou et al., Epidemiology, 2017
24. Expected Question from Reviewer 1Q. How are you convinced that important variables are adequately matched?I’ll show that large-scale propensity score does match not only thousands of unseen variables but also cardiovascular important variables24
25. Result: baseline characteristics after matching in Korean NHIS) A+C vs A+DC+D vs A+C C+D vs A+D A+C (n=4751)A+D (n=4751)SDC+D (n=1739)A+C (n=1739)SDC+D (n=2382)A+D (n=2382)SDFemale, n (%)2065 (43.5)1932 (40.7)0.06859 (49.5)882 (50.8)-0.031340 (56.4)1354 (57.0)0.01DM, n (%)1593 (33.5)1581 (33.3)0.01264 (15.2)243 (14.0)0.03591 (24.9)532 (22.4)0.06CKD, n (%)111 (2.3)79 (1.7)0.0530 (1.7)21 (1.2)0.0433 (1.4)20 (1.0)0.05Dyslipidemia, n (%)2249 (47.3)2252 (47.4)0.00577 (33.3)510 (29.4)0.08706 (29.7)655 (27.6)0.05CCI, mean2.62.50.032.11.90.08 1.91.70.11DM, diabetes mellitus; CKD, chronic kidney disease; AF, atrial fibrillation; CCI, Charlson comorbidity index
26. Result: Age distribution before and after matchingBefore matchingBefore matchingBefore matchingAfter matchingAfter matchingAfter matching
27. Result: Inclusion year distributionBefore matchingBefore matchingBefore matchingAfter matchingAfter matchingAfter matching
28. Expected Question from Reviewer 2Q. You analyzed hypertensive patients without baseline blood pressure. How can I believe this result? 28
29. Expected Question from Reviewer 2Q. You analyzed hypertensive patients without baseline blood pressure. How can I believe this result? My question: Can large-scale propensity score matching of claim data decrease the difference of baseline blood pressure between the target and comparator cohorts?29
30. Method: study population NHIS-national sample cohort (NHIS-NSC) DBConsecutive observation for 1M patients who were randomly sampled from whole Korean population between 2002-2013The NHIS instituted the biennial national health examination program, which is recommended for all of the insured employees or self-employed individuals over 40 years old and their dependentsLee et al., Int J Epidemiol. 2016
31. Expected Question from Reviewer 2Q. You analyzed hypertensive patients without baseline blood pressure. How can I believe this result? My question: Can large-scale propensity score matching of claim data decrease the difference of baseline blood pressure between the target and comparator cohorts?I’ll show the difference of baseline blood pressure and other values from health examination among two cohorts before and after large-scale PSM(https://github.com/ohdsi/studyprotocolsandbox/tree/chan.validation)31
32. AC vs AD32
33. CD vs AC33
34. CD vs AD34
35. 35