1 OHDSI Methods Benchmark 2 OHDSI Methods Benchmark 3 Two types of tasks Estimate effect of one exposure Casecontrol SCCS Casecrossover Cohort study when comparator is nonactive Comparison of two exposures ID: 1044220
Download Presentation The PPT/PDF document "Method benchmark update Martijn Schuemie" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
1. Method benchmark updateMartijn Schuemie1
2. OHDSI Methods Benchmark2
3. OHDSI Methods Benchmark3Two types of tasks:Estimate effect of one exposureCase-controlSCCSCase-crossoverCohort study when comparator is non-active Comparison of two exposuresCohort studyComparative SCCS
4. OHDSI Methods Benchmark4Three types of data sets:Real negative controlsSynthetic positive controlsRCTs
5. Negative controlsAdvantagesReal exposures and outcomes, so real confounding (measured and unmeasured)Effect size is known (RR = 1)DisadvantageOnly null effects5
6. Negative controls: where to start?Selected four exposures of interestDiclofenacCiprofloxacinMetforminSertralineSelected four outcomes of interestAcute pancreatitisGastrointestinal (GI) bleedingStrokeInflammatory Bowel Disease (IBD)6
7. LAERTES to find controlsFor each exposure (outcome) of interest, list controls where there wasNo evidence in literatureNo evidence in labelsNo evidence in spontaneous reportsPrevalent in an observational databaseEvidence for both exposure and outcome, just not both combinedManually review evidence (Google search), select 25 most prevalent controls per exposure (outcome).Refinement: there should be a lack of evidence of causality. So individual reports in FAERS or eHealthMe.com do not count, as long as PRR not statistically significant greater than 1.7
8. Including comparators8For every exposure-outcome pair, find a comparator exposure that is also a negative control.Heuristics for finding valid comparators:A 2-armed trial in clinicaltrials.gov comparing the two drugs, orFirst four digits of the ATC code match (same indication) but the 5th doesn't (different class).Implemented in The Amazing Comparator Finder: https://schuemie.shinyapps.io/ComparatorFinder/
9. Including comparators9Selected four exposures of interestDiclofenac vs CelecoxibCiprofloxacin vs AzithromycinMetformin vs GlipizideSertraline vs VenlafaxineSelected four outcomes of interestAcute pancreatitisGastrointestinal (GI) bleedingStrokeInflammatory Bowel Disease (IBD)Select a comparator for each exposure control
10. Including nesting cohortsSome methods such as the nested case-control design require a nesting cohort to be defined.For example: a case-control study for metformin may well be nested in T2DM, meaning both cases and controls should have prior diagnose of T2DM.Heuristic for picking nesting cohorts: condition or procedure codes that appear more often on date of treatment initiation than expected.(also implemented in The Amazing Comparator Finder)10
11. Including nesting cohorts11Selected four exposures of interestDiclofenac vs Celecoxib nested in arthralgiaCiprofloxacin vs Azithromycin nested in otitis mediaMetformin vs Glipizide nested in T2DMSertraline vs Venlafaxine nested in depressionSelected four outcomes of interestAcute pancreatitisGastrointestinal (GI) bleedingStrokeInflammatory Bowel Disease (IBD)Select a nesting cohort for each exposure control.
12. First draft of negative control set200 exposure-comparator-outcome-nesting-cohort combinationsPotentially stratified by 4 outcomes and 4 exposures of interestNeed help in reviewing!12
13. Decisions on outcomesRequire inpatient visit or not?First occurrence or all?13
14. Next step for method benchmarkVetting of the negative control setSignal injection Acute effects or long term?Selection of RCTsVojtech, Soledad, and Jill have volunteered to helpWill start with a single RCT14
15. CaseCrossover packageMartijn Schuemie
16. Case-crossover design16ExposureOutcomeFor every case (person with the outcome) define n control windows prior to the outcome (usually n = 1)Determine exposure status on index date and control windowsCase 1Case 2Case 3ExposureFixed interval (e.g. 30 days)OutcomeOutcomeAt risk windowControl windowAt risk windowControl windowAt risk windowControl window
17. Case-time-control design17ExposureOutcomeAdjust for trend in prescribing by picking matched controlsMatching typically on age, gender, and calendar timeCaseControlAt risk windowControl windowAt risk windowControl window2001-02-03Exposure
18. OHDSI Methods LibrarysNew-user cohort studies using large-scale regression for propensity and outcome modelsCohort MethodsSelf-Controlled Case Series analysis using few or many predictors, includes splines for age and seasonality.Self-Controlled Case SeriessA self-controlled cohort design, where time preceding exposure is used as control.Self-Controlled CohortsA self-controlled design, but using temporal patterns around other exposures and outcomes to correct for time-varying confounding.IC Temporal Pattern Disc.sBuild and evaluate predictive models for user-specified outcomes, using a wide array of machine learning algorithms.Patient Level PredictionsUse negative control exposure-outcome pairs to profile and calibrate a particular analysis design.Empirical CalibrationsUse real data and established reference sets as well as simulations injected in real data to evaluate the performance of methods.Method EvaluationsConnect directly to a wide range of database platforms, including SQL Server, Oracle, and PostgreSQL.Database ConnectorsGenerate SQL on the fly for the various SQL dialects.Sql RendersHighly efficient implementation of regularized logistic, Poisson and Cox regression.CyclopssSupport tools that didn’t fit other categories, including tools for maintaining R libraries.Ohdsi R ToolsEstimation methodsPrediction methodsMethod characterizationSupporting packagesUnder constructionsAutomatically extract large sets of features for user-specified cohorts using data in the CDM.Feature ExtractionsCase-control studies, matching controls on age, gender, provider, and visit date. Allows nesting of the study in another cohort.Case-controlsCase-crossover design including the option to adjust for time-trends in exposures (so-called case-time-control).Case-crossover
19. A single CaseCrossover studycaseCrossoverData <- getDbCaseCrossoverData(connectionDetails = connectionDetails, cdmDatabaseSchema = cdmDatabaseSchema, oracleTempSchema = oracleTempSchema, outcomeDatabaseSchema = cohortDatabaseSchema, outcomeTable = cohortTable, outcomeId = 1, exposureDatabaseSchema = cdmDatabaseSchema, exposureTable = "drug_era", exposureIds = 1124300, useNestingCohort = TRUE, nestingCohortDatabaseSchema = cohortDatabaseSchema, nestingCohortTable = cohortTable, nestingCohortId = 2, useObservationEndAsNestingEndDate = TRUE, getTimeControlData = TRUE)matchingCriteria <- createMatchingCriteria(controlsPerCase = 1, matchOnAge = TRUE, ageCaliper = 2, matchOnGender = TRUE)subjectsCtc <- selectSubjectsToInclude(caseCrossoverData = caseCrossoverData, outcomeId = 1, firstOutcomeOnly = TRUE, washoutPeriod = 183, matchingCriteria = matchingCriteria)exposureStatus <- getExposureStatus(subjects = subjects, caseCrossoverData = caseCrossoverData, exposureId = 1124300, firstExposureOnly = FALSE, riskWindowStart = -30, riskWindowEnd = 0, controlWindowOffsets = c(-60))fit <- fitCaseCrossoverModel(exposureStatus)19
20. A single CaseCrossover studycaseCrossoverData <- getDbCaseCrossoverData(connectionDetails = connectionDetails, cdmDatabaseSchema = cdmDatabaseSchema, oracleTempSchema = oracleTempSchema, outcomeDatabaseSchema = cohortDatabaseSchema, outcomeTable = cohortTable, outcomeId = 1, exposureDatabaseSchema = cdmDatabaseSchema, exposureTable = "drug_era", exposureIds = 1124300, useNestingCohort = TRUE, nestingCohortDatabaseSchema = cohortDatabaseSchema, nestingCohortTable = cohortTable, nestingCohortId = 2, useObservationEndAsNestingEndDate = TRUE, getTimeControlData = TRUE)matchingCriteria <- createMatchingCriteria(controlsPerCase = 1, matchOnAge = TRUE, ageCaliper = 2, matchOnGender = TRUE)subjectsCtc <- selectSubjectsToInclude(caseCrossoverData = caseCrossoverData, outcomeId = 1, firstOutcomeOnly = TRUE, washoutPeriod = 183, matchingCriteria = matchingCriteria)exposureStatus <- getExposureStatus(subjects = subjects, caseCrossoverData = caseCrossoverData, exposureId = 1124300, firstExposureOnly = FALSE, riskWindowStart = -30, riskWindowEnd = 0, controlWindowOffsets = c(-60))fit <- fitCaseCrossoverModel(exposureStatus)20Get the data from the CDM database:Specified 1 outcome in the cohort tableSpecified a nesting cohort in the cohort tableNesting cohort ends on observation endGet data for selecting controls
21. A single CaseCrossover studycaseCrossoverData <- getDbCaseCrossoverData(connectionDetails = connectionDetails, cdmDatabaseSchema = cdmDatabaseSchema, oracleTempSchema = oracleTempSchema, outcomeDatabaseSchema = cohortDatabaseSchema, outcomeTable = cohortTable, outcomeId = 1, exposureDatabaseSchema = cdmDatabaseSchema, exposureTable = "drug_era", exposureIds = 1124300, useNestingCohort = TRUE, nestingCohortDatabaseSchema = cohortDatabaseSchema, nestingCohortTable = cohortTable, nestingCohortId = 2, useObservationEndAsNestingEndDate = TRUE, getTimeControlData = TRUE)matchingCriteria <- createMatchingCriteria(controlsPerCase = 1, matchOnAge = TRUE, ageCaliper = 2, matchOnGender = TRUE)subjectsCtc <- selectSubjectsToInclude(caseCrossoverData = caseCrossoverData, outcomeId = 1, firstOutcomeOnly = TRUE, washoutPeriod = 183, matchingCriteria = matchingCriteria)exposureStatus <- getExposureStatus(subjects = subjects, caseCrossoverData = caseCrossoverData, exposureId = 1124300, firstExposureOnly = FALSE, riskWindowStart = -30, riskWindowEnd = 0, controlWindowOffsets = c(-60))fit <- fitCaseCrossoverModel(exposureStatus)21Specify matching criteria for case-time-controlSelect cases and controls
22. A single CaseCrossover studycaseCrossoverData <- getDbCaseCrossoverData(connectionDetails = connectionDetails, cdmDatabaseSchema = cdmDatabaseSchema, oracleTempSchema = oracleTempSchema, outcomeDatabaseSchema = cohortDatabaseSchema, outcomeTable = cohortTable, outcomeId = 1, exposureDatabaseSchema = cdmDatabaseSchema, exposureTable = "drug_era", exposureIds = 1124300, useNestingCohort = TRUE, nestingCohortDatabaseSchema = cohortDatabaseSchema, nestingCohortTable = cohortTable, nestingCohortId = 2, useObservationEndAsNestingEndDate = TRUE, getTimeControlData = TRUE)matchingCriteria <- createMatchingCriteria(controlsPerCase = 1, matchOnAge = TRUE, ageCaliper = 2, matchOnGender = TRUE)subjectsCtc <- selectSubjectsToInclude(caseCrossoverData = caseCrossoverData, outcomeId = 1, firstOutcomeOnly = TRUE, washoutPeriod = 183, matchingCriteria = matchingCriteria)exposureStatus <- getExposureStatus(subjects = subjects, caseCrossoverData = caseCrossoverData, exposureId = 1124300, firstExposureOnly = FALSE, riskWindowStart = -30, riskWindowEnd = 0, controlWindowOffsets = c(-60))fit <- fitCaseCrossoverModel(exposureStatus)22Define at-risk and control windowsDetermine exposure status in windows
23. A single CaseCrossover studycaseCrossoverData <- getDbCaseCrossoverData(connectionDetails = connectionDetails, cdmDatabaseSchema = cdmDatabaseSchema, oracleTempSchema = oracleTempSchema, outcomeDatabaseSchema = cohortDatabaseSchema, outcomeTable = cohortTable, outcomeId = 1, exposureDatabaseSchema = cdmDatabaseSchema, exposureTable = "drug_era", exposureIds = 1124300, useNestingCohort = TRUE, nestingCohortDatabaseSchema = cohortDatabaseSchema, nestingCohortTable = cohortTable, nestingCohortId = 2, useObservationEndAsNestingEndDate = TRUE, getTimeControlData = TRUE)matchingCriteria <- createMatchingCriteria(controlsPerCase = 1, matchOnAge = TRUE, ageCaliper = 2, matchOnGender = TRUE)subjectsCtc <- selectSubjectsToInclude(caseCrossoverData = caseCrossoverData, outcomeId = 1, firstOutcomeOnly = TRUE, washoutPeriod = 183, matchingCriteria = matchingCriteria)exposureStatus <- getExposureStatus(subjects = subjects, caseCrossoverData = caseCrossoverData, exposureId = 1124300, firstExposureOnly = FALSE, riskWindowStart = -30, riskWindowEnd = 0, controlWindowOffsets = c(-60))fit <- fitCaseCrossoverModel(exposureStatus)23Fit model
24. Evaluation of case-crossoverThree analyses variants:Simple case-crossover (1-day windows, control window 30 days prior)Nested case-crossoverNested case-time-controlExecuted against OHDSI benchmark negative controls24
25. Using all negative controls25Simple case-crossover Nested case-crossoverNested case-time-control
26. Stratified by exposure26Nested case-time-control
27. Stratified by outcome27Nested case-time-control
28. OHDSI Symposium coming!Posters?OHDSI methods benchmarkResults of large set of methods on benchmark…28
29. Topic of next meeting(s)??29
30. Next workgroup meetingWestern hemisphere: June 86pm Central European time12pm New York9am Los Angeles / StanfordEastern hemisphere: May 313pm Hong Kong / Taiwan4pm South Korea4:30pm Adelaide9am Central European time8am UK timehttp://www.ohdsi.org/web/wiki/doku.php?id=projects:workgroups:est-methods