Real-world evidence (RWE) framework - PowerPoint Presentation

1. Real-world evidence(RWE) framework MedTech case study 2: Sleepio Impact on patient and system outcomesTechnologySleepio is a digital health technology designed to support user's self-management of insomnia. It is accessed mainly through its website with iOS and FitBit device integration. The programme is structured around a sleep test, weekly interactive cognitive behavioural therapy (CBT‑I) sessions and regular sleep diary entries.Effect on patient outcomesResearch question: Does using Sleepio improve sleep and quality of life? (Stott et al. 2021)Study design and data sourceThis was a comparative cohort study.Data were from the Improving Access to Psychological Therapies (IAPT) audit in Buckinghamshire. All individuals meeting sleep criteria were offered Sleepio. The study compared those who adopted Sleepio with those who did not using propensity score matching.Compared patient reported outcomes at 1 year including the PHQ-9 for general mood and the GAD-7 for anxiety.Table 1: example reporting characteristicsParameterSleepio groupUnmatched controlsMatched controlsStandardised difference with unmatched controlsStandardised difference with matched controlsAge (years)40.4640.6940.73−0.015−0.017Gender (female)66.5%66.6%69.0%−0.0040.053Diagnosis – depression43.9%41.7%42.0%0.0450.039Diagnosis – anxiety52.2%53.7%54.3%−0.032−0.043Diagnosis – other3.9%4.6%3.7%−0.0320.01PHQ-915.2014.1315.210.184−0.001GAD-713.5212.9113.660.123−0.03Sleep item2.431.942.440.5670.038ReferenceStott R, Pimm J, Emsley R, Miller CB, Espie CA. Does adjunctive digital CBT for insomnia improve clinical outcomes in an improving access to psychological therapies service?. Behaviour research and therapy. 2021 Sep 1;144:103922.What was done well?Presented a flow diagram describing number of people meeting eligibility criteria, offered the intervention, and completing follow-up.Clearly listed variables used to estimate propensity scores and time of measurement and described matching procedure.Clearly reported study results.Reported absolute values of each variable and standardised differences before and after matching (Table 1).Effect on system outcomesResearch question: Does using Sleepio reduce health resource use? (Sampson et al. 2022)Table 2: example regression coefficientsCategoryPreferred modelModel 2Model 3Model 4Model 5Time0.002*0.002*0.002*0.001*0.004Intervention−0.038*−0.036*−0.033*0.033*0.263*Post−0.002*−0.002*−0.002*−0.000−0.003Seasonal adjustmentYesYesYesNoNoAge bandsYesYesNoNoNoPractice random effectsYesYesYesYesNoReferenceSampson C, Bell E, Cole A, Miller CB, Marriott T, Williams M, Rose J. Digital cognitive behavioural therapy for insomnia and primary care costs in England: an interrupted time series analysis. BJGP open. 2022 Mar 9.Study design and data sourceThis was an interrupted time series study.Primary care data were from 9 GP practices in Buckinghamshire collected through the EMIS systemEstimated change in total primary care costs after the rollout of Sleepio in Buckinghamshire.Controlled for a range of observed patient level characteristics and for seasonality to account for differences in primary care activity across the calendar year.What wasdone well?What else could have been done?Evidence reviewers would also have benefited from:Access to a study protocol prepared in advance of study conduct. More detailed assessment of data suitability including provenance, quality, and relevance. The RWE framework describes best practices for planning, conducting and reporting real-world evidence studies.How did the studies add to other evidence?The NICE medical technologies guidance on Sleepio for treating insomnia was developed using evidence from both randomised controlled trials and real-world data.Much of the effectiveness data came from randomised controlled trials, and was supplemented by analysis of Sleepio in local NHS settings. *Indicates statistical significance (p<0.001)Clear description of study design and statistical models and rationale for them..Performed extensive sensitivity analyses including a model with a different identification mechanisms (Table 2).