Extensions to traditional statistical meta-analysis - PowerPoint Presentation

Slide1

Extensions to traditional statistical meta-analysis

James ThomasSystematic Reviews for Complicated and Complex Questions, ESRC Methods Festival, St Catherine’s College, Oxford, 10th July 2014

EPPI-Centre

Social Science Research UnitInstitute of EducationUniversity of London18 Woburn SquareLondon WC1H 0NRTel +44 (0)20 7612 6397Fax +44 (0)20 7612 6400Email eppi@ioe.ac.ukWeb eppi.ioe.ac.uk/

The EPPI-Centre is part of the Social Science Research Unit at the Institute of Education, University of LondonSlide2

OutlineWhat is meta-analysis, and why are ‘extensions’ needed?What is complexity? And why is it a challenge in systematic reviews?Extensions to traditional meta-analysis when complexity is encounteredAcknowledgement: presentation draws on: O’Mara-Eves A, Thomas J (2013) Methods for configurational synthesis: extensions to traditional meta-analysis for addressing intervention complexity and contextual variation in reviews. 21st Cochrane Colloquium: Quebec 19-23 SeptemberSlide3

Form review team (involve ‘users’)Formulate review question, conceptual framework and inclusion criteria (develop ‘protocol’)

Search for and identify relevant studiesDescribe studiesAssess study quality (and

relevance)

Synthesise findingsCommunicate and engage

Map

Synthesis

The common stages of a systematic review; focus here on synthesisSlide4

Typically are used to address three key research questions:What is the overall estimate of the size of the effect and its precision?Is there heterogeneity across the study effects?What (if any) variables explain differences across the study effects (if heterogeneity is present)?Traditional meta-analytic modelsSlide5

Aggregative approaches in researchAggregative reviews predominately add up (aggregate) findings of primary studies to answer a review question…… to indicate the direction or size of effectSlide6

Newman M, Bird K, Tripney J, Kalra N, Kwan I, Bangpan M, Vigurs C (2010) Understanding the impact of engagement in culture and sport: A systematic review of the learning impacts for young people. London: Department for Culture, Media and Sport. http://culture.gov.uk/images/research/CASE-systematic-review-July10.pdf RCT forest plot: Does children’s participation in structured arts activities improve their cognitive learning outcomes? Slide7

Current popular meta-analytic methods are limited to:Questions of differences between two groups or correlations or variablesOne causal / relational proposition at a timeOne outcome at a timeAre we limiting ourselves?Slide8

Policymakers and practitioners usually do not ask a single narrow aggregative question They begin with a particular problem and ask “what is the best way to achieve outcome X?”They also ask “does it vary according to…?” and “What does X mean to Y?”And they ask these questions relating to complex problems -Whose questions are we addressing?Slide9

A complex interventionDefined in MRC guidance as: “interventions with several interacting components… Many of the extra problems relate to the difficulty of standardising the design and delivery of the interventions, their sensitivity to features of the local context, the organisational and logistical difficulty of applying experimental methods to service or policy change, and the length and complexity of the causal chains linking intervention with outcome.”Craig P et al (2008): Developing and evaluating complex interventions: the new Medical Research Council guidance.

BMJ 337Some would say the above is merely complicated…Slide10

Complicated and complexTruly complex interventions are best conceptualised as dynamic processesVirtuous circlesFeedback loopsNon-linear step changes in responses / outcomesMultiple ‘routes’ to effectivenessRogers PJ. Using Programme Theory to Evaluate Complicated and Complex Aspects of Interventions. Evaluation. 2008;14(1):29-48Slide11

Complex reviews of social research:Start from a given ‘problem’often a known outcome and populationAim to identify a range of possible ‘solutions’Rarely aim to come to a single answerAcknowledge that there are rarely replications of interventions in social researchUse heterogeneity to better understand factors that influence the impact of interventionsContain detailed and complex conceptual frameworks (programme theories etc)So to configure findings (as well as aggregate)Slide12

Configurative approaches in researchConfigurative reviews predominately arrange (configure) the findings of primary studies to answer the review question….… to offer a meaningful picture of what research is telling us Slide13

Sub-group analysisMeta-regressionNetwork meta-analysisMultivariate meta-analysisPath analysisFactor analysisQualitative comparative analysis…?Other tools in our toolboxSlide14

Subgroup analysisDiCenso et al (2002) BMJ;324:1426 Slide15

Meta-regressionCatalá-López et al. BMC Psychiatry 2012, 12:168Slide16

Network meta-analysisFacilitates an estimate of the relative effectiveness of interventions – even when they have not been directly compared with one another in a trialThorlund and Mills Systematic Reviews 2012, 1:41Slide17

Problems with ‘traditional’ approachesThe above approaches work well in some situations, BUTThere are rarely replications of complex interventions (is it possible to have a genuine replication?); leads to a lack of dataEven when an analysis has many studies, interventions, contexts etc. all differLots of unexplained heterogeneitySymmetrical nature of correlational analysisSlide18

Complexity and correlational analysisCorrelation is symmetricWhen testing for a connection between cause and effect, also tests equally for absence of cause and absence of effectCorrelation therefore cannot detect multiple causal pathways*E.g. Asserting that ‘interventions which are delivered by peers tend to be effective’ should not require that those not delivered by peers are not effective* There are usually too few studies to use interaction variablesSlide19

Another approach: qualitative comparative analysis (QCA)Originally developed by Charles Ragin in political science and historical sociologyNot a correlational approachUseful for small numbers of studiesFocus can be on combinations of intervention componentsMore inductive mode of analysis than above

Thomas J, O’Mara-Eves A, Brunton G (2014) Using Qualitative Comparative Analysis (QCA) in systematic reviews of complex interventions: a worked example. Systematic Reviews. 3: 67Suggests that intensity & qualityare sufficient to gain effective outcomeSlide20

SummaryMeta-analysis can address a single focused question, requiring a straightforwardly aggregative answer“Real world” questions tend to require configuration AND aggregationExtensions to meta-analysis are able to configure and aggregate study findings successfullyBut few current methods can cope with genuinely complex situationsSlide21

WebsitesEPPI-Centre Website http://eppi.ioe.ac.ukTwitter @James_M_Thomas

@EPPICentreEmailj.Thomas@ioe.ac.uk

Thank you for your attention

EPPI-Centre

Social Science Research UnitInstitute of EducationUniversity of London18 Woburn SquareLondon WC1H 0NR

Tel +44 (0)20 7612 6397

Fax +44 (0)20 7612 6400

Email eppi@ioe.ac.uk

Web eppi.ioe.ac.uk/

The EPPI-Centre is part of the Social Science Research Unit at the Institute of Education, University of

London