Introduction to Systematic Reviews - PowerPoint Presentation

Slide1

Introduction to Systematic Reviews

Fiona Beyer on behalf of the Evidence Synthesis Team

Institute of Health & Society

fiona.beyer@ncl.ac.ukSlide2

Aims of this session….

What is a systematic review?

How to formulate an appropriate review question?

Why and how to write a protocol?

How to find relevant studies?How to extract appropriate data? How to critically appraise studies? Introduction to synthesis

BRIEF

introduction and signposting!Slide3

What is a systematic review?

SYSTEMATIC

:

Done or acting according to a fixed plan or system: methodical

REVIEW: A critical appraisal of a book, play or other workSlide4

“A

systematic review

is a review in which there is a

comprehensive search

for relevant studies on a specific topic, and those identified are then appraised and synthesized according to a predetermined and explicit method.” (Klassen 1998)A systematic review attempts to collate all empirical evidence that fits pre-specified eligibility criteria in order to answer a specific research question. It uses explicit, systematic methods that are selected with a view to

minimizing bias

, thus providing more reliable findings from which conclusions can be drawn and decisions made (Antman 1992, Oxman 1993)

What is a systematic review?Slide5

Why

use systematic reviews?

Minimise the impact of bias/errors

Can help to end confusion

Highlight where there is not sufficient evidence

Combining findings from different studies can highlight new findings

Can mitigate the need for further trialsSlide6

Facilitate

rational decision making

Health care providers, researchers and policy makers are inundated with unmanageable amounts of information

Over 20 million citations in PubMed

Approx. 75 to 100 RCTs published dailyUsually impossible to consider all relevant individual primary research studies in a decision making context

Enable practitioners to keep up to date and practice evidence-based medicine

Why

use systematic reviews?Slide7

Why not

traditional

reviews?

‘Traditional’ reviews

Systematic reviewsUnclear how included studies were selectedProtocol specifies exact research question/inclusion criteriaBased on incomplete set of studiesComprehensive search to find all relevant studiesMay not consider quality of included studiesStandardised critical appraisal across included studiesUnclear

how conclusions follow from included studies

Methods transparent and reproducibleSlide8

Hierarchy of evidenceSlide9

Who undertakes systematic reviews?

Cochrane/Campbell Collaboration

NICE/Regulatory bodies

Health Technology Assessment

Academics/researchers/clinicians

MSc/PhD studentsSlide10

Multidisciplinary teams

Clinicians

Health services researchers

Information scientists

Statisticians Health economists Patient and public involvement

Who undertakes systematic reviews?Slide11

Systematic Reviews -

the

process (1)

Define review question

– very precisely; in partnership with commissioners, clinicians, patients (as appropriate)Develop protocol – provides transparency; defines exact inclusion criteria and methodsIdentify relevant studies – usually comprehensive search across multiple bibliographic databases plus reference checkingAssess eligibility – careful matching of studies against inclusion criteriaSlide12

Systematic Reviews -

the

process (2)

Extract

relevant data – only what is required to answer the questionCritically appraise studies – use a published tool to compare methodological features across studiesSynthesise appropriately – depending on type of question and study designs quantitative or qualitative synthesis

Disseminate

to appropriate audience – full transparent write-up plus as appropriate short report to funder, journal article, patient leaflet… Slide13

Formulating the review questionSlide14

Formulating review

question

Questions may be broad or narrow

Well-formulated questions will guide many aspects of the review process

Searching strategy

Inclusion/exclusion criteria

Data extraction

Choice of synthesis method

Presentation/dissemination of findingsSlide15

Quick Activity

Discuss a very broad question and how you might narrow it? (10 mins)

Discuss the potential limitations of your review questions

If time and resource were not a limitation – consider how useful would the answer to your review question be?Slide16

Current guidance

a clear and concise statement of a review's objectives (or questions) is critical and should begin with a precise statement of the primary objective, including the interventions reviewed and the targeted problem; ideally, this would be presented in a single sentence

Cochrane & PRISMA StatmentSlide17

Current guidance - Cochrane

“To assess the effects of [

intervention or comparison

] for [

health problem

] in [

types of people, disease or problem, and setting if specified

].” Slide18

Formulating review question

Determining the scope is a decision dependent upon multiple factors:

Perspectives regarding a question’s relevance and potential impact;

Supporting theoretical, biologic and epidemiological information;

The potential generalizability and validity of answers to the questions;

Available resources;

The wider literature base – has a recent high-quality SR been conducted?Slide19

Formulating review question

Often dealing with complex interventions

Might be a need to develop working definitions of the intervention of interest

Several options on how to do this (pragmatic real world v theoretical, logic models, etc.)

Use content experts outside the review team to ensure that the resulting definitions are likely to be robust and meaningful

Need to carefully consider appropriate study design which will answer your particular questionSlide20

Formulating review question

Advantages

Disadvantages

Broad

questionComprehensive summary of all evidenceMay ‘lump’ interventions that are heterogeneous – apples/ pearsFindings are likely to be generalisableMay be a lot of studies to manage (difficult for narrative synthesis)Narrow questionMore discrete and manageable inclusion criteriaMay be too few studies to provide meaningful result (although this can be useful evidence of ‘gap’)Less heterogeneity

May

not be generalizable at allSlide21

Formulating review

question: tools (1)

PICO is for effectiveness questions which usually use controlled trials

Glasziou et al, 2003Slide22

1. Setting

what is the context of the question?                               

e.g. European

countries

2. Perspective who are the users/potential users of the outcomes?                                e.g. chronically ill3. Intervention what is being done to them?                                e.g. increased cost-sharing4. Comparison what are the alternatives?                                e.g. no increase5. Evaluation how will you measure if the intervention is successful?                                           e.g. access to health services    

SPICE is designed for questions around public health interventions

Booth 2004

Formulating review

question: tools (2)Slide23

Sample a description of what is likely to be a smaller group of participants

Phenomenon

(of)

may not be an intervention or exposure

Interest Design details of the design will help to make decisions about the robustness of the research Evaluation what are you measuring?Research type qualitative, mixed methods?  Cooke et al 2006

Formulating review

question: tools (3)Slide24

Any questions??Slide25

Developing the protocolSlide26

Developing protocol

A protocol is an essential component of the systematic review process

Helps to ensure careful

a priori

planning of

whole process

Consistency

Transparency

Integrity

Journals often require a protocol registration number before publishing systematic reviewsSlide27

One of the features that distinguish a systematic review from a traditional review is the pre-specification of criteria and methods

W

hat studies will be

included

and excluded? (use previous question formulation tools)How (exactly) will each stage be conducted? (search strategy – consult librarian, study selection, critical appraisal, synthesis method – consult statistician)What are the primary and secondary outcomes of interest?

Developing protocolSlide28

PROSPERO – CRD initiative

Search for ongoing reviews

Register planned review online (& update as work progresses)

Avoids duplication of reviewsSlide29

Finding eligible studiesSlide30

Searching for eligible studies

Can use the same tools to help define search as for question formulation (PICO, SPICE, SPIDER)

Use only the items needed to define search (usually at minimum population and intervention/phenomenon of interest), sometimes outcomes

Consult information specialist or librarian for advice (the search is the first potential major source of bias in your review)Slide31

Thesaurus headings (e.g. MeSH in MEDLINE)

Controlled vocabulary thesaurus used for indexing articles

Standardised search terms regardless of how author has described the study

Each database has a different thesaurus – must translate

Searching for eligible studiesSlide32

Searching for eligible studies

Keywords

Also search title and abstract fields in case of imprecise or non existent indexing

Need synonyms and truncation to search word stems (e.g. child* retrieves child, children, childless, childbirth,

etc.)Slide33

Where to searchElectronic databases according to the topic area: MEDLINE, EMBASE, Cochrane databases, PsycInfo, etc.

Grey literature, dissertations, theses, conference proceedings, national bodies (NICE, HTA), clinical trial database (

www.clincialtrails.gov

/

)University library web pages give advice about databases available in different topic areasCheck existing systematic reviews in a similar area for ideasLook at the databases own guidance for searching they vary!Download results to reference management software (e.g. University supports Endnote, alternatively Mendeley)Searching for eligible studiesSlide34

Boolean operatorsSlide35

Select potentially eligible abstracts (in Endnote)

Assess titles/abstracts against your predetermined inclusion/exclusion criteria, use groups or custom fields to record

Retrieve

full text articles of initial selections

Assess full text for inclusionRequires judgement (>1 reviewer)Check reviewer agreement (3rd review to resolve)Use a selection form to ensure consistency and record decisionsSelecting included studiesSlide36

Any questions??Slide37

Data

ExtractionSlide38

Data

Extraction (1)

Pilot a data extraction form

See options later

Collect

the following information:

Citation details to identify the study

Characteristics of population/intervention

Outcomes at baseline and follow-up

Methodological details according to chosen critical appraisal toolSlide39

What effect measures do you want to calculate?

What data do you need to do this?

How are you planning to group studies for the analysis

?

By intervention, study design??What information do you need to extract to enable you to organise and analyse the way you want?

Data

Extraction (2)Slide40

Extract

sufficient

information:

to describe studies

to allow you to undertake the planned analysisso you do not need to return to the full text papersBut not too muchDon’t waste time extracting ‘nice to have’ but unnecessary detail

Data

Extraction (3)

REMEMBER YOUR PROTOCOL

IT

IS YOUR ROADMAP, FOLLOW IT!Slide41

There is a wide selection of software to choose

from

Consider:

What

are you are familiar with?What package best suits your data?How many included studies do you have?

Data

Extraction (4)Slide42

Software Options

Word

Excel

Access

EPPI reviewer

COVIDENCE

REVMAN

????Slide43

Consistency/Standardisation

Data extractors need to collect and interpret data in a standard way

Essential

>one reviewer is extracting

data

Independent

piloting of data extraction forms – always one standardised form

Regular

discussion of progress/disagreements

Regular comparison of data extraction – don’t wait till the endSlide44

Efficient data extraction

Once data extraction is complete you may need to

:

Sort/search your data

Filter

data

Calculate frequencies

Transform data (e.g. SE to SD)

Categorising/coding data will make these tasks easier:

Needs to be implemented with consistency by the whole team Slide45

Things to consider

You

may

need to update the form, or have more than one

form

Any changes need to be agreed and made consistently

Are you including more than one study design

?

You may need separate forms for each study

design

However, you are still answering the same question, so make sure the core information extracted is the

sameSlide46

Things to consider

You

may

need to update the form, or have more than one

form

Any changes need to be agreed and made consistently

Have one or a few studies reported data differently from the others

?

Will the data still be useful?

Should you include it?

Make sure

the core information extracted is the

sameSlide47

Any questions??Slide48

Critical AppraisalSlide49

Critical Appraisal

Why - what are we trying to achieve?

Not all published and unpublished literature is

rigorous!

being in a journal doesn’t mean it is good

Quality may be used as an explanation for differences in study results or to guide interpretation of findings, strength of inferencesSlide50

Critical

Appraisal

Quantitative studies

Internal validity

Bias: selection

;

performance

;

detection

;

attrition

;

reporting

External validity

Better to

use domain based

assessment not numerical scores

Cochrane Risk of Bias 2.0 – RCTs (Higgins 2016)

QUADAS 2 – diagnostic accuracy (Whiting 2006)

ROBIS for systematic reviews (Whiting 2016)Slide51

Qualitative studies

Three broad categories

Rigour

: has a thorough and appropriate approach been applied to key research methods in the study?

Credibility: are the findings well presented and meaningful? Relevance: how useful are the findings to you and your organisation?

Critical

AppraisalSlide52

CASP appraisal checklist

Clear aims of research (goals, why it is important, relevance)

Appropriate methodology

Sampling strategy

Data collection

Relationship between researcher and participants

Ethical issues

Data analysis

8.

Findings

9

.

Value of research (context dependent) Slide53

Any questions??Slide54

Data synthesisSlide55

Data Synthesis

Building up; putting together; making a whole out of the parts; the combination of separate elements of thought into a whole; reasoning from principles to a conclusionSlide56

Data Synthesis

Results from different studies types need to be

synthesised

appropriately

Quantitative data meta-analysis (if enough homogeneity in interventions/ outcomes/ study designs) AND/ORnarrative synthesisQualitative dataVarious methods depending on review purpose, e.g. meta-ethnography, thematic synthesis Slide57

Meta-analysis/forest plotSlide58

Heterogeneity

Differences

Methodological heterogeneity: study design/ biases

(Statistical) heterogeneity: intervention effects are more different than expected due to chance

Clinical heterogeneity: variability in participants or interventions

Shouldn’t statistically pool if too much heterogeneity

ask for advice (early)!Slide59

Narrative synthesis

Instead of/alongside meta-analysisPotential bias in presentation

Can be more difficult to provide a take home messageSlide60

Narrative synthesis - tools

Partly informed by methodological work in qualitative synthesis

Tabulation

Groupings and clusters

Vote counting as a descriptive toolExamination of moderator variables (elements of e.g. setting, population)Slide61

Qualitative synthesis

Some qualitative synthesis methods are based on those used in primary qualitative research

Qualitative synthesis generates theories and explanations from analysis of textual data (rather than effect size)

Critical appraisal is not used for discarding ‘unfit’ studies but still important for understanding weight of evidenceSlide62

Reporting your review

Ensure you follow guidance for reporting your protocol (PRISMA-P) and systematic review

PRISMA for meta-analysis of RCTs, MOOSE for observational studies, ENTREQ for qualitative syntheses

See Equator network for full set of reporting guidelinesSlide63

Be organised!!!

Maintain an audit trail; keep good records of all decisionsSlide64

Version control

Dates – YYYYMMDDVersion numbering v0.1 = first draft

v1.0 = final version

v1.1 = minor amendments to final version

v2.0 = major revisionAvoid using draft, draft 1, final , etc.Clear naming conventionE.g. Date_project_title 20/11_Autism HTA_resultsv1.2.docSlide65

Gantt chartsSlide66

Other help available

FMS Systematic review group

Informal monthly session where methods are discussed and issues can be raised (Baddiley-Clark building)

fiona.beyer@ncl.ac.uk

or jenni.hislop@ncl.ac.ukMSc in Public Health and Health Services Research (~October 2017)10 credit module ‘Introduction to systematic reviewing and critical appraisal’pghealth@newcastle.ac.ukSlide67

Any questions??Slide68

Thanks for your attention!

Contact details:

Fiona Beyer

Institute

of Health & Society fiona.beyer@ncl.ac.uk Slide69

ReferencesAntman, E. M. et al (1992) A

comparison of results of meta-analyses of randomized control trials and recommendations of clinical experts. JAMA 268:240-248

Booth

, A. and Brice, A. (2004), “Appraising the evidence”, in Booth, A. and Brice, A.

(eds), Evidence Based Practice for Information Professionals, Facet Publishing, London, pp. 104-18.CASP Critical appraisal tools http://www.casp-uk.net/checklists Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011], (2011) Higgins, J.P.T. and Green, S. (eds.) The Cochrane Collaboration. http://handbook.cochrane.org/Cooke A, Smith D, & Booth A. "Beyond PICO: the SPIDER tool for qualitative evidence synthesis." Qualitative Health Research 22.10 (2012): 1435-1443.Covidence https://www.covidence.org/ Centre for Reviews and Dissemination (2009) Systematic Reviews: CRD's guidance for undertaking systematic reviews in health care. www.york.ac.uk/inst/crd/pdf/Systematic_Reviews.pdf Equator network (reporting guidelines) https://www.equator-network.org/reporting-guidelines/ Glasziou P & Del Mar C. Evidence-based medicine workbook: finding and applying the best evidence to improve patient care. University of Queensland, 2003.Higgins, J.P.T. et al (2016) 'A revised tool for assessing risk of bias in randomized trials', in Chandler, J., McKenzie, J., Boutron, I. and Welch, V. (eds.) Cochrane Database of Systematic Reviews.Klassen et al. Guides for reading and interpreting systematic reviews. Arch Pediatr Adolesc Med 1998;152:700-704Moher,

D. et al (2009

) 'Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement',

Annals of Internal Medicine, 151(4), pp. 264-269

.Oxman AD, Guyatt

GH.

(1993) The

science of reviewing research.

Annals of the New York Academy of

Sciences

703: 125-13

Prospero

https://www.crd.york.ac.uk/PROSPERO

/

Whiting,

P.F. et al

(2006) 'Evaluation of QUADAS, a tool for the quality assessment of diagnostic accuracy studies',

Medical Research Methodology, 6(9), p. 8

.

Whiting,

P. et al

(2016) 'ROBIS: A new tool to assess risk of bias in systematic reviews was developed',

Journal of Clinical Epidemiology, 69, pp. 225-234.