Dawn Craig Institute of Health amp Society dawncraignclacuk dawncraig Aims of this session Outline what a systematic review is To discuss scope and the formulation of a review question ID: 585646
Download Presentation The PPT/PDF document "Introduction to Systematic Reviews" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Introduction to Systematic Reviews
Dawn Craig
Institute of Health &
Society
dawn.craig@ncl.ac.uk
@
dawn_craigSlide2
Aims of this session….
Outline what a systematic review is
To discuss scope and the formulation of a review question
The purpose and format of a protocol
Systematic searching/screening of
studies
Data extraction/quality appraisal and intro to evidence synthesisSlide3
What is a systematic review?
SYSTEMATIC
: Done or acting according to a fixed plan or system: methodical
REVIEW:
A critical appraisal of a book, play or other workSlide4
What is a systematic review?
“A
systematic review
is a review in which there is a comprehensive search for relevant studies on a specific topic, and those identified are then appraised and synthesized according to a predetermined and explicit method.”*
(*Klassen et al. Guides for reading and interpreting systematic reviews. Arch
Pediatr
Adolesc
Med 1998;152:700-704.)
A
systematic review
attempts to collate all empirical evidence that fits pre-specified eligibility criteria in order to answer a specific research question. It uses explicit, systematic methods that are selected with a view to minimizing bias, thus providing more reliable findings from which conclusions can be drawn and decisions made
(
Antman
1992,
Oxman
1993)Slide5
Why we need systematic reviews
Minimise the impact of bias/errors
Can help to end confusion
Highlight where there is not sufficient evidence
Combining findings from different studies can highlight new findings
Can mitigate the need for further trialsSlide6
Why we need systematic reviews
Facilitate
rational decision making
Health care providers, researchers and policy makers are inundated with unmanageable amounts of information
Over 20 million citations in PubMed
Approx. 75 to 100 RCTs published daily
Usually impossible to consider all relevant individual primary research studies in a decision making context
Enable practitioners to keep up to date with evidence accumulating in field and to practice evidence-based medicineSlide7
Why not traditional
reviews
‘
Un
scientific’
rarely pre-specify or make methods explicit
Rarely transparent or reproducible
Usually
qualitative
, subjective,
opinions of individual
Of
ten incomplete
,
filing cabinet
or MEDLINE
review
Difficult
to make sense across groups of studies, especially when conflicting based on qualitative reading aloneSlide8
Hierarchy of evidenceSlide9
Who undertakes systematic reviews?
Cochrane/Campbell Collaboration
NICE/Regulatory bodies
Health Technology Assessment
Academics/researchers/Clinicians
MSc/PhD studentsSlide10
Who undertakes systematic reviews?
Multidisciplinary teams
Clinicians
Health services researchers
Information scientists
Statisticians
Health Economists
Patient and public involvement – particularly for guidelinesSlide11
Key Stages in a Systematic Review- the process
Define research/review question
In consultation/collaboration with the clinical community, commissioners and patient/public representatives
Identify relevant studies
Develop a comprehensive search strategy and undertake systematic searches of the literature
Assess eligibility
Select those studies which meet the pre-defined inclusion criteria
Data extraction /checking
Develop data extraction from into which study
information
and
outcome data
can be extracted, checked
& verified
Synthesis
Narratively and/or statistically summarise/describe the data, exploring similarities
and differences
between studies.
Develop review protocol
Pre-specify
the type
of studies to be included
, the
methods of
collating, appraising and analysing data
Knowledge translation
Review details and results are disseminated to relevant target audiences using appropriate formats
Study assessment/appraisal
Assess the quality
and
validity of the included studies using the pre-defined method. Slide12
Define research/review question
Questions may be broad or narrow
Well-formulated questions will guide many aspects of the review process
Searching strategy
Inclusion/exclusion criteria
Data extraction
Choice of synthesis method
Presentation/dissemination of findingsSlide13
Quick Activity
Discuss a very broad question and how you might narrow it? (10 mins)
Discuss the potential limitations of your review questions
If time and resource were not a limitation – consider how useful would the answer to your review question be?Slide14
Current guidance
a clear and concise statement of a review's objectives (or questions) is critical and should begin with a precise statement of the primary objective, including the interventions reviewed and the targeted problem; ideally, this would be presented in a single sentence
Cochrane &
Prisma
StatmentSlide15
Current guidance
“To assess the effects of [
intervention or comparison
] for [
health problem
] in [
types of people, disease or problem, and setting if specified
].”
Several criteria/frameworks proposed to help guide question
developmentSlide16
Question formulation
Determining
the scope
is
a decision dependent upon multiple
factors:
P
erspectives
regarding a question’s relevance and potential impact;
Supporting
theoretical, biologic and epidemiological information
;
The
potential generalizability and validity of answers to the
questions;
Available resources;
The wider literature base – has a recent high-quality SR been conducted?Slide17
Question formulation
Advantages
and disadvantages to both broad and narrow
questions
The
validity of very
broad question may
be criticized for ‘mixing apples and
pears’; but advantages might include
Comprehensive summary of the
evidence
Generalizability
of findings
Most obvious advantage of narrow focus is clarity of objectives and ease of reading; but disadvantages might include
Sparse
evidence may limit findings/usefulness
Generalizability of findingsSlide18
Question formulation
Often dealing with complex interventions
Might be a need to develop working definitions of the intervention of interest
Several options on how to do this (pragmatic real world v theoretical, logic models, etc.)
Use content experts outside the review team to ensure that the resulting definitions are likely to be robust and meaningfulSlide19
Protocol Development
A protocol is an essential component of the systematic review process
Helps to ensure careful
a priori
planning
Consistency
Transparency
Integrity
Integral part of the process for leading organisations/publication processSlide20
Protocol Development
One of the features that distinguish a systematic review from a narrative review is the pre-specification of criteria
Inclusion
Exclusion
Methods
Outcomes to be synthesised
Etc.Slide21
PROSPERO – CRD initiative
Search for existing current reviews
Register their planned review online
Publish protocol online
Update record on Prospero website as the review progresses
Avoids duplication of reviewsSlide22
Searching for Information
Types of Studies (RCTs, non-RCTs, cohort/case-controlled)
Population and setting
Interventions
Outcome measures
Cochrane Handbook and CRD Guidelines
Both provide explanations re the difference study designs, likely biases and issues to consider when including them
www.york.ac.uk/inst/crd/pdf/Systematic_Reviews.pdf http://handbook.cochrane.org/Slide23
Searching for Information
MESH terms and key words/synonyms Medical Subject Heading – controlled vocabulary thesaurus used for indexing articles
young;
adoles
*; teen*; child*...................
*
end of the ‘stem’ of the word it will automatically search for all the endings for that word stem
Child* will also return children, childbearing, childbirth and so on…Slide24
Searching for Information
Word variantsAIDSacquired immunodeficiency syndromeacquired immuno-deficiency syndrome
acquired immune deficiency syndrome
acquired immune-deficiency syndrome
Synonyms
e.g. Newborn: infant, toddler, baby, etc.
Plurals
e.g. child : children OR teenager : teenagersSpelling variants (UK vs US) e.g. randomise/randomizeSlide25
Searching for Information
Where to searchElectronic databases: Medline, Embase, Cochrane, PsycInfo, etc.
Grey literature, dissertations, theses, conference proceedings, national bodies (NICE, HTA), clinical trial database (
www.clincialtrails.gov/
)
Look at the databases own guidance for searching they vary!Slide26
Boolean operatorsSlide27
Searches in medline
2006-2010 database
Alcohol – subject (all subheadings)
5565
Alcohol – subject or keyword
12052
([young or
adoles
* or teen*] [review or system*]
19180
Rows above combined with
or
6791Slide28
More specific – get help!Slide29
Selection of Studies
Reference manager software packageEndnote – RefMan – ProCite – MendeleyImport results and screenAssess titles/abstracts against your predetermined criteria
If in doubt include
Retrieve full text articles of initial selections
Assess full text for inclusion
Requires judgement (>1 reviewer)
Check reviewer agreement (3
rd review to resolve)Use a selection form to ensure consistency and record decisionsSlide30
Data
Extraction/Quality AppraisalSlide31
Data Extraction
Be clear what information you want
from the
studies:
Study details
Data for
your analysis
Information will need to be collected relating to:
Methodology
Population
Interventions being compared
Outcomes evaluated Slide32
Give consideration to….
What effect measures
you are
you going to
calculate
What data do you need to do this?
How
are
you planning to group studies for
the analysis?
By intervention?
By study design?
What
information do you need to
extract to enable you to organise
and analyse
the
way you want
?
REMEMBER YOUR PROTOCOL – IT IS YOUR ROADMAP, FOLLOW IT!Slide33
How much to extract??
Level of judgement is required
Sufficient to describe studies
Sufficient to allow you to undertake the planned analysis
Sufficient so you do not need to return to the full text papers
However
You need to limit unnecessary detailSlide34
Data extraction software?
There is a wide selection of software to choose from
Selection depends on a number of factors
Main
considerations are
probably
What are you
are
familiar with?
What package best suits your data?
How many included studies do you have?Slide35
Which software?
Word
Excel
Access
EPPI reviewer
COEVIDENCE
REVMAN
????Slide36
Consistency/Standardisation
We all have to be doing the same thing
Essential >one
reviewer is extracting
data
Data must be interpreted in the same way by all reviewers
Independent piloting of
data
extraction
forms –
always
one standardised form
Regular discussion of progress/disagreements
Regular comparison of
data
extraction – don’t wait till the endSlide37
Efficient data extraction
Once data extraction is complete you may need to:
Sort/search
your data
Filter data
Calculate frequencies
Transform data (e.g. SE to SD)
Categorising/coding data will make these tasks
easier
Needs to be implemented with consistency by the whole team
A database can be designed to have this
functionalitySlide38
Things to consider
Are you including more
than one study
design?
You may need separate forms for each study design
However, you
are still answering the same
question, so make
sure the core information extracted is the same
Have one or a few studies reported data differently from the others?
Will the data still be
useful?
Should you include it?
Make sure the core information extracted is the same
You
may
need to update the form, or have more than one
form
Any changes need to be agreed and made consistentlySlide39
Stay on track……
Be careful about collecting ‘extra’
data
It is very tempting
to collect data that are not directly relevant to the review question
The data needed to answer the review question should have already been
decided (REMEMBER YOUR PROTOCOL)
Collect data for good reasons – stay focused and don’t get side-tracked
Time and
effort to collect, only to find it is not usefulSlide40
Quality Assessment & Critical Appraisal
Why bother????
What are we trying to achieve?
Not all published and unpublished literature is
rigorous!
being in a journal doesn’t mean it is good
Quality
may be used as an explanation for differences in study results
or to guide interpretation
of
findings, strength
of inferencesSlide41
Quality Assessment & Critical Appraisal
Quantitative studies
Internal validity
Bias: selection
;
performance
;
detection
;
attrition
;
reporting
External validity
Move away from checklists/numerical scores to domain based assessment
Cochrane Risk of Bias - RCTs
QUADAS 2 – diagnostic accuracy
ROBIS for systematic reviewsSlide42
Quality Assessment & Critical Appraisal
Qualitative studies
Three broad categories
Rigour
: has a thorough and appropriate approach been applied to key research methods in the study?
Credibility
: are the findings well presented and meaningful?
Relevance
: how useful are the findings to you and your organisation? Slide43
CASP appraisal checklist
Clear
aims of research (goals, why it is important, relevance)
Appropriate methodology
Sampling strategy
Data collection
Relationship
between researcher and participants
Ethical
issues
Data
analysis
8
. Findings
9
.
Value
of research (context dependent) Slide44
Data Synthesis
Building up; putting together; making a whole out of the parts; the combination of separate elements of thought into a whole; reasoning from principles to a conclusionSlide45
Data Synthesis
Results from different studies need to be synthesisedAre studies and results similar enough to be combined into a single numerical result?NO – qualitative descriptive/narrative summaryYES – quantitative meta-analysis
Heterogeneity
Difference in results can arise due to differences in study design, population, selection, intervention delivery
How similar is similar? Results from heterogeneous studies should not be pooledSlide46
Narrative synthesis
Instead of/alongside meta-analysisPotential bias in presentationLack of a take home messageSlide47
Tools for narrative synthesis
Partly informed by methodological work in qualitative synthesisTabulationGroupings and clustersVote counting as a descriptive tool
Examination of moderator variables (elements of e.g. setting, population)
Rodgers et al Evaluation 2009 15 49-72Slide48
Meta-analysis/forest PlotSlide49
Most important thing:
Be organised!!!Slide50
Gantt chartsSlide51
Reference management
Use a reference manger to sift and storeKeep all citations retrieved
Add in those you can’t download
Use to de-duplicate results
Sift citations for inclusion/exclusion
Can use codes/notesSlide52
Version control
Dates – YYYYMMDDVersion numbering v0.1 = first draft v1.0 = final versionv1.1 = minor amendments to final version
v2.0 = major revision
Avoid using draft, draft 1, final final, etc.
Clear naming convention
E.g. Date_project_title
20/11_Autism HTA_resultsv1.2.docSlide53
Other help available
FMS Systematic review group Informal monthly session where methods are discussed and issues can be raised
fiona.beyer@ncl.ac.uk
or
jenni.hislop@ncl.ac.uk
alternatively you have my contact details
MSc in Public Health and Health Services Research (~October 2016)
10 credit module ‘Introduction to systematic reviewing and critical appraisal’pghealth@newcastle.ac.ukSlide54
Thanks for your attention!
Contact details:
Dawn Craig
Institute
of Health &
Society
dawn.craig@ncl.ac.uk @dawn_craig