Workshop: Thematic Synthesis and Framework Synthesis - PowerPoint Presentation

Slide1

Workshop: Thematic Synthesis and Framework Synthesis

Parts 1-4 – Data Extraction, Quality Assessment, Synthesising Across Studies,

Completing the AnalysisSlide2

Workshop: Framework Synthesis, Meta-Ethnography and Realist Synthesis Slide3

Shared Topic: Adherence to Antiretroviral therapy (ART) for HIV in

Zambia

BACKGROUND:

Antiretroviral

therapy (ART) has significantly improved

morbidity

and mortality of individuals infected with

HIV.

However, lack of adherence to highly active

antiretroviral therapy

(HAART) remains

a key

challenge to

successful management

of patients with HIV/AIDS.

Adherence

rates lower than 95% are associated with

development of

viral resistance to antiretroviral

medications.

‘Efforts

to sustain

adherence in

Africa and elsewhere remain important goals to

optimize outcomes

for individuals and global HIV treatment.’

(Mills

,

Nachega

,

Buchan

,

Orbinski

,

Attaran

, Singh et al., 2006).

Slide4

Shared Topic: Adherence to Antiretroviral therapy (ART) for HIV in

Zambia

Different Emphases

Barriers and Facilitators to ART

(Framework

Synthesis)

Theory Explaining Adherence to ART

(Thematic Synthesis) Slide5

Reading and Data ExtractionSlide6

Data extraction

What is it?

An attempt to reduce a mass of material (your included papers) to a much smaller body of text and numbers, amenable to analysis and the interpretation of findingsSlide7

Data extraction form

Location

Setting

Sample (n)

Age

Gender

Ethnicity

Socio-economic status

Intervention (if any)

Quality assessment criteria

Results?

Further citations

See

Handouts

1-3Format?????Slide8

What results do you extract?

What is your question?

Keep the question in mind as you read

:

Are the data relevant to this question?

Is the question answered by the data?Slide9

Data Extraction

Framework

Synthesis

Thematic Synthesis

Extracts

data against framework. Coding framework with definitions

provided

to increase consistency. Data not explained by framework

is “

parked” for subsequent inductive stage. Distinction typically made between original

data extracts and

author’s analysis.

Key themes and concepts extracted and

reviewed

for inclusiveness. Distinction preserved between original (participant) extracts and (author’s analysis) findings. Findings coded in duplicate. Discrepancy between codes resolved by third person. Slide10

Quality Assessment

Andrew Booth, Reader in Evidence Based Information Practice, Co-Convenor – Cochrane Collaboration Qualitative Methods Group Slide11

Before You Begin…

Consider how you will use judgements of quality (cp. 50% of published Cochrane

Quantitative

Reviews performed quality appraisal but did not make it clear how judgements were used!)

To exclude or to moderate?

Will chosen instrument militate against certain types of research?

Quality of

reporting

or quality of

study

? Slide12

Variability in Practice - 1

21 papers did not describe appraisal of candidate studies

6 explicitly mentioned not conducting formal appraisal of studies

5 papers did a critical appraisal, but did not use a formal checklist

7 described modifying existing instruments

1 used an existing instrument without modification

Dixon-Woods, Booth & Sutton (2007)Slide13

Variability in Current Practice - 1

23 papers

did not describe

critical appraisal

5

papers explicitly pleaded against

quality

assessment of papers or provided

valid

reason for not conducting

quality

appraisal.

Criteria

used varied between detailed descriptions of relevant items in existing or modified checklists to a set of broad criteria evaluating, for example,

rich description of data, credibility or relevance of the original study.

Hannes

and

Macaitis

(2012)Slide14

Variability in Current Practice - 2

One

team used

overall

judgement

(

Smith et al., 2005).

Five opted

for

self-developed

assessment

instrument

Three

used

previously developed checklists to create own. Two mentioned critical appraisal, but did not specify tool. Most used existing instruments/frameworks. 24 different assessment tools identified: Critical Appraisal Skills Programme

(

CASP) (

n

=

18)

Mays

and Pope criteria

(

n

=

6)

Popay

criteria

(

n

= 6)

Joanna

Briggs Institute

(

n

= 4).

Hannes

and

Macaitis

(2012)Slide15

Appraising research quality

1. Epistemological criteria:

Judgement of ‘trustworthiness’ requires criteria tailored to different research ‘paradigms’.

2. Theoretical Criteria:

Explicit theoretical framework shaping the design of the study and informing claims for generalisability

3. Prima facie ‘Technical’ criteria:

Used to assess ‘quality’ common to all research traditions e.g.:

Sufficient explanation of background;

Method appropriate to question;

Succinct statement of objectives/research questions;

Full description of methods include approach to analysis;

Clear presentation of findings including justification for interpretation of data etc.

Noyes J (2005)Slide16

Two dimensional approach to appraising qualitative research

Technical markers –CASP

Epistemological and theoretical markers – Popay et al

Technical Quality High

Description – thicker

Privileges Subjective experience and meanings

Use of theory to build explanations

Technical Quality Low

Description - thinner

Imposed pre-determined framework on respondents narratives.

Limited/no/inappropriate use of theory, little explanatory insight

(Noyes, 2005)Slide17

Available Tools - 1

CASP –

10 questions to help you make sense of qualitative research

http://www.casp-uk.net/wp-content/uploads/2011/11/CASP_Qualitative_Appraisal_Checklist_14oct10.pdf

Joanna Briggs Institute -

Critical Appraisal Checklist for Interpretive & Critical Research

http://www.jbiconnect.org/agedcare/downloads/QARI_crit_apprais.pdf

National Centre for Social Research.

Quality in Qualitative Evaluation: A Framework for Assessing Research Evidence.

London: National Centre for Social Research/UK Cabinet Office, 2003

http://www.civilservice.gov.uk/wp-content/uploads/2011/09/a_quality_framework_tcm6-38740.pdfSlide18

Available Tools - 2

Dixon-Woods M, Shaw RL, Agarwal S & Smith JA (2004) The problem of appraising qualitative research.

Quality & Safety in Health Care,

13, 223-5.

Hannes K, Lockwood C, Pearson A (2010). A comparative analysis of three online appraisal instruments' ability to assess validity in qualitative research.

Qualitative Health Research

. 20(12):1736-43.

Popay J, Rogers A & Williams G (1998) Rationale & standards for the systematic review of qualitative literature in health services research.

Qualitative Health Research,

8, 341-51.

Seale C & Silverman D (1997) Ensuring rigour in qualitative research.

European Journal of Public Health, 7, 379-84. Slide19

CASP – Technical/Procedural ToolSlide20

JBI – Theoretical

Tool Slide21

Key Issue

How are you going to use the quality assessment?

From quantitative assessment we know authors frequently say they do it – but they don’t incorporate it into results

Is it technical proceduralism gone mad?

Or can we use the assessments to improve our synthesis and subsequent interpretation?Slide22

Quality Assessment

Framework

Synthesis

Thematic Synthesis (e.g. as first stage of Meta-Ethnography)

Pragmatic so tends to include all studies

. Focuses explicitly on

quality of reporting.

Qualitative sensitivity analysis

used to test robustness of synthesis.

Quality Assessment as

Hurdle (often used when plenty of studies to draw upon):

Studies

using qualitative design and analysis

method included. Studies assessed for relevance first to continue to full-text review. Studies passing quality appraisal (are retained. Slide23

Data SynthesisSlide24

What is Data Synthesis?

Process of moving from focus on single studies (cp. Data Extraction and Quality Assessment) to focus on cross-study analysis

Requires identification of patterns across data, including contradictory findings and data that does not fit

Iterative and requires

ongoing

refinement

Acts as prelude to Analysis which seeks to explain patterns, contradictions and differences Slide25

Thematic synthesis; Critical Interpretive Synthesis; Meta-ethnography

Only include “good” qualitative studies (?)

Constant comparison; iterative; interpretations generated from the data by reviewers

Create a theory

Inductive

(theory-generating)

Examples:

Thomas J, Harden A

. Methods for the thematic synthesis of qualitative research in systematic reviews.

BMC Medical Research Methodology

2008

; 8.

Campbell R et al

. Evaluating meta-ethnography: a synthesis of qualitative research on lay experiences of diabetes and diabetes care. Social Science & Medicine 2003; 65:671-684.

Methods

of

qualitative

evidence

synthesisSlide26

Framework synthesis:

Only include “good” qualitative studies (?)

Map data from included studies onto an existing framework to test the framework/theory (a role for theory)

Build a conceptual model or framework

Deductive

(theory-testing)

Examples:

Oliver S

et al: A multidimensional conceptual framework for analysing public involvement in health services research.

Health Expectations 2008

, 11:72-84.

Brunton

G, Oliver S, Oliver K,

Lorenc

T. A Synthesis of Research Addressing Children’s, Young People’s and Parents’ Views of Walking and Cycling for Transport London. London, EPPI-Centre, Social Science Research Unit, Institute of Education, University of London; 2006.

Methods of qualitative evidence synthesisSlide27

“Best-fit” framework synthesis

Identify relevant pre-existing conceptual models or frameworks

Identify and extract all relevant qualitative studies satisfying review’s inclusion criteria

Code data from included studies against framework

Use secondary thematic analysis/synthesis to generate completely new themes to supplement the framework’s themes

Create new framework and conceptual model or theory

Deductive

and

Inductive

Framework

and

Thematic synthesis

Carroll C, Booth A, Cooper K

. A worked example of “best-fit” framework synthesis: A systematic review of views concerning the taking of potential

chemopreventive agents, BMC Medical Research Methodology 2011; 11: 29

Methods of qualitative evidence synthesisSlide28

Data Synthesis

Framework

Synthesis

Thematic

synthesis (may be taken forward as

Meta-Ethnography)

Original best fit framework is expanded to include new themes. Relationship between themes is examined and the data is used to reconstitute a new model. Particular attention is directed at discrepant cases.

Second-order constructs pertinent to adherence identified and cross-compared and presented in results section. Key themes consolidated into

line of argument (third-order analysis), presented in the synthesis ⁄ discussion section.Slide29

Booth et al, 2011Slide30

Some Practicalities

Tabulation of data

– looking for and explaining differences (e.g. majority…, split…, exception…)

Post-Its

– arranging according to patterns or clusters

Mapping

e.g. Mind Map, Process Maps (e.g. Pathways of Care), Logic Models

Integration

(with quantitative) – congruence, contradictions, gaps with explanation