Thematic analysis Part 4: Avoiding common problems

Lecture overview Part 1: What is thematic analysis?
Part 2: Thematic analysis is uniquely flexible
Part 3: Six phases of reflexive thematic analysis
Part 4: Avoiding common problems<br>

Conceptualising the process and researcher positioning You are not a mechanic…
Proceduralism vs. active, reflexive, deliberative and systematic engagement in a flexible, recursive process.
TA is a ‘recipe’ for starting your (reflexive, knowing) adventure!
Not a prescriptive recipe that must be followed exactly as specified to guarantee a good outcome.<br>

Some common problems we encounter… TA fails to address the research question.
Unconvincing or under-developed analysis:
Too many or too few themes (the former is more common).
Too much overlap between themes or themes are unrelated.
Themes are vague or not internally consistent or coherent.<br>

Some common problems – slide 2 Analysis fails to provide a rich or ‘thick’ description and interpretation of the data.
Little or no analytic work has been carried out (e.g. using data collection questions as ‘themes’).
The data are not contextualised, or acknowledged as situated or located.
Mismatch between data and analytic claims or extracts are not compelling.
Too few or too many data extracts (and little or no analytic commentary).
Paraphrasing not analysing and interpreting data.
Failure to consider other obvious readings of the data and variation and contradiction in the data.<br>

Some common problems – slide 3 Arguing with the data.
The analysis and theoretical frameworks are contradictory or the analysis is theoretically inconsistent.
In research conducted by psychological practitioners: confusing clinical interpretation with qualitative data analysis.
Confusing themes and topic-summaries (or naming themes poorly).
Unacknowledged and unknowing ‘mash-ups’ of different types of TA procedures and philosophy.<br>

Examples of common problems – confusing topic summaries and themes Bacsu et al. (2012) – exploring the key determinants of healthy aging in rural communities in Canada.
Interviews with 42 seniors.
Eight ‘themes’: housing; transportation; healthcare; finances; care giving; falls; rural communities; and support systems (note the one or two word theme names that only identify the topic not the story).
Each ‘theme’ is a summary of the main points raised around the topic.
‘Themes’ rather thin – not rich and multifaceted; each theme = small number of observations.
E.g. falls are a barrier, one-third experienced falls and sustained injuries, seriousness of injuries are downplayed. Latter observation related to existing literature – cultural values may influence downplaying and under-reporting of fall related injuries.<br>

Falls (Bacsu et al., 2012) “Falls and fall related injuries were key barriers that emerged from the interviews with the rural seniors. Approximately, one-third of women (39%) and men (31%) had experienced falls with injuries which ranged from physical injuries to functional limitations. Some of the respondents’ comments made in relation to their injuries included: “I fell on the ice and had a concussion”, “I fell, I tripped over my cane and I fell on the sink but not here in my house, hit my hand here and had two black eyes, that was it.” Respondents tended to downplay the severity of injuries. As one respondent commented, “I fell backwards down the steps, and I don’t feel any of the effects of that. … He [doctor] said I cracked a bone in my back here.” None of the respondents identified falls where they did not sustain injuries. The finding to downplay the severity of injuries might be indicative of a larger issue related to seniors underreporting falls (Allen, 2004; Roe et al., 2009). To date, there is limited research that examines how the rural context influences seniors’ health care decisions especially in relation to reporting fall and fall related injuries. However, within the rural context, cultural values of hardiness, independence and self-efficacy (Craig, 1994) may significantly influence older adults’ decision to down-play or not report a fall or fall-related injury.” (p. 83)<br>

Examples of common problems – unknowing ‘mash-ups’ "The first author closely read a subset of transcripts and created an initial codebook of themes, using the thematic analysis approach outlined by Braun and Clarke [28]. The creation of the codes was an iterative process. Two transcripts were read, and a codebook was created; after this, the additional transcripts were read, and the codebook was changed as necessary to better fit the themes that emerged... Researchers met to discuss the coding of each of the small-group discussions, and discrepancies in coding were dealt with by the researchers coming to consensus." (Roditis & Halpern-Felsher, 2015, pp. 180-182)<br>

Activity 1: Reflecting on examples of common problems Pause the recording and read the two examples of ‘mash-ups’ and the example of a thematic structure on the following slides. For the ‘mash-ups’, reflect on what is problematic in the described analytic process. What evidence is there of unknowing ‘mash-ups’? For the thematic structure, reflect on the conceptualisation of themes and the number of themes reported.
Our reflections follow.<br>

More ‘mash-ups’ – slide 1 “Analysis of the open-ended responses pertaining the reasons for emoticon usage was undertaken using thematic analysis. In line with Braun and Clarke (2006) analytic strategy the written responses were repeatedly read by two naïve coders to enable familiarity with the responses. Subsequently, each coder independently identified initial themes, which were then scrutinised by identifying codes within the data. A theme was determined as characterising a response pattern in which the phenomena of interest is at least described within the narratives (Boyatzis, 1998). From this, it is possible to gain an insight into the extent of frequency of each theme, to enable additional quantitative analysis. Following this, the data was reviewed again in line with these themes to increase the validity of the coding procedure. This review process was undertaken concurrently with the development of a “codebook” as a means of organising the narratives for a more full interpretation (Crabtree & Miller, 1999). Finally, relevant extracts were selected to represent these.” (Kaye et al., 2016, p. 464)<br>

More ‘mash-ups’ – slide 2 “Thematic analysis was selected because it unearths patterns in the data, allowing for the discovery of the true meaning of the data (Boyatzis, 1998; Grbich, 2007). This involved repeated listening to the digital recordings of the focus groups, and reading the transcripts to become familiar with the data collected; comparing and contrasting this data; identifying, analysing and reporting common themes within the data; coding and naming these themes; and reporting the results of this analysis (Boyatzis, 1998; Green and Thorogood, 2005; Braun and Clarke, 2006). To help reduce the potential for bias, all four team members individually undertook a thematic analysis, and then met together to compare results, resolve any points of divergence in interpretation of the data, and reach consensus with the themes. On completion of this process, the team reached agreement and all concurred that the resultant well-defined themes accurately represented the data collected. This process was crucial to establishing dependability of the themes (Boyatzis, 1998).” (Stevens et al., 2011, p. 511)<br>

EG of a thematic structure – slide 3 Lewis et al. (2012) - the perceived benefits and disadvantages of non-invasive prenatal diagnosis (NIPD) for fetal sex determination.
Interviews with 44 participants. “Three themes emerged” but 19 themes reported:
Technical aspects of technology/testing procedure (slightly different themes used at different points) – the following headings also described as themes: Safety, Ease of testing procedure.
Timing: Preparation, Control, Peace of mind, Delayed attachment to pregnancy, Normalisation of pregnancy.
(Enhanced) decision-making: Continue without further testing, Invasive testing if necessary, Undecided.
Disadvantages of NIPD: NIPD increases bonding at a time when miscarriage risk is high/miscarriage risk, Increased anxiety, Connection to a fetus when the pregnancy may be terminated, (Being) robbed of a surprise, Misuse of technology.<br>

Lewis et al. (2012)’s thematic structure – slide 4 This is the Safety theme reproduced in full:
“Safety. Safety was found to be a key value of NIPD. For those women who had experienced both non-invasive and invasive testing, the advantage of NIPD was that the procedure posed no risk to the mother or fetus.‘Having an amniocentesis, obviously there’s a risk of miscarriage. With the non-invasive blood test there’s none of those associated risks so it’s much better.’ Di – CMT carrierDecision-making around invasive testing was identified as a distressing choice participants had to make owing to the associated risk of miscarriage.
‘Being told that you are doing something that could risk the pregnancy, it’s your choice and you are choosing to have a test, that’s quite distressing.’ Zoe – X-SCID carrierIn comparison, no such emotion was experienced by participants when deciding whether to use NIPD.” (p. 1128)<br>

Our reflections on the ‘mash-ups’ – slide 1 Example 1 (Kaye et al., 2016)
B&C cited for familiarisation but confusingly suggests B&C recommend “naïve coders” when this notion is strongly associated with positivist forms of TA and a desire to control for “researcher bias”.
Coders working independently – again associated with positivist forms of TA.
Analysis moves from themes to codes, as in codebook and coding reliability TA.
Frequency of themes determined.
Codebook developed for data charting.
Seems to be an unacknowledged and unexplained ‘mash-up’ of reflexive TA, coding reliability and codebook.
We question whether the authors understand the philosophical and procedural differences in the approaches they have combined.<br>

Our reflections on the ‘mash-ups’ – slide 2 Example 2 (Stevens et al., 2011)
Assumption that data have a true meaning/truth resides within data and themes can be an accurate representation of data, rather than an emphasis on researcher subjectivity and interpretation.
Description of analytic process seems to be a ‘mash-up’ of the different approaches cited but no justification/rationale for this or acknowledgement of fundamental differences.
Positivist concern for reducing bias – independent coders, consensus coding.<br>

Our reflections on the thematic structure – slide 3 Example 3 (Lewis et al., 2012)
Lots of themes! All rather thin, and closer to codes than multifaceted, rich and complex themes (e.g. Safety consists of two analytic observations – around safety and distress).
There’s a lack of clarity around the thematic structure – a table/map would have clarified things. Confusing when theme names change slightly.
Themes are often poorly named – what about safety? What’s the story here?
Many ‘themes’ are closer to topic summaries than fully realised, shared meaning themes. The ‘sub-themes’ (not described as such, just as ‘themes’) is where there is more evidence of shared meaning.
A method like framework analysis or template analysis might have been a better fit – as these emphasise a hierarchal thematic structure and the conceptualisation of themes as topic summaries.<br>

Good practices in reporting a TA A good balance between analytic narrative and data extracts (half and half is a useful starting guide – but not a rule!).
The analytical commentary provides original and novel insights into the meaning of the data.
There is a good fit between the data and analytic claims.
Each theme has a clear central organising concept and is distinctive.
An appropriate number of themes are presented – analysis is not fragmented and ‘undercooked’.<br>

Good practices in reporting a TA – slide 2 Each theme is discussed in sufficient depth and detail.
The themes work together to tell a story about the data.
The analytic claims fit with the overall theoretical position of the analysis and are consistent.
There is strong evidence of a systematic and thorough analytic process.
The analysis explains why the data are interesting and important in relation to the research question.<br>

Good practices in reporting a TA – moving towards more complex TA The analysis moves beyond description to interpretation and conceptual engagement.
The analysis provides a critical analysis of the assumptions underpinning the data and the implications of the data.
The data is contextualised, and acknowledged as situated and located.
The analysis captures some of the different ‘stories’ in the data and considers some other plausible interpretations of the data.
The analysis makes an argument.<br>

Our 15-point checklist for good quality TA (Braun & Clarke, 2006) 1-5 and 11 for reflecting on your own practice; 6-10, 12-15 can be used to both reflect on your own practice and evaluate reports of TA.
Transcription – appropriate level of detail and ‘accuracy’.
Coding – each data item given equal attention.
Coding – thorough, inclusive and comprehensive.
Coding – all relevant extracts for each theme have been collated.
Coding – themes have been checked against each other and the original data set.
Coding – themes are internally coherent, consistent and distinctive.
Analysis – data have been interpreted/analysed rather than just paraphrased or described.<br>

Our 15-point checklist continued… Analysis and data match, extracts illustrate analytic claims.
Analysis – tells a convincing and well-organised story about the data and topic.
Analysis – good balance between extracts and analytic narrative.
Overall – enough time allocated to complete all phases adequately, no rushing or once-over-lightly.
Written report – assumptions and specific approach explicated.
Written report – good fit between what claim to do and what have done.
Written report – language and concepts consistent with theoretical position.
Written report – the researcher is active in the research process.<br>

Reporting standards for reflexive TA (Braun & Clarke, 2021) Use a writing style that brings your voice into the text (e.g. first person in the methodology).
Introduction provides a contextualisation and rationale for the research – referencing research, theory and the wider context. Not necessarily a conventional literature review and not about finding a “gap” in the literature.
Clearly articulated and conceptually aligned research question.
Methodology rather than method – demarcate the conceptual underpinnings of the research and the orientation to TA, provide evidence of reflexivity, richly situate the participant group/dataset, rationale for dataset size (not saturation!) and constitution, describe your analytic process in a situated and specific way.<br>

Reporting standards – slide 2 Analysis rather than results or findings – start with a brief overview of the analysis, the analytic narrative should explain the meaning and significance of the data, themes should tell a story about the data.
Discuss 2-6 themes (including subthemes) in a single report – each theme should be rich, complex and multifaceted and organised around a central concept.<br>

Reporting standards – slide 3 Use subthemes judiciously to highlight an aspect of the central concept of the theme – an overly elaborate thematic structure can be a sign of a fragmented and underdeveloped analysis.
Theme names should convey something of the “essence” of each theme, one word theme names should be avoided.
Balance of data extracts and analytic narrative – select vivid and compelling data extracts from across the dataset.
Conclusions should arise from or cut across the themes – themes are not analytic conclusions.
Theme-by-theme contextualisation should be avoided (this indicates the analysis and discussion should have been combined).<br>

A tool for evaluating thematic analysis (TA) manuscripts for publication: Twenty questions to guide assessment of TA research quality (Braun & Clarke, 2020). Adequate choice and explanation of methods and methodology1. Do the authors explain why they are using TA, even if only briefly?2. Do the authors clearly specify and justify which type of TA they are using?3. Is the use and justification of the specific type of TA consistent with the research questions or aims?4. Is there a good ‘fit’ between the theoretical and conceptual underpinnings of the research and the specific type of TA (i.e. is there conceptual coherence)?5. Is there a good ‘fit’ between the methods of data collection and the specific type of TA?<br>

Evaluation tool slide - 2 6. Is the specified type of TA consistently enacted throughout the paper? 7. Is there evidence of problematic assumptions about, and practices around, TA? These commonly include:
Treating TA as one, homogenous, entity, with one set of – widely agreed on – procedures.
Combining philosophically and procedurally incompatible approaches to TA without any acknowledgement or explanation.
Confusing summaries of data topics with thematic patterns of shared meaning, underpinned by a core concept.
Assuming grounded theory concepts and procedures<br>

Problematic assumptions and practices continued (e.g. saturation, constant comparative analysis, line-by -line coding) apply to TA without any explanation or justification.
Assuming TA is essentialist or realist, or atheoretical.
Assuming TA is only a data reduction or descriptive approach and therefore must be supplemented with other methods and procedures to achieve other ends.
8. Are any supplementary procedures or methods justified, and necessary, or could the same results have been achieved simply by using TA more effectively?
9. Are the theoretical underpinnings of the use of TA clearly specified (e.g. ontological, epistemological assumptions, guiding theoretical framework(s)), even when using TA inductively (inductive TA does not equate to analysis in a theoretical vacuum)?<br>

Evaluation tool continued – slide 3 10. Do the researchers strive to ‘own their perspectives’ (even if only very briefly), their personal and social standpoint and positioning? (This is especially important when the researchers are engaged in social justice oriented research and when representing the ‘voices’ of marginal and vulnerable groups, and groups to which the researcher does not belong.)11. Are the analytic procedures used clearly outlined, and described in terms of what the authors actually did, rather than generic procedures?12. Is there evidence of conceptual and procedural confusion? For example, reflexive TA (e.g. Braun and Clarke 2006) is the claimed approach but different procedures are outlined such as the use of a codebook or coding frame, multiple independent coders and consensus coding, inter-rater reliability measures, and/or themes are conceptualised as analytic inputs rather than outputs and therefore the analysis progresses from theme identification to coding (rather than coding to theme development).<br>

Evaluation tool continued – slide 4 13. Do the authors demonstrate full and coherent understanding of their claimed approach to TA?A well-developed and justified analysis14. Is it clear what and where the themes are in the report? Would the manuscript benefit from some kind of overview of the analysis: listing of themes, narrative overview, table of themes, thematic map?15. Are the reported themes topic summaries, rather than ‘fully realised themes’ – patterns of shared meaning underpinned by a central organising concept?
If so, are topic summaries appropriate to the purpose of the research?
If the authors are using reflexive TA, is this modification in the conceptualisation of themes explained and justified?
Have the data collection questions been used as themes?<br>

Evaluation tool continued – slide 5 Would the manuscript benefit from further analysis being undertaken, with the reporting of fully realised themes?
Or, if the authors are claiming to use reflexive TA, would the manuscript benefit from claiming to use a different type of TA (e.g. coding reliability or codebook)?
16. Is non-thematic contextualising information presented as a theme? (e.g. the first 'theme' is a topic summary providing contextualising information, but the rest of the themes reported are fully realised themes). If so, would the manuscript benefit from this being presented as non-thematic contextualising information?17. In applied research, do the reported themes have the potential to give rise to actionable outcomes?<br>

Evaluation tool continued – slide 6 18. Are there conceptual clashes and confusion in the paper? (e.g. claiming a social constructionist approach while also expressing concern for positivist notions of coding reliability, or claiming a constructionist approach while treating participants’ language as a transparent reflection of their experiences and behaviours)19. Is there evidence of weak or unconvincing analysis, such as:
Too many or two few themes?
Too many theme levels?
Confusion between codes and themes?
Mismatch between data extracts and analytic claims?
Too few or too many data extracts?
Overlap between themes?
20. Do authors make problematic statements about the lack of generalisability of their results, and or implicitlyconceptualise generalisability as statistical probabilistic generalisability (see Smith, 2017)?<br>

Activity 2 – Evaluate a published TA paper Use the common problems, good practices, the relevant points from the 15-point checklist, and the evaluation tool to critically evaluate an example of published TA. You can use the example we have provided* or select your own.
What has been done well? Where – if at all – is there scope for improvement?
Hold in mind the constraints of journal publishing – limited word counts, stylistic norms etc.
*Fahs, B. (2014). Perilous patches and pitstaches: Imagined versus lived experiences of women’s body hair growth. Psychology of Women Quarterly, 38(2), 167–180.<br>

References for part 4 – slide 1 Bacsu, J. R., Jeffery, B., Johnson, S., Martz, D., Novik, N., & Abonyi, S. (2012). Healthy aging in place: Supporting rural seniors’ health needs. Online Journal of Rural Nursing and Health Care, 12(2), 77–87.
Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77-101.
Braun, V., & Clarke, V. (2020). One size fits all? What counts as quality practice in (reflexive) thematic analysis? Qualitative Research in Psychology. Advance online publication.
Fahs, B. (2014). Perilous patches and pitstaches: Imagined versus lived experiences of women’s body hair growth. Psychology of Women Quarterly, 38(2), 167–180.
Kaye, L. K., Wall, H. J., & Malone, S. A. (2016). “Turn that frown upside-down”: A contextual account of emoticon usage on different virtual platforms. Computers in Human Behavior, 60, 463-467.<br>

References for part 4 – slide 2 Lewis, C., Hill, M., Skirton, H., & Chitty, L. S. (2012). Non-invasive prenatal diagnosis for fetal sex determination: Benefits and disadvantages from the service users’ perspective. European Journal of Human Genetics, 20, 1127–1133.
Roditis, M. L. & Halpern-Felsher, B. (2015). Adolescents’ perceptions of risks and benefits of conventional cigarettes, e-cigarettes, and marijuana: A qualitative analysis. Journal of Adolescent Health, 57, 179-185.
Smith, B.(2018). Generalizability in qualitative research: Misunderstandings, opportunities and recommendations for the sport and exercise sciences. Qualitative Research in Sport, Exercise and Health, 10(1), 137-149.
Stevens, J., Dahlen, H., Peters, K., & Jackson, D. (2011). Midwives' and doulas' perspectives of the role of the doula in Australia: A qualitative study. Midwifery, 27(4), 509–516.<br>