Large Language Models for Thematic Analysis in Healthcare Research: A Blinded Mixed-Methods Comparison with Human Analysts
1Primary Care Research Centre, University of Southampton, Southampton, UK
*Corresponding author: Callum Hill, ch1g22@soton.ac.ukAbstract
Large language models (LLMs) are increasingly used for qualitative thematic analysis, yet evidence on their performance in analysing focus-group data, where polyvocality and context complicate coding, remains limited. Given the increasing role of such models in thematic analysis, there is a need for methodological frameworks that enable systematic, metric-based comparisons between human and model-based analyses.
We conducted a blinded mixed-methods comparison of two general-purpose LLMs (ChatGPT-5 and Claude 4 Sonnet), an LLM-based qualitative coding application (QualiGPT), and blinded human analysts on an in-person focus-group transcript informing an AI-enabled digital health proposal. We evaluated deductive coding using a 10-code, 6-theme codebook against an expert consensus adjudication; inductive coding with a structured Likert-scale comparison to a reference-standard set of inductive themes generated by expert consensus; and manual quote verification of LLM segments to define LLM hallucination (evidence absent or non-supportive) and error rate (including partial matches and speaker-coded segments).
During deductive coding against an expert consensus adjudication, large language models (LLMs) yielded a mean agreement of 93.5% (95% CI 92.5–94.5) with κ = 0.34 (95% CI 0.26–0.40); blinded human coders achieved 92.7% (95% CI 91.6–93.9) agreement with κ = 0.34 (95% CI 0.26–0.41). Mean Gwet’s AC1 was 0.92 (95% CI 0.90–0.93) for the blinded human analysis, and 0.93 (95% CI 0.92–0.94) for the LLM-assisted deductive analysis, reflecting high agreement despite the low overall code prevalence (7.8%, SD = 3.2%). Only one model achieved non-inferiority in inductive analysis of the transcript (p = 0.043). The strict hallucination rate in inductive analysis was 1.2% (SD = 2.1%). LLMs were non-inferior to human analysts for deductive coding of the focus-group data, with variable performance in inductive analysis. Low hallucination but significant comprehensive error rates indicate that LLMs can augment qualitative analysis but require human verification.
Article notes
Competing Interest Statement
The authors have declared no competing interest.
Funding Statement
Yes
Introduction
Due to recent developments in data computation and scaling, large language models (LLMs) have advanced significantly since their inception, now demonstrating the capacity to analyse human text and infer both explicit and implicit meaning.[1,2] Understanding whether LLM-assisted analysis can reliably surface patient and clinician-centred requirements that inform clinical decision support design is directly relevant to their deployment within qualitative clinical research settings. There is growing interest in the existing literature surrounding the utility of LLMs in thematic analysis, due to their rapid processing of data, as well as their potential to mitigate intrinsic researcher biases in qualitative research-an important consideration when exploring patient experiences, clinician perspectives, and healthcare decision-making processes. [3–5]
Existing work has demonstrated the performance of LLMs in a range of qualitative analysis tasks. [6,7] Bennis et al demonstrated high researcher agreement with LLMs when analysing survey responses. [8] Kornblith et al compared LLMs against parallel human analysts in both sentiment classification and thematic categorisation, finding comparable inter-human and LLM-human agreement. [9]
Previous studies have focused largely on interview data, with minimal exploration of focus group transcripts. [10–12] To our knowledge, there have been no studies comparing the use of LLMs to human researchers in thematic analysis of focus group data in a healthcare context. Focus groups are widely used in healthcare research to capture co-constructed meaning, patient-clinician interactions, and shared understanding of care processes.
Qualitative coding of focus group data presents several challenges to LLMs, as compared to interview data. Interview transcripts are typically linear, dyadic, and contextually consistent. In contrast, focus groups are polyvocal, featuring a greater prevalence of co-constructed and co-dependent meaning which relies more heavily on interpersonal and contextual meaning. [12] Prior work has suggested that LLMs are more limited in their ability to identify latent meaning, interpersonal dynamics, and contextual meaning. [13] Finally, focus groups can often generate “noise” from tangential dialogue. [12]
Methodological approaches comparing human and LLM interpretations have varied widely. Studies differ in their use of common global coding rules, researcher blinding, and evaluation metrics, ranging from primarily quantitative comparisons to subjective qualitative impressions. [13–16] Input datasets in LLM analyses are often simplified for ease of analysis, meaning that results are not always representative of results in practical analytic tasks. [17] Finally, it is well documented that LLMs “hallucinate” data or text in their responses, as this property has been described as intrinsic to their design. [18,19] Few studies have quantified the effect of this property on qualitative analysis.
This study aims to assess the performance of LLMs using an existing database of qualitative focus group and PPI sessions focused on the development of an AI tool to support physical activity in the management of chronic conditions. To our knowledge, this study represents the first mixed-methods study in a healthcare focus group context, comparing LLMs and humans in thematic analysis with formalised evaluations of inductive, deductive, and error-based performance.
Materials and Methods
Study design, data sources, and participants
We conducted a comparative mixed-methods analysis to evaluate the performance of LLMs and humans in thematic analysis using secondary data of a transcript from a focus group to support the development of an AI-integrated tool to support physical activity recommendations in people living with multiple chronic conditions.
A mixed-methods design was chosen to obtain both quantitative evidence of analytic performance, as well as qualitative insight into areas requiring interpretive nuance. The mixed-methods methodology was chosen to provide methodological triangulation by combining these domains after each was completed to provide an overall synthesis of performance in thematic analysis (Figure 1). This study adheres to GRAMMS (Good Reporting of a Mixed Methods Study) requirements (S1 text). [20]
We used an existing database of focus groups and PPI sessions from a previous programme of work around patient engagement in grant development on the topic of AI tools for physical activity. The details of this work have been reported previously. [21] The focus group was conducted in July 2025 and included seven patient and public contributors aged 18 years and older with multiple long-term conditions able to provide informed consent. The focus group lasted for 2 hours and was conducted in line with National Institute of Health Research (NIHR) guidelines for patient and public involvement.
Data processing and analytic procedures
The dataset was analysed according to a pre-defined analytic framework (S2 text). The recording of the focus group was transcribed by JK, a physician with experience in qualitative methods and AI implementations in healthcare. The transcript was anonymised at this stage, with any personal identifiable information redacted. No raw identifiable information was processed by any LLM. To ensure comparability and reproducibility, all inputs and outputs for the human and LLM analysis were standardised in JavaScript Object Notation (JSON).
We defined a set of reflexive Braun and Clarke aligned global coding rules to be followed by both the LLM and human parallel analytic streams (S2 text)t. [22] Evidential requirements included the need for both verbatim quote and segment ID when creating a code. Researcher segments were not coded. Each transcript segment could be used to support one, several, or no codes. For each analytic task, both human qualitative analysts and LLMs were blinded to each other’s output.
All LLMs were used according to the pre-defined analytic procedure. Claude 4 Sonnet and ChatGPT-5 were accessed via their respective APIs in October 2025. QualiGPT, a role-based prompt framework that repurposes ChatGPT to conduct systematic inductive and deductive coding, was accessed during the same period through the ChatGPT web interface under identical task specifications. [23,24] Default decoding parameters (temperature = 1.0) were retained to reflect typical, real-world model behaviour and to maintain compatibility across API and web-interface environments. Full prompt templates, JSON formatting, and sample outputs are provided (S2, S3, S4 Files), with LLM inputs and data processing files publicly available on GitHub. [25]
We compared blinded human analysts and LLMs to a reference-standard consensus derived by adjudication. An expert adjudication panel was purposively constituted with researchers in our multidisciplinary research group, who had established backgrounds in qualitative methods, and knowledge of applying AI in qualitative analysis (CH, GS, AD).
For the deductive analytic task, 10 codes were developed and agreed upon by an expert adjudication group after reviewing the transcript. These codes were grouped into 6 themes. The codes and themes were based on both prevalence, and utility in assessing the LLM’s performance in both descriptive and inferential domains of analysis.
The transcript was analysed deductively and reviewed by two researchers (CH and GS). Each participant segment could be labelled either, “true” or “false” in each of the 10 codes, for a total of 1240 data values for each analytic iteration of the transcript. Differences in interpretation were adjudicated by a third researcher (AD) to mediate an expert consensus of the transcript. Mean prevalence of the codes in the transcript according to the expert panel analysis was 7.8% (SD=3.2%). The LLMs tested were tasked with analysing the transcript deductively using an identical set of coding rules to the human analysts.
For the inductive task, the expert consensus panel created an adjudicated inductive analysis consisting of 14 codes grouped into 5 themes, reflecting data prevalence and conceptual relevance according to the transcript. Each code was identified according to its label, definition, and supporting evidence.
Both the LLMs and the blinded human analysts were then tasked with performing an identical inductive analysis task according to the pre-defined coding requirements. Strict output formatting in JSON was enforced to enable comparisons between outputs.
Comparative analysis involved comparing the LLMs to humans across the deductive and inductive analytic tasks. The human analysts were blinded to the results of the reference-standard panel analysis and were not involved in drafting the methodology of this study.
We computed and quantified agreement between the LLM outputs and blinded human researchers with the panel human deductive analysis. Agreement was quantified according to several metrics, including percentage agreement, Cohen’s Kappa, Gwet’s AC1, F1, and Jaccard index. [26]
For the comparative inductive analysis, a qualitative researcher compared the LLM outputs to the human inductive analysis. We created a 5-point Likert scale to quantify agreement between the LLM and human analysis (S2 text), with 5 representing a high-level of agreement, and 1 representing poor agreement. The outputs were also reviewed and compared qualitatively to detect differences in interpretation not appreciated by the numerical scale.
Non-inferiority thresholds were specified as 0.03 for AC1, reflecting approximate inter-human variation in pilot deductive analysis of the same transcript, and 0.5 points on a 5-point Likert scale for the inductive analysis, representing an acceptable half-step shift in conceptual framing for thematic analysis. Likert scores (1–5) were treated as approximately interval data for calculation of means and confidence intervals. We additionally assessed superiority in each analytic task for the LLMs. We applied a Holm adjustment to control for multiplicity across the three model comparisons. [27]
Quote verification involved manually parsing through the LLM inductive analysis to identify whether each segment quote given by the LLM output matches with quotes from the transcript. The reviewing researcher also determined at this stage whether the quote given supports the code assigned by the LLM. Each segment given as evidence for a corresponding code was judged to either be a full match, partial match, or a non-match to the transcript and corresponding code, with non-matching segments counted as a hallucination. Instances where the LLMs coded researcher speech were counted as errors rather than hallucinations.
Reflexivity and Ethics
The research team comprised clinicians, qualitative researchers, and researchers with experience in computational methods. The adjudicated panel comprised a senior qualitative researcher (GS), and two other researchers with experience in qualitative research and applications of artificial intelligence (CH, AD). The blinded human analysts included a researcher with experience in qualitative studies and multimorbidity (CP), and a researcher with experience in qualitative research in the context of primary care (DH). To reduce reflexive bias, the human analysts were blinded to the background of the study and the LLM outputs.
Ethical approval was granted by Southampton University Faculty of Medicine ethics committee, ERGO approval number: 106517. Participants gave informed written consent to participate in the grant panel feedback focus group and were separately offered the option to decline inclusion of their transcript in any further research analysis. Data handling, storage, and processing adhered to university policy and GDPR standards.
The transcript was fully anonymised prior to analysis. LLM processing was conducted within secure computing environments approved by the host institution.
Results
Comparative deductive analysis
We compared the performance of two LLMs (ChatGPT-5 and Claude 4 Sonnet), and one LLM-based qualitative analysis model (QualiGPT), against two blinded human analysts using human adjudicated analysis, created by an expert consensus adjudication panel of researchers with qualitative backgrounds and experience in AI applications (Table 1). A total of 6200 binary data values were coded in the qualitative analysis: 2480 by human analysts and 3720 by LLMs. Overall agreement was high across all coders. For the 1240 coded data points for each deductive analysis, mean overall agreement was 92.7% (95% CI 91.6-93.9) for the human blinded analysts, and 93.5% (95% CI 92.6-94.5) for the LLMs. Inter-rater reliability expressed as Gwet’s AC1 was 0.92 (95% CI 0.90–0.93) for human analysts and 0.93 (95% CI 0.92–0.94) for the LLMs. Values of Cohen’s κ were lower with mean values of 0.34 for both groups, representing the attenuation of κ when applied to low-prevalence datasets. Jaccard index was computed as a measure of overlap of positive codes within the dataset and was similar for both the human and LLM analyses.
Non-inferiority and superiority testing (Figure 2A) were performed in the deductive analysis according to a Holm-adjusted, non-parametric, one-sided bootstrap test with segment-level resampling of the difference in AC1 between the LLM results and the mean of human analysts. All LLMs were non-inferior to human analysts (p < 0.0001) relative to the predetermined non-inferiority value (0.03). ChatGPT-5 (p = 0.048) and Claude 4 Sonnet (p = 0.039) achieved statistical superiority over human analysts in the deductive analysis.
Measures of coding accuracy included sensitivity, specificity, and F1, with comparable results between the blinded human analysts and LLMs according to each measure. The assessed LLMs demonstrated a high specificity (0.98, 95% CI 0.97-0.98) when tested on the focus group transcript, suggesting that the LLMs are inclined to interpret the transcript conservatively, rarely assigning codes unless clear textual evidence is present. Sensitivity was lower for both groups, reflecting the challenge of consistently identifying infrequent or implicit themes within the data.
Thematic-level agreement analysis (Table 2) demonstrated variable performance across the codes tested, with Cohen’s κ ranging from –0.05 to 0.54. Human blinded analysts achieved the highest mean concordance within the codes of Privacy and Data Concerns, Consent and Transparency, and Need for personalisation. LLMs demonstrated highest concordance in the codes of Human and AI Interactions in Healthcare, Privacy and Data Concerns, and Consent and Transparency. Lower concordance was observed in themes requiring interpersonal or inferential reasoning such as Patient Expertise and Collaborative Involvement of Patients in Design. Despite these differences, concordance was similar between human analysts and LLMs, indicating broadly similar concordance across all codes.
Comparative inductive analysis
The comparative inductive analysis (Table 3 and S5 file) was conducted by an expert consensus adjudication group. Agreement was assessed qualitatively and using a 5-point Likert scale (S2 text), where 5 indicates complete alignment. Performance of LLMs and human analysts on the Likert scale was comparable. Only one LLM (ChatGPT-5) reached non-inferiority at the pre-specified margin (Figure 2B). We observed the highest level of congruence in descriptive themes expressed in the transcript. These included the need for personalisation, difficulty accessing healthcare, harm from generic advice, and scepticism surrounding data security.
Across both human analysts and LLMs, a high degree of overall thematic convergence emerged. Analysts identified speech segments such as the following:
T037: “We as individuals monitor how we respond. I mean, there are at least three of us in this room”.
This was universally detected as reflecting the desire for individual treatment in the context of healthcare and multimorbidity. Although was sometimes framed in analogous or overlapping codes such as “individualised treatment needs”, the underlying emphasis on personalisation was preserved between analysts.
Significant agreement was identified in the theme of Truth, misinformation, and guidance. For example:
T002: “you can often read things and then you do a little bit of research around it and you find that actually the truth is perhaps even the opposite”
This statement was widely recognised as an expression of misinformation and trust degradation in everyday healthcare interactions.
While there was strong overall agreement in thematic identification, key divergences emerged in how human analysts and LLMs conceptualised and framed these themes. Human analysts tended towards a relational conceptualisation of personalisation with codes such as “personable approach”, and “patients as individuals”. Although both human analysts and LLMs identified the need for personalisation in healthcare, LLMs tended to frame the lack of such personalisation as a systemic design issue needing to be addressed, while human analysts framed the problem at a more personal, affective level.
Despite the expert adjudicated panel agreeing that the theme of care fragmentation featured prominently in the transcript, there was variation in how well it was addressed, specifically by the LLMs. Although accessibility barriers were frequently commented upon, the code of care continuity and fragmentation was an area of variable LLM performance. We noted this domain as a specific example of weakness in inferential reasoning, with the LLMs seemingly struggling to conceptualise the speech within the code.
Peer support was another theme that illustrated divergence. All analysts recognised it as a significant social factor, but the depth of interpretation differed. Human analysts performed strongly in this domain, demonstrating inferential capacity, while the LLMs seemingly struggled to infer peer support and connection beyond what was explicitly stated in the text, for example:
T040: “The community aspect of it, about people being able to connect with people locally, maybe with the same condition or maybe just with the same exercise routines.”
This was widely identified by the LLMs as an explicit example of community support. This theme, relying heavily on inferential and latent meaning, was a domain in which human analysts added significant additional insight, while the LLMs produced a much more descriptive analysis.
The LLMs contributed distinctive analytical insights by framing participants’ experiences through a more systemic and process-oriented lens than human analysts. Whereas human coders tended to interpret negative experiences as rooted in structural or interpersonal factors, the LLMs linked poor experience more systemically to outdated systems, training, and guidance. Interestingly, AI implementations featured more heavily in the inductive analysis by the LLMs, even when speech segments seemingly had less relation to AI.
Trust and feedback in AI development had variable interpretations. The LLMs diverged significantly in their analysis, with QualiGPT particularly focusing on language use within AI implementations. Claude 4 Sonnet coded “AI learning potential”, with some overlap in thematic analysis. LLM codes such as “compliance language” reflected their tendency to focus on practical tangible implementations and improvements.
Further evidence of additional insight appeared in the domain of flexibility and adaptation. For example:
T042: “For fluctuating conditions, can it change like that?”
This quote was more readily translated by the LLMs into specific design requirements, coded as “need for flexible, adaptive programs.” Human analysts, by contrast, showed less consensus in coding this segment, with only one identifying it distinctly. This suggests that LLMs may exhibit a comparative advantage in abstracting technical or system-level implications from participants’ remarks, whereas human analysts prioritised the subjective, experiential meaning of those same expressions.
Result verification
We quantified the accuracy and validity of the LLM coding according to the metrics of strict hallucination rate, expanded hallucination rate, and comprehensive error rate (Table 4). The strict hallucination rate included segments which did not support the code assigned by the LLM, aligning with the definition of hallucination in machine learning. The overall hallucination rate of the models tested was 1.2%(SD=2.1%), with two out of the three models tested having no hallucinations in the inductive analysis. The expanded hallucination rate included both segments identified as erroneous evidence, and researcher segments coded in the inductive analysis. The mean expanded hallucination rate was 8.6%(SD=5.1%) for the models tested, significantly higher than the strict definition of hallucination. Finally, the comprehensive error rate included partial matches of segment-code meaning in its definition. The mean comprehensive error rate was 12.4% (SD=5.1%) for the models tested.
Discussion
In this mixed-methods comparison, we observed comparable performance between LLMs and blinded human analysts. The strengths of the LLMs included their application to descriptive qualitative codes, as well as their high specificity in coding the transcript. Our dataset, reflective of a practical, unrefined focus group transcript, had a low overall prevalence of qualitative codes, enabling evaluation performance under minimally curated conditions. It is well-documented that Cohen’s κ and Jaccard index are significantly affected by the prevalence within a sample. [28,29] Although we report high raw agreement values, the “prevalence effect” of such metrics leads to lower agreement metrics than have been reported elsewhere. Comparative analysis according to deductive codes and themes did not yield significant differences between humans and models tested; although we observed a trend toward stronger model performance in descriptive coding.
Our results introduce a novel methodological approach to the comparison of humans and LLMs in thematic analysis. Each of the models tested performed well as defined by strict hallucinations, with most models outputting no erroneously coded sections. However, performance declined in speaker delineation according to the inductive analysis prompt, despite researcher speech segments being clearly labelled in the transcript. Although the strict hallucination rate of the LLMs was relatively low (1.2%), the comprehensive error rate of 12.4% was considerably higher. This has significant implications as to the utility of LLMs in qualitative analysis of focus groups and arose largely from the LLMs coding facilitator speech segments in the transcript. Although researcher speech segments could be removed in data pre-processing, it could be argued that this removes part of the transcript data which may have implications for its overall meaning.
This finding highlights a broader consideration in applying LLMs to qualitative data: while pre-processing can improve technical performance metrics, it also risks undermining context important for interpretive validity. Furthermore, the use case for LLMs within qualitative research necessitates the ability of such models to accurately perform predetermined instructions and adhere to coding requirements set by researchers. Our findings illustrate an epistemic issue in applying LLMs to coding tasks: LLMs do not “understand” coding instructions as a human analyst but approximate them using probabilistic pattern-recognition.
Our comparative inductive analysis aligns with prior research suggesting that LLMs can reproduce prominent themes and sometimes expand upon or offer additional insights to human coded data. [30–32] Prior research has questioned the ability of LLMs to detect latent meaning and affective responses within qualitative datasets. [33–35] Comparative inductive analysis revealed weaknesses in domains such as peer support and co-production. Further research could explore this in more detail.
Our results affirm the emerging consensus that LLMs perform best in constrained, deductive coding tasks, and less effectively in inductive analysis requiring sensitivity to interpersonal nuance and latent meaning. [2,36] This mirrors findings in broader research suggesting that human-AI hybrid implementations are best placed to balance efficiency with accuracy and validity. [37,38]
Prior methodologies have measured agreement between LLMs and human in qualitative analysis using quantitative agreement metrics and Likert-scale agreement. [4,39] Deiner et al., 2024 provides a methodological treatment of hallucinations when applying LLMs to thematic analysis. [30] While our findings similarly indicate low overall hallucination rates, we extend this framework by introducing a comprehensive error rate. Furthermore, although previous studies have reported the use of blinded analysts, to our knowledge, no studies have applied a formal non-inferiority framework. As such, our study assesses functional comparability rather than descriptive similarity alone.
Strengths and limitations
This study’s strengths include a blinded-mixed methods design to enable for a robust comparison of model performance compared to qualitative analysts across a variety of metrics. By integrating quantitative agreement metrics with qualitative interpretation, our study offers a reproducible framework for evaluating AI performance in thematic analysis. We utilised a minimally curated focus group dataset to ensure an accurate assessment of the practical performance of LLMs in thematic analysis, while enforcing a uniform analytic framework which allowed for direct comparison between the LLMs and human analysts. Additionally, we tested the models according to both deductive and inductive qualitative analysis, while also reporting hallucination and error rates of the LLMs tested. Finally, this study offers methodological transparency through the open sharing of prompts, task-specifications, and LLM outputs.
However, several limitations should be acknowledged. The study was based on a single focus group transcript with seven participants discussing the topic of AI implementations in social care and physical activity. Subsequently, model performance may vary when applied to external datasets. Our findings are also model and version-specific and may shift according to future model updates. It should be noted that the adjudicated human reference-standard panel which acted as the comparator for the primary findings of this study was the interpretation of a panel of researchers, and not an objective proof. As such we evaluated whether the outputs were aligned with this panel-consensus, rather than if they were epistemically correct.
Our non-inferiority analysis treated two blinded human analysts as a benchmark for human performance. Given the interpretive nature of qualitative analysis, this benchmark is subject to intrinsic variation according to the skills, style, and interpretive depth of the analysts chosen. We also note that the scope non-inferiority testing was confined to two specified analytic tasks applied to a single focus group dataset.
Finally, limitations in the statistical analysis of the data should also be acknowledged. Our deductive analysis produced narrow confidence intervals, in part due to the larger number of data points, and significant effect separation from the pre-specified margin. However, our comparative inductive analysis involved fewer codes, reducing the statistical power to detect a difference at the pre-specified non-inferiority margin. Further studies should develop multi-group sampling to obtain stable estimates of inductive analysis performance.
Conclusions
This study introduces a reproducible and blinded evaluation framework for comparing large language models (LLMs) and human researchers in thematic analysis of healthcare data. Through structured data formatting and analysis, we demonstrate how computationally rigorous approaches can complement qualitative interpretation We suggest several methodological contributions for future consideration in mixed-methods research on the application of LLMs to qualitative data. The use of global and local coding rules with strict output formatting requirements improved the auditability of the comparison. The use of blinded human researchers further reduced reflexivity and expectancy bias, allowing for valid comparisons between human analysts and LLMs. Quote verification provided a grounded view of error modalities when applying LLMs to qualitative data. Finally, by testing the LLMs across both inductive and deductive analytic tasks, we identified variation in their interpretive capacity while also quantifying their relative performance.
From a clinical informatics perspective, these methodological refinements support the responsible integration of LLMs into health research workflows. The implementation of transparent and reliable qualitative analysis underpins patient and clinician-centred design of decision-support tools, healthcare implementations, and policy frameworks. LLMs that can deliver reproducible qualitative analysis while maintaining low error rates have the potential to accelerate the synthesis of patient and clinician insight into system design and clinical decision-making frameworks, while preserving the necessity of human oversight.
This study’s findings highlight an opportunity for the development of specialised, clinically contextualised LLMs trained to capture the relational and affective dimensions of patient narratives. Whereas general-purpose models perform well in descriptive or procedural coding, healthcare decision-making frequently depends on understanding empathy, trust, and interpersonal meaning, domains where current models demonstrate limitations. Future research should explore the fine-tuning of LLMs on clinical datasets, enabling the emergence of relationally competent LLMs capable of detecting trust, affect, and empathy.
Collectively, this study introduces a reproducible, blinded evaluation framework for comparing LLM and human performance in thematic analysis in a healthcare setting, incorporating formal non-inferiority testing and error auditing. While LLMs demonstrated non-inferior performance in deductive analysis of a polyvocal transcript, their higher comprehensive error rate and weaker performance in affective coding supports their use as an adjunct, rather than a replacement for human qualitative analysts.
Supporting information
S1 Text. GRAMMS checklist
S2 Text. Study protocol
S3 File. Deductive Results
S4 File. Inductive Results
S5 File. Comparative Inductive Analysis
Declarations
Availability of data and materials
The primary data that support the findings of this study are not publicly available due to privacy and ethical considerations, as despite anonymisation, the transcript contains potentially identifiable information. Requests for controlled access to the raw data may be directed to the University of Southampton Faculty of Medicine Ethics Committee (ERGO 106517) for researchers who meet the criteria for confidential data access.
Full results of thematic analysis including excerpts from the transcript are available in the supporting material. Code used in data processing and analysis is publicly available on GitHub: https://github.com/CHill887/Large-Language-Models-for-Thematic-Analysis-Mixed-Methods-Comparison
Data Availability
The primary data that support the findings of this study are not publicly available due to privacy and ethical considerations, as despite anonymisation, the transcript contains potentially identifiable information. Requests for controlled access to the raw data may be directed to the University of Southampton Faculty of Medicine Ethics Committee (ERGO 106517) for researchers who meet the criteria for confidential data access. Full results of thematic analysis including excerpts from the transcript are available in the supplementary material. Code used in data processing and analysis is publicly available on GitHub: https://github.com/CHill887/Large-Language-Models-for-Thematic-Analysis-Mixed-Methods-Comparison
https://github.com/CHill887/Large-Language-Models-for-Thematic-Analysis-Mixed-Methods-Comparison
Acknowledgements
We would like to thank all the patients and participants who contributed to this work, and also to Lucy Smith for leading the original focus group.
Abbreviations
- AI
- Artificial intelligence
- AC1
- Agreement Coefficient 1 (Gwet’s)
- CI
- Confidence interval
- GDPR
- General Data Protection Regulation
- GRAMMS
- Good Reporting of a Mixed Methods Study
- LLM
- Large language model
- NIHR
- National Institute for Health and Care Research
- NHS
- National Health Service
- PPI
- Patient and Public Involvement
- SD
- Standard deviation