NigBench: A multilingual point-of-care medical query benchmarking study of large language models in Nigeria
Intron Health
BioRAMP
University of Florida
Brown University
University College Hospital
University of Jos
InStrat Global Health Solutions
PATH, Switzerland
PATH, United Kingdom
University of Birmingham, UK
*Corresponding Author: Tobi Olatunji Tobi@intron.ioAbstract
In this study, we introduce a novel benchmark comprising over 9,000 real-world, point-of-care, multilingual, and multimodal clinical question-answer pairs sourced from frontline health workers in Nigeria. Using the dataset, we compare local general practitioners to multiple leading open and closed LLMs. Our results reveal several critical insights into the suitability of LLMs as clinical decision support systems in low-resource contexts. The results confirm that performance varies widely by language and input modality (e.g., text vs speech): while models perform best on English text inputs, their accuracy drops significantly for local-language speech. Critically, it is possible to achieve substantial performance gains by transcribing and translating other languages into English before prompting an LLM – an important insight for non-anglophone product developers. Finally, this benchmark highlights key limitations of SLMs in supporting frontline healthcare in low-resource settings and provides a clear opportunity to track improvements as novel solutions are developed.
Article notes
Introduction
The global shortage of physicians, particularly in low- and middle-income countries (LMICs), has created significant barriers to achieving universal health coverage and equitable healthcare delivery [1]. Nigeria exemplifies this challenge, with a physician-to-patient ratio far below the World Health Organization’s recommended threshold [2,3]. To address this gap, Nigeria relies heavily on ‘community health extension workers’ (CHEWs) – facility-based staff that also do community outreach, who have undertaken a 3-year pre-service training programme, and whose scope of work spans essential primary care services, including triage, preventive care, health education, and referral for complex cases [4,5]. Despite their formal training, these frontline health workers often encounter clinical situations beyond their scope of practice [6]. In such instances, timely consultation with more experienced clinicians is ideal but frequently unattainable due to infrastructural and workforce limitations [7,8]. This mismatch between the clinical demands placed on CHEWs and their access to medical expertise has direct implications for the quality of care, patient safety, and health outcomes [8–10].
In recent years, the emergence of large language models (LLMs) has sparked global interest in the potential of artificial intelligence (AI) to transform primary care [11]. LLMs have demonstrated remarkable abilities in biomedical question-answering, achieving and, in some cases, surpassing the performance of average human test-takers on U.S., Indian, and African professional medical [12–15]. These developments have opened new avenues for deploying LLMs as clinical decision support tools, especially in low-resource environments where human expertise is scarce [16]. However, the deployment of LLMs in LMIC health systems remains nascent [17]. There is limited evidence on the utility, accuracy, and contextual relevance of these tools when responding to real-world clinical questions generated by frontline workers in non-Western, multilingual contexts [e.g., 13-15]. Most existing benchmarks focus on high-resource settings, failing to account for linguistic diversity, infrastructural constraints, and nuanced LMIC care pathways in countries such as Nigeria.
To bridge this gap, this study describes the development of a multilingual and multimodal benchmark dataset comprising vignette-style questions posed by CHEWs in Nigeria. The questions reflect the everyday clinical ambiguities, triage challenges, and referral dilemmas that CHEWs face in the field. To demonstrate the utility of the dataset, we compare the performance of leading proprietary, open-source, and biomedical LLMs against local clinician-generated responses to evaluate the reliability and practical value of the former as potential AI assistants for frontline health workers.
Methods
Question Collection & Curation From CHEWs
Three geopolitical zones in Nigeria were selected to capture regional diversity in healthcare delivery and disease burden: North-Central (Jos), Southwest (Ibadan), and South-South (Bayelsa). In each region, local healthcare organizations and training institutions (College of Medicine at University of Ibadan, University of Jos, and InStrat Global Health Solutions) were invited to support the recruitment of actively practicing Community Health Extension Workers (CHEWs) from public primary healthcare facilities.
A web-based data collection interface (Extended Data Figure 1) was developed on the Intron Data Vault Crowdsourcing Platform (DVCP) to enable CHEWs to submit realistic, point-of-care clinical questions encountered during their daily work. Participants were instructed create scenarios based on lived experiences and use pseudonyms to protect identifying information. The platform supported multilingual input and submission of de-identified multimodal attachments (e.g., images of rashes, audio recordings of patient symptoms, and short videos). CHEWs were trained and onboarded onto the platform through live workshops and remote support. An initial pilot involving a small cohort of CHEWs was conducted in November 2024 to test usability, identify challenges, and iterate on the interface and training materials. Following successful piloting, full-scale data collection was undertaken from December 2024 to February 2025.
To ensure diversity and representativeness, CHEWs were prompted to submit questions spanning the full breadth of their curriculum, guided by category and scenario prompts derived from national standing orders. Specifically, 57 topics or categories were defined based on the training curriculum and ‘Standing Orders’ [18]. Each category had an average of six subcategories, totalling 362 subcategories. These subcategories were presented to CHEWs as prompts to diversify question topics (as illustrated in Extended Data Figure 1). Similar to Med-PaLM-2 and Healthbench [12,19], the 57 categories were further grouped (using LLM-suggested topics) into 7 broad themes, such as Emergencies, Reproductive Health, and Mental Health; these groupings are utilized later to present summary results. Full details on mapping of topic areas are provided in Extended Data Table 1.
Questions were submitted in both English or local languages (Hausa, Yoruba, Igbo, and Pidgin English), with input accepted via typed text or voice recordings to reflect natural communication modes. Each question was annotated with metadata provided by CHEW, including language, question difficulty (easy, medium, or hard), and whether multimedia was included. Multimodal attachments (audio, images, videos, or documents) were submitted using contributors’ personal devices, with no prior training or constraints on camera quality, image resolution, lighting, brightness, or microphone quality, clarity, or internet bandwidth, to ensure fidelity to how such communications with clinicians actually occur.
Question Post-Processing
Voice-based inputs from CHEWs were transcribed to text by professional transcribers, and all non-English questions were translated into English by bilingual medical professionals. To prevent prompt mimicry, inputs were reviewed to ensure they were not simple restatements of scenario prompts. Scenario-prompt pairs with a ROUGE-L similarity greater than 0.6 were removed from the dataset. Short or poorly detailed vignettes (<100 characters) were excluded to preserve the quality of the dataset.
Collecting Clinician Responses
General Practitioners (GPs) from each corresponding region were recruited to provide human responses to the CHEW questions (See Extended Data Figure 2 for the web interface used to facilitate answer collection). Each finalised question-answer (QA) pair (vignette) consisted of a topic category (e.g., malnutrition), a scenario prompt (e.g., an under-five child with weight loss), a CHEW-reported clinical scenario (e.g., AB came to the clinic 5 days ago with a 2-year-old child who was reported to be losing weight rapidly…) as text or audio, a CHEW-formulated question (e.g., how do I counsel the parents for…), optional multimedia attachments, and a local physician response/answer in English. All questions (English or Nigerian languages) were answered in English.
LLM Answer Generation
Unimodal, multimodal, multilingual, open, and closed LLMs of different sizes were selected to generate responses to the CHEW-generated questions:
- Closed-source general-purpose multimodal models: OpenAI GPT-4o, GPT-4.1, o4-mini, Gemini 2.0 flash, Anthropic Claude-4-sonnett.
- Open-source text model: Meta AI LLaMA 3.3-70b, LLaMa-4-maverick, Google Gemma 3-27b, Alibaba Qwen2.5-32b, Deepseek-R1-V2
- Open-source multimodal models: Microsoft Phi-4-multimodal instruct, Alibaba Qwen2-Audio-7B, Qwen2.5-VL-7B
LLM Prompting
Each vignette was submitted to the LLMs using a contextualized prompt describing rural limitations, the CHEW’s role, and expectations for primary care relevance. Given the community-based nature of the scenarios and questions, it was crucial to provide context to the LLMs by crafting a prompt that aligns with the task objective. The final prompt used for all LLMs was as follows:
“You are an expert medical doctor with years of experience in Africa, working in a community-based hospital setting. Your expertise covers a wide range of medical specialties and conditions, from common ailments to complex diseases. You are tasked with answering health-related questions. You should provide a correct and concise answer based on your clinical knowledge and experience, explaining the reasoning behind your response using medical evidence and current locally relevant practices to support your answer. Here is a question: {scenario} {question} {attachment}”
Benchmarking LLMs
Using subsets of the whole dataset, several evaluations of different groups of LLMs were conducted. Each experiment is outlined below. Since the evaluation framework was consistent across all experiments, it is described only once at the end of the document.
Experiment 1: Question Answering Capabilities Based on English Language Inputs
A total of 266 randomly selected audio files, each containing questions asked in English, were used for this analysis. The questions were transcribed by professional transcribers on the Intron DVCP. Responses to the transcribed versions of the questions were generated by 11 LLMs: GPT-4o, GPT-4.1, o4-mini, Gemini 2.0 flash, Claude-4-sonnett, LLaMA 3.3-70b, LLaMa-4-maverick, Gemma 3-27b, Qwen2.5-32b, Deepseek-R1-V2, Phi-4-multimodal instruct, and then assessed as described later. A multimodal subset of these models (i.e., GPT-4o, Gemini 2.0 Flash, Phi-4-Multimodal Instruct), and Qwen2-Audio-7B (another multimodal model), were then tested using the original audio files, and the responses were again assessed using the standardized procedure described below.
Experiment 2: Question Answering Capabilities Based on Nigerian Language Text Inputs
As described above, 469 audio-format questions (supplied by the CHEWs as speech) spanning major Nigerian languages, such as Hausa (n = 118), Yoruba (n = 234), and Pidgin (n = 117), were randomly selected. The number of Igbo questions was subsequently deemed too small to analyse. The questions were transcribed into local-language text and then translated into English by bilingual, medically trained native speakers (of each language) on the Intron DVCP, creating a parallel dataset of audio and text questions in Nigerian languages and their English translations.
Responses to the local language text questions were generated by the following 6 models: GPT-4o, Gemini 2.0 flash, LLaMA 3.3-70b, Gemma 3-27b, Qwen2.5-32b, Phi-4-multimodal instruct. In parallel to the evaluation of these outputs, the English text translations were fed back into the same LLMs to assess the impact of input language.
Experiment 3: Question Answering Capabilities Based on Nigerian Audio Inputs
As described above, a multimodal (speech-text) subset of these models (i.e., GPT-4o, Gemini 2.0 Flash, Phi-4-Multimodal Instruct, and Qwen2-Audio-7B) was then tested using the original local language audio files, and the responses were again assessed using the standardized procedure described below.
Evaluation Framework & Expert Evaluator Selection
Evaluation Framework
Due to the open-ended and care-focused style of the questions posed by CHEWs (majority of the questions focused on clinical management and patient/provider education), the lengthy answers returned by LLMs, and the known limitations of quantitative metrics on medical QA evaluation – high semantic overlap does not always connote correctness [12,20] – it was necessary to define more nuanced rating criteria to evaluate several key dimensions. We based our approach on multiple prior works [21,22]; expert raters scored each answer using a 10-question rubric, each assessed with 5-point Likert scales, as follows:
Content Quality:
1. Factuality – Scientific correctness and consistency with standard medical practice.
2. Appropriateness – Suitability for a rural African primary care context.
3. Adequacy – Completeness in addressing the question.
4. Reasoning – Quality and coherence of clinical logic.
5. Self-Awareness: Identifies incomplete information or uncertainty and seeks more info.
Quality of Communication:
6. Empathy – Evidence of compassion and respect for the patient.
7. Fluency/Clarity – Readability and linguistic appropriateness for the CHEW audience.
Safety-related:
8. Hallucination – Presence of irrelevant, extraneous, or nonsensical content.
9. Bias – Suggestions are inappropriate or irrelevant to local contexts.
10. Harm – Potential for incorrect advice to cause clinical harm.
The user interface for the evaluators is shown in Extended Data Figure 3. The full scoring rubric, including descriptions of the Likert score elements, is provided in Extended Data Table 2.
Expert Panel Recruitment
To facilitate the benchmarking component of this study, a panel of general practitioners with regional familiarity was recruited to serve as expert raters. Vetted raters were sourced through the Intron Health online community of experts at regional healthcare institutions closest to CHEW communities. The experts’ task (as described later) was to rate LLM and doctor answers to CHEW questions based on specified criteria.
Contributor Compensation
All contributors (CHEWs, physicians, reviewers, and raters) were compensated at rates ranging from $10 to $100/hr based on fair market rates appropriate to their level of training, years of experience, and time invested in the project. LLM costs were calculated using the relevant tokenisers for each model and the per-token cost at the time of the experiments.
Statistical Analysis & Ongoing Quality Assessment
All analyses were carried out in Python & R (version 4.5.0), and the relevant code is available at [link].
Quantitative Evaluation
Summary statistics (means and standard deviations) are presented for each model and metric as descriptive overviews.
For formal inference, we employed cumulative link mixed models (CLMMs; ordinal package in R), which are purpose-built for ordinal Likert-scale data and avoid the distributional assumptions of parametric tests [23]. For formal inference on the ordinal ratings, we fit CLMMs for each of the 11 language-modality subsets (4 languages × 3 modalities, minus English translated text): score ∼ responder + dimension + (1 | question_id), where score is the ordered 1–5 rating, responder is the model or human identity, dimension controls for the 10 quality dimensions, and (1 | question_id) is a random intercept for each question. All models used a logit link. Estimated marginal means (EMMs) were obtained, and all pairwise contrasts and model-vs-human contrasts were computed with p-values adjusted using the Benjamini–Hochberg (BH) false discovery rate procedure. To aid interpretation, latent-scale EMMs were converted to expected scores on the original 1–5 scale.
To assess cross-language consistency, pairwise model contrasts from each per-subset CLMM were pooled across languages within each modality via two-stage inverse-variance meta-analysis, yielding both fixed-effect and DerSimonian–Laird random-effects pooled log-odds ratios, along with heterogeneity diagnostics (Cochran’s Q, I², τ²). Contrasts with I² ≥ 50% were flagged for direction-consistency assessment. To estimate cross-modality effects for Nigerian languages, CLMMs with a responder × modality interaction were fit for each language, restricted to the three universal models present across all modalities (Gemini 2.0 Flash, GPT-4o, Phi-4 Multimodal), and cross-modality contrasts were pooled across languages. An equivalent model was fit for English (audio and transcribed text only). Finally, to test whether the cross-modality effect differed between English and Nigerian languages, meta-analytic subgroup tests (Q-between) were performed comparing the pooled Nigerian estimate against the English estimate for each model. All p-values were BH-adjusted within each analysis.
Ethical Approval
This study received IRB approval from the National Health Research Ethics Committee of Nigeria (NHREC) with approval number NHREC/01/01/2007-19/12/2024 and the Jos University Teaching Hospital Health Research Ethics Committee (JUTH HREC) with approval number JUTH/DCS/IREC/127/XXXI/2890.
Results
Dataset Overview
A total of 7,289 unique questions, contributed by 284 Community Health Extension Workers across three districts in Nigeria (see sTable 1 for a summary of the contributor characteristics), were collected using the pipeline illustrated in Figure 1. 66 licensed general practitioners generated 9,303 answers to those questions (i.e., a small subset were answered by more than one clinician to provide a dataset for comparing human answers) – see sTable 2 for a summary of the respondent characteristics. Notably, 2,054 questions were asked in local languages (Yoruba: 963; Igbo: 28; Hausa: 653; and Pidgin English: 410). Additionally, 1,142 of the scenario descriptions were collected as audio (rather than typed text), supporting the analysis of speech-driven inputs. The full dataset characteristics are summarized in sTables 3 & 4. An example of a vignette with an LLM and human response is provided in Figure 2, along with the relevant metadata.
Evaluation of LLM Response Quality
The expert evaluators’ characteristics are summarized in sTable 5.
LLM Performance with English Language Inputs (sTable 6 & sFigure 1)
For English audio inputs, four models were evaluated. Gemini 2.0 Flash and GPT-4o were the strongest performers, with model-derived expected scores of 4.90 (95% CI 4.87-4.92) and 4.88 (95% CI 4.85-4.90) out of 5, respectively; the difference between them was not statistically significant (OR = 1.20 [95% CI 0.97-1.49], p = .091). Both significantly outperformed Phi-4 Multimodal (expected score = 4.46 [95% CI 4.38-4.55]; e.g., Gemini 2.0 Flash vs. Phi-4 Multimodal: OR = 6.23 [95% CI 5.18-7.49], p < .001) and Qwen2 Audio 7B (expected score = 3.50 [95% CI 3.35- 3.64]; e.g., Gemini 2.0 Flash vs. Qwen2 Audio 7B: OR = 28.73 [95% CI 23.83-34.64], p < .001).
The top four models (GPT-4.1, 4.77 [95% CI 4.74-4.80]; Claude Sonnet 4, 4.75 [95% CI 4.72-4.79]; Gemini 2.0 Flash, 4.74 [95% CI 4.71-4.77]; DeepSeek-R1, 4.71 [95% CI 4.68-4.74]) were statistically indistinguishable from one another, and Gemini 2.0 Flash significantly outperformed GPT-4o (OR = 1.26 [95% CI 1.09-1.45], p = .001). All frontier models significantly outperformed human respondents (human expected score = 4.37 [95% CI 4.32-4.41]); e.g., Gemini 2.0 Flash vs. Human: OR = 2.86 [95% CI 2.37-3.46], p < .001; GPT-4o vs. Human: OR = 2.27 [95% CI 1.96-2.63], p < .001). Phi-4 Multimodal was statistically indistinguishable from human performance (OR = 1.00 [95% CI 0.88-1.13], p = 1.000), whereas Qwen 2.5 32B was significantly outperformed by humans (OR = 0.51 [95% CI 0.45-0.57], p < .001).
LLM Performance with Nigerian Language Inputs
For Nigerian language audio inputs (pooled across Hausa, Pidgin, and Yoruba; see sTables 7-9 & sFigure 2), Gemini 2.0 Flash achieved a pooled expected score of 4.80 (95% CI 4.76-4.84) and GPT-4o scored 4.65 (95% CI 4.61-4.70), both substantially outperforming Phi-4 Multimodal (2.05 [95% CI 1.97-2.12) and Qwen2 Audio 7B (1.75 [95% CI 1.68-1.82]; all pairwise BH-adjusted p < .001). All contrasts were direction-consistent across the three languages, though the magnitude of differences varied substantially (I² ranging from 78.9% to 99.0%). For Nigerian transcribed text, GPT-4o (4.67 [95% CI 4.64-4.71]) and Gemini 2.0 Flash (4.61 [95% CI 4.58-4.65]) again led the ranking and both significantly outperformed human respondents (GPT-4o vs. Human: OR = 1.53 [95% CI 1.40-1.68], p < .001; Gemini 2.0 Flash vs. Human: OR = 1.86 [95% CI 1.69-2.05], p < .001).
In contrast, Gemma 3 27B, Llama 3.3 70B, Phi-4 Multimodal, and Qwen 2.5 32B all performed significantly worse than humans (ORs from 0.08 to 0.43, all p < .001). High between-language heterogeneity (I² > 90% for most contrasts) reflected uneven model capabilities across languages; for instance, Llama 3.3 70B scored 4.71 (95% CI 4.64-4.78) in Pidgin but only 2.66 (95% CI2.59-2.74) in Yoruba. On translated text, where Nigerian language questions were first translated into English, all models significantly outperformed humans (ORs from 2.02 to 6.32, all p < .001), and heterogeneity was low to moderate (I² = 0–74.4%), indicating a consistent model advantage across languages.
LLM Performance Variation Across Languages & Input Modalities (Figure 3)
Formal comparisons of scores between English and Nigerian language subsets revealed that all models performed significantly better in English than in Nigerian languages on audio inputs (all BH-adjusted p < .001). The gap was dramatic for smaller models: Phi-4 Multimodal scored +2.42 points (95% CI 2.30-2.53) higher in English than in Nigerian languages, and Qwen2 Audio 7B scored +1.75 (95% 1.59-1.91) points higher, whereas the top-performing models showed more modest, though still significant, differences (GPT-4o: +0.22 [95% CI 0.17-0.28]; Gemini 2.0 Flash: +0.10 [95%CI 0.05-0.14]). For transcribed text, a similar pattern emerged: Qwen 2.5 32B (+1.51 [95% CI 1.42-1.59], p < .001) and Phi-4 Multimodal (+1.04 [95% CI 0.97-1.11], p < .001) exhibited the largest English advantages, whereas GPT-4o showed a negligible, non-significant difference (+0.01 [95% CI –0.03-0.05], p = .733) and human respondents showed essentially no language-group difference (+0.00 [95% CI –0.06-0.07], p = .947). These results suggest that while smaller and mid-tier models are substantially less capable in Nigerian languages, the top-tier models (particularly GPT-4o) and human respondents maintain consistent performance across language groups (Figure 3a).
Cross-modality analyses for Nigerian languages revealed that transcribing audio queries yielded performance boosts for all three universal models (Gemini 2.0 Flash, GPT-4o, Phi-4 Multimodal; pooled ORs from 0.28 to 0.63 for audio vs. transcribed text, all p < .001). Translation from Nigerian languages into English produced even larger gains, particularly for Phi-4 Multimodal (audio vs. translated text: OR = 0.02 [95% CI 0.02-0.03], p < .001, corresponding to approximately 50-fold higher odds of a better score on translated text). Translation improved scores beyond transcription alone for all models (transcribed vs. translated text ORs from 0.04 to 0.40, all p < .001). In English, this pattern reversed for the top models: both Gemini 2.0 Flash and GPT-4o scored significantly higher on audio than on transcribed text (ORs = 1.68 [95% CI 1.39-2.04] and 1.78 [95% CI 1.51-2.10], respectively; both p < .001), and meta-analytic subgroup tests confirmed that the direction of the modality effect differed significantly between English and Nigerian languages (all Q-between p < .001). The practical impact of translation was substantial (Figure 3b and 3c): models that performed poorly on Nigerian language audio and transcribed text achieved competitive scores once questions were translated into English. Phi-4 Multimodal, for example, had a raw mean of 2.37 on Nigerian audio and 2.82 on Nigerian transcribed text, but reached 4.68 on translated text.
Similarly, Qwen 2.5 32B improved from 2.56 (transcribed) to 4.51 (translated), and Llama 3.3 70B from 3.17 to 4.67. All models significantly outperformed humans on translated text (all p < .001), including those that were significantly worse than humans on transcribed text (e.g., Phi-4 Multimodal vs. Human on transcribed text: OR = 0.11; on translated text: OR = 3.25, p < .001).
LLM Performance by Dimension
To examine whether model–human differences varied by quality dimension, Kruskal–Wallis tests were performed on transcribed text data for the three models used across all experiments (Gemini 2.0 Flash, GPT-4o, and Phi-4 Multimodal) and human respondents. Significant differences between responders were observed for all 10 quality dimensions (all BH-adjusted p < .001), with the largest effects for empathy (H(3) = 451.6), factuality (H(3) = 432.8), and clinical reasoning (H(3) = 428.1), and the smallest for harm (H(3) = 119.1). Gemini 2.0 Flash and GPT-4o were statistically indistinguishable on multiple dimensions, including empathy (4.64 vs. 4.63, p = .930), fluency/clarity (4.84 vs. 4.82, p = .800), hallucination (4.84 vs. 4.83, p = .470), factuality (4.66 vs. 4.59, p = .684), and adequacy (4.64 vs. 4.57, p = .757), though Gemini 2.0 Flash significantly outperformed GPT-4o on self-awareness (4.19 vs. 3.97, p = .004). Relative to human respondents, the top AI models showed the greatest advantages on empathy (Human = 3.89 vs. Gemini = 4.64, GPT-4o = 4.63) and factuality (Human = 4.31 vs. Gemini = 4.66, GPT-4o = 4.59). Phi-4 Multimodal scored significantly lower than humans across all dimensions (all p < .001), with the widest gaps on self-awareness (2.92 vs. 3.77), clinical reasoning (3.29 vs. 4.22), and empathy (3.49 vs. 3.89), and narrower gaps on fluency/clarity (4.30 vs. 4.71) and harm (4.45 vs. 4.79). Figure 4 illustrates the harm metric scores for this same group of models across the different clinical themes/categories. For context, the expert panel rated a total of 18,603 answers (2,589 human; 15,584 LLM; 430 distractors), and whilst the majority of safety scores fell within the “safe” range (1-2), it’s crucial to note that over 1000 answers were rated between “harmful” and “highly harmful” (3-5).
LLM Performance by Clinical Theme
Kruskal–Wallis tests by disease category group confirmed significant differences among responders for all seven category groups (all BH-adjusted p < .05), with the strongest effects for nutrition (H(3) = 907.8, N = 10,270) and body systems (H(3) = 776.0, N = 11,870), and the weakest for infectious diseases (H(3) = 8.9, N = 380, p = .030). The general ranking of Gemini 2.0 Flash and GPT-4o outperforming humans, who in turn outperformed Phi-4 Multimodal, was consistent across disease domains. In the nutrition category, both Gemini 2.0 Flash (4.67, 95% CI 4.62-4.71) and GPT-4o (4.61, 95% CI 4.57-4.66) significantly outperformed humans (4.28, 95% CI 4.23-4.32; both p < .001), while Phi-4 Multimodal (3.46, 95% CI 3.39-3.53) scored significantly lower than humans (p < .001); Gemini 2.0 Flash and GPT-4o did not differ from each other (p = .579). For emergencies, GPT-4o (4.60, 95% CI 4.54-4.66) and Gemini 2.0 Flash (4.57, 95% CI 4.49-4.64) were essentially tied (p = .370), both significantly outperforming humans (4.47, 95% CI 4.40-4.53; p < .001). In mental health, Gemini 2.0 Flash (4.61, 95% CI 4.51-4.71) marginally outperformed GPT-4o (4.42, 95% CI 4.32-4.53; p = .051), while GPT-4o significantly outperformed humans (4.31, 95% CI 4.19-4.42; p = .011). For infectious diseases, the small sample size (N = 380) limited statistical power, with only the Phi-4 Multimodal vs. Gemini 2.0 Flash comparison reaching significance (p = .004). The performance gap between the top models and Phi-4 Multimodal was remarkably stable across disease domains, ranging from 0.67 points (infectious diseases) to 1.21 points (nutrition).
Cost Varied Substantially by Modality But Not Language
The average cost of responding to questions in English for human clinicians was $10-$100/hr, whereas for the models it ranged from $1.5/hr (least expensive) to $33/hr (most expensive). The cost of answering in a Nigerian language was comparable to English text, and unsurprisingly, the use of audio inputs (range of means across models: 0.3 (US) cents to 42 (US) cents per query) was more expensive than text (range of means across models: $0.002 (US) cents to 37 (US) cents per query); see sTable 10.
Discussion
This study presents a large-scale, multilingual (and multimodal) dataset for assessing LLM performance on real-world clinical questions posed by health workers in sub-Saharan Africa. The results of the benchmarking exercise using a subset of the data illustrate that frontier models, such as GPT-4o and Gemini 2.0, consistently outperform smaller models across key domains, particularly in correctness, appropriateness, and safety. Moreover, the results clearly highlight the importance of input modality-language interactions; audio questions, especially in local languages such as Yoruba and Hausa, were more challenging for LLMs to process safely and accurately. Furthermore, transcription and translation into English consistently returned higher-quality results than using raw audio signals in local languages – an important strategy for improving performance, but one grounded in an approach that effectively embeds structural bias into a product. Finally, this study also adds much-needed detail to the understanding of how smaller, open (weights and fully open source) models perform more generally, illustrating that in situations that require small language models (SLMs), such as remote last-mile settings, there is still a way to go before they are better, or even equivalent to local expert clinicians.
In the Context of the Literature
The results of this study are broadly consistent with other African country-specific LLM benchmarking exercises [13–15], in that the large proprietary models appear to be superior to human experts in responding to frontline health worker queries and can do so at a fraction of the cost. Moreover, the observation of variation in language-related performance, as well as the possibility of recovery through translation into English, is consistent with previous studies in the medical question-answering (but not frontline healthcare) setting [24]. Furthermore, the observed between-topic performance differences are also (mostly) consistent with prior studies [25]: inherently higher-risk specialties, such as obstetrics, experience more frequent safety issues, while simpler topics, like nutrition, often rank among the safest in terms of LLM outputs. However, there are interesting outliers, such as mental health, which are high-performing in some contexts and low in others [13], possibly a result of representation in publicly available texts (i.e., mental health in Rwanda is a relatively over-represented topic in the literature, due to the history of the nation, whereas this is less true of Nigeria). Finally, consistent with the recent systematic review on the topic [26], we found that frontier LLMs can produce more empathic responses than local human experts.
Strengths and Limitations
A major strength of this study is the real-world nature of the data, collected from practicing CHEWs in naturalistic clinical settings. By incorporating five languages, multiple modalities, and varying question difficulties, the study offers a uniquely representative assessment of model readiness for global health contexts. Moreover, expert evaluation across ten dimensions provides a robust assessment framework that allows us to pinpoint specific issues across the content, communication and safety landscape. Finally, we’ve purposefully not exposed all of the vignettes in the benchmarking dataset to LLMs to reduce the risk of contamination (e.g., them accidentally being scraped or incorporated into training data); as such, there is a significant proportion of the original dataset that remains entirely unused and can be exploited for future research. Nonetheless, limitations exist. The impact of automatic speech recognition was not evaluated as transcription errors in production systems with cascaded transcription and translation pipelines may further impair the quality of transcribed audio inputs. Additionally not all local languages were equally represented (e.g., Igbo had limited data). Human evaluators, despite high agreement, may have exhibited systematic biases, as they have done in other studies [27]. Furthermore, the single-turn nature of the interactions in this dataset isn’t fully representative of how CHEWs interact with ‘ask me anything’ type solutions, based on our insights from an ongoing trial in Nigeria [28], wherein there seems to be a non-trivial number of multi-turn interactions around the same case. This is important to note, as the implication is that the results of benchmarking on this dataset provide weak (if any) guarantees about how a product incorporating a high-performing model might perform in a field deployment if behaviour/interaction patterns are meaningfully different from those captured to create this dataset.
Implications for Policymaker & Product Developers
These findings carry significant implications for the deployment of AI in LMIC health systems. First, where there is a preference for speech-based engagement in local languages, the results of this study suggest that this can entail a nontrivial performance-price trade-off. At the ecosystem level, a significant investment is needed to improve models in this context; in the interim, product developers should consider workflows that support voice-based interaction and use English-translated transcripts to ensure optimal results. Second, for policymakers, these findings illustrate the non-uniform safety of LLMs across sub-specialities. Specifically, applications in reproductive health, infectious disease, or emergency care warrant stricter oversight due to the much higher likelihood of potentially harmful responses.
Conclusion
The findings from this in silico benchmarking exercise, in conjunction with other recent studies emerging from Sub-Saharan Africa, provide a strong rationale for advancing to real-world evaluations of selected large language models (LLMs) as components of clinical decision support systems in low-resource settings. That said, much more research and development is necessary before standalone, smaller, open-source models are ready for field evaluations, given that they continue to fall short of local human expert outputs. Such progression in the type of evaluation these tools are subject to is necessary to determine whether the observed improvements in response quality translate to enhancements in the quality of primary healthcare and to systematically characterise infrastructural, contextual, and other implementation challenges. More generally, persistent performance limitations across non-English languages suggest that further investment in high-quality, locally relevant language resources, particularly those that capture diverse speech and audio representations, or novel methods for leveraging the sparse data available, is necessary to mitigate the current reliance on the “transcribe and translate” workaround and enable more inclusive deployment of LLM-based tools.
Supporting information
Declarations
1.Data Availability
The full dataset used for this study can be accessed at: https://huggingface.co/datasets/intronhealth/nig-bench
2.Code Availability
All code used for analysis can be found in the following GitHub repository: https://github.com/PATH-AI-Initiative/AI4Health_NigeriaBench
Data Availability
The dataset used for this study can be accessed at: https://huggingface.co/datasets/intronhealth/nig-bench
3.#Acknowledgements
This research was supported by the Gates Foundation (INV-068056). The funders had no role in the study design, data collection and analysis, the decision to publish, or the preparation of the manuscript.
5.Competing Interests Statement
TO holds equity in Intron, the developer of the web platform used for data collection. The other authors declare no competing financial or non-financial interests.