This study provides the first systematic, reproducibility-aware evaluation of frontier reasoning LLMs on core orthodontic diagnostic computations. All three models achieved high overall accuracy (94.0 to 99.0%), but beneath these headline figures lies a complex pattern of task-specific vulnerabilities, divergent error profiles, and a previously undocumented trade-off between accuracy and consistency.

The overall accuracy levels substantially exceed those reported in previous orthodontic LLM evaluations, where ChatGPT-4 achieved entirely correct responses in only 40.5% of interceptive orthodontics cases⁴ and crowding classification from intraoral images reached only 50%.⁶ The markedly superior performance likely reflects two factors: the use of the latest reasoning-enhanced models, and the text-based, numerically specified task format that removed the confound of visual interpretation known to degrade LLM performance by approximately 10 percentage points across model families.²⁶

The most striking finding was ChatGPT 5.4 (Thinking)’s systematic failure on lateral cephalometric interpretation (76% accuracy), with 12 items misclassified with perfect determinism across all four trials. Qualitative error analysis revealed a dual ANB threshold error: nine items with ANB values of 4-5° were over-classified as “Mild Class II” because the model applied a threshold at the ANB mean (3°) rather than at the upper limit of the normal range (5°), whilst three items with ANB 0° were under-classified as Class I. The thinking mode allocated additional inference-time computation to every item, yet this deliberation did not correct the underlying threshold error; the model reasoned carefully and consistently toward the wrong answer, a pattern that might be termed rigorous incorrectness. The term “Mild Class II” does not appear in formal cephalometric classification systems^23,25 but is widely used in informal clinical discourse across the internet, which forms part of the unstructured text on which LLMs are trained. The model appears to have internalised this label without acquiring the corresponding rule that ANB values within one standard deviation of the mean are classified as Class I. This represents a form of hallucination²⁷ in which the output is linguistically plausible but factually unsupported by the classificatory framework.

Gemini 3 (Thinking)’s six IOTN-DHC errors reveal a qualitatively different failure mode. All six items were correctly assigned to Grade 4, but the sub-classification letter was incorrect, reflecting failure in applying the worst-feature hierarchy when multiple Grade 4 features coexisted. Whilst these errors would not alter treatment eligibility, they indicate a deficiency in the granularity of clinical rule application that would compromise referral documentation.

Claude Opus 4.6 (Extended Thinking)’s Bolton errors demonstrated a third failure mode: stochastic arithmetic variability, with accuracy ranging from 82% to 96% across trials on items involving minor decimal discrepancies of 0.5 percentage points. This pattern is consistent with the non-deterministic token-sampling process inherent to autoregressive architectures and suggests that extended thinking, by generating longer deliberation traces with more sampled tokens, may paradoxically increase arithmetic noise as a byproduct of deeper reasoning.¹³

Perhaps the most novel contribution is the documentation of an accuracy-consistency paradox. The most accurate model (Claude, 99.0%) demonstrated the lowest consistency (93.0%), whilst the least accurate (ChatGPT, 94.0%) exhibited the second-highest consistency (98.0%) because its errors were perfectly deterministic. This paradox carries direct implications for patient safety: a single-trial evaluation cannot distinguish a model that is reliably correct from one that is reliably wrong. The overwhelming majority of published LLM evaluations are structurally unable to detect this pattern.¹⁰

Across all four tasks, errors were concentrated at or near the normal–abnormal classification boundary, forming a gradient from cross-boundary misclassification in cephalometric interpretation, through within-grade discrimination errors in IOTN-DHC, to sub-threshold arithmetic noise in Bolton analysis. This suggests that LLMs are most vulnerable in tasks requiring precise threshold-based judgements and least vulnerable in straightforward arithmetic with unambiguous clinical direction.

The clinical significance of these findings cannot be separated from the problem of automation bias.⁷ ChatGPT 5.4 (Thinking) achieved 100% accuracy on Bolton analysis, yet the same model, with identical confidence and zero uncertainty signalling, systematically misclassified 24% of cephalometric items, with 9 of 50 items (18%) receiving an incorrect sagittal classification capable of altering the treatment plan. A clinician who has verified the model’s flawless computational performance will rationally develop trust in its diagnostic capabilities, and this earned trust provides no basis for suspecting that the same model’s cephalometric interpretation is systematically wrong. This asymmetry between perceived and actual reliability across task domains constitutes the primary clinical risk identified by this study.

Two remediation strategies emerged from the findings. One was multi-model verification. Across the entire dataset, no single item was misclassified by more than one model, suggesting that consensus-based querying would have identified and corrected every error. The other was context-rich prompting, which yielded 100% accuracy and consistency across all three models by embedding explicit classification rules within the prompt. The elimination of all errors indicates that the original failures reflected imprecise parametric knowledge rather than a fundamental limitation in reasoning.^16,17,18 These errors were corrected once authoritative in-context knowledge was provided, which is analogous to the mechanism of retrieval-augmented generation.²⁸

Several caveats temper these findings. The context-rich prompting result was achieved on purpose-built items with clean numerical inputs; real clinical data introduces measurement noise, mixed dentition complexities, and anomalous presentations that were not tested. Furthermore, context-rich prompting presupposes that the clinician or system designer already possesses the correct classification rules, creating a paradox whereby those who would benefit most from LLM assistance are least equipped to construct the prompts that would ensure accuracy. The use of consumer-facing web interfaces, whilst reflecting realistic clinical access modalities, precludes control over inference parameters and introduces reproducibility threats from undisclosed server-side updates.²⁹ The absence of a non-thinking baseline condition means that the specific contribution of extended reasoning cannot be isolated. The binary scoring rubric does not capture error magnitude, and the synthetic data may not fully generalise to clinical practice.