Patient satisfaction feedback is a crucial metric for determining areas of improvement in hospital services, directly impacting quality of care [1][2]. In French hospitals, patient feedback is collected systematically through multiple channels to capture diverse patient experiences. Feedback is primarily gathered via the national e-Satis survey, administrated by the French National Health Authority (HAS), which is emailed to patients three weeks after discharge if consent is provided. Additionally, institutional satisfaction questionnaires are completed by patients before discharge, and spontaneous patient communications—either complaints or expressions of satisfaction—are received through the hospital’s Patient Relations Department (Figure 1).

Patient feedback offers clinical and organizational insights in three domains—Fluidity & Patient Rights, Health Care, and Hospitality. Sub-topics span for example emergency-service responsiveness, staff empathy and availability, patient rights, and comfort factors such as room temperature. Precise topic labeling lets staff target improvement priorities, but manual classification is too time-consuming for large-scale deployment.

Montpellier Regional Hospital analyzes feedback with a regex-based keyword dictionary in Sphinx to produce the quarterly “Patient Experience Barometer.” Even with periodic updates, the method’s reliance on fixed terms overlooks nuanced, context-rich narratives and complex clinical sentiments. Like other legacy NLP approaches, it needs replacement by a model that can grasp semantically rich, clinically relevant content. In 2022, an unsupervised classification of 2.5 million feedbacks provided a comprehensive overview of patient concerns and defined 20 categories [3,4]. However, models like Naive Bayes, BERT and convolutional networks struggle to accurately classify nuanced feedback because they cannot effectively handle complex language contexts [5–7][6].

Large Language Models (LLMs) like GPT-4 and Llama-3.1 405B offer a promising alternative due to their superior ability to understand natural language and detect subtle nuances in patient feedback [6,8],[9]. Moreover, they might offer the needed explainability of their classifications, which is mandatory in high stakes language processing[10]. They have been recently applied in exploratory studies for quality measurement purposes [11] and specific language handling tasks in healthcare[12], including text classification[13]. However, since these models have been shown to exhibit inaccuracies in the medical field [14], there is a pressing need to improve the reliability of text classification. Recent work shows that evaluating the self-consistency of LLM predictions — that is, ensuring the model provides the same classification over multiple independent attempts — greatly enhances performance in classification tasks [15]. However, these models often produce hallucinations, which are factual mistakes or logical flaws in generated text [16]. Although minor factual hallucinations may be tolerable in certain hospital contexts, logical hallucinations—also known as “extrinsic faithfulness hallucinations”—introduce illogical reasoning that undermines explainability and is therefore incompatible with sensitive healthcare applications. While these logical hallucinations directly impact reliability in a critical way, they also clearly jeopardize the explainability of resulting classifications[17]. Ethically, healthcare choices must be traceable from data collection to decision, grounded in explainable, verifiable evidence. When a patient-experience barometer presents nonsensical categories, clinicians lose confidence, breaking the feedback loop with patients. Because AI lacks the interpersonal trust granted to humans, it must offer explicit, transparent explanations for its outputs[18]. Since Chain of Thought (CoT) generation methods improve the explainability of these models [19], validating CoT logical structures might be a potential solution to the issue of logical hallucinations.

To address this challenge, we developed Self-Logical Consistency Assessment (SLCA), a method designed to improve the reliability of LLM-generated predictions. The SLCA combines Self-Consistency Assessment (SCA) [15] — the evaluation of the LLM ability to provide the same predictions in similar conditions — together with an original method called Logical Consistency Assessment (LCA), which appraises the capability of an LLM to produce a logically structured CoT. The aim of this study is to evaluate whether SLCA can achieve human-level performance and provide a practical option in the automated classification of patient feedback.