Proposed Context-of-Use Evaluation Framework for Medication Management Tasks Completed by Generative Artificial Intelligence
1University of Colorado School of Medicine, Department of Biomedical Informatics, Aurora, CO
2Wellstar MCG Health, Department of Pharmacy, Augusta, GA
3ATLAS Institute, University of Colorado Boulder, Boulder, CO, USA
4Department of Computer Science, University of Georgia, Athens, GA
5University of Georgia College of Pharmacy, Department of Clinical and Administrative Pharmacy, Augusta, GA
*corresponding author: Email: Andrea.sikora@cuanschutz.eduAbstract
Background
Standardized evaluation of agentic artificial intelligence (AI) for medication management is lacking. Given the potential lethality of medication errors endorsed or missed by AI, performance evaluation constructs are essential. The purpose of this evaluation was to develop a standardized grading framework for performance evaluation of medication management tasks.
Methods
A mixed-methods approach was undertaken that included literature evaluation for standards and best practices of comprehensive medication management (CMM), panel discussions, and iterative application to set of cases. The goal was to develop a grading framework that effectively evaluated domains like safety, factuality, and clinical relevance that can be employed for a broad range of medication domains (i.e., electrolyte replacement, antibiotic selection). Inter-rater reliability with intraclass Krippendorff’s Alpha was the primary outcome.
Results
A total of 5 panelists developed the CMM Evaluation Framework, which includes 4 dimensions: safety, factuality, completeness, and preference. These dimensions are applied to three CMM skills: collecting patient data, analyzing information, and designing regimens. Each dimension is rated from 1-5. An additional dimension evaluated the presence of hallucinations and errors with high harm scores (i.e., “absolute failure” criteria regardless of an overall score). The Krippendorff’s Alpha was highest in the medication therapy problem and medication therapy format categories, for 50 pneumonia cases, run in triplicate (150 total).
Conclusions
This framework is informed by national standards for CMM and the healthcare professionals dedicated to the provision of this service. These domains allow for the possibilities of practice variation via the preference domain while also having strong guardrails against the commission of medication errors. Further analyses beyond pilot testing are necessary.
Article notes
Competing Interest Statement
The authors have declared no competing interest.
Introduction
Comprehensive medication management (CMM) provided by clinical pharmacists integrated with interprofessional teams is the standard of care across the spectrum of care given it improves time to appropriate drug treatment, prevents medication errors and ADEs, and reduces mortality.(1–4) Defined as “a patient-centered approach to optimize medication use and improve patient health outcomes…that ensures patients’ medications (both prescription and nonprescription) are individually assessed to determine each has an appropriate indication, is effective and achieving defined patient and/or clinical goals, is safe given comorbidities and other medications being taken, and that the patient is able to take the medication as intended and adhere to the prescribed regimen,” CMM is a cognitive service aimed at individualizing medication therapy to maximize benefit and minimize harm.(5) Multiple interprofessional societies have the importance of CMM.(1, 3, 6–8)
The advent of generative and agentic artificial intelligence (AI) opens the door for AI-assisted CMM. However, a vast majority of AI achievements are centered around diagnosis (i.e., prediction modeling, differential diagnosis list generation, image interpretation) and surrounding healthcare tasks (i.e., ambient listening, note summarization).(9–27) Treatment poses a very different domain in that there are not clear-cut training datasets (i.e., a radiology image that does or does not show pneumonia), and medications in particular do not obey natural language constructs, instead consisting of structured alphanumeric combinations absolutely essential for safe interpretation. Additionally, the ability of AI to perform actual reasoning tasks, rather than list-generation has been called into question and represents an opportunity for improvement.(12, 28–31)
Recently, several papers have evaluated different large language models (LLMs) for medication tasks; however, the lack of rigorous evaluation of the outputs leaves open substantial potential for patient harm (i.e., suggesting an agent is capable of potassium dosing despite the agent having no understanding that renal dysfunction could result in overdose and fatal arrhythmias).(32–37)
A result of the lack of rigorous grading frameworks is false confidence in the knowledge and reasoning skills of an agent that has the potential to lead to patient harm. Here, our goal was to develop a clinician-designed, standardized framework for grading LLM performance on CMM-related tasks.
Methods
Study Design
This project was reviewed and approved by the University of Colorado Institutional Review Board (COMIRB #25-1631). All methods were performed in accordance with the ethical standards of the Helsinki Declaration of 1975.(38) This evaluation followed the transparent reporting of a multivariable model for individual prognosis or diagnosis (TRIPOD–LLM) extension reporting frameworks, as applicable (Supplemental Appendix).(39, 40)
CMM Framework Development
We developed a standardized evaluation framework for CMM tasks. A panel of five clinical pharmacists conducted a literature review and developed a standardized grading framework that included three tasks of CMM, as well as domains for evaluation. Resources used for creation of the framework included the American Society of Health-System Pharmacy (ASHP} Required Competency Areas, Goals, and Objectives for Postgraduate Year Two (PGY2) Critical Care Pharmacy Residencies and previous studies that have implemented a multi-part clinician grading scale.(41, 42) Additionally, the Harm Associated with Medication Error Classification (HAMEC) was incorporated into the framework to aid in categorization of potential medication-related harm as part of the generative AI evaluation.(43) This score ranges from 0 to 4 (Table 1).(43)
LLM Testing
After creation of the grading framework, we applied the framework to an analysis of LLM outputs when prompted about pneumonia treatment. A total of 50 cases involving community-acquired pneumonia (n=17), hospital-acquired pneumonia (n=19), and ventilator-associated pneumonia (n=14) were created following an institutional treatment protocol. Source material was taken from LexiDrug for each medication, with review by 3 board-certified pharmacists to ensure accuracy and clinical relevance.(44)
GPT5-Chat was provided with the institutional pneumonia pathway, a renal dosing protocol, and an aminoglycoside dosing protocol. The prompt used for the query is provided in Table 2. Each prompt was tested in triplicate with standard decoding parameters temperature = 1.2 and top-p = 0.95 (max_tokens = 4000).
Table 2 presents the prompt used to test the large language model (LLM) on pneumonia patient cases. The prompt instructs the LLM to act as a clinical pharmacist, review each hospitalized patient case, apply the provided antimicrobial and dosing guidelines, and generate an appropriate antibiotic treatment plan. The response is required to follow a structured JSON format with six fields: diagnosis, recommended antimicrobial therapy, relevant patient information, clinical goal, monitoring parameters, and duration of therapy. The prompt also requires the antimicrobial recommendation to be written in complete MedMatch format to ensure consistency and standardization across model outputs, where LLM = large language model; CAP = community-acquired pneumonia; HAP = hospital-acquired pneumonia; VAP = ventilator-associated pneumonia; PNA = pneumonia; RASS = Richmond Agitation-Sedation Scale.
Statistical Analysis
A total of three clinicians reviewed the outputs and graded them according to the proposed CMM evaluation framework. Descriptive statistics were used to summarize the evaluation results. Inter-rater reliability was assessed using Krippendorff’s Alpha, as it is well suited for studies involving multiple raters and can accommodate different types of rating scales while remaining robust to incomplete ratings if they occur. Krippendorff’s Alpha ranges from −1 to 1, where values closer to 1 indicate stronger agreement beyond chance. Given the hypothesis generating nature of this exploration, no attempt was made to calculate sample size.
Data Availability
The datasets are available upon reasonable request from the authors but are not posted publicly to avoid LLM training on the cases.
Results
CMM Evaluation Framework
The proposed CMM Evaluation Framework is displayed in Table 3 and visually in Figure 1. This was created and evaluated by a panel of 5 board-certified pharmacists using resources including ASHP Required Competency Areas, Goals, and Objectives for Postgraduate Year Two (PGY2) Critical Care Pharmacy Residencies, previous studies, and the HAMEC.(41–43) There are 3 CMM tasks evaluated (collecting patient information, analyzing information, and designing medication therapy regimens, monitoring parameters, and care plans). Within collecting information, there is 1 subtask (identification of relevant information). Within analyzing information, there are 2 subtasks (therapeutic goal and medication therapy problem(s)). Within design medication regimens, there are 5 subtasks (accuracy, clinician preference, medication therapy format, monitoring parameters, and duration). Finally, there are two automatic fail criteria, including any hallucinations of information and any proposed treatment that could result in serious or significant harm to the patient (HAMEC 3 or 4).(43) At minimum, accuracy, alignment with clinician preference, and format should be assessed when assessing a medication order. Other aspects (duration, monitoring, etc.) can be assessed if prompted. All portions of the proposed framework were tested to determine Krippendorff’s Alpha. Figure 2 shows the CMM evaluation framework and how it fits with the LTARC framework.(45, 46)
LLM Testing
GPT5-Chat was tested on the 50 pneumonia cases. These cases were graded by three board-certified pharmacists and rated for interrater reliability. GPT5-Chat performance is described in Table 4. Overall, GPT5 had variable performance, based on clinician assessment.
Interrater reliability was assessed among three pharmacists grading 50 pneumonia cases run in triplicate using the CMM rubric, with agreement quantified by Krippendorff’s α and 95% bootstrap confidence intervals. Interrater reliability between clinicians was highly variable depending on the category being evaluated. Reliability was strongest for Medication Therapy Problem and Medication Therapy Format (α = 1.00 for both), and moderate for Medication
Therapy Regimen Accuracy (α = 0.60; 95% CI, 0.48–0.70) and Clinician Preference (α = 0.58; 95% CI, 0.47–0.66). Duration showed fair agreement (α = 0.44; 95% CI, 0.22–0.61), while Identification of Relevant Information and Therapeutic Goal showed weak agreement (α ≈ 0.21 and 0.21, respectively). Monitoring Parameters demonstrated poor reliability (α = −0.21; 95% CI, −0.26 to −0.16), indicating disagreement beyond chance, and agreement on automatic-failure criteria was low (Serious or Severe Harm, α = 0.03; Hallucinated Information, α = 0.15). Overall, graders were most consistent on structural medication-therapy elements and least consistent on monitoring parameters and failure adjudication. Results are graphed in Figure 3.
Discussion
The CMM grading framework represents one of the first attempts to create a grading framework that takes into account multiple aspects of medication therapy and is aligned with the LTARC framework.(45, 46) Using standardized pneumonia cases that follow a comprehensive guideline, including antibiotic selection based on symptoms, allergies, and diagnosis, as well as renal dosing and aminoglycoside dosing guidelines, we sought to evaluate the reliability of this framework.
The proposed CMM grading framework builds upon prior work that evaluated LLM responses through the lens of safety, completeness, factuality, and preference.(42) However, this framework defines the components of CMM and divides the tasks into subtasks for evaluation. Additionally, each subtask has specific criteria for grading, in an attempt to improve standardization between clinician graders. Finally, this framework is unique because it includes two automatic fail criteria, for either proposing something that has the possibility to cause significant or serious harm or including any hallucination in the response. This is a key component to evaluation of LLMs, as they have been known to frequently hallucinate, and they are subject to less oversight compared to other medical tools, such as devices or medications.(30, 31, 34, 36, 45–49) Employment of these automatic fail criteria results in a definitive minimum competency, the ability of an LLM to not fabricate information and to provide a response that will not result in serious patient harm. Future directions would include definition of “good” or “excellent” responses and analysis of what is minimum competency for an LLM to be deployed in a healthcare setting. If an LLM can appropriately provide comprehensive medication management on 99% of cases, but recommends a medication regimen that could cause serious patient harm on 1% of edge cases, is that “safe enough” for regular use?(45, 46, 48–52)
Limitations of this analysis include a limited number of clinician graders, all of whom had similar training (post-graduate year 1 and postgraduate year 2 critical care pharmacy residency training). Future directions include analysis of other healthcare provider types as well as different case types. Additionally, these cases were designed to test every component of the CMM grading framework, but ideally the framework could be applied to any case involving medications, with accuracy, alignment with clinician preference, and medication therapy format being evaluated at a minimum. Further evaluation to ensure that the interrater reliability for these required components is high enough to ensure utility when used in silo is warranted.
Conclusion
This represents one of the first task-specific evaluation frameworks for CMM. Our CMM grading framework exhibited variable interrater reliability when tested on a set of pneumonia cases using GPT5-Chat.
Supporting information
Data Availability
All data produced in the present study are available upon reasonable request to the authors.
Conflicts of Interest
The authors have no conflicts of interest.
Funding
Funding through Agency of Healthcare Research and Quality for Dr. Sikora was provided through R21HS028485 and R01HS029009. Funding through National Library of Medicine for Dr. Gao was provided through R00LM014308.