In this study, we demonstrated that large language models (LLMs) are capable of generating executable, often fully functional Python code for data analysis tasks relevant to aggregated health datasets provided to the public. In many cases, the models produced correct outputs without requiring manual modification, successfully interpreting the CSV schema and applying appropriate analytical logic.

One of the most important lessons learned for Objective 1 is that state-of-the-art models such as ChatGPT-4o and GPT-4.1 mini handled typical analytical tasks (e.g., filtering, aggregation, calculating weighted averages) with high reliability, even in a non-English context. The presence of diacritics, local column names, and Czech-specific medical terminology posed no substantial barrier. Most models processed the few-shot examples correctly and adapted to the structure of the data with remarkable accuracy. A surprising and encouraging finding was that the majority of outputs were not only syntactically valid but also executable without errors. This suggests that LLMs may already be suitable as assistants in the early stages of analytical pipeline development, particularly in contexts where users need support in transforming natural-language questions into executable code. Indirectly consistent with our findings, a recent evaluation on Brazilian medical aptitude exams (non-English context) similarly found that GPT-4o achieved the best performance, and in addition to the observed results herein, large-scale proprietary models substantially outperformed open-source models such as Qwen and LLaMA [16]. In another 2025 study [17], the authors showed that when using carefully designed prompt-engineering strategies, GPT-4o ranked among the top performers in validating drug-repurposing hypotheses.

Nevertheless, performance varied across models. Claude and Gemini, for example, frequently failed to apply the required filtering for primary diagnoses, despite this instruction being explicitly stated in the input prompt. This limitation illustrates that even when models can parse code structure, they may overlook essential logic embedded in the task description. Interestingly, Gemini 1.5 Pro was able to correct such mistakes in subsequent queries, suggesting potential for intra-session adaptation. DeepSeek-R1, in contrast, consistently generated responses in English rather than Czech, which may limit its utility in public sector or healthcare contexts where local language use is essential. It is important to note that, although DeepSeek-R1 is currently prohibited for production use under many institution policies, we still considered it essential to include the model in our benchmarking because it is one of the few high-performing LLMs that can be fully self-hosted. Nevertheless, real-world adoption in our context should prioritize models that combine self-hosting capability with robust local-language support and compliance with institutional security requirements (e.g., secure variants of LLaMA, Qwen, or other models validated for privacy-preserving deployment).

Some tasks further revealed that models could generate syntactically valid and executable code that nonetheless failed to match the expected results. This was particularly evident in Query 1 and Query 2, where the outputs “looked correct” but returned erroneous answers. These subtler forms of failure highlight the necessity of systematic validation processes, even when the generated code executes without error.

For Objective 2 (query customization), we observed a moderate degree of variability across models. In practice, it is often necessary to explicitly instruct an LLM to perform certain operations, such as calculating a weighted average, if this is not done automatically. Encouragingly, providing clarifications in Czech often resulted in immediate corrections, demonstrating that models can adapt rapidly within a single session. For non-programmer analysts, this type of interaction is not a significant barrier, but rather a natural part of iterative query refinement.

For Objective 3, we explored a more open-ended, end-to-end use case, where the model was not provided with a predefined dataset but instead had to identify suitable resources, acknowledge their limitations, and propose an analytical strategy. ChatGPT-4o demonstrated the ability to navigate the NZIP catalog, retrieve relevant datasets, and generate both descriptive limitations and executable Python code tailored to the research question. The returned outputs were not technically incorrect but were contextually relevant, incorporating epidemiological reasoning such as COVID-related diagnostic effects, screening program initiation, and the constraints of aggregated data.

To our knowledge, this is the first systematic comparison of LLMs applied to real-world public health queries over structured healthcare datasets. Related work has evaluated LLMs in adjacent areas, such as their performance in medical education. For example, a recent comparison of DeepSeek-R1 and ChatGPT-4o on the Chinese National Medical Licensing Examination [18] illustrates the growing interest in benchmarking LLMs within healthcare-specific contexts.

An important methodological question concerns the reproducibility of results across repeated runs. Since ChatGPT-4o emerged as one of the best-performing models in our study, we examined whether its outputs remained stable when identical prompts were re-issued. A previous study [19] reported that while the free version of ChatGPT showed variability over time, the paid versions remained stable, with no significant changes across updates. Consistent with these findings, another study on information extraction from electronic health records (EHR) [20] found that GPT-4o produced perfectly reproducible outputs across repeated runs at temperature 0. In our own non-systematic, selectively chosen retest attempts, issuing the same prompt from scratch generally produced nearly identical code and the same numerical outputs, supporting the view that averaging over multiple runs may be unnecessary for this class of analytical tasks. However, broader testing would be needed to generalize this observation. Since temperature=0 was used, results should be deterministic.

A key limitation of this study is the scope and complexity of the analytical queries. We focused primarily on well-defined, moderately complex tasks over a single class of public health datasets. Future work should expand the set of tasks to include more advanced analytical operations, such as multi-step aggregations, time-series modeling, and predictive analyses, in order to better assess the upper limits of LLM capabilities in applied public health informatics. While execution time was uniformly fast (under one minute per query), our study did not systematically evaluate performance trade-offs under conditions involving larger datasets, high-dimensional tables, or real-time constraints. Several recent studies highlight the potential for resource-efficient LLMs in data analysis. For example, López Espejel et al. (2024) demonstrated that smaller, open-source models can achieve competitive performance when properly configured [21]. Their findings suggest that lower-cost models could be leveraged for public health analyses, particularly when local deployment or privacy considerations preclude the use of commercial cloud models.

Another important consideration is knowledge verification. Generative LLM outputs, while syntactically and logically plausible, must be validated before adoption in clinical or epidemiological analyses. Recent work in biomedical generative AI [22] demonstrates that models can generate disease-centric associations and structured biomedical knowledge, but ontologies and curated vocabularies are essential to verify terms for diseases, drugs, symptoms, and genetic relationships. Translating this principle to public health data analysis implies that generated code and aggregated results should be checked against official coding standards (e.g. ICD-10), population registries, or other authoritative references.

Related software solutions in the EHR domain illustrated practical approaches to end-to-end reasoning and query generation. For example, ClinQuery [23] implemented a pipeline from named entity recognition to SQL query generation and natural language reasoning, enabling retrieval of insights from complex clinical datasets. Similarly, graphRagSqlator [24] can explore database schemas automatically and generate valid SQL queries from natural language prompts. These systems are designed to handle large databases with heterogeneous table structures, i.e. possibly challenges analogous to those faced when working with NZIP or other multi-table public health registries. By contrast, our work demonstrates that LLMs can already generate fully functional Python code over structured CSV datasets, providing a foundation for future extension to larger, relational patient-level datasets.

Our findings suggest that contemporary LLMs are increasingly capable not only of generating technically correct analytical code, but also of interpreting the clinical meaning of aggregated health indicators and identifying variables that are relevant to complex associative questions. Remarkably, several models demonstrated an ability to reformulate natural language requests into multi-step analytical strategies that approximate human reasoning about disease patterns, diagnostic pathways, or temporal dynamics. Even when working with aggregated data, where clinical nuance is inherently constrained, the models were often able to infer which dimensions (e.g., age, sex, diagnosis type, region, or temporal trends) were likely to carry meaningful epidemiological information. This capability has profound implications. As public health agencies continue to expand the availability of open aggregated datasets, LLMs could substantially empower citizens, journalists, clinicians, and policymakers who lack formal training in analytics, effectively lowering the threshold for accessing and interpreting population health trends. At an institutional level, these technologies raise the possibility of shifting human analysts away from manual coding and routine exploratory tasks toward higher-value work such as designing richer and more semantically coherent datasets, curating metadata, and developing synthetic datasets that mirror patient-level structures such as disease history. In such a future ecosystem, LLMs may function as scalable analytical engines, while human experts focus on ensuring data integrity, ontology alignment, and interpretability. Ultimately enabling more sophisticated, reproducible, and clinically meaningful analyses across the public health data landscape.