Among all the included papers, few of them propose solutions to improve the reliability of LLMs. First, future research work should focus more on how to improve the accuracy of LLMs’ responses in the healthcare domain. More specifically, domain-specific health data are demanded for training and fine-tuning of LLMs to improve the performance of LLMs in various tasks in the healthcare domain. Therefore, data harmonization and consortia established for LLM training are potential directions that can benefit the broad research community. Qualified medical professionals can contribute to the creation of the dataset for LLM training. This, however, will be expensive in terms of time and effort [2]. Alternatively, using retrieval augmented generation (RAG) to augment LLM with external knowledge that is up to date might be a solution for scenarios where accurate in-depth professorial knowledge is required. Second, to prevent the hallucination issue, LLMs should be limited to making responses based on validated references. Blockchain technology can be used in this process to provide validation and traceability. Moreover, a holistic system, or a keep-experts-in-the-loop framework, that efficiently facilitates the expert validation process becomes important in order to improve the accuracy and safety of health LLMs. Third, clinical trials based on health outcomes such as mortality and morbidity rates should be conducted in clinical settings to validate the utility of LLM applications formally [1].

How conversational LLMs lead to bias or privacy issues in healthcare research was not thoughtfully examined with experiments in our reviewed papers. Future studies should first focus on investigating the mechanisms of how LLMs caused bias and privacy issues with stringent experiments and then developing practical solutions.

Regarding bias issues, it is suggested that systematic monitoring is necessary to ensure the impartial functioning of LLMs. However, all these sources discuss bias only with mere sentences and superficial summaries without any experimental investigation. Hence, it is worth noting that further work should also focus more on conducting experiments to understand how bias impacts the responses of LLMs in information, diagnosis, recommendation, and surveillance. More specifically, all applications of LLMs in healthcare should be tested regarding the exhibitions of bias, and the bias mitigation strategies such as data augmentation and targeted recruitment (e.g., the All of Us Research Program targets the collection of data from historically underrepresented populations [77]).

Regarding privacy issues, two technical approaches to mitigate the privacy risk while training LLMs are data anonymization [78] and synthetic data generation [79]. For deep learning models, model inversion attacks can potentially infer training data giving model weights [80]. Considering the exponentially increased open-sourced LLMs with published model weights, a sensitive patient dataset needs to be de-identified [81] or replaced with a synthetic dataset before being used to train or fine-tune an LLM. Otherwise, the patients with whom the data are associated should be informed about their participation in the training or fine-tuning process [82]. To solve the privacy issues, legal, social, and technical protection approaches need to be implemented together to ensure the privacy and security of the whole process of training and using LLMs for healthcare applications.

To raise the public acceptance level of LLMs, explainable AI should be employed to address the interpretability issues of LLMs by making the training data and model architecture transparent. More rigorous experimental studies using LLMs are encouraged in the “AI in medicine” research community to demonstrate or improve the reliability of LLM applications in healthcare. Moreover, stakeholders and decision-makers can propose new policies or regulations to manage the accountability and transparency of AI-generated content including the responses from LLMs.

There appears to be research that is beginning to address some of these raised issues. For example, Zack et al. assessed the potential of GPT-4 to perpetuate racial and gender biases in healthcare [83]. Hanna et al. assessed racial and ethical bias of ChatGPT in healthcare-related text generation tasks [84]. However, more research studies in these directions are needed to validate these findings and conduct more comprehensive and transparent assessments.

Moreover, almost all the research studies LLMs’ responses in one language. For example, 62 out of 65 study English, one focuses on Korean [2], one focuses on Chinese [45], and one focuses on Japanese [34]. Their findings cannot be extrapolated to other languages directly. Considering that many patients or people around the world or even in the US do not speak English, it is necessary to guarantee that LLMs are usable universally or equitably and conduct more research to investigate the performance of LLMs in other languages.