The doubling year over year growth rate of AI in healthcare publications continues, with total publication numbers exceeding 23,000 based on our search methodology (Figure 1). Similarly, the growth rate of mature publications based on our methodology has also been sustained with mature publications more than doubling in number compared to the year prior (Figure 1). With the introduction of ChatGPT and other LLMs since 2022, for the first time significant rise in use of text data(11.6%) and large language models(8.2%) was observed this year. These have historically ranged between 0% to 8.5%[7].

Our methodology was modified from the years prior as it was no longer possible to manually curate such a large number of publications for exclusions, speciality classification or provide subject matter expert viewpoints[7]. Publication in some of the key specialities such as Radiology could possibly have exceeded 10,000. We also believe the trends are reflected in mature publications for each speciality and are adequate to provide substantial information for the purpose of this analysis. We did try various model based classification methods including BERT classifier, LLMs, including prompt engineering LLMs and fine tuning them on our prior years data for the purpose of speciality classification but their results remained suboptimal with classification accuracy parameters ranging from 30-68%.

In 2023, Imaging, GI, Ophthalmology, Oncology, Surgery, Head & Neck, General categories continue to lead the healthcare specialties in the number of mature publications. These specialities represent the healthcare specialities which use images as data type and deep learning models,mostly convolutional neural networks. These healthcare specialties also have a significant number of open source datasets, datathon like competitions and support from their respective societies towards provision of resources to foster AI research[10]. Surgical specialties relative to other medical healthcare specialties tend to outperform in research and publication trends, again primarily related to image based data and application models. COVID-19 publications,as expected, continue to decrease as the pandemic is resolved. Genetics related mature publications also decreased, but the reasons are less clear. Anesthesiology, Critical Care, Nephrology, Rehabilitation continue to have low volumes of publications as they lack image based data for research. Also important to note that in 2021, we observed a drop in publications, which was hypothesized to be due to COVID-19, as in prior publications[7].

Imaging publications continue to focus on various imaging modalities including interpretation of X-rays, CT scans, MRIs and ultrasounds[10–12]. Majority of this research work utilizes deep learning, more specifically and not unexpectedly convolutional neural networks[13,14]. There were a few publications which focus on optimizing image processing and clinician workflows[15]. Most of the Gastroenterology publications continue to focus on trials related to application of deep learning models for endoscopy for diagnosis or to measure treatment response[16,17]. Surgical publications including those for Head and Neck, Orthopedics are focused on image analysis using deep learning to guide surgical diagnosis or decision making.Its interesting to observe a significant number of dentistry research work related to use of image interpretation of dental X-rays or CT scans[18,19]. Oncology publications use both ML and DL to guide diagnosis and treatment decision making. For oncologic diagnosis,use of AI in various cancer screenings such as for breast,cervical and prostate cancer[20]. For management many of the oncology publications focused on radiotherapy planning and predicting or measuring treatment effects for various cancers[21]. Pathology publications continue to focus on use of AI on whole slide images or cytopathology for diagnosis using deep learning[22,23]. Cardiology publications use multimodal data including EKG, CT angiograms amongst others for diagnosis and prediction of arrhythmias or ischemic heart disease,respectively[24–26].

Similar to the prior years, imaging data emerged as the top data type used this year as well, and deep learning models remained the most utilized model type. A significant increase in the utilization of text data was also observed, which may be attributed to the rapid advancement of large language models (LLMs). This advancement was evident in the distribution of model types across all papers, with LLMs being utilized in 8.2% of cases. Additionally, we have witnessed the development of Large Vision Models (LVMs), indicating progress towards more robust multimodal model development, which is essential for future AI healthcare advancements. Among all mature articles, two articles from the LVM category stood out: one in the medical specialty of Pathology, where the author built a Hybrid Vision Transformer for learning gastric histology[8,27], and the other in the Emergency Medicine, where authors developed multimodal models to facilitate the interpretation of chest radiographs in the Emergency Department[28]. Since the distribution of utilized data types and model types is a primary and key indicator of understanding the diversity in overall AI research development, we hope to see more diverse datasets and models utilized to enhance overall AI in healthcare development

Despite large language models being present prior to November 2022 (Figure 6), their use in research related to healthcare was insignificant. Since the release of ChatGPT, there has been a tremendous interest in the use of GPT and large language models in healthcare. Initial publications are related to trials of mostly GPT[29]. These trials have focused primarily in a few defined areas: a.) comparison against clinician performance at various tests, b.) summarization tasks, c.) question and answering tasks for patient or clinician questions or recommendations[30–34]. Hence, publications related to comparison between clinician experts’ performance on proficiency tests such as board examinations were significant[35,36].Almost all these publications though used GPT for comparison and without its modification,i.e.,without further significant prompt engineering or fine tuning.

We acknowledge certain limitations of our analysis based on our search or analysis methodology. Search was limited to PubMed and with the restrictions mentioned in the methodology section. It is possible that some of the significant publications might have been missed due to our methodology.In prior years, we have manually excluded many publications from the PubMed search and classified publications into various select healthcare specialities.This year due to the volume of publications(23,306), it was not possible to perform any manual evaluation of all the publications. We did trial various methods including BERT and GPT for automated classification including fine tuning on prior data but the results were not satisfactory. Analysis of mature publications provides a more focused review and a consistent methodology also allows us to provide year over year comparative evaluations[7].

Since, our maturity model is primarily based on the BERT architecture[37], which has limitations such as a lack of understanding of out-of-vocabulary words. BERT’s vocabulary is limited to the tokens it was trained on, so it may struggle with out-of-vocabulary words or rare terms. Therefore, our maturity classification is sensitive if abstracts of publications have used terms that are new for the model. Additionally, the BERT model has a limit in maximum token size of 512, so if the combined length of the title and abstract exceeds 512 tokens, the model may not be able to understand the entire context[37].

We have also evaluated alternate approaches to learn maturity, specifically focusing on methodologies derived from scholarly literature and journal metrics, such as no. of citations, journal impact factors, and journal H-index etc. However, it is important to note that using these metrics to evaluate maturity can be misleading when applied to the analysis of articles from the most recent year due to the unpredictable dynamic trends of these metrics[38–40]. The current BERT-based approach, despite its limitations, remains content-aware, reproducible even with changing factors such as article citation count and journal impact factor, and fine-tunable with new data.

LLM search methodology is limited to search using “LLM” and “GPT” only. This was our preferred method as our trials with other words and sequences yielded a high number of false positive searches.