Generative AI in a new era of computer model-informed tobacco research: a short report
Department of Population and Public Health Sciences, University of Southern California, Los Angeles, California, USA
Center for Precision Psychiatry, Massachusetts General Hospital, Boston, Massachusetts, USA
Department of Psychiatry, Harvard Medical School, Boston, USA
Department of Quantitative Social Science, Dartmouth College, Hanover, New Hampshire, USA
Dr Julia Vassey, Department of Population and Public Health Sciences, University of Southern California, Los Angeles, California, USA; vassey@usc.eduAbstract
Background
Social media influencers who promote e-cigarettes on Instagram or TikTok for tobacco brands use marketing tactics to increase the appeal of their promotional content, for example, depicting e-cigarettes alongside healthy lifestyle or entertainment imagery that could decrease youths’ risk perceptions of e-cigarettes. Monitoring the prevalence of such content on social media using computer vision and generative AI (artificial intelligence) can provide valuable data for tobacco regulatory science (TRS).
Methods
We selected 102 Instagram and TikTok videos posted by micro-influencers in 2021–2024 who promoted e-cigarettes alongside posts featuring four themes: cannabis, entertainment, fashion or healthy lifestyle. We used OpenAI’s GPT-4o multimodal large-scale visual linguistic model to detect the presence of nicotine vaping, cannabis vaping, fashion, entertainment and healthy lifestyle. The model did not require any additional training and improved its performance as we modified the text prompt.
Results
The model’s accuracy was 87% for nicotine vaping, 96% for cannabis vaping, 99% for fashion, 96% for entertainment and 98% for healthy lifestyle.
Conclusions
Generative AI can achieve accurate object detection with zero-shot learning (no additional training of the pretrained model). This model can be applied to big data-scale sample sizes of images and videos to detect e-cigarette-related and other substance-related promotional content and contexts (eg, healthy lifestyle) used for the promotion of these products on social media, providing valuable data for TRS.
Boxed Text
WHAT IS ALREADY KNOWN ON THIS TOPIC
- Research has shown that computer vision models can be trained to detect e-cigarette objects in social media videos, including e-cigarette promotional content posted by influencers. Generative artificial intelligence (AI) models have not yet been applied to detect e-cigarette or cannabis products and different lifestyle contexts such as entertainment, fashion or healthy lifestyle, depicted alongside e-cigarette promotions, in social media videos.
WHAT THIS STUDY ADDS
- A generative AI model (GPT-4o) was able to accurately detect the presence of nicotine and cannabis vaping as well as entertainment, fashion and healthy lifestyle themes in Instagram and TikTok videos featuring micro-influencers. The model did not require any additional training and improved its performance as we modified the text prompt guiding the model to detect these themes in videos.
HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY
- The pilot study demonstrated that GPT-4o and related models can now be applied to big data-scale sample sizes of images and videos to detect not only e-cigarette or other substance-related (eg, cannabis) promotional content but also contexts (eg, healthy lifestyle) used for promotion of these products on social media, providing valuable data for tobacco regulatory science.
Introduction
Social media influencers who promote e-cigarettes on Instagram or TikTok for tobacco brands use diverse marketing features to increase audiences’ appeal to their promotional content. For example, micro-influencers (content creators with∼10 000 to ∼100 000 followers)1 often promote e-cigarettes alongside posts featuring positively perceived lifestyle contexts or themes (eg, healthy lifestyle, gaming) or alongside cannabis use.14 Such marketing may affect youth perceptions of influencer relatability and contribute to normalisation of e-cigarette and other substance (eg, cannabis) use.58 While some social media platforms’ community guidelines (eg, Instagram and TikTok) prohibit influencer e-cigarette-related promotional posts, they still exist on social media.9 The platforms detect and remove some of this content10 but do not use efficient algorithms to eliminate all e-cigarette promotional posts.
Data on the prevalence of tobacco-related promotional posts is valuable for tobacco regulatory science. Prior studies used human qualitative coding1115 and automated machine learning approaches1620 to analyse tobacco-related images and videos. Automated approaches include computer vision, a field of artificial intelligence (AI) that applies machine learning and artificial neural networks to analyse visual content of image-based and video-based e-cigarette-related promotional content on social media. Prior research16 20 used convolutional neural networks-based models (that gradually learn information from simple to more complex patterns of an image)21 to classify e-cigarette products and nicotine warning labels in e-cigarette-related Instagram images. Another study19 applied a transformer-based model (that selectively focuses on important image regions)22 to identify e-cigarette-related products and attributes (ie, mod and pod devices, e-juices and their flavour names, nicotine warning labels and e-cigarette brand names). These models required large training samples to detect research-specific objects.19
Newer, more efficient transformer-based models such as OpenAI’s GPT-4 (generative pretrained transformers) can often solve human-level object detection tasks without additional model training.2325 These models, originally trained on text and visual elements from numerous online open sources, can detect objects accurately without building a custom model fine-tuned on a training data set (ie, ‘zero-shot’ learning).26 GPT-4 outperformed over 20 other multimodal models (eg, BLIP-2, LLaVA-1.5) that process text and image-based data.27 GPT-4’s capability of analysing images and videos is an upgrade from the earlier GPT-3 version that analysed text data.25 28 In this pilot proof-of-concept study, we applied GPT-4o, a multimodal large-scale visual linguistic model (LVM), to detect nicotine or cannabis vaping, and contexts (ie, healthy lifestyle, entertainment and fashion) appearing alongside promotions of these substances, in Instagram and TikTok videos featuring micro-influencers.1
Methods
Description of themes
We selected the influencer videos featuring nicotine vaping, one of the four other themes, or a combination of those in one video; 22 videos contained nicotine vaping (disposable or refillable pod devices), while 20 videos each featured cannabis (disposable pod devices, blunts or bongs (water pipes)), healthy lifestyle (exercising, hiking, playing tennis, swimming or skateboarding), entertainment (DJ-ing, dancing) or fashion (modelling); 12 videos contained multiple themes.
Model architecture
We applied a novel model that uses the transformer architecture and OpenAI’s GPT-4.23 24 28 GPT-4o, a multimodal LVM, combines text and visual comprehension.25 It takes images and videos (broken into frames that are analysed as images) as input and, using text instructions in a prompt we engineered to guide the model, provides textual responses based on the visual content. Model performance gets improved with prompt modifications. Figure 1 shows an example of an input video, the prompt we engineered to guide the model, and the output. The prompt guided the model to provide a probability of a specific theme’s presence in a video. We iterated through multiple prompt revisions until we found reliable model outputs that could be extracted using simple text processing. To verify the robustness of the results, we ran the same prompt three times30 and the results remained consistent. For theme detection in each video, we selected four frames evenly distributed over the whole duration of a video. The decision to analyse four frames instead of the entire video was driven by efficiency and to minimise the cost per token (the amount of information processed per query) set by OpenAI’s API. The output included one probability per theme in a video based on the prompt (figure 1).
Model performance evaluation
The model accuracy was evaluated against the true content observed in the videos, as described by human coders (a graduate student at the University of Southern California and an author of this manuscript (JV)). Inter-rater reliability (Kappa) was 0.99. We assessed GPT-4’s accuracy, the area under the curve (AUC), precision, recall and F1 score for all themes. To validate GPT-4’s performance on the nicotine vaping theme, where it showed the lowest accuracy, we used the DyHead model developed in our prior research specifically for e-cigarette object detection.19
Results
GPT-4o’s accuracy was 87% for nicotine vaping, 96% for cannabis vaping, 99% for fashion, 96% for entertainment and 98% for healthy lifestyle, with AUC, precision, recall and F1-score ranging from 0.75 to 1 across the themes (online supplemental table 1 and online supplemental table 2). The DyHead validation model achieved 81% accuracy for e-cigarette (nicotine vaping) theme detection—comparable but slightly lower than the GPT-4o model’s 87% accuracy for nicotine vaping. The GPT-4o and DyHead models’ AUC (online supplemental table 1), precision, recall and F1-scores (online supplemental table 2) for nicotine vaping detection were also comparable but slightly lower overall for DyHead. Online supplemental table 3 and online supplemental table 4 present the GPT-4o model’s predicted probabilities for theme detection across all videos and for multiple themes in 12 videos, respectively. Online supplemental figure 1 provides error analyses for both models, while online supplemental figure 2 displays the distribution of confidence scores from the validation model.
Discussion
This proof-of-concept study showed that GPT-4o can accurately identify e-cigarettes, cannabis, entertainment, fashion and healthy lifestyle. GPT-4o and related models can now be applied to big data-scale sample sizes of images and videos to detect not only nicotine or cannabis products but also contexts appearing alongside promotions of these substances on social media. Estimating the prevalence of these marketing features is important since they could increase audiences’ (including youth) appeal to e-cigarette products and contribute to normalisation of e-cigarette and other substance use.
GPT-4o achieved high accuracy with zero-shot learning, that is, without explicit training on a labelled data set of images or videos. This high accuracy may reflect the scope of GPT-4o’s pretraining and potential exposure to similar content. Performance improved as we modified the text prompt guiding the model. Zero-shot learning was not possible in prior studies that used non-generative AI transformer-based models which required a large training sample of images. Fine-tuning of GPT-based models might still be needed for some research tasks; however, their zero-shot performance will likely only improve over time as these models get constantly upgraded.
GPT-4o’s performance on small object detection such as nicotine vaping was lower than on the other themes but slightly higher than DyHead’s performance on nicotine vaping detection. The non-OpenAI DyHead model (developed in our prior research)19 was specifically chosen for validation to compare OpenAI’s GPT-4o performance with non-OpenAI approaches, particularly for small object detection, such as e-cigarettes, where GPT-4o demonstrated its lowest performance among the evaluated themes. DyHead is an e-cigarette object detection model focusing on spatial object recognition whereas GPT-4o identifies broader themes by analysing data without spatial localisation. While using DyHead model predictions as a comparator for the GPT-4o model’s results is reasonable, the DyHead model provides more detailed frame-level inference not currently possible via GPT and was intended for a different use case than video-level theme identification. As such, comparing theme-detection accuracy between the two models captures only one aspect of their respective utilities.
GPT-style models provide new self-service capabilities to tobacco researchers due to the web-based interface for inputting research questions in written language and generating model predictions. Researchers without deep learning expertise are now empowered to define marketing themes, iterate on prompts and take advantage of these models without coding directly or creating custom models that need to be maintained and updated over time. GPT-4 can also be useful to analyse data sets with no protected health information, (as is the case with publicly available social media imagery used in this study). Overall, this hints at a new era of model-informed tobacco research broadly available throughout the research community. However, the goal of this pilot study was not to identify the best suitable model but rather to explore an alternative to the custom models used in prior research.1620 This manuscript offers the scientific community the opportunity to see the strengths and limitations of the GPT-4 and make a decision about which model suits best to their research question.30
Limitations
GPT-4o sometimes misclassified cannabis vaping as nicotine vaping, likely due to the high resemblance of disposable devices,4 which could only be distinguished by cannabis logos or brand names that the model occasionally missed. False negatives for nicotine vaping detection for both GPT-4o and DyHead were caused by obscured objects (a white pod on a white background) and missed objects likely due to sampling only four frames (in one video a pod did not appear throughout the whole clip). GPT-4o failed to detect one older-generation pod device, likely due to the absence of such examples in the pretraining data set, whereas DyHead identified it accurately. Future research could evaluate GPT-4o’s performance on out-of-distribution samples (eg, niche topics or unique objects) to better understand its generalisability. DyHead was used only for nicotine vaping detection. We treated any disposable vaping device as nicotine vaping while assessing DyHead’s accuracy, since the model was specifically designed for e-cigarette object detection and was not trained to distinguish between identically shaped disposable nicotine and cannabis vape devices based on logos or brand names shown on the device. Unlike GPT-4o, DyHead cannot detect broad themes (eg, fashion). Given GPT-4o’s high performance on broad theme detection in this study and its superior performance compared with other models (eg, LLaVA-1.5) in prior research,27 we did not apply alternative models to validate its performance on broad theme detection. We used only 102 videos for this study. Although small sample sizes are typical for pilot studies, future work should evaluate larger samples to estimate performance more precisely.
Conclusion
This proof-of-concept study showed that the GPT-4o model was capable of accurate e-cigarette-related and non-e-cigarette-related theme detection featured in influencer marketing without any fine-tuning of the model (additional training of the pretrained model to improve accuracy). The study adds evidence to the existing literature1620 that computer vision and generative AI models can now be applied to big data-scale sample sizes of images and videos to detect not only nicotine or cannabis products but also contexts appearing alongside promotions of these substances on social media.
Supplementary material
Data availability statement
Data and the python code underlying this article are shared on GitHub: https://github.com/e-cigarette-marketing-ml/vaping-gpt