This study presents the evaluation for High Alert using Step 6 of the Intervention Mapping framework and reports findings from the formative, process, outcome, and acceptability evaluations. Given that Step 6 is often underreported compared to earlier steps in this framework, this study extends the current DUIC prevention literature and serves as a practical guide for researchers employing Intervention Mapping for evaluation.

The formative and acceptability evaluations underscore High Alert’s positive reception among youth, with most participants willing to engage with it and recommend it to peers. The program achieved high content ratings that were significantly greater than the various static infographics displayed to the active control. Examining the findings from each evaluation together provides additional insights. For instance, Module 4 on risky cannabis use received the lowest pre-test ratings and participant engagement scores, which suggests a need for potential refinements before future testing and implementations.

The outcome evaluation provided initial support for High Alert’s potential to reduce DUIC behaviours compared to no contact, particularly when considering driving after cannabis co-use. However, no differences were found when examining the DUIC (cannabis-only) variable, prompting a need for additional testing and better outcome variable measurement. High Alert participants demonstrated significantly greater knowledge gains when compared to the active control group and were more likely to report expectations of reducing DUIC afterwards. This suggests that High Alert’s contents and interactive elements (e.g., reflective exercises, real-life cases, multimedia) might have been more effective at reinforcing learning compared to static materials in the active control. However, knowledge gains did not translate into significant differences across other behavioural outcomes, highlighting a common gap between knowledge acquisition and engaging in risky behaviour [50,75,76].

Aside from knowledge, no significant changes were observed in other behavioural DUIC determinants. One explanation is that the theory-based methods and practical strategies used to deliver content (as described in [30]) may not have effectively influenced the targeted determinants. However, High Alert participants demonstrated the greatest improvements in mean change scores and had the highest proportion of individuals with improved attitudes across most determinants, even though they were not statistically significant. Alternatively, the statistical methods used to assess changes in determinants across groups (i.e., analyzing change scores with the Kruskal-Wallis H test or Mann-Whitney U tests) might have been insufficient to detect subtle changes over time. Although these non-parametric tests were chosen a priori due to the large number of Likert-type variables, violations of normality, and their use in similar studies [77–80], future research could benefit from more robust analytical techniques, such as repeated measures mixed modelling, which Teeters and colleagues [11] used to identify changes in perceptions of DUIC dangers over time.

The process evaluation recognized several recruitment and implementation challenges, offering insights for future improvement. Recruitment was hindered by skepticism about the study’s intentions, limited targeting options in paid advertisements, and the prevalence of bot responses on public social media. Bots are increasingly problematic in online research, and many published fraud prevention methods were ineffective in this trial [69–74]. While mass email recruitment was effective and attracted the least bot attention, it limited the sample diversity. Future research should focus on private recruitment strategies and establish early partnerships (e.g., academic institutions, high schools, driving schools) for mass outreach to verified participants. Public social media could be valuable for future widespread implementation but should be avoided in the interim efficacy testing to mitigate bot exposure. Retention was another challenge, with pilot trial attrition (57%) exceeding that of comparable studies [26,81,82] and highest among those reporting frequent cannabis use and DUIC [31]. This echoes findings from previous studies [83] and highlights the need for tailored strategies to improve retention among higher-risk participants. Another issue was adherence to the 7-day spacing between Sessions 1 and 2, a decision informed by expert and youth recommendations [12,13,30]. This split may have led to greater attrition, as 20% (n = 5) of those who started High Alert did not start Session 2. Unfortunately, we were unable to identify why participants did not adhere to or start the second session. No technical issues were reported, and emails may have been flagged as junk or ignored until the reminder was sent. Future research could also send session reminders via text messaging [84] and explore whether redesigning High Alert into a single session leads to better adherence and fewer dropouts than two separate sessions [85].

Integrating findings from all four evaluations offers a holistic view of High Alert’s initial implementation, preliminary efficacy, and areas for improvement in program delivery and testing. Reporting these findings from Step 6 provides valuable guidance for researchers conducting similar studies, enabling them to develop comprehensive evaluations and proactively address many of the implementation challenges we faced. Step 6 evaluations are often underreported [28–30,32–43], yet they are essential for understanding the complexities of program delivery. Without this data, there is a risk of misinterpreting the effect of interventions, as it overlooks critical elements such as fidelity, participant engagement, and contextual influences. By underscoring the importance of these evaluations, we advocate for more detailed and transparent reporting in intervention research, especially for studies targeting high-risk behaviours like youth cannabis use and DUIC.