To improve data quality and reduce invalid responses for an online survey a real-time comparison of response consistency was trialed during data collection by using two administrations of the ASSIST cannabis scale in an Internet survey. Nearly half (45.3%) of individuals who met initial eligibility criteria were later screened out for inconsistent responses. These individuals tended to be slightly older, used cannabis less frequently in the last 30 days, and had lower total ASSIST scores at screening and baseline compared to those who were screened in. There are many possible explanations underlying these differences including, recall bias or even falsifying data in an attempt to avoid detection. Nonetheless, future research should try to understand whether this is a result of careless responding, recall bias, or due to greater reliability issues of web-based administration and if these vary among different groups.

Without the two-step screening process, the sample would have consisted of 329 (43.5%) participants with inconsistent responses. This sample would have been slightly older with lower total ASSIST scores at screening and baseline. It is possible that the screening process excluded valid respondents and as a secondary analysis, it is not possible to determine how this would have affected the results of the RCT. It was not expected that nearly half of the individuals screened would have inconsistent data, however, if these cases were discarded post-hoc during data cleaning, the study would have been severely underpowered jeopardizing the legitimacy of the results. Alternatively, keeping such cases during data cleaning could have had equally devastating impacts by significantly inflating or reducing potential associations between intervention condition assignment and cannabis use outcomes at follow-up [16].

The finding that nearly half of potential participants were screened out is concerning and suggests that invalid responding may occur at a high frequency in Internet surveys and conservative approaches for identifying invalid responding needs to be balanced with the risk of removing valid cases, and Type I /II error [1, 2, 5, 7, 10]. Establishing the point where invalid cases cause more harm than the risk of removing them [9] will be important for future research, however, including at least one measure to identify invalid cases is nonetheless recommended for online surveys [1, 2]. All methods may not measure the same construct [17]. but given the multiple underlying causes for invalid responding, any one method could still improve data quality. A two-step recruitment process, using a consistency check like the one implemented in this study merits further investigation. The ability to program the consistency checks automatically in survey programs is especially useful and provides an alternative to post-hoc data cleaning methods.

The advantages of conducting Internet-based survey research has contributed to the increased use of this modality in many fields. Internet surveys are generally affordable, easy to administer, accessible, and can accommodate complicated question skip patterns [10]. Regrettably, Internet survey research is not without its unique challenges. Invalid responding is a recognized data quality issue and screening for this issue needs to become common practice [5] in addictions research. Furthermore, although findings from this study need to be replicated, there may be unexpected benefits to implementing this two-stage method of screening. We speculate that the extra screening undertaken during recruitment may have reduced fraudulent responses and resulted in a more conscientious sample, which subsequently contributed to the high follow-up rates (92% at 3-months; 91% at 6-months) post baseline.

As a secondary analysis, this project was not powered to address invalid responding and its design did not include a control group using another mode of administration. Similarly, the associations discussed here may not generalize beyond Internet surveys of individuals who use cannabis. Another limitation of this analysis is that motivations or reasons underlying the inconsistent responses were not investigated and thus, other explanations or biases may be involved. During development testing of the ASSIST, participants were asked to explain the reasons for their inconsistent responses. In addition to inattention, other common responses included not understanding the question and remembering or forgetting information between testing [15]. While these issues may still contribute to the inconsistent responses, some issues such as question clarity have been improved in subsequent revisions of the ASSIST, including the version used in this study. Nonetheless, we are unable to speculate further as to the source of the invalid responding observed. That is, it is unclear as to whether the inconsistent responding was a limitation of the measure used, the mode of administration (i.e. Internet survey), or a result of participant carelessness or deliberate attempts to be eligible for the study. Whether related to inattention or another variable, inconsistent answers should nonetheless be identified and used with caution in analyses. Finally, time to compete each survey was not recorded, but should be included in future studies as completion time has been associated with invalid responding.

The inconsistent responses observed in this project could be the result of measurement error or an individual making a mistake on a response; however, the inconsistencies could also be related to carelessness, fraudulent responders, or the randomness of bots. Thus, some change in answers is expected between repeated administrations of a tool (e.g., a participant may remember new information). On the ASSIST specifically, such changes can alter the total score by one to four points per question. Placing an upper limit of 10 points on the changes balances the expected measurement error while attempting to reduce the risk of invalid responding. Furthermore, previous research has demonstrated that the web-based measures of the ASSIST has less internal consistency compared to paper-and-pencil administrations [18]. A 10-point difference on the cannabis ASSIST scale demonstrates significant behavioural changes and could potentially have an impact on outcome data [19] and a primary concern for this project was related to the eligibility criteria of moderately risky use. Brief interventions (like the one being tested in the RCT) are recommended for individuals scoring in the moderate range on the ASSIST (4–26). Therefore, it was of particular concern that a 10-point decrease could change a participant from moderate to low risk. Conversely, a 10-point increase was less likely to move a participant from moderate to high risk where a brief intervention would possibly be less beneficial. Nonetheless, future research should systematically investigate the impact of a 10-point cut-off for inconsistencies on the ASSIST.