Measurement validity is an important consideration when designing studies and interpreting findings that rely on self-reported nicotine use measures. As market-places grow in complexity, so does the assessment of nicotine use. This Special Communication considers ramifications of a diversifying market-place on survey research. Two areas of concern are most evident from our observations:

1. Measuring HTP use is challenging where HTPs are new to the market but vaping products are already well established, and also in jurisdictions where verbs are used interchangeably when referring to HTPs and cigarettes.
2. Modalities for administering cannabis are converging with those for nicotine, making it important to capture information not only about ‘product classes’ but also the substances being administered.

As the latter concern is already being discussed within the scientific community,^40–43 we will largely focus on the first issue below. This is not to say the latter concern has been addressed to the fullest extent. Many national or international surveillance systems have yet to modify questionnaires to collect information about which substances are being administered during use of various products/devices; few validation studies have quantified the extent of misclassification introduced by this issue; and the relative prevalence of cannabis and NVP use versus that of HTPs in many regions makes this a pressing concern. Moreover, the two issues are interconnected, and increasingly so as cannabis products that emulate HTPs continue rolling out (eg, Omura’s Series X). Some survey strategies noted below are relevant to the broader challenge of disentangling substances from devices or products. Avoiding measurement error when assessing electronic inhalable nicotine product use will require awareness of trends and products in the cannabis market.

That aside, we contend that the challenges of measuring HTP use have yet to be appropriately recognised. This notion is supported by a handful of studies scrutinising self-reported HTP measures. Brose et al ⁴⁴ performed split sample tests to compare two survey strategies for collecting self-reported awareness of HTPs. Half of a 2017 general adult population sample in England reported their HTP awareness and ever use status after being shown the preamble ‘Heat-not-burn tobacco products use a technology whereby tobacco is being heated as opposed to being burnt’, while the other half’s survey further included ‘Some of the popular brands of heat-not-burn tobacco products include Ploom and IQOS’. The prevalence of self-reported HTP awareness and ever use was 7.8% and 1.3%, respectively, among the respondents who were shown written HTP device brand examples, compared with 10.8% and 2.2% for the other group. Lee et al ⁴⁵ examined the test–retest reliability of nicotine use measures among 121 self-reported nicotine users from a 2019 web survey of Korean adults. While use patterns involving cigarettes and/or NVPs demonstrated moderate to high agreement, HTP use patterns (eg, exclusive use, HTP–cigarette concurrent use and HTP–NVP–cigarette concurrent use) were highly discordant. In a 2018 sample of nicotine users from England, Canada, Australia and the USA, Miller et al ¹⁸ evaluated responses from self-reported ever users of HTPs about the device brands they had used. Approximately 6% of self-reported HTP ever users reported a brand that was an NVP or cannabis vapouriser and 24% selected ‘don’t know’.

Even slight differences in survey structure can impact a survey’s ability to obtain accurate information on HTP use. For instance, using a ‘check all that apply’ item to simultaneously assess multiple nicotine products (eg, ‘Which of the following products have you ever used…’) is likely more susceptible to underestimating product use than stand-alone forced-choice survey questions (eg, ‘Have you ever used an HTP?’…Yes/No…; ‘Have you ever used an NVP?’… Yes/No…; etc).⁴⁶ Surveys may benefit from incorporating descriptive preambles that aid respondents in disentangling product classes from one another.^{47 48} This could take the form of written product class definitions, examples of specific product brands, device images embedded throughout the survey and so forth. Depending on the jurisdiction in which a survey is being administered, it might be preferable to enquire about individual HTP brands rather than the overarching product class (eg, IQOS is currently the only Food and Drug Administration-authorised HTP device in the USA). Other considerations include the nomenclature used to differentiate product classes (eg, referring to HTPs as heat-not-burn; distinguishing HTPs from NVPs with ‘HTPs contain actual tobacco leaf…NVPs do not contain actual tobacco leaf’); the order in which product classes are asked about (eg, does the survey first ask questions about cigarettes, NVPs, HTPs, etc); the formatting of questionnaire items (eg, for ascertaining device brand information, should surveys use a dropdown list (eg, similar to the Miller et al survey¹⁸) or open-ended format?); and the overarching survey structure (eg, is the survey largely sequenced around individual product classes or more so around intention/perception/opinion/etc constructs?).

There is an urgent need to evaluate measurement validity of self-reported HTP use, both to understand the extent of misclassification in surveys and to discern which of the above methods most effectively mitigate misclassification. While the criteria frequently relied on for validating self-reported tobacco use (biomarkers of exposure to nicotine (eg, cotinine) and tobacco (eg, tobacco-specific nitrosamines)) are not always specific to individual product classes, other strategies can provide insight, as demonstrated by Brose et al,⁴⁴ Lee et al ⁴⁵ and Miller et al.¹⁸ Survey developers must consider incorporating items that probe beyond a singular initial question about product use. This should involve ascertaining information about the device brand(s) that a respondent has used, and potentially other factors, such as device ownership. Another tactic for web-based surveys is to request that self-reported users of a specified product submit an image of their device. Additionally, cohort studies are uniquely situated to retrospectively examine response consistency over time (ie, do baseline self-reported ever users of HTPs still report HTP ever use at follow-up?).

Crucially, the quality of information ascertained from a survey will depend on the development steps taken prior to implementing the survey on the field. How potential respondents interpret questions and instructions is a major determinant of survey data validity. Insight on terminology tendencies and perceived differences between different products (eg, a product containing tobacco (HTP) vs a product containing liquid nicotine (NVP)) can be gained through qualitative research approaches such as focus groups and cognitive testing studies. For HTPs specifically, there is a particular need for focus groups and cognitive testing among current users of HTPs, as well as among users of other nicotine products (eg, current cigarette smokers).

Until more qualitative and quantitative evaluations of measurement validity are conducted, there is not solid grounding to explicitly recommend specific protocols for assessing self-reported use of electronic inhalable nicotine products. As such, caution is warranted when interpreting survey results that rely on self-reported HTP use until more is known about measurement validity. Comparisons of HTP prevalence rates and correlates of use across multiple data sources should consider methodological differences as a potential explanation of results. With longitudinal data, researchers must consider whether HTP measurement validity was consistent over time. It is possible that a baseline wave of HTP data collected in a jurisdiction where HTPs had recently entered the market is more prone to misclassification than a follow-up wave, either due to modified survey methods or a change in awareness of HTPs within the sampled population.