Validity of oral fluid test for Delta-9-tetrahydrocannabinol in drivers using the 2013 National Roadside Survey Data

Article type: Research Article

Keywords: Blood samples, Cannabis, Delta 9-tetrahydrocannabinol, Driving under the influence, Marijuana, Oral fluid samples

Authors: Huiyan Jin, Sharifa Z. Williams, Stanford T. Chihuri, Guohua Li, Qixuan Chen ⚬

Affiliations: grid.21729.3fDepartment of Biostatistics, Columbia University Mailman School of Public Health, 722 West 168th Street, New York, NY 10032 USA; grid.21729.3fDepartment of Anesthesiology, Columbia University College of Physicians and Surgeons, 622 West 168th Street, New York, NY 10032 USA; grid.21729.3fDepartment of Epidemiology, Columbia University Mailman School of Public Health, 722 West 168th Street, New York, NY 10032 USA

License: © The Author(s). 2018 CC BY 4.0 Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Article links: DOI: 10.1186/s40621-018-0134-2  |  PubMed: 29457201  |  PMC: PMC5817052

Relevance: Relevant: mentioned in keywords or abstract

Full text: PDF (755 KB)

Background

Marijuana is a commonly used drug in the United States. In 2015, there were an estimated 22.2 million Americans aged 12 years or older reporting current marijuana use and 19.8% (about one in every five) young adults aged 18 to 25 years were current users (Center for Behavioral Health Statistics and Quality ref. 2016). Experimental and epidemiological studies showed that recent marijuana use is associated with decreased driving performance—e.g., decreased reaction time, reduced lateral control, driving slower speeds, and impairment of psychomotor skills—and thus increased motor vehicle crash risk (Brady and Li ref. 2013; Brady and Li ref. 2014; Hartman and Huestis ref. 2013; Chihuri et al. ref. 2017). Driving under the influence (DUI) of marijuana is of increasing concern in the United States (Berning et al. ref. 2015; Li et al. ref. 2016; Davis et al. ref. 2016; Johnson et al. ref. 2012; Edwards et al. ref. 2017). The annual number of fatal automobile crashes attributable to impaired driving has been on the rise and marijuana is the most frequently detected non-alcohol drug in drivers in the United States (Brady and Li ref. 2013; Brady and Li ref. 2014; Romano and Voas ref. 2011; Tefft et al. ref. 2016; Downey et al. ref. 2013; Hartman et al. ref. 2015; Lennéa et al. ref. 2010; Li et al. ref. 2012). From 1999 to 2010, the prevalence of marijuana detected among drivers involved in fatal car crashes tripled from 4.2% to 12.2% (Brady and Li ref. 2014).

Delta 9-tetrahydrocannabinol (THC) is the primary psychoactive compound in marijuana and its presence in blood samples is viewed as an indicator of recent (1–8 h after smoking or oral intake) marijuana use (Karschner et al. ref. 2012; Hartman et al. ref. 2016; Jones et al. ref. 2008). Because blood THC test is a more accurate measure of THC-induced impairment (Hartman et al. ref. 2016; Grotenhermen et al. ref. 2007), many US states use blood THC concentrations as the gold standard for determining marijuana-related DUI (Center for Behavioral Health Statistics and Quality ref. 2016; Berning et al. ref. 2015). Different cut-off points of blood THC concentrations have been proposed for establishing per se laws, which make operating a motor vehicle a criminal offense for an individual if he or she has a specific amount of drug or metabolite in his or her body. This threshold concentration is a legal limit, and exceeding this threshold serves as proof of legal impairment (Grotenhermen et al. ref. 2007; Wong et al. ref. 2014; National Conference of State Legislatures ref. 2017). Currently, 20 states have established per se limit laws on marijuana-impaired driving, with four states using 5 nanograms per milliliter (ng/ml) as the threshold for THC and the others having zero tolerance (National Conference of State Legislatures ref. 2017).

Compared to collecting whole blood samples, measuring THC in oral fluid is a less intrusive and less costly method of testing (Grotenhermen et al. ref. 2007; Bosker and Huestis ref. 2009; Drummer ref. 2005; Drummer ref. 2006). Although no accurate predictive function has been established between the oral fluid THC concentration and the whole blood THC concentration, studies have showed a correlation between these two test results (Lee and Huestis ref. 2014). Hence oral fluid THC testing has potential utility as a standard screening tool for detecting marijuana-related DUI (Gjerde et al. ref. 2014; Langel et al. ref. 2014; Gjerde et al. ref. 2015; Edwards et al. ref. 2017). In this study, we assess the validity of oral fluid THC test versus blood THC test using sensitivity and specificity analysis and a logistic regression, and estimate the quantitative relationship between the oral fluid THC concentration and the blood THC concentration using a correlation analysis and a linear regression on the log-transformed THC concentrations.

Methods

Data source

Data for this study came from the 2013 National Roadside Survey of Alcohol and Drug Use by Drivers (NRS) conducted by the National Highway Traffic Safety Administration, the National Institute on Drug Abuse, and the Insurance Institute for Highway Safety. The aim of the NRS was to estimate the prevalence of driving under the influence of alcohol and/or drugs in the United States (Berning et al. ref. 2015). The first NRS was conducted in 1973; since then, it has been conducted in 1986, 1996, 2007 and 2013. The 2013 NRS was a national field survey based on voluntary and anonymous random stops of non-commercial drivers at 300 locations across the 48 contiguous states (Kelley-Baker et al. ref. 2016). The sample was selected using multistage sampling method and only verbally consented drivers were included in the survey. Data were collected from drivers in 60 locations during the hours of 9:30 AM to 11:30 AM and 1:30 PM to 3:30 PM on Fridays and another 240 locations 10 PM to 12 AM and 1 AM to 3 AM on Fridays and Saturdays from June 7, 2013 through March 30, 2014. Detailed information on the 2013 NRS methodology can be found elsewhere (Kelley-Baker et al. ref. 2016).

The survey included questions on driver demographic characteristics such as age and race, drinking and drug use habits such as time of last marijuana use, trip information including trip origin and destination, seatbelt usage, and vehicle information such as number of passengers and vehicle type. All information was collected from the drivers via electronic tablet except for gender, which was recorded via officer observation. In addition, breath alcohol, oral fluid alcohol, and oral fluid drug concentration tests were administered and whole blood specimens were collected during the survey process (Kelley-Baker et al. ref. 2016). Among 11,100 eligible drivers, 8802 (79.3%) voluntarily participated in the 2013 NRS, and 4669 (42.1%) consented to provide both blood and oral fluid samples for drug and alcohol testing. For this study, we used the data from 4596 drivers aged 16 years and older with both blood and oral fluid THC concentration test results. Compared to all the drivers who participated in the 2013 NRS, these 4596 drivers had a similar age distribution and proportion of females, but included fewer whites (59.1% vs. 62.0%) and were more likely to report either never having used marijuana or the last time of use to be beyond a year (79.4% vs. 69.5%).

Measures of interest

In the 2013 NRS, 1 ml (ml) of oral fluid sample and 10 ml of whole blood specimen were collected from each consented driver. The presence of THC in oral fluid and whole blood samples were first screened by Enzyme-linked Immunosorbent Assay (ELISA) (Moore et al. ref. 2006). The minimum detectable screening concentration of THC in oral fluid, as measured by ELISA, was 4 ng/ml. Samples screened as having THC concentrations ≥4 ng/ml were further tested using liquid/gas chromatography and mass spectrometry (LC/MS/MS) technology with a minimum detectable THC concentration of 2 ng/ml in oral fluid. A similar screening and confirmation process was carried out for measuring THC concentrations in whole blood samples. The minimum detectable concentration of THC in blood was 10 ng/ml using the ELISA screening test and 1 ng/ml using the LC/MS/MS confirmation test. Results from the LC/MS/MS test were recorded as the final results for both oral fluid and blood samples (Kelley-Baker et al. ref. 2016).

Defining positivity of THC in blood and oral fluid samples

We defined a blood THC test as positive if the THC concentration was greater than 0 ng/ml (or ≥ 1 ng/ml) in the whole blood sample for two reasons. First, it is the minimum detectable concentration of THC in the whole blood. Second, it is the cutoff point for establishing per se laws by many states (National Conference of State Legislatures ref. 2017). This cut-off was consistent with the NRS methodology (Kelley-Baker et al. ref. 2016). To define positivity for oral fluid THC test that best predicted positivity of THC in blood, we dichotomized the oral fluid THC concentrations using a series of cut-offs greater than or equal to the minimum detectable concentration (≥ 2 ng/ml, ≥ 3 ng/ml, etc.). We conducted a series of logistic regression of the binary blood THC test (positive vs. negative) on the binary oral fluid THC test defined by each cut-off value. Each logistic regression model controlled for driver age (16–20 years, 21–34 years, 35–54 years, and 55 years or older), gender (female or male), race (white, black, or other), time of last marijuana use (beyond a year/never, over a month, within the past month to 2 days, or in the past 24 h), and blood alcohol concentration (BAC; 0 mg/dl, 1–39 mg/dl, and 40 or more mg/dl). We used a random sample with 80% of the 2013 NRS data to build logistic regression models and the remaining 20% to calculate accuracy defined as the proportion of true positive or true negative cases and area under receiver operating characteristic curve (AUC) (Robin et al. ref. 2011). Higher values of accuracy and AUC indicate better predictive ability of the dichotomized oral fluid THC concentrations at the selected cut-off in distinguishing between positive and negative blood THC tests. The highest accuracy (0.966) was found in the cut-off values of ≥ 2 ng/ml and ≥ 3 ng/ml for oral fluid test. Although the cut-off value of ≥ 3 ng/ml yielded a slightly greater AUC (0.865) than the cut-off value of ≥ 2 ng/ml (0.863), we defined positivity of THC in oral fluid using ≥ 2 ng/ml because it was the minimum detectable THC concentration (i.e., same as using > 0 ng/ml) and was consistent with the NRS methodology (Kelley-Baker et al. ref. 2016).

Statistical analysis

We first calculated positive rates of THC in both blood and oral fluid samples and computed sensitivity and specificity of the oral fluid THC test in predicting the positivity of blood THC. The analysis was conducted using the whole sample and then stratified by driver characteristics.

We then fit three logistic regression models of the positivity in blood THC on oral fluid THC, which was included in the model using both a binary covariate (> 0 ng/ml vs. = 0 ng/ml) and the natural-log of the continuous THC concentration. For samples with 0 ng/ml oral fluid THC concentrations, 0.5 ng/ml was used to enable natural-log transformation. The first model assessed the crude association. The second model adjusted for BAC and time of last marijuana use. The third model further controlled for demographic characteristics of driver age, gender, and race. We tested the interaction terms between BAC and oral fluid THC test, but the interaction terms were excluded from the final models because they were not statistically significant.

We also examined the association between the continuous blood THC concentration and the continuous oral fluid THC concentration among drivers with positive THC in either blood or oral fluid. We calculated Pearson correlation coefficient between these two concentrations in both the original and natural-log transformed scales. We then built linear regression models of the natural-log transformed blood THC concentration on the natural-log transformed oral fluid THC concentration. Similar to the logistic regression, we used a value of 0.5 ng/ml for 0 ng/ml THC concentrations to enable natural-log transformation and considered both the unadjusted and the two adjusted models.

Finally, we conducted two sensitivity analyses. In the first sensitivity analysis, we multiply imputed the missing data on gender and time of last use of marijuana using the multiple imputation by chained equations algorithm to create complete data for the covariates in the adjusted models (Van Buuren and Groothuis-Oudshoorn ref. 2011). The second sensitivity analysis accounted for complex survey design in the 2013 NRS survey. Analyses were conducted using R version 3.2.4 (R Development Core Team, Vienna, Austria).

Results

Figure 1 shows the percentages of positive and negative blood THC tests at different oral fluid THC concentrations (0, 2, 3, 4, and ≥5 ng/ml). Most of the drivers with fluid THC concentrations under the minimum detection limit (< 2 ng/ml or = 0 ng/ml) were negative in the blood THC tests. The percentage of positivity in blood THC increased as the concentration of oral fluid THC increased.

Table 1 provides the positive rates of THC in blood and oral fluid samples as well as the sensitivity and specificity of the positivity of oral fluid THC in predicting the positivity of blood THC. Overall, 9.4 and 8.9% of the samples tested positive for THC in blood and oral fluid, respectively, yielding a sensitivity of 79.4% with a 95% confidence interval (CI) (75.2%, 83.1%) and a specificity of 98.3% (95% CI: 97.9%, 98.7%) for the oral fluid test in predicting positivity in blood THC. The oral fluid test had the highest sensitivity among drivers with positive BAC but lowest sensitivity among drivers older than 55 years of age. The specificity was higher than 90% in all groups, except drivers with the last use of marijuana in the past 24 h (66.7%, 95% CI: 52.1%, 79.2%).

Estimated odds ratios (ORs) and associated 95% confidence intervals from the logistic regression of the positivity in blood THC given oral fluid THC are presented in Table 2. Both the unadjusted and adjusted regression models showed that positivity in oral fluid THC was significantly associated with positivity in blood THC. After adjusting for driver BAC, last time use of marijuana and demographic characteristics, having a positive oral fluid THC test was associated with an 11-fold increase in the odds of having a positive blood THC test. The odds of positivity in blood increased as the concentration of the oral fluid THC increased, where every 10% increase in oral fluid THC was associated with an additional 6% (95% CI: 5.1%, 7.7%) increase in the odds of positive blood THC. Other covariates significantly associated with positivity in blood THC included BAC, race, and last time use of marijuana. By including BAC and last time use of marijuana, the predictive ability of the logistic regression improved with AUC increased from 0.87 (95% CI: 0.82, 0.91) to 0.95 (95% CI: 0.91, 0.98).

Figure 2 shows the scatter plots and estimated Pearson correlation coefficients for continuous THC concentration in blood and in oral fluid on both the untransformed and log-transformed scales. Log-transformed values were associated with a higher Pearson correlation coefficient (0.53 vs. 0.37), indicating a stronger linear association and as such, we used these transformed values in linear regression models. The linear regression models examining the association between the natural-log blood THC concentration and the natural-log oral fluid THC concentration shows that a 10% increase in oral fluid THC concentrations was associated with a 2.4% (95% CI: 2.1%, 2.8%) increase in blood THC concentrations (Table 3). Drivers who used marijuana in the past 24 h had a higher blood THC concentration than drivers who never used marijuana or who last used marijuana a year or more prior to testing. We compared the point estimates and 95% confidence intervals of the coefficients in Tables 2 and 3 to those with multiple imputation and survey weighting adjustment and found that the results were consistent.

Discussion

This study showed that oral fluid tests are reasonably accurate in detecting blood THC positivity with a 79.4% of sensitivity and 98.3% of specificity. The sensitivity and specificity are similar across subgroups of drivers with different characteristics, except for drivers older than 55 years of age (a low sensitivity of 54.6%) and drivers who reported using marijuana in the past 24 h (a low specificity of 66.7%). Positivity in oral fluid THC is strongly associated with positivity in blood THC (OR = 11.36). Information on BAC and last time use of marijuana may be used to enhance our ability in predicting the positivity in blood THC. These results suggest that the oral fluid test is a valid method for detecting the presence of THC in the blood. Our study reaffirms the findings reported by Kelley-Baker et al. (ref. 2016).

There exists a fairly strong linear relationship between the log-transformed oral fluid THC concentrations and the log-transformed blood THC concentrations (Pearson correlation coefficient = 0.53, p = < 0.001). This finding adds to the existent evidence that oral fluid THC is associated with blood THC (Langel et al. ref. 2014; Gjerde et al. ref. 2014). We improved the prediction models by including covariates and using the log-transformation to yield a better linear correlation. However, given that oral fluid THC concentration only explained 29% of variation in the blood THC concentration, it is not advisable to derive blood THC concentrations from oral fluid THC concentrations.

We defined blood THC positivity using the detection limit (≥ 1 or > 0 ng/ml) as the cut-off. Alternatively, we conducted a sensitivity analysis by using other cut-off points with > 1, 2, 3, and 5 ng/ml. Because there was no driver with a blood THC concentration of 1 ng/mL in the study sample, the sensitivity-specificity analysis using > 1 ng/ml as the cut-point yielded the same results as that using > 0 ng/ml. When a cut-point of > 2 ng/ml was used, the sensitivity improves significantly from 79.4% to 91.7% but the specificity reduces from 98.3% to 95.3% when the positivity in the oral fluid sample was defined as > 0 ng/ml. This sensitivity analysis supports our conclusion that the oral fluid test is a valid alternative to detect the positivity of THC in the blood. However, given that many states have adopted a zero-tolerance policy toward drugged driving, the analysis using the cut-off point of > 0 ng/ml to define blood THC positivity provides the most relevant results for assessing the validity of using oral fluid test as a screening tool for detecting marijuana-impaired DUI.

We then identified optimal cut-off concentration in oral fluid that provided the highest possible diagnostic accuracy for predicting positivity of THC in blood using accuracy and AUC. We found that a cut-off value of ≥ 2 and ≥ 3 ng/ml in oral fluid achieved the best diagnostic accuracy. We chose to use the cut-off value of ≥ 2 ng/ml (same as using > 0 ng/ml) because it is also the minimum detectable level in the oral fluid sample. This is consistent with other studies using a cut-off value of ≥ 2 ng/ml (Kelley-Baker et al. ref. 2016; Verstraete ref. 2005), although a cut-off of 38 ng/ml was used in a population with low prevalence of drugs use (Gjerde et al. ref. 2014).

It is worth noting that less than half of the eligible drivers (41.4%) consented to provide both whole blood and oral fluid samples and these consenting drivers were less likely to be white and report using marijuana recently than those who refused to provide specimens. Therefore, our study results are potentially susceptible to bias. Driver age, race and last time use of marijuana were reported by the drivers via electronic tablet and thus can also be subject to bias. Moreover, the measurement of blood and oral fluid THC concentrations was limited by the detection limits of ELISA and LC/MS/MS screening and confirmation tools. With the improvement of the detection limits in both oral fluid and whole blood samples in the future, we expect to observe a higher degree of agreement in the positivity of THC tests between the oral fluid and blood samples. Although THC is the most important psychoactive constituent of cannabis, there are other chemical compounds in marijuana that may impair driving safety. Currently, marijuana test based on oral fluids is limited primarily to detecting THC only.

Notwithstanding, the 2013 NRS provided valuable data in assessing the validity of the oral fluid test as a less invasive and less costly alternative to blood test in detecting the presence of THC in the blood. Further research is needed to better understand the relationship between blood and oral fluid THC concentrations and to develop a more reliable algorithm for calculating blood THC concentrations based on oral fluid testing results.

Conclusions

Oral fluid test is a highly valid method for detecting the presence of THC in the blood and thus could be used as a screening tool for detecting marijuana-related DUI. Although there is a significant linear association between the log-transformed blood THC concentrations and the log-transformed oral fluid THC concentrations, log-transformed oral fluid THC concentrations can explain only 29% of the variation in log-transformed blood THC concentrations. Therefore, the oral fluid test is not an accurate method for estimating THC concentrations in the blood.

References

1. Berning A, Compton R, Wochinger K. Results of the 2013–2014 National RoadsideSurvey of alcohol and drug use by drivers. (Traffic Safety Facts Research Note. Report No. DOT HS 812118). Washington, DC: National Highway Traffic Safety Administration; 2015.
2. WM Bosker, MA Huestis. Oral fluid testing for drugs of abuse. Clin Chem, 2009. [DOI | PubMed]
3. JE Brady, G Li. Prevalence of alcohol and other drugs in fatally injured drivers. Addiction, 2013. [DOI | PubMed]
4. JE Brady, G Li. Trends in alcohol and other drugs detected in fatally injured drivers in the United States, 1999–2010. Am J Epidemiol, 2014. [DOI | PubMed]
5. Center for Behavioral Health Statistics and QualityKey substance use and mental health indicators in the United States: results from the 2015 National Survey on drug use and health2016
6. S Chihuri, G Li, Q Chen. Interaction of marijuana and alcohol on fatal motor vehicle crash risk: a case-control study. Inj Epiemiol, 2017. [DOI]
7. KC Davis, J Allen, J Duke, J Nonnemaker, B Bradfield, MC Farrelly. Correlates of marijuana drugged driving and openness to driving while high: evidence from Colorado and Washington. PLoS One, 2016. [DOI | PubMed]
8. LA Downey, R King, K Papafotiou, P Swann, E Ogden, M Boorman. The effects of cannabis and alcohol on simulated driving: influences of dose and experience. Accident Accid Anal Prev, 2013. [DOI | PubMed]
9. OH Drummer. Review: pharmacokinetics of illicit drugs in oral fluid. Foren Sci Int, 2005. [DOI]
10. OH Drummer. Drug Testing in Oral Fluid. Clin Biochem Rev, 2006. [PubMed]
11. LD Edwards, KL Smith, T Savage. Drugged driving in Wisconsin: oral fluid versus blood. J Anal Toxicol, 2017. [DOI | PubMed]
12. H Gjerde, K Langel, D Favretto, AG Verstraete. Estimation of equivalent cutoff thresholds in blood and oral fluid for drug prevalence studies. J Anal Toxicol, 2014. [DOI | PubMed]
13. H Gjerde, K Langel, D Favretto, AG Verstraete. Detection of illicit drugs in oral fluid from drivers as biomarker for drugs in blood. Forensic Sci Int, 2015. [DOI | PubMed]
14. F Grotenhermen, G Leson, G Berghaus, OH Drummer, HP Krüger, M Longo. Developing limits for driving under cannabis. Addiction, 2007. [DOI | PubMed]
15. RL Hartman, TL Brown, G Milavetz, A Spurgin, DA Gorelick, G Gaffney. Controlled vaporized cannabis, with and without alcohol: subjective effects and oral fluid-blood cannabinoid relationships. Drug Test Anal., 2016. [DOI | PubMed]
16. RL Hartman, TL Brown, G Milavetz, A Spurgin, RS Pierce, DA Gorelick. Cannabis effects on driving lateral control with and without alcohol. Drug Alcohol Depend, 2015. [DOI]
17. RL Hartman, MA Huestis. Cannabis effects on driving skills. Clin Chem, 2013. [DOI | PubMed]
18. MB Johnson, T Kelley-Baker, RB Voas, JH Lacey. The prevalence of cannabis-involved driving in California. Drug Alcohol Depend, 2012. [DOI | PubMed]
19. AW Jones, A Holmgren, FC Kugelberg. Driving under the influence of cannabis: a 10-year study of age and gender differences in the concentrations of tetrahydrocannabinol in blood. Addiction, 2008. [DOI | PubMed]
20. EL Karschner, DM Schwope, EW Schwilke, RS Goodwin, DL Kelly, DA Gorelick. Predictive model accuracy in estimating last Δ9-tetrahydrocannabinol (THC) intake from plasma and whole blood cannabinoid concentrations in chronic, daily cannabis smokers administered subchronic oral THC. Drug Alcohol Depend, 2012. [DOI | PubMed]
21. T Kelley-Baker, JH Lacey, A Berning, A Ramirez, C Moore, K Brainard. 2013–2014 National Roadside Study of alcohol and drug use by drivers: methodology, 2016
22. K Langel, H Gjerde, D Favretto, P Lillsunde, EL Øiestad, SD Ferrara. Comparison of drug concentrations between whole blood and oral fluid. Drug Test Anal, 2014. [PubMed]
23. D Lee, MA Huestis. Current knowledge on cannabinoids in oral fluid. Drug Test Anal., 2014. [DOI | PubMed]
24. MG Lennéa, MP Dietze, JT Triggsa, S Walmsley, B Murphy, JR Redmand. The effects of cannabis and alcohol on simulated arterial driving: influences of driving experience and task demand. Accid Anal Prev, 2010. [DOI | PubMed]
25. K Li, B Simons-Morton, B Gee, R Hingson. Marijuana-, alcohol-, and drug-impaired driving among emerging adults: changes from high school to one-year post-high school. J Saf Res, 2016. [DOI]
26. MC Li, JE Brady, CJ DiMaggio, AR Lusardi, KY Tzong, G Li. Marijuana use and motor vehicle crashes. Epidemiol Rev, 2012. [DOI | PubMed]
27. C Moore, M Vincent, S Rana, C Coulter, A Agrawal, J Soares. Stability of Δ9-tetrahydrocannabinol (THC) in oral fluid using the Quantisal™ collection device. Forensic Sci Int, 2006. [DOI | PubMed]
28. National Conference of State Legislatures. 2017 [cited 2017 Aug 14]. Available from: http://www.ncsl.org/documents/transportation/per_se_chart_Feb_2017.pdf.
29. X Robin, N Turck, A Hainard, N Tiberti, F Lisacek, JC Sanchez. pROC: an open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinformatics, 2011. [DOI | PubMed]
30. E Romano, RB Voas. Drug and alcohol involvement in four types of fatal crashes. J Stud Alcohol Drugs, 2011. [DOI | PubMed]
31. Tefft BC, Arnold LS, Grabowski JG. Prevalence of marijuana involvement in fatal crashes: Washington, 2010–2014. Washington, DC; 2016.
32. S Van Buuren, K Groothuis-Oudshoorn. Mice: multivariate imputation by chained equations in R. J Stat Softw, 2011
33. Verstraete AG. Oral fluid testing for driving under the influence of drugs: history, recent progress and remaining challenges. Forensic Sci Int 2005;150:143–150.
34. Wong, JE Brady, G Li. Establishing legal limits for driving under the influence of marijuana. Inj Epidemiol, 2014. [DOI | PubMed]