We investigated the impact of supplementing administrative claims with NLP-generated features from EHR free-text notes when using various propensity score-based weighting methods for high-dimensional proxy adjustment. After fitting LASSO PS models, we found that supplementing prespecified variables and claims codes with additional NLP-generated features using n-grams performed best in terms of treatment prediction. Interestingly, the overall covariate balance in all 3 empirical studies also improved and moved treatment effect estimates furthest from the unadjusted estimates in a direction consistent with prior expectation in 2 of the 3 studies. In our experiments, NLP tools more sophisticated than n-grams, such as MTERMS, and word or sentence embeddings, did not further improve covariate balance or confounding adjustment.

The use of NLP technology to leverage information from unstructured free-text EHR notes to improve confounding adjustment in healthcare database studies is a growing area of work.^39–43 To our knowledge, this study is the first to apply and compare the scalability and performance of several alternative NLP tools for purposes of high-dimensional proxy adjustment. These findings indicate that large-scale NLP-enabled feature generation is quite feasible and may provide additional confounder information when conducting high-dimensional proxy adjustment. This may result in improved effect estimates, although differences in effect estimates were modest in our example studies, and it supports automated approaches to confounding control.

When controlling for a large number of variables, we found that weighting methods that down-weight the tails of the PS distribution (matching weights and overlap weights) were more robust in balancing covariates compared to inverse probability weighting. This finding was consistent across all 3 empirical studies and is also consistent with previous studies that have compared alternative weighting approaches.⁴⁴ Because people with extreme PS values represent those for whom the confounding may be refractory, PS weighting approaches that down-weight the tails of the PS distribution may also mitigate unmeasured confounding that is likely stronger in the tails of the PS distribution.⁴⁵ However, the tradeoff is that the target population for matching weights and overlap weights cannot be defined a priori and is dependent on the distribution of the estimated PS. This creates additional challenges when comparing effect estimates across different PS models as it can be unclear if movement in the estimated treatment effect is due to better confounding control or heterogeneity in the treatment effect caused by changing target populations. Ideally, we would target the same population across all analyses by weighting to the same population (e.g., using inverse probability weights); however, across all 3 empirical studies we were only able to adequately balance covariates when using methods that down-weight the tails of the PS when controlling for high-dimensional sets of variables. Consequently, we did not focus on results from analyses using inverse probability weights.

Some additional limitations deserve attention. First, other factors including random chance and hidden biases not captured through PS weighting could also contribute to unpredictable movement in estimated effects and a true ‘gold standard’ is never known with certainty. Consequently, as discussed previously, methodological comparisons of estimated treatment effects in real-world data examples are inherently limited. However, such comparisons are widely used and can be informative, particularly when consistent trends are observed across multiple studies increasing confidence in the robustness of findings.

Second, while we explored several NLP methods for generating structured features from free-text notes, there are many NLP tools that were not considered or customized to our EHR data due to resource constraints. Future work could explore additional NLP tools such as large language models, for extracting confounder information from EHR data, as well as training embeddings on our local EHR data for better representing features in EHRs. In addition, different ways of modeling the NLP-generated features may also improve the performance. For example, in this study we simply created dichotomous variables from the n-gram and other NLP output. Other options could consider modeling the NLP output as categorical or continuous features, such as using the term frequency and the reciprocal document frequency (tf-idf) of the n-gram output or the weights of embedding clusters. These approaches help integrate the fine-grained contribution weights of NLP-generated features. In this study, we chose to focus on generating binary features from the NLP output to simplify the functional form of the PS model and reduce the likelihood of model misspecification. While more flexible nonparametric models could potentially be used to model complex relationships between continuous/categorical features and treatment, both theory and simulations have shown that using flexible nonparametric models for PS estimation comes at a cost of slow convergence rates, which can harm the properties of causal estimators, particularly in high-dimensional models.^46–48 A thorough comparison of modeling more complex NLP-output with flexible nonparametric models is beyond our scope.

In conclusion, we found that off-the-shelf NLP tools can scale to generate large numbers of structured features from free-text EHR notes. We found that supplementing administrative claims with large numbers of NLP-generated EHR features improved overall covariate balance in PS-weighted analyses, but the observed impact on estimated treatment effects was incremental and more complex NLP tools had no advantage over simple n-grams.