Development of a natural language processing application to extract and categorize mentions of violence from mental healthcare records text
1King’s College London, United Kingdom
*Corresponding author; Email: sharon.sondh@kcl.ac.ukAbstract
Background
Experiences of violence are reported frequently by mental health service users, victims of violence are at a greater risk of mental health disorders, and violence may sometimes occur as a consequence of a mental disorder. Electronic health records (EHRs) are an important source of information about healthcare, and its social context. Occurrences of violence are not routinely recorded as structured data in EHRs but are however recorded in the free text narrative.
Objective
Our objective was to address this research gap by creating a natural language processing (NLP) application that extracts information related to various forms of violence (physical (non-sexual), sexual, emotional, and financial) from the EHR of a large south London mental health service. Additionally, we aimed to extract features concerning the patient’s role (victimization vs. perpetration), timing (recent vs. historic), domestic context, presence (actual, threat, or unclear), and polarity (affirmed, abstract, or negated) of the violent behaviors.
Methods
Two raters independently annotated 6,500 randomly selected segments of clinical notes containing violence-related keywords from a large mental healthcare provider in South London, each containing 400 characters (with approximately 200 characters before and after the keyword) after rigorous training using a pre-defined and approved coding book provided by senior professionals. We utilized 90% of the annotated data for fine-tuning a multi-label BERT model (employing 5-fold cross-validation) with the remaining 10% of data reserved for a blind test.
Results
The model performed well on the blind test set for emotional violence (F1= 0.89), financial violence (0.88), physical (non-sexual) violence (0.84), and unspecified violence (0.81), and the patient role (0.89 as perpetrator; 0.84 as victim), polarity (0.89 for affirmed behavior), presence (0.95 for actual violence), and domestic settings (0.88). We were unable to achieve satisfactory results in capturing temporal aspects (0.65 for past violence).
Conclusions
We were able to improve substantially on previously developed NLP for ascertaining violence in routine mental health records, providing novel opportunities for both surveillance and research.
Article notes
Competing Interest Statement
The authors have declared no competing interest.
Funding Statement
This research was supported by the UK Prevention Research Partnership (Violence, Health and Society; MR-VO49879/1), which is funded by the British Heart Foundation, Chief Scientist Office of the Scottish Government Health and Social Care Directorates, Engineering and Physical Sciences Research Council, Economic and Social Research Council, Health and Social Care Research and Development Division (Welsh Government), Medical Research Council, National Institute for Health and Care Research, Natural Environment Research Council, Public Health Agency (Northern Ireland), The Health Foundation, and Wellcome. The views expressed in this Article are those of the authors and not necessarily those of the UK Prevention Research Partnership or any other funder.
1.Introduction
The impact of abusive experiences on mental health is profound. Victims of emotional intimate partner violence (IPV) are more susceptible to suicidal ideation, post-traumatic stress, anxiety, depressive symptoms, psychological distress, physical pain, trauma, anger, shame, poor physical health, and somatic symptoms [1]. Specifically, women who have experienced sexual violence during their childhood are more likely to suffer from mental illness in adult life [2]. Victims of domestic violence (DV) may also nurture negative beliefs and attitudes towards others, making it challenging to maintain healthy relationships [3]. Psychological (emotional) abuse has further significant effects on the risk for posttraumatic stress disorder (PTSD) and depression [4]. Additionally, one systematic review found that depression, anxiety, and PTSD were all notable factors associated with both experiencing and perpetrating intimate partner violence [1].
In addition to physical and sexual violence, emotional (psychological) and financial abuse may also be significant components of domestic violence or intimate partner violence [9]. Financial or economic abuse may often go unnoticed, as it occurs within intimate partner relationships [10] [11], and involves tactics such as restricting access to finances, withholding contributions for necessities, limiting access to financial information, and exerting control over household spending [11]. This can have detrimental effects on physical and mental health, as well as on self-esteem and ability to participate in work, social activities, and community engagement [12]. Emotional abuse, a broader experience encountered in daily life, has also proven to be harmful [13], for example as a significant risk factor for partner violence [9, 14], and in the strong correlations of parental emotional abuse with PTSD symptoms in children and adolescents [15], and depressive symptoms in adulthood.
Violence is therefore an important feature of presentations to mental health care. However, there remains a dearth of studies evaluating this from the clinical notes of mental health patients, despite the opportunities offered by these growing digital information resources. In other sectors, NLP has been deployed to open up information resources, notably in the extraction of abuse-related narratives from domestic violence reports provided by the New South Wales Police Force [13]. However, studies to date have often focused on single binary classification models, which may overlook label dependencies and lead to suboptimal results.
To address these research gaps, we applied a multi-label classifier to capture the dependencies and correlations between labels. Also, we sought to develop and improve capability to characterize mentions of violence and abuse from mental healthcare notes using NLP techniques using the Maudsley Clinical Record Interactive Search (CRIS) platform [5]. CRIS was developed in 2008 to enable researcher access to full but de-identified information, within a robust data governance framework, from the electronic health record (EHR) of the South London and Maudsley NHS Foundation Trust, a large provider of mental health care to a catchment population of 1.3m residents in South East London [6,7]. CRIS currently contains data from over 500,000 patients, has supported over 300 published research papers, and has been replicated as a model by several other mental healthcare providers in the UK, expanding the possibilities for multi-site evaluations [8]. Previous work with CRIS resulted in the development of an NLP tool to ascertain violence experiences, sexual abuse, physical abuse, and domestic abuse which was successful but limited in scope [5]. This work reports on the development of a second application, with a greater range of functionality. This redevelopment recognizes that patient role, domestic context, temporality, polarity, and the manifestation of violent behavior are crucial and intersecting features, given that patients may be victims, perpetrators, witnesses, or a combination of these roles [19], and identifying their role is particularly challenging in a domestic context [20]. Our development aimed to provide improved capability for researchers utilizing the CRIS database to explore associations between diverse forms of violence and specific mental health conditions. Furthermore, we anticipated that better recognition of the links between violence and mental health, drawn from research in routine data, might encourage better routine documentation and thus improve information resources as a by-product.
2.Methods
2.1Data sources
Our development work comprised two stages. In the first stage, reported in Botelle et al (2022)[5], we collected 3771 instances of potential general violence, sexual violence, and physical violence using the words and word fragments"abus", " assault", " attack", " beat", " chok", " fight", " fought", " hit", " punch", " push", " rape", " slap", " strangl", " struck", " threw", and " violenc". In the second stage, focusing on ascertainment of emotional/financial abuse, text was collected using a search for the following list of word and word fragments “coerciv, emotional abus, emotional manipulat, emotional violen, emotionally abus, gaslight, psychological abus, psychological manipulat, psychological violen, psychologically abus, economic abus, financial abus, financially abus, battered, maltreated”. For each of these strings, we randomly extracted instances up to a maximum number of 300. This resulted in 3500 instances. For these two data collection stages, we excluded resulting in 6,685 instances.
2.2Data labelling
The 6685 instances collected in the previous step are potential mentions of violence. Some of these instances, however, will not be true mentions of violence, having resulted from ambiguous use of the matching strings and keywords. In order to create a dataset for model training and testing, all instances were therefore manually labeled, i.e. annotated, as to the presence or absence of different types and features of violence.
2.2.1Annotation guidelines
Prior to annotation, we created annotation guidelines aligned with our research objectives, and which defined: These guidelines served as the foundation for our annotation process, providing consistency and rigor in our data collection. Specific definitions of each violence type and feature are summarized in Table 1.
- ● Violence Types: We provided clear definitions for different types of violence, including emotional, financial, and others. Additionally, we furnished illustrative examples for each violence category.
- ● Patient Roles and Context: We elucidated the roles that patients could assume—victim, perpetrator, witness, or a combination thereof. Furthermore, we addressed the polarity (affirmed, negated, or abstract), presence (actual, threatened, or unclear), temporality (present, past, or uncertain), and setting (domestic, non-domestic, or ambiguous) associated with violence mentions.
We list some examples and the corresponding violence type and values in Table 2.
2.2.2Annotation
We used Label Studio to annotate the instances. Out of all the instances, 185 were randomly chosen for a pilot study, aimed at familiarizing annotators with the annotation guidelines. We randomly selected 700 instances for double-annotation from the remaining 6316 instances. Of these 700 instances, 643 underwent successful double-annotation, and the remaining 57 instances were reserved for wider discussion to resolve disagreements. In this round of double-annotation, 372 instances exhibited at least one discrepancy across six annotation categories. To address this, we conducted an additional round of double-checking, where two annotators collaborated to reach a final decision on these instances. All 700 instances from this double annotation step with resolved consistency were later used in developing the NLP model.
For the remaining 5,800 instances, we conducted single annotations in the majority, with 20% double-annotated overlap agreement checks (n=1,008 instances). For 691 instances with at least one discrepancy across six annotation categories, we conducted a double-check as before. Subsequently, all 5,800 instances with resolved consistency from this step were later used to develop the NLP model. In summary, a total of 6,500 instances were annotated and used in NLP model development.
2.2.3NLP model development
We utilized a pre-trained BERT model as the foundation for NLP model development. This pre-trained model was further fine-tuned using 90% of the annotated datasets, with the remaining 10% of the instances set aside for the blind test. The training data and the blind test data were balanced according to the ethnicity, gender, and age groups of the patients from which they were extracted.
Dataset balancing was based on previous studies finding significant associations of gender and ethnicity with emotional violence and IPV [28], as well as DV experiences and the need for social support [29]. As a result, the training dataset consisted of 5849 instances and the blind test dataset consisted of 651 instances.
The BERT for Multi-label Sequence Classification model was used. It was trained for violence type, patient role, polarity, presence, temporality, and setting. The annotated data were divided into two sets before training the model. During training, we applied 5-fold cross-validation to find model performance (Bert tokenizer, max_lenth=256, and 4 epochs for each fold; the 4th epoch results in each fold achieved the best performance when using AUROC as the evaluation matrix). To make full use of the annotated training dataset, we trained a full model on all the training data for practical use, and evaluated this with the blind test set. For detailed data processing procedures, see Figure 1.
3.Results
3.1Annotation agreement
Table 3 summarizes agreement for double annotation instances. Inter-annotator agreement (IAA) and Cohen’s kappa value demonstrate robust agreement between the two annotators. Specifically, Cohen’s kappa values ranged from 0.78 to 0.84, indicating substantial concordance.
3.2Data summary
Of all 6500 instances, 5849 were applied for training and validating and 651 (10%) remained for the blind test. Table 4 summarizes the datasets and Table 5 summarizes label frequency in the training and test sets.
3.3Model performance on validation and blind test datasets
Table 6 summarizes the results of 5-fold cross-validation with 5849 instances as training and validation data. Table 7 summarizes the model performance on the blind test dataset.
Table 6 indicates strong model performance in identifying emotional violence, financial violence, physical violence (non-sexual), irrelevant instances, and unspecified violence. However, regarding sexual violence, while precision was high (mean=0.85), recall was not as satisfactory. The model effectively distinguished between perpetrator and victim roles among patients, as well as the polarity of affirmed and abstract violence. Additionally, it accurately detected the presence of actual violence, temporal aspects labeled as ‘Not-past’, and instances of domestic violence. Table 7 findings were similar, showing that the classification performance for violence type is good. Specifically, on the blind test data the model could correctly identify emotional violence with F1=0.89, financial violence with F1=0.88, physical (non-sexual) violence with F1=0.84, sexual violence with F1=0.67, and unspecified violence with F1=0.81.
Considering the patient role, perpetrator and victim were ascertained with good performance (F1 0.89 and 0.84 respectively). However, performance was unsatisfactory for witnessed violence, and the co-occurrence of various patient roles due to insufficient representation. For polarity, the model could predict abstract violence and affirmed violent behavior well with F1 scores larger than 0.89. For presence, this model performed well in identifying the actual violent experiences of the patients (F1=0.95). Temporality prediction was challenging for the model, as indicated by low precision, recall, and F1 score of the blind test set. The model showed good performance for predicting the domestic setting (F1=0.88).
4.Discussion
4.1Principal findings
Following an initial development of NLP to ascertain violence experiences from mental health records [5], this study developed new models with greater functionality. Our objective was to extend detectable violence types, from physical and sexual to encompass emotional and financial abuse, and to develop type-specific characterisations of the patient role (victim, perpetrator, witness), polarity (affirmed, negated, abstract), presence (actual, threat, unclear), temporality (past, not-past, unclear), and setting (domestic, not domestic, unclear). Using 6500 annotated instances we were able to fine-tune a multi-label BERT model with promising results in nearly all respects where there were sufficient annotations available, aside from temporality.
This study describes what we believe to be an important innovation in applying NLP to extract information about violence. Although previous NLP development has been described that extracts information on physical assault, threat, sexual assault, emotional/verbal abuse, stalking, financial abuse, social abuse, and property damage, this was developed for text from police reports and not from health records [13]. Furthermore, NLP previously developed for extracting violence-related information from the Maudsley CRIS mental healthcare data resource was focused on mentions of violence, domestic violence, sexual violence, physical violence, perpetration and victimisation as discrete entities without any functionality to combine these features - for example, to ascertain receipt of domestic sexual violence; in addition, financial and emotional violence were not previously captured [5]. The developments presented here address this research gap and applying the results of this model should have a high potential to transform the ascertainment of violence as an exposure and outcome in mental healthcare, allowing novel research in this sector.
Considering methodology, existing studies have tended to focus on single binary classification models. Training binary classifiers independently may not account for label dependencies, potentially leading to suboptimal results. In contrast, the multi-label classifier we have trained can capture dependencies and correlations between labels. Integrating this information is important for characterizing types of violence [16,26] - most importantly in distinguishing between receipt (victimization) and perpetration of violence, as well as the setting, and other features, all of which are important in the interpretation and application of NLP-derived data in research. For example, it has been found that female emotional abuse victims experiencing abuse more than once a week have an increased risk of depression [27]. Applying the NLP developed here across large data resources will enable further investigation of these exposures as predictors of mental disorder course and outcome.
4.2Limitations
Although successful in many respects, our attempts to create an NLP model that could characterize references to temporality, and capture witnessed violence faced significant challenges. In part, this endeavor was hampered by insufficient sample size, particularly for witnessed violence, preventing the successful development of the intended model. Also, because health records text is inevitably recorded by clinicians in retrospect, the near-universal use of the past tense limited ascertainment of temporality, compounded by text on timing which was either absent or too highly varied for accurate capture; inter-rater agreement was also the lowest for temporality as a construct in both evaluations which may reflect ambiguity in its description.
In addition, we focused on four specific types of violence namely physical (non-sexual) violence, sexual violence, emotional violence, and financial violence, with a further unspecified violence category to capture other types. This characterisation might be improved by adding more types of violence salient to mental health, as well as potentially by more granular categorisation within each type and better distinguishing between threats and actions of violence. In annotations, we sought to assign single violence types for each instance without considering the co-occurrence of various types of violence in one description. In this respect, emotional abuse may co-occur with threats and/or acts of physical and/or sexual violence from intimate partners, constituting acts of violence in their own right. Research indicates that the deployment of various forms of psychological violence is linked to a heightened likelihood of physical and sexual violence against female partners, multifaceted forms of abuse that can significantly impact women, irrespective of whether other forms of violence are present [23]. Future studies could try to overcome this by allowing multiple labels; however, of the 6500 instances annotated in our development, less than 10 described co-occurring types.
As a final consideration particularly applicable to mental healthcare text, in our annotation process, no attempt was made to infer the truth or not of recorded violence instances. Thus, delusional or hallucinated experiences of violent acts or threats were annotated in the same way as non-delusional experiences, as they tended to be described as such in records with only more distant, qualifying text indicating the suspected origin as a false belief or perception. This will be an important consideration when inferences are made from cross-diagnostic prevalence studies and further development would be required for this characterisation to be deployed.
5.Conclusion
In this study, we sought to improve capabilities in NLP for structured data extraction of received and perpetrated violence in mental health electronic health records. We were successful in developing a suite of NLP applications using a BERT model to extract information on various types of violence (physical, sexual, emotional, financial) and to characterize the polarity, presence, patient role, and setting. Ascertaining the temporality of the described violence in the same model was more challenging and needs further development. The model performed well in other respects and holds the potential to advance both research and clinical practice.
Data Availability
The data analysed in this study is subject to the following licenses/restrictions: All the relevant aggregate data are found within the article. The data used in this work have been obtained from CRIS. It provides authorised researchers regulated access to anonymised information extracted from SLaM's electronic clinical records system. Individual-level data are restricted in accordance with the strict patient-led governance. Data are available for researchers who meet the criteria for access to this restricted data: (i) SLaM employees or (ii) those having an honorary contract or letter of access from the trust. Requests to access these datasets should be directed to CRIS administrator: cris.administrator@kcl.ac.uk
Acknowledgements
RS is part-funded by: i) the NIHR Maudsley Biomedical Research Centre at the South London and Maudsley NHS Foundation Trust and King’s College London; ii) the National Institute for Health Research (NIHR) Applied Research Collaboration South London (NIHR ARC South London) at King’s College Hospital NHS Foundation Trust; iii) UKRI – Medical Research Council through the DATAMIND HDR UK Mental Health Data Hub (MRC reference: MR/W014386); iv) the UK Prevention Research Partnership (Violence, Health and Society; MR-VO49879/1), an initiative funded by UK Research and Innovation Councils, the Department of Health and Social Care (England) and the UK devolved administrations, and leading health research charities; v) the NIHR HealthTech Research Centre in Brain Health at King’s College London and South London and Maudsley NHS Foundation Trust.
AR is funded by i) Health Data Research UK, an initiative funded by UK Research and Innovation, Department of Health and Social Care (England); ii) devolved administrations, and leading medical research charities and iii) the UK Prevention Research Partnership (Violence, Health and Society; MR-VO49879/1), an initiative funded by UK Research and Innovation Councils, the Department of Health and Social Care (England) and the UK devolved administrations, and leading health research charities.
HKS is a fully funded PhD student at the National Institute for Health Research (NIHR) Maudsley Biomedical Research Centre.
SS is a fully funded PhD student at the NIHR HealthTech Research Centre in Brain Health.
This research was supported by the UK Prevention Research Partnership (Violence, Health and Society; MR-VO49879/1), which is funded by the British Heart Foundation, Chief Scientist Office of the Scottish Government Health and Social Care Directorates, Engineering and Physical Sciences Research Council, Economic and Social Research Council, Health and Social Care Research and Development Division (Welsh Government), Medical Research Council, National Institute for Health and Care Research, Natural Environment Research Council, Public Health Agency (Northern Ireland), The Health Foundation, and Wellcome. The views expressed in this Article are those of the authors and not necessarily those of the UK Prevention Research Partnership or any other funder.