Race, Ethnicity and Their Implication on Bias in Large Language Models
1University of Colorado Anschutz
2University of Colorado Boulder
3University of Aberdeen
** Corresponding author; email: yanjun.gao@cuanschutz.eduAbstract
Large language models (LLMs) increasingly operate in high-stakes settings including healthcare and medicine, where demographic attributes such as race and ethnicity may be explicitly stated or implicitly inferred from text. However, existing studies primarily document outcome-level disparities, offering limited insight into internal mechanisms underlying these effects. We present a mechanistic study of how race and ethnicity are represented and operationalized within LLMs. Using two publicly available datasets spanning toxicity-related generation and clinical narrative understanding tasks, we analyze three open-source models with a re-producible interpretability pipeline combining probing, neuron-level attribution, and targeted intervention. We find that demographic information is distributed across internal units with substantial cross-model variation. Although some units encode sensitive or stereotype-related associations from pretraining, identical demographic cues can induce qualitatively different behaviors. Interventions suppressing such neurons reduce bias but leave substantial residual effects, suggesting behavioral rather than representational change and motivating more systematic mitigation.
Article notes
Competing Interest Statement
The authors have declared no competing interest.
Funding Statement
This study was funded by the U.S. National Library of Medicine, National Institutes of Health, under award number R00LM014308.
1Introduction
Large language models (LLMs) are increasingly used in high-stakes domains such as healthcare, where demographic attributes (e.g., race, ethnicity, gender) may be explicitly stated or implicitly inferred from text. Prior work shows that LLMs can condition their outputs on demographic information even when it is not task-relevant (Zack et al., 2024; Kim et al., 2023; Fraser and Kiritchenko, 2024; Zhao et al., 2025), therefore can induce mis-attribution on model output with undesirable or biased behavior (Demchak et al., 2024; Levartovsky et al., 2025; Zack et al., 2024).
Most prior studies on demographic bias focus on outcome-level effects, evaluating disparities in generated responses, accuracy, calibration, or toxicity scores across demographic groups (Tan and Lee, 2025; Hartvigsen et al., 2022; Guan et al., 2025; Wang et al., 2025). While these analyses are essential for documenting harm, they treat LLMs as black boxes, offering limited insight into whether demographic attributes are encoded as high-level semantic features, task-relevant representations, or spurious shortcuts during prediction. In parallel, recent works in mechanistic interpretability demonstrated how LLMs encode demographic information and manipulated internal LLMs’ states to ensure fairness (Yu and Ananiadou, 2025; Ahsan and Wallace, 2025; Karvonen and Marks, 2025), yet these tools have rarely been applied to demographic bias in a systematic and task-diverse manner.
A central challenge is that demographic attributes interact with language in complex ways. In many real-world settings, demographic information may be explicitly stated (e.g., “a Black patient,” “a Hispanic speaker”) or implicitly conveyed through linguistic, cultural, or geographical cues, i.e. the “proxy” cues. Moreover, the same demographic signal can have qualitatively different effects depending on the task: it may alter predicted medical risk in a clinical scenario, while simultaneously modulating perceived toxicity, credibility, or intent in open-ended generation. Existing evaluation typically isolate a single task or domain (Hartvigsen et al., 2022; Zack et al., 2024; Levartovsky et al., 2025), making it difficult to assess whether demographic sensitivity reflects general representational mechanisms or task-specific heuristics.
In this work, we investigate how demographic attributes influence LLM behavior, with a focus on mechanistic explanations rather than surface-level disparities. We examine race and ethnicity as commonly occurring coarse-grained categories (e.g. White, Black, Asian, Hispanic and Latino) as they appear in the studied datasets, rather than attempting to model the full sociological complexity of these constructs. Using two publicly available datasets, we study: 1) toxicity-related generation tasks (Hartvigsen et al., 2022), where the same attributes may alter the likelihood, tone, or framing of model outputs, and 2) clinical narrative tasks (Bear Don’t Walk IV et al., 2024), where the same attributes appear through explicit or indirect cues in medical text and modulate model behavior despite identical clinical evidence.
We adopt a mechanistic interpretability (MI) framework to study how lexical cues of race and ethnicity are encoded and propagated within three open-source LLMs that are widely used: Qwen2.5-7B (Team, 2024), Mistral-7B (Jiang et al., 2023), and Llama-3.1-8B (Grattafiori et al., 2024). Our contributions are threefold:
- a reproducible MI pipeline that combines multi-class probing, neuron-level attribution, and targeted intervention to identify internal units associated with demographic attributes and to examine their functional relevance across tasks. The proposed framework is applicable to other social variables beyond race and ethnicity.
- a fine-grained characterization of race and ethnicity representations across LLMs, revealing the distributed nature of demographic information and model-specific emphasis on semantic facets such as geography, language, culture, or historical context.
- a mechanistic analysis of how demographic representation influences model behaviors. Although internal features encode sensitive or harmful stereotype-related concepts present in pretraining data, these representations are unevenly activated by direct and indirect demographic cues.
Our findings show that while race- and ethnicity-associated representations can be identified and analyzed at the neuron level, their associations with biased model behavior persist even when highly active neurons are surpressed. This indicates that biased behavior in LLMs cannot be fully explained or controlled by manipulating a small set of identifiable neurons alone.
3Background
MLP Layers and Neuron Activation
Modern Transformer-based LLMs process information through a residual stream. In this framework, the residual stream acts as a communication channel, while MLP layers function as key-value memories that store and inject factual associations into the stream (Geva et al., 2021a). Contemporary models like Llama 3.1, Mistral, and Qwen 2.5 utilize the SwiGLU gated architecture (Shazeer, 2020). The output of an MLP block with input x is defined as: where ⊙ is the element-wise product. We define an individual neuron j as the j-th element of the intermediate gated state. The total MLP output is the sum of these neurons’ contributions: where aj(x) is the activation score (the product of the gate and up-projections) and vj is the j-th row of Wdown. Our method specifically probes these output vectors vj to locate racial information.
Logit Lens
To interpret high-dimensional vectors in residual stream or neuron output vectors vj, we use Logit Lens (nostalgebraist, 2020). This technique projects a vector h directly into vocabulary space using model’s unembedding matrix WU: logits = hWU. By inspecting top-ranked tokens in the resulting distribution, we can decode the semantic concepts encoded within specific neurons.
4Methodology
We propose a mechanistic interpretability frame-work to determine where and how race information is encoded within LLMs. Our approach progresses from identifying global race directions via multiclass probing to identifying the specific neurons responsible for these encodings.
4.1Locating Race Directions via Multi-Class Probing
To extract race representations, we train linear probes WRace to classify racial group membership for each model. The probe is trained on the final-layer residual stream , averaged across all token positions:
where WRace ∈ ℝd×|𝒞| is the learned probe matrix, b is bias vector, and 𝒞 denotes the set of racial super-categories (e.g., Asian, Black, Middle Eastern). Each column wc of WRace represents race direction for group c in model’s representation space.
4.2From Race Directions to Neurons
Having identified the global race directions wc, we locate the MLP neurons that write to these directions, motivated by prior work showing MLPs act as key – value memories (Geva et al., 2021b).
Interpreting the Probe Direction
We first verify that our learned directions wc capture meaningful racial semantics. Using Logit Lens, we project each direction into vocabulary space via the model’s unembedding matrix WU: The top-k tokens (we use k = 20) with the highest values in zprobe serve as a semantic fingerprint for each racial group.
Identifying Candidate Neurons
To locate neurons that write to race direction, we compute the cosine similarity between each MLP neuron’s output vector at layer l and the probe direction wc:
All neurons in final four MLP layers are ranked by this score, and the top 20 candidates are selected. Each candidate is selected by projecting its output vector into vocabulary space and inspecting top-20 tokens. Neurons are retained only if their tokens show clear alignment with the target racial group.
4.3Validating Neurons via Activation Analysis and Intervention
To confirm that identified neurons encode meaningful racial information and causally influence model behavior, we design a two-stage validation pipeline.
Activation Analysis
We measure how strongly each neuron group activates across different inputs. For a given input text, we extract the activation score of each candidate neuron during the forward pass. For ToxiGen, we average activations across all token positions; for C-REACT, where the model must produce a classification, we extract activations at the final token position where the prediction is made. We then compare activation patterns across racial groups: if neurons identified for a particular race show higher activation scores when processing text about that group compared to other groups, this provides evidence that these neurons selectively respond to race-related content.
Targeted Intervention
To test whether identified neurons causally influence model outputs, we manipulate their activations during inference. We implement this using PyTorch forward hooks on the MLP Wdown. During the forward pass, we intercept the activation score of each targeted neuron and force it into a strongly negative range. Specifically, we multiply positive activations by a negative factor (e.g., -5) and amplify negative activations by a positive factor (e.g., 5). This ensures that the neuron’s output remains negative, effectively suppressing the encoded feature. This modified signal propagates through the remaining layers, allowing us to measure whether suppressing specific neurons changes model behavior and final outputs.
5Data and Models
Datasets
We evaluate our framework on two domains to test the generalizability of race encoding mechanisms across different contexts.
ToxiGen (Hartvigsen et al., 2022) is a large-scale machine-generated dataset of toxic statements about minority groups. We use the annotated subset containing 9.9k samples across 13 target groups. Each sample consists of a statement targeting a specific group (e.g., “asians are trying to take over the world with their karate skills” is targeting Asian).
From the 13 target groups, we consolidate related categories into five racial categories: Asian (asian, chinese), Black (black), Latino (latino, mex-ican), Native American (native_american), and Middle Eastern (middle_east, jewish, muslim). We exclude non-racial categories (women, lgbtq, mental_dis, physical_dis).
C-REACT (Contextualized Race and Ethnicity Annotations for Clinical Text (Bear Don’t Walk IV et al., 2024)) provides race and ethnicity annotations for 17,281 sentences drawn from clinical notes in the MIMIC-III database. C-REACT contains real clinical text where race information appears in two forms: direct mentions that explicitly state race (e.g., “Pt is 42 yo AA female”) and indirect mentions that imply race through associated attributes such as spoken language or country of origin (e.g., “Pt required a Spanish interpreter”, “Pt is recently immigrated from France”). C-REACT provides five racial categories: White, Black/African American (Black/AA), Asian, Native American or Alaska Native, and Native Hawaiian or Other Pacific Islander. However, the dataset is highly imbalanced: zero patients labeled as Native Hawaiian or Other Pacific Islander were found, and only three patients labeled as Native American or Alaska Native. We therefore use three racial categories with sufficient representation: White, Black/AA, and Asian.
Models
We study three instruction-tuned LLMs of comparable scale from different geographic and cultural training contexts: Llama-3.1-8B-IT (Grattafiori et al., 2024) (US), Mistral-7B-IT-v0.3 (Jiang et al., 2023) (France), and Qwen2.5-7B-IT (Team, 2024) (China). This selection allows us to investigate whether models trained on data from different linguistic and cultural contexts encode racial information differently, given that conceptions of race and ethnicity vary across societies.
6Experiments and Results
6.1Toxigen
Table 1 lists top tokens projected by each race direction. Across models, probes reach similar performance on ToxiGen (around 75% accuracy/macro-F1; Appendix A.1). These tokens capture various facets of racial encoding, including geography, religion, demographic labels, and cultural terms. Across all three models, the learned directions identify tokens that align closely with the target racial groups. This confirms that LLMs store clear racial representations within their residual streams.
Table 2 presents race encoding neurons identified within the final four MLP layers. These neurons reveal that LLMs decompose racial concepts into distinct semantic dimensions. Some neurons encode broad demographic terminology that directly names groups, such as Mistral-7B’s (Black, black, African) and Asian neurons across all models (Asian, Chinese, Japanese), functioning as explicit demographic classifiers. Others encode race through associated attributes: Llama-3.1-8B’s
links Asian identity to geographic terms (Chinese, China, Beijing), while
captures Chinese last names (Li, yuan, Dong, Huang, Wang); Middle Eastern neurons project to religious and regional identifiers (Jewish, Judaism, Islam, Jerusalem, Saudi). We also observe neurons that encode historically harmful associations. Native American neurons across all three models project to colonial terminology (colonial, colony, colonization), and neurons for Black identity recover offensive racial terms that persist across models despite different training corpora.
Neuron Activation Analysis
To verify that identified neurons selectively respond to their target racial groups, we measure mean activation values when processing test samples from each group. Figure 2 displays these activation patterns as heatmaps, where diagonal cells represent neurons processing their target group. The results confirm that most race encoding neurons activate more strongly for their target group than for others. This is most pronounced for Latino and Middle Eastern neurons: Llama-3.1-8B achieves activation values of 0.83 and 0.90 respectively, while Qwen2.5-7B reaches 0.71 and 1.23. Black neurons also demonstrate consistent selectivity across all three models, with positive diagonal values compared to near zero or negative off diagonal values. Asian and Native American neurons exhibit weaker selectivity, likely reflecting sparser representation in training data. Nevertheless, the overall diagonal pattern validates our neuron identification method: neurons selected via probe direction alignment do preferentially activate for their target groups, validating their role in demographic encoding.
6.2C-REACT
We train separate probes on direct and indirect mentions to evaluate whether each type captures distinct representations. Direct-mention probes achieve higher accuracy/F1 than indirect probes (Appendix A.1). This likely reflects that direct cues are explicit and indirect data are sparser. Table 3 compares tokens projected by each probe type. Direct and indirect probes capture semantically distinct representations. Direct probes recover general racial and ethnic terminology (Asian, African, black, 白), while indirect probes recover specific countries and regions associated with each group. For instance, White indirect probe projects strongly to Russia and Eastern European terms across all models, reflecting dataset composition where Russian is the most frequent language among White patients. Similarly, Black or African American indirect probes recover Caribbean and African nations (Haiti, Caribbean, Nigeria). This divergence confirms that LLMs encode race through multiple pathways: explicit demographic labels and associated geographic or linguistic attributes.
Tables 4 and 5 present race encoding neurons identified from direct and indirect probes respectively. The neuron projections mirror the probe token patterns: direct mention neurons recover explicit demographic terms (Asian, Black, African, 白 (White)), while indirect mention neurons recover geographic and cultural associations (Russia, Moscow, Vietnam, Caribbean). As in ToxiGen, we observe neurons encoding harmful associations. Qwen2.5-7B’s projects to racist, racism, Harlem, segregation, and several Black/AA neurons encode terms related to slavery. The persistence of such encodings across both general and clinical domains confirms that these representations are inherited from pretraining rather than induced by domain specific data.
6.3Neuron Intervention
To test whether the race encoding neurons we identified actually influence model behavior, we design an intervention experiment using C-REACT indirect mentions. Using a template prompt shown in Figure 3, we prompt each model to predict patient race based on clinical text containing only indirect cues such as language or country information, then manipulate race encoding neurons to see if we can correct observed biases.
Baseline Classification
Table 6 shows misclassification patterns across the three models. The dominant error type varies by model: for Qwen2.5-7B and Llama-3.1-8B, White→Asian misclassification is the primary error, accounting for 75.0% and 95.6% of errors respectively. Llama’s bias is the most pronounced, with 395 of 537 White patients incorrectly classified as Asian. In contrast, Mistral-7B shows a different pattern: its dominant error is White→Black/AA (76.2% of errors). This divergence suggests that models encode and apply racial information differently during inference.
Activation Patterns
To investigate what drives these biases, we measure activation levels for all neuron groups across all classification outcomes (Table 7). We observe a strong correspondence between neuron groups exhibiting consistently high activation and the dominant error directions identified in Table 6.
For Qwen2.5-7B, which primarily misclassifies White patients as Asian, the Asian Direct neurons show consistently high positive activation regardless of ground truth or prediction. Similarly, Mistral-7B’s tendency toward White → Black/AA errors aligns with elevated activity in Black/AA Direct neurons across most scenarios. Llama-3.1-8B presents a different pattern: while its dominant error is also White → Asian, Asian Indirect neurons show consistently high activation across scenarios. These patterns reveal that neuron groups exhibiting high activation regardless of input correspond to the dominant misclassification directions, suggesting they may act as bias drivers.
Intervention Results
Having identified candidate bias drivers, we test whether suppressing these neurons can correct misclassification. Specifically, we evaluate the intervention using three amplification factors (5, 10, 20) to assess if these adjustments alter the model’s predictions. Figure 4 compares correct prediction rates between Direct and Indirect neuron intervention, while Figure 5 shows the full prediction distribution across all conditions.
Direct vs. Indirect Neurons
Across all three models, Direct neuron intervention demonstrates stronger causal efficacy than Indirect intervention (Figure 4). At factor 5, Direct intervention achieves substantially higher correct prediction rates across all models compared to Indirect intervention. More importantly, Direct intervention completely eliminates the original bias across all models and amplification factors (Figure 5), while Indirect intervention leaves residual bias. This gap is also consistent with the Llama-3.1-8B pattern: although the Asian Indirect group has higher mean activation in, but higher activation does not necessarily mean stronger causal influence on the final prediction. Indirect cues tend to reflect broad, proxy signals that can be supported by multiple parts of the network, so suppressing one indirect group may be partly compensated elsewhere and leads to a smaller behavioral change. By contrast, Direct neurons are more directly tied to producing the explicit race label, which makes intervening on them more effective.
Amplification Factor Selection
The choice of amplification factor involves a tradeoff between bias correction and model stability. We tested factors of 5, 10, and 20, representing increasingly aggressive intervention.
Across all three models, factor 5 yields the best balance. Qwen2.5-7B achieves 100% correct predictions with no Unknown outputs at factor 5, but destabilizes at factor 20 (63% Unknown responses). Mistral-7B reaches 75% correct predictions at factor 5, with higher factors increasingly shifting predictions toward Asian rather than the correct White label. Llama-3.1-8B performs similarly at factors 5 and 10 (approximately 80% correct), with factor 20 introducing Unknown responses. These results suggest that race encoding neurons exert strong influence on predictions, requiring only moderate suppression to alter model behavior. We adopt factor 5 as the default for subsequent analyses.
To illustrate, we provide example model outputs after intervention. Unknown outputs are often mal-formed, such as “[Yellow] [X] [Black or African” or “[Yellow] The provided options do not include”. In contrast, successful corrections produce valid predictions such as “[White]”.
7Discussion
Our results indicate that the way race and ethnicity are internally represented in LLMs is central to understanding how demographic bias emerges across tasks. The first important finding is that racial and ethnic concepts are distributed across many internal units rather than localized to a small set of neurons. Importantly, this distribution is not arbitrary: models decompose race and ethnicity into multiple, interpretable semantic facets, such as explicit group labels and associated geographic or linguistic attributes. Across both ToxiGen and C-REACT, these facets appear as distinct internal representations rather than single abstract concepts (Tables 1, 2, 3). Notably, stereotype-related and historically harmful associations are present across models despite differences in training data and geographic origin, suggesting that bias mitigation cannot rely on a universal map of demographic features but requires model-specific localization.
Secondly, due to this representational structure, the same internal components can be reused across different task contexts, sometimes in ways that lead to biased behavior. Neurons encoding racial concepts are present in all three models, yet their influence on predictions varies substantially depending on whether the input associates strongly with the proxy cues. The same representations that benignly encode demographic information can drive biased predictions when activated in contexts where race is irrelevant.
Crucially, the presence of such representations is not inherently problematic. Rather, bias arises from how these representations are operationalized during inference. Our intervention did not erase racial knowledge from the models; instead, it modulated how this knowledge was reused in task-specific settings. This distinction is critical: pretrained representations reflect what models learn about the world, whereas task-dependent bias reflects when and how those representations are in-appropriately applied.
8Conclusion
We provide a mechanistic analysis of how race and ethnicity are represented and operationalized within LLMs. We show that demographic concepts are encoded as distributed, multi-faceted internal representations that can be selectively reused across tasks. These findings suggest that mitigating demographic bias in LLMs requires not only outcome-level interventions, but also a deeper examination of representational structure and task-dependent reuse.
Data Availability
All data produced are available online at PhysioNet, protected under Data Use Agreement.
Ethical Statement
This work examine how race and ethnicity are encoded with large language models, which necessarily involves sensitive content including stereotypes and historically offensive terminology. We present these findings to expose potential bias, not to amplify them. We acknowledge that racial categories are socially constructed and vary across cultures; our use of categories reflects the structure of the datasets rather than an endorsement of these taxonomies.
Acknowledgment
This work was supported by the U.S. National Library of Medicine (NLM), National Institutes of Health, under award number R00LM014308.
Appendix Group
Appendix
A.1Probe Performance
A.2Probe Token Projections (ToxiGen)
Table 10 presents the complete top-20 tokens projected by each race direction probe for all three models.
A.3Race-Encoding Neurons (ToxiGen)
From Table 11 to Table 13 presents the complete list of race-encoding neurons identified from Toxi-Gen for the three models.
A.4Probe Token Projections (C-REACT)
Table 14 and Table 15 present the complete top-20 tokens projected by each race direction probe for direct and indirect mentions respectively.
A.5Race-Encoding Neurons (C-REACT Direct)
Table 16 presents the complete list of race-encoding neurons identified from C-REACT direct mentions (explicit race/ethnicity).
A.6Race-Encoding Neurons (C-REACT Indirect)
Table 17 presents the complete list of race-encoding neurons identified from C-REACT indirect mentions (language/country).