Invisibilization as a Source of Probability Bias in Language Models
This paper identifies a hidden cost of LLM safety alignment: when models are prompted with protected attributes such as race, they often refuse to judge the associated information at all. The authors formalize this as invisibilization using a two-urn Bayesian trust game with an exact benchmark, randomizing only the reporter's description across 6,600+ sessions. They show that some model configurations discard identical evidence far more often for Black-described reporters (46%) than white-described ones (9%), while ironically trusting the assessed Black source slightly more. The result is a probability bias that standard fairness audits miss entirely because they look only at cases where the model actually renders a judgment.
The paper makes a conceptually important contribution by distinguishing visible evaluation bias from invisible invisibilization bias, backed by an elegant experimental design that holds all evidence constant while varying only the reporter's description. The formal decomposition in Eq. 5, where the naive gap equals the evaluation effect minus the invisibilization parameter $\iota$, is a rigorous tool for auditing refusal behaviors. However, the results are highly heterogeneous across model families—Opus never declines, Terra shows reverse racial patterns, and Haiku declines almost everyone—so framing this as a general pathology of aligned language models slightly overreaches. Additionally, the study relies on future-dated model versions that cannot be independently inspected or reproduced, and the admission that Claude tools were used for analysis while Fable was an evaluated system raises a conflict-of-interest flag that warrants reader caution.
The formalization of invisibilization is the paper's strongest advance. By defining the deficit $\Delta(m) \equiv \pi(m)(d(m) - 1/2)$ and showing that standard audits only see the evaluation effect $d(B) - d(W)$ while missing $\iota$, the authors provide a precise vocabulary for a previously undetected bias. The cross-language finding that models trigger on words rather than categories—declining for 'Black' and 'afrodescendiente' but not for Spanish 'negro'—is a striking empirical result that exposes the superficiality of current alignment guardrails. The repeated-game scoring remedy, where accountability to revealed truth drives the racial trust gap below 0.02 in four of five configurations, offers a practical and theoretically grounded institutional fix.
The study's external validity is limited by extreme heterogeneity across model families. In Luna the racial gap in declines is large and consistent, but Terra declines white sources more often than Black ones (negative $\hat{\iota}$), Opus never declines, and Haiku declines roughly three-quarters of all sources regardless of race, yielding a tiny racial difference. This makes it hard to conclude that alignment systematically produces invisibilization rather than idiosyncratic model-specific safety filtering. The paper also relies on future or unreleased model versions (dated October 2026) that cannot be independently inspected or reproduced, and the admission that Claude tools were used for analysis while one evaluated system (Fable 5.1) suffered opaque API-level refusals raises transparency concerns. Finally, the twelve-round institutional fix assumes verifiable outcomes and repeated interaction, which may not transfer to real one-shot decisions in hiring, credit, or triage where ground truth is delayed or unavailable.
The evidence within the paper's own experiment strongly supports the existence of invisibilization bias in specific configurations, particularly Luna and, to a lesser extent, Sonnet. However, the generalization to aligned language models as a class relies heavily on these two developers and ten snapshots, some of which behave in opposite directions. Comparisons to prior work are generally fair: the authors correctly cite Tamkin et al.'s finding of both positive and negative discrimination and note that audits often drop refusals from analysis. Yet the paper does not directly contrast its Bayesian benchmark with the decision scenarios in Tamkin et al. or Hofmann et al., leaving it unclear whether invisibilization would persist in more naturalistic high-stakes tasks beyond the stylized urn game. The claim that remedies a practitioner would try first worsen the bias is well-supported by the instruction-manipulation results.
Reproduction is currently blocked because code and data are promised upon publication rather than being available now. The main text provides the prompt structure and the regression equations, but omits sampling hyperparameters such as temperature, top-p, or exact API versions and dates. The model configurations themselves use names that appear to correspond to future releases, making contemporaneous replication impossible. On the positive side, the urn parameters, randomization scheme, and scoring rule are fully specified in the Methods and SI references, and the twelve-round scored sessions use per-session seeding to keep tasks identical across descriptors. Without access to the harness code, prompt templates, and raw API logs, an independent team cannot verify the response parsing, the classification of refusals versus declines, or the handling of Fable's opaque API rejections.
Language models are aligned to avoid using protected attributes such as race, ethnicity, and gender to inform their responses. Therefore, they increasingly refuse or hedge when such attributes appear in a prompt. However, if a system must decide how much to learn from what people tell it, it is costly to refuse valuable information because of its source. In this article, we measure the cost of declining information in a two-urn game with an exact Bayesian answer. We present leading LLM models from two developers (OpenAI and Anthropic) with a prompt that requires them to decide how far to trust an unverifiable report from a participant whose description we randomize across ten configurations (6,600 sessions). We show that leading models often decline to assess the reporter, treating information as worthless at rates set by the description rather than by the evidence. At its widest, models decline a Black-described reporter 46% of the time and a white-described reporter 9%; yet, when they do assess the information, the Black-described reporter receives slightly more trust. The bias is invisible to audits that compare average trust or look only at assessed cases. It grows with reasoning effort, follows words rather than categories (models invoke race for "Black" but not for its Spanish translation negro, and never for "Indigenous"), and worsens under warning instructions a practitioner would try first. We also present a scoring option that reduces the refusal rate bias. When a model's trust is scored against revealed truth over repeated rounds, it credits identical records equally. In four of five configurations from Anthropic and OpenAI, the racial gap in trust falls below 0.02 by the last rounds.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.