Cross-Model In-Context Neurofeedback Reveals (Limited) Activation Control in LLMs
Probes on hidden activations are a promising tool for detecting deception and harmful intent, but they assume models cannot volitionally alter their own representations. Baldelli et al. ask whether open-weight LLMs can control their activations through in-context neurofeedback, devising a cross-model writer-subject protocol to disentangle genuine self-access from in-context pattern matching. Across 42 semantically opaque readouts and six safety monitors, they find that free-generation effects are largely explained by sentence-selection competence, while fixed-sentence control is tiny and monitor evasion is essentially absent. The results rigorously bound the current threat but flag a capability that may grow with model scale.
The paper presents a carefully designed, large-scale empirical study that credibly bounds in-context activation control in open-weight LLMs. Its central negative finding—that free-generation effects are driven by in-context pattern matching rather than privileged self-access, with only three of ten models showing statistically significant but tiny fixed-sentence effects and no model reliably lowering a safety-monitor score—is well-supported by the data. While the authors are appropriately cautious in framing these results as not yet a safety threat, the practical significance of sub-two-point shifts on a 100-point scale remains debatable.
The experimental design is the clearest strength. By separating in-context pattern matching, privileged generative access, and processing-time control via a cross-model writer-subject grid and a fixed-sentence condition, the authors directly address the confounds that plagued earlier work. The use of 42 semantically opaque readouts—principal component bands, encoder residuals, and trajectory geometry—ensures that free-generation success is nontrivial, and the scale of 168,000 sessions across five model families lends statistical weight to the negative finding that diagonal excess is never significantly positive.
The fixed-sentence control effects are statistically significant for only three models, but at 0.4 to 1.8 points on a 100-point scale they are an order of magnitude smaller than free-generation effects and of doubtful practical relevance.
The uniformly random feedback control is another methodological limitation: because real scores are temporally correlated while random ones are independent, the objective difference $\tau_t^{\max} - \tau_t^{\min}$ may not fully cancel non-objective-related drifts. The authors acknowledge this, noting that "we observe small drifts on some readouts or monitors, both against and in favour of the instructed direction," and concede that they therefore only trust effects that are large and sustained—a standard that the fixed-sentence and monitor results only marginally meet.
The evidence strongly supports the claim that free-generation performance is largely due to in-context pattern matching, as external writers of matched competence move subject readouts just as much, yielding no significant diagonal excess. The reconciliation of Li et al. (2025) and Aoki et al. (2026) is convincing at a conceptual level: Baldelli et al. confirm Aoki et al.'s suspicion that prior positive results rely on superficial mechanisms rather than genuine metacognitive control. The monitor experiments align with Kowalski et al. (2026) in showing an asymmetry—raising a concept's signature is easier than suppressing it—but the observed effects are much smaller and limited to a single model raising (not lowering) scores.
Reproducibility is generally strong: the authors release code and data, use greedy decoding, exhaustively document prompts and readout calculations in the appendices, and identify all model checkpoints. The primary barrier to independent reproduction is computational cost, with 168,000 sessions in Experiment 1 alone and additional activation extraction via vLLM-Lens. However, the exact model revision hashes are not consistently reported, and because several models are instruction-tuned variants with potentially changing behavior across API updates, locking the precise weights used would improve confidence in replication.
Probes on activation vectors are widely used to detect misbehavior in language models, but their success relies on the assumption that models cannot control what their activations encode. Humans can learn, through neurofeedback, to control the firing of even single neurons, and recent studies investigated whether this ability extends to language models, reaching conflicting conclusions. No experimental design so far disentangles competence at the task from genuine access to one's own activations. We fill this gap with in-context neurofeedback sessions, in which a "subject" model is asked to maximize (or minimize) a score derived from its activations, shown back to it at every turn. In our first experiment, the sentences are generated either by the subject or by a second "writer" model, while the score always derives from the activations of the subject. We test ten open-weight models from five families, with sizes from 9B to 32B. Some models move their own scores considerably, but no more than a writer of comparable competence moves them. Is this because the task requires no privileged access, or because the subject controls its activations while processing the sentences, whoever wrote them? To answer this, we prefill a fixed sentence as the subject's reply at every turn and ask it to increase or decrease the score. Three out of ten models show small but consistent control effects. We further ask whether models can control their activations to the point of fooling an activation monitor. We make models repeat sentences a monitor would flag, such as a deceptive answer for a deception monitor, and ask them to raise or lower the monitor's score. No model learns to lower it, but one model can raise it. In-context control of a model's own activations might not yet be a threat to activation monitors, but it exists in a few models and needs to be tracked as models evolve.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.