What Lies Beyond Perception: A Partition Theory of the Limits of AI Evaluation and Oversight
Mahran asks what AI evaluation can certify when both examinee and evaluator have perceptual limits. The paper models each agent as a partition over stimuli—inputs it cannot distinguish—and proves a one-sided bound: a sharper examinee can always emulate a coarser evaluator's expected behavior and hide extra capability, while a coarser examinee cannot hide its blind spots under exhaustive probing. This yields predictions about sandbagging, monitor evasion, and weight noise, plus a worrying corollary: exact imitation of a weaker model is undetectable by any behavioural evaluator. The results suggest that behavioural evaluation alone can only establish lower bounds on discrimination, never upper bounds on capability.
The paper delivers an elegant, formally rigorous theory of evaluator limits rooted in partition structures. Its thesis is that "Judgement is bounded by perception" (Lemma 4, Section 2), and its main theorems—emulation from above (Theorem 7), detection from below (Theorem 9), and the discrimination asymmetry (Corollary 11)—are cleanly proved and geometrically intuitive. However, the model's power relies on exact partitions, deterministic behavior modulo seeds, and the assumption that an emulator knows its target perfectly; when these fail, as they do in real systems with noise, state, and bounded computation, the sharp undetectability claims soften to tendencies rather than certainties. The conclusion that "A safety case that argues from behavioural evaluation alone that a model lacks an ability must therefore assume that cooperation" (Abstract) is important but ultimately leans heavily on idealised exactness conditions that are rarely met in practice.
The echo test (Definition 6) is a novel and natural construction, and Proposition 10's completeness result—that it certifies everything any gradable test can, "much as a complete problem stands for its complexity class"—is a genuine conceptual contribution. The specificity control in Corollary 12 is striking and practically important: "a detector that flags concealment whenever the examinee fails the echo test therefore fires on Bh and never on Bc. Against these two arms it does worse than chance: its flags run opposite to the property it is meant to detect" (Section 3, Scope). The extensions to multiple observers via the join (Proposition 21) and the redundancy bound for controlled channels (Proposition 23) are elegant applications of the same partition machinery, giving concrete guidance on when added monitors help.
The undetectability results assume exact emulation, which requires the examinee to know both the evaluator's partition and the target behaviour precisely. Mahran acknowledges in Remark 8 that "The emulator must model the evaluator" and that "No behaviour is evidence of concealment on its own," yet the paper never quantifies how demanding this modelling is. In practice, slight mismatches due to sampling temperature, context limits, or scaffolding changes would make exact sandbagging detectable in principle, undermining the sweeping claim that "Exact imitation of a weaker model is undetectable by any behavioural evaluator" (Abstract). The noise analysis rests on Assumption 24, which idealises that "Each distinction in Π_B and each part of E_B, links included, has a breaking point along it: a scale below which it is present and from which on it is removed." Real weight noise degrades representations gradually and unpredictably, so Proposition 25's sharp jump at σ_e is an idealization. The paper also elides the operational gap between discrimination and competence: it notes that "discrimination is necessary for competence but not sufficient" (Corollary 13, Section 3.1), yet a system that copies its input attains the finest partition with zero competence, meaning the partition framework can mislead about capability if taken alone.
The comparison to prior work is generally fair and accurate. Barnett and Thiergart (2024) are correctly cited: as the fetched source confirms, evaluations "can establish lower bounds on AI capabilities" while facing "an inability to establish upper bounds on capabilities" (Barnett and Thiergart, Abstract and Sec. 2.1). Burns et al. (2024) on weak-to-strong generalization is accurately characterized as showing a substantial residual gap, and the contrast with Engels et al. (2025) as qualitative versus quantitative is apt. However, Mahran's empirical implications in Section 5 are post-hoc readings of existing studies rather than independent validations. The paper admits this limitation itself: "Section 5 reads published results through the model and offers predictions as tests, not as results" (Section 7), which weakens the empirical support for the framework.
The paper is purely theoretical, containing no code, experiments, or new datasets. Mathematical proofs are relegated to appendices A, B, and C, which are referenced but not fully reproduced in the main text; reproducing them requires checking those appendices. The empirical predictions (P1–P3) lack implementation details: "Assumption 24 idealises weight noise as removing distinctions and parts of the emission outright; real noise degrades them gradually" (Section 7). The paper also notes that "The model neither lists these changes nor says how any of them acts" (Section 5.1). Without code, hyperparameters, or benchmark specifications, an independent team would have to design the predicted experiments from scratch. The purely mathematical results are verifiable in principle, but the appendices must be inspected to confirm the proofs.
Replacing the judge can move a benchmark score, and models can be trained to hide capabilities. Both put the evaluator in question: what can it certify? We develop a partition theory of evaluation from one principle: judgement is bounded by perception. An agent's discrimination is the partition of its inputs into classes it cannot tell apart. Discrimination is necessary for capability but not sufficient, so bounds on it bear directly on whether evaluation can show that a model lacks an ability. We prove a one-sided bound. An examinee whose partition refines its evaluator's has the discrimination to pass every test the evaluator can grade, provided each input has an acceptable reply. We define the echo test, which accepts any reply the evaluator perceives as identical to the input, and show that it is complete for the tests the evaluator can grade. Passing therefore certifies at most a floor on discrimination, never a ceiling, and only a sharper evaluator can raise it. Used to detect concealment, echo-test failure points the wrong way: under exhaustive probing it flags every honestly weaker examinee and no concealer that presents as the evaluator's peer. Exact imitation of a weaker model is undetectable by any behavioural evaluator. We extend the bound to ensembles of monitors and to channels a model controls, explain published findings on judge disagreement and sandbagging, and derive testable predictions. A safety case that argues from behavioural evaluation alone that a model lacks an ability must assume the model's cooperation.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.