What and Where in Identifying and Quantifying Conventions in Humans and AI Agents
Cooperative success depends on implicit conventions, yet conventions are usually inferred only indirectly from score drops when partners are swapped. This paper introduces the convention gap $\Delta = \operatorname{mean}(P(\text{life loss})) - \text{actual life-loss rate}$ in the game Hanabi, using it to identify which situations carry implicit meaning for humans and for a hierarchy of off-belief learning agents, and to measure what removing those conventions costs. By comparing populations on a single exact literal baseline, the authors turn conventions from inferred causes of failure into directly measurable and causally testable objects.
The paper makes a solid methodological advance by replacing black-box cross-play scores with a fine-grained, interpretable convention gap. The finding that humans match level 2 on shared conventions while exceeding all levels on indirect reference is striking and well-supported by two independent corpora. However, the causal claims rest entirely on agent interventions; the human-specific conventions cannot be ablated, and the observational gap-to-importance link is explicitly broken by the rerouting result.
The decomposition of the convention gap by situation and turn is rigorous and careful. The replication across the hanab.live and AH2AC2 corpora ($r = 0.97$) strengthens the generalizability of the human profile, and the intervention design with frequency-matched random masks sets a high bar for causal claims about convention value. The induced taxonomy cleanly shows that higher OBL levels shift from hand-position splits to knowledge-state splits.
The paper's headline comparisons between humans and agents describe a narrow, highly trained subpopulation rather than typical human play. Although the authors acknowledge that the hanab.live corpus reflects a taught convention system, the framing still generalizes to "humans" broadly. The OBL hierarchy is limited to one training paradigm, so the claim that no agent level approaches the human gap on indirect reference may not extend to other architectures. Moreover, the causal intervention analysis is restricted to agents because no level enters indirect reference often enough to mask, leaving the most important human-specific convention causally untested.
The evidence supports the observational claims well: the shared taxonomy, held-out estimation halves, and pair-clustered bootstrap intervals show careful statistics. The comparison to related work is fair, positioning the convention gap as a more direct diagnostic than cross-play score or modular policy separation. However, the comparison to the prior Fukushima et al. (2026) study is mostly confirmatory, and the paper does not benchmark against other Hanabi interpretability methods that might locate conventions without an exact literal baseline.
Reproducibility is strong on several fronts. The metric code is public from the previous study and pinned to a specific commit; every game is replay-verified against ground-truth scores; all numbers were recomputed by a separate audit pipeline; and the bootstrap intervals reproduce exactly from recorded seeds. The main limitation is that the key intervention experiments rely primarily on one level-5 seed model, with partial replication on a second seed, and the exact model weights as well as the human corpora are needed for an independent reproduction.
Cooperative success rests in part on implicit communication. Partners converge on conventions: regularities that give an action meaning beyond its direct effect. Yet conventions are usually observed only indirectly: a score drop when partners are swapped shows that a convention existed, but not what it was or where it was used. In the card game Hanabi, the exact probability that a play fails can be computed from the hints received and the cards in view. Players who use conventions succeed more often than this prediction allows; the resulting margin is the convention gap. Off-belief learning provides a hierarchy of agents (levels 1-5) whose first level uses no conventions by construction and whose later levels use progressively more. For humans and these agents on one scale, we ask what the conventions are and where they are used. On what: in two independently collected corpora of human games, humans show a larger gap than every agent level. On the conventions both populations use, however, humans do no better than level 2. The human advantage is concentrated in a situation that no agent level learns to use: hints that point to a card other than the ones they touch. On where: conventions depend on what the player already knows about the card, not on how recently it was hinted, and they are used most in the mid-game; humans use them earlier and later in the game than the agents. In interventions on the agents, blocking a single high-gap situation costs little, because meaning reroutes to other situations, but blocking several together costs far more than the sum of the single costs. Thus, the convention gap turns conventions from an inferred cause of score loss into a measurable, comparable, and causally testable object.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.