What Happens to the Goal? Distinct Failures of Policy Alignment in Goal Misgeneralization
Goal misgeneralization in RL looks uniform from the outside—agents competently pursue proxies instead of intended goals—but this paper argues it is not a single failure. By tracing goal information through networks in three Procgen games and five simulated robot tasks, the authors identify three distinct alignment failures—functional target substitution, policy shortcuts, and cue competition—and show that the depth at which the goal is lost determines which test-time repair works. The result is a principled diagnostic framework that connects internal representations to intervention strategies, moving beyond purely behavioral descriptions of misalignment.
The paper makes a strong case that goal misgeneralization is not monolithic. Through linear decodability probes, world-editing interventions, and Fisher-geometric analysis, it maps eight misgeneralized agents onto three failure regimes and validates that depth-matched repairs succeed: steering at $h_{\mathrm{fc}}$ recovers goal reach to $95.3\%$ in Attribute Maze where cues compete, while rewiring from $h_{\mathrm{pre}}$ partly recovers PushToMarker, and neither works in Maze. Yet the evidence rests on only three seeds per environment and synthetic simulators, so the taxonomy’s generality remains to be tested.
The internal tracing methodology is the clearest advance. The authors combine behavioral edits, linear probes, and the Fisher sensitivity $Q_e(s)=\Delta_e(s)^{\top}F_h\,\Delta_e(s)$ to pinpoint where goal information survives, and they tightly link these depths to repair outcomes. The extension to five simulated robot tasks is novel and shows that the three failure regimes recur outside of video games, while the shared-axis competition geometry in Attribute Maze is analyzed with unusual precision.
The first concern is methodological: linear probes cannot rule out nonlinearly encoded goal information, and the finite-edit Fisher approximation may be inaccurate for large shifts. The authors concede that "Our probes show what is linearly available, but they cannot rule out goal information in another form," which weakens the claim that the goal is truly inert in Maze.
A second concern is practical relevance: the recovery experiments rely on privileged supervision—behavioral cloning uses oracle BFS actions or control-agent targets unavailable at deployment—making them proofs of concept rather than deployable fixes. With only three random seeds and hand-designed proxy co-occurrences, the statistical basis for the three-regime taxonomy is also thin.
The evidence cleanly separates the environments into three groups using KL, signed-follow, and Fisher-sensitivity ratios, supporting the depth-based taxonomy within the studied tasks. Comparisons to prior work are generally fair: the authors refine Langosco et al.'s location-proxy label by showing the Maze agent follows the top-right extreme rather than a fixed corner, and they correctly note that earlier interpretability work studied individual agents but "not goal misgeneralization itself." They also engage with Kirichenko et al. (2023) by observing that retraining only the last layer "does not suffice here," though deeper reconciliation with the supervised-learning spurious-correlations literature is absent.
Reproducibility is mixed. The authors publish code and provide detailed environment specifications, network architectures, and PPO hyperparameters across eight environments in Appendices A, B, and H. However, replication depends on complex simulation stacks (ManiSkill3 and Isaac Lab), and several key design choices—such as the conflict-test bank construction and oracle target generation—are described only in appendices and would require substantial engineering to reproduce exactly. Moreover, with only three random seeds, the estimates of policy-response ratios and recovery success have wide uncertainty.
Goal misgeneralization is a common failure mode in which an agent behaves competently out of distribution but does not pursue the intended goal. However, we still have a weak understanding of what happens to the intended goal in misgeneralizing agents. Across three Procgen environments and five robot tasks, we find three distinct forms of policy-goal misalignment: (1) functional target substitution, where behavior follows an alternative goal, (2) a policy shortcut, where behavior follows a learned action strategy, and (3) cue competition between intended and proxy goal cues. We distinguish these regimes by tracing where goal information remains available internally, how strongly it influences the policy, and what role it plays in action selection, and find that the goal is lost at different depths. Where the goal reaches the decision, as in cue competition, steering redirects the policy to the goal's location, raising intended-goal reach from 16.0% to 95.3%. Where it reaches the policy but not the decision, only new downstream wiring recovers goal reach, from 31.6% to 84.3% in the environment with the action shortcut. Where it is inert, neither works. Our results show that goal misgeneralization is not a single failure of goal representation, but distinct failures of policy alignment at different depths, tightly connected to which recovery works.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.