Learning Perception to Action Rules through Reinforcement Learning for Theory-of-Mind Reasoning
Theory-of-Mind reasoning requires connecting an agent's perception to their belief and then to their action, yet large language models often fail to compose these inferences even when each part is known. This paper proposes rule-based reinforcement learning: using Group Relative Policy Optimization to train models to emit concise if–then rules that bridge perception, belief, and action before answering. The authors show that this approach outperforms free-form chain-of-thought reasoning and supervised fine-tuning across four ToM benchmarks, while using roughly half the tokens, and that rules from larger models improve downstream GPT-4o-mini accuracy.
The paper presents a clear empirical case that structured, rule-conditioned GRPO improves ToM benchmark scores over both SFT and unconstrained chain-of-thought. The token-efficiency and cross-model transfer results are encouraging. However, the work does not engage with recent evidence that benchmarks such as Hi-ToM contain severe shortcuts—Zhong et al. (2026) showed that Hi-ToM can be solved by simple heuristics like tracking the object location when the outermost agent leaves the scene—leaving open whether the gains reflect genuine mental-state reasoning or more reliable exploitation of spurious correlations. The authors' own caveat that their reward does not verify semantic rule correctness further tempers causal claims.
The core empirical finding is well supported: across four datasets and scales from 0.5B to 14B, GRPOr raises average accuracy by 7.2 percentage points over free-form GRPO thinking ($p<0.01$) and cuts trace length from 52.1 tokens to 25.2. The result replicates on the Llama family and survives an ablation without SFT warm-up, suggesting robustness. The short-thinking control experiment is a particularly useful addition, demonstrating that the gain is not merely brevity: on HiToM, GRPOr still beats a length-matched free-form baseline (60.0% vs. 46.1% for Qwen2.5-7B).
The largest flaw is the reward function. Because $r_{answer}$ depends only on final answer correctness and $r_{format}$ only on JSON validity, the model can receive a high total reward while generating semantically vacuous rules. Examples in Table 11 confirm this: Qwen2.5-0.5B produces the nonsensical rule "If Noah enters the patio, he will enter the bucket" yet answers correctly. The authors acknowledge this explicitly, but the limitation undermines the central claim that rules causally mediate reasoning.
A second concern is benchmark validity. Zhong et al. (2026) recently demonstrated that Hi-ToM is highly shortcut-prone, yet the paper uses it as a core evaluation without shortcut audit or discussion; the strongest gains on HiToM (up to +16.7 pp) may therefore reflect shortcut learning rather than improved ToM. The SFT degradation—rule generation before answer hurts relative to answer-only SFT—further hints that the GPT-5-mini-generated reference rules may not be optimal scaffolds for smaller models.
Within the paper, the evidence supports the comparison against internal baselines: GRPOr > GRPOt > SFTa > SFTr. However, the comparison to related work is selective. The paper cites Lu et al. (2025) but does not directly address their finding that SFT alone is competitive and that RL on small models can induce "reasoning collapse" with truncated, shallow outputs; the authors' small models do produce short rules, but the qualitative analysis in Appendix F shows some of them are incoherent, partially aligning with Lu et al.'s observation. The GPT-4o-mini transfer experiment (Figure 2) is a nice touch, but the negative transfer from ≤1.5B rules complicates the claim that rules encode universally transferable task knowledge—transfer only works once the generator is large enough, which suggests scale-dependent signal rather than clean logical structure.
Reproducibility is partially blocked. Although hyperparameters are reported in detail—batch size 8, learning rate $5\times 10^{-6}$, KL coefficient 0.001, eight completions at temperature 0.8, and a clearly specified reward $r = r_{\mathrm{format}} + r_{\mathrm{answer}}$—the authors state "Our code will be publicly open upon acceptance." The ground-truth rules are generated with gpt-5-mini-2025-08-07, which may not be publicly accessible, making exact replication of the SFTr and GRPOr training pipelines difficult. Data splits are documented, but final test-set results are averaged over only three random seeds {3, 4, 42} and model selection uses development-set accuracy, a common practice but one that should be reported with variance estimates. Compute types (H100, A100, A40) are disclosed but total GPU-hours are not.
In Theory-of-Mind (ToM) reasoning, large language models may correctly infer the agent's mental state but fail to use that inference when predicting or judging the agent's subsequent actions. In this work, we investigate whether explicitly connecting perception, belief, and action with rules improves ToM reasoning. We generate concise \textit{if--then} rules that express relations between perception to action and use Group Relative Policy Optimization (GRPO) to train Qwen2.5 and Llama3 models across four ToM benchmarks. GRPO with rule generation outperforms prompting, supervised fine-tuning, and think reasoning. Moreover, rules generated by larger models ($\geq$3B) provide transferable knowledge that improves downstream performance of GPT-4o-mini. These findings suggest that explicit rules help models bridge the reasoning gap in ToM and support knowledge transfer across models.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.