Function-Structured Reinforcement Learning with Executable Verifiers for Mathematical Reasoning
This work tackles algorithmic mathematical reasoning, where final-answer rewards fail to expose intermediate errors and successful code execution does not guarantee correctness. The proposed Function-Structured Graph Reinforcement Learning (FSG-RL) framework connects typed subproblem graphs to executable Python nodes, then trains a policy with supervised fine-tuning followed by Group Relative Policy Optimization (GRPO) using multi-verifier rewards and span-level credit assignment. The core claim is that verifying interface compliance, local functional behavior, and final consistency together pushes a 9B parameter model past a 27B API baseline, while also providing statistically significant gains over a strong SFT stage.
The paper presents a well-motivated staged SFT-to-RL pipeline that ties structured decomposition to executable verification. The multi-verifier reward design is principled, with explicit hyperparameter bounds ensuring that any format-valid correct answer outranks any wrong one. The empirical lift from SFT to GRPO is large and statistically significant under an exact McNemar test. However, the evaluation pool is modest, and the headline victory over a much larger API reference is weakened by mismatched inference protocols.
The paper’s strongest contribution is the tight coupling of graph-structured verifiers with span-level credit assignment. By decomposing the response into tagged nodes and assigning each token the advantage of its corresponding node, the method traces learning signals back to specific code spans. The reward hyperparameters are carefully justified: with $\eta=0.25$, $\lambda_Y=1.50$, and the remaining weights bounded so that $R_{\mathrm{wrong}} \leq 1.45 < 1.60 \leq R_{\mathrm{correct}}$, any correct answer outranks any wrong one, creating a clean total-reward ordering that anchors the RL objective.
The benchmark scale is small—only 866 GRPO training problems and 400 held-out evaluation problems—and teacher-supervised FC-GRPO is trained on merely 60 additional hard problems, making those gains suggestive rather than conclusive.
The comparison with Qwen3.8-27B API is not a controlled baseline: the API uses temperature 0 and a 180-second timeout, whereas the local policies use greedy decoding with a 2,048-token limit. The authors qualify the result with "under these settings," but the framing still invites an apples-to-oranges reading.
The staged results in Table 1 provide solid evidence that verification-guided RL advances correctness beyond format compliance. Execution success plateaus between SFT (86.00%) and GRPO (90.25%), whereas final-answer accuracy jumps from 43.25% to 67.50% and full success from 32.25% to 52.25%, supporting the claim that the gains extend beyond producing valid code. The paper is also careful to distinguish program–answer consistency ($B=1$) from true correctness ($Y=1$). However, the absence of head-to-head baseline runs on the curated benchmark—no competing methods such as ToRA, PAL, or MathCoder are actually evaluated—limits the empirical strength of the comparison.
The authors announce code availability and release annotated training records with public evaluation inputs. Yet exact reproduction is hampered because the private verifier specifications and target calls are withheld, and the paper omits several implementation details required to rebuild the harness from scratch, such as the exact LoRA rank and full prompt templates. Training hyperparameters are partially reported, but without the private checks the reward function cannot be fully replicated.
Algorithmic mathematical reasoning requires reliable decomposition, computation, and aggregation. Final-answer rewards provide limited guidance on intermediate errors, while successful execution does not guarantee mathematical correctness. This work proposes Function-Structured Graph Reinforcement Learning (FSG-RL), connecting subproblem graphs and Python implementations with multi-verifier feedback. The policy first learns to generate code from function graphs through supervised fine-tuning (SFT). Group Relative Policy Optimization (GRPO) then optimizes the policy using answer-gated rewards and span-level credit assignment. The framework also supports teacher supervision and structured memory. A benchmark curated from Grade School Math 8K (GSM8K), MathQA, MATH, and Omni-MATH pairs public function graphs with private verification specifications. Under a unified evaluation protocol, GRPO improves final-answer accuracy from 43.25% to 67.50% and full solution success from 32.25% to 52.25% over SFT. Continued reinforcement learning (RL) with teacher supervision yields additional gains. The gains extend beyond producing correctly formatted code, supporting verifier-guided reinforcement learning for mathematical reasoning. Code is available at https://github.com/ZihanLiummyycc/FSG-RL.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.