The Missing Primitive: Diagnosing and Repairing Mathematical Reasoning in Large Language Models
LLMs can hit high accuracy on hard math problems while lacking genuine structural insight. This paper proposes the Mathematical Primitive---a concise, non-procedural statement of the key idea that makes a solution possible---and introduces Prim, a benchmark that disentangles structural understanding from procedural execution across Discovery, Generation, Digestion, and Execution. The authors show that independent Discovery is the dominant bottleneck and introduce Absorb, a primitive-privileged self-distillation framework that selectively transfers structural guidance into the student model without requiring primitives at inference time.
The paper makes a strong conceptual case that final-answer accuracy conflates structural discovery with downstream execution. Its four-dimensional decomposition is elegant, and the empirical finding that discovery-limited failures dominate is robust across a diverse model suite. Absorb's bounded override mechanism is a principled response to regressions seen in standard SFT and OPSD, yielding consistent average gains of 2.42--4.48 points. Nevertheless, the work is handicapped by a minuscule evaluation set---just 182 problems---and heavy reliance on hypothetical future models (e.g., GPT-5.4-High) for annotation, teaching, and judging, which makes independent verification impossible.
The diagnostic framework is the paper's strongest asset. The four-dimensional taxonomy cleanly separates structural understanding from execution, and the data reveal dramatic asymmetries: for instance, Qwen3.6-27B achieves 24.73% Discovery but 92.31% Digestion, and providing a gold primitive boosts Execution by up to 29.67 percentage points over Generation. The contrast between Teacher Plan and Teacher Primitive is a particularly convincing empirical argument that structural insight is not reducible to procedural scaffolding.
First, statistical power and coverage are limited. With only 182 problems drawn from a single high-difficulty source, the benchmark is too small to support reliable ranking of frontier models or strong claims about lineage-level differences; the absence of confidence intervals or variance estimates exacerbates this.
Second, the entire pipeline---primitive generation, teacher guidance, and automated judging---depends on GPT-5.4-High, a model that does not currently exist, making the work impossible to replicate independently. The primitive scoring rubric permits only a single gold primitive per problem, which the authors conservatively admit may under-credit valid alternative conceptual routes. Finally, Absorb's clamping bound and top-K support sets are introduced heuristically; while ablations are relegated to Appendix E, the main text offers little intuition for why this specific functional form is optimal.
The evidence largely supports the paper's core claims. The dominance of discovery-limited failures is established through a clear decomposition and rescue experiments showing that primitives convey far more than task decomposition. Comparisons to SFT and OPSD baselines are fair within the paper's setup, though the omission of RLVR-based methods is notable. The related work section adequately situates the primitive concept among prior process-supervision approaches, but the paper could engage more deeply with literature distinguishing insight from routine problem solving.
Reproducibility is the weakest aspect of the paper. The authors commit to releasing code and model weights, but every stage of the pipeline is gated behind proprietary future models such as GPT-5.4-High that are inaccessible to the community. Training details are partially specified, yet the exact judge prompts depend on unreleased model APIs. The tiny benchmark size means that reported point differences could be highly sensitive to a handful of examples; without error bars, readers cannot assess statistical stability. Independent reproduction would require reconstructing the entire closed annotation pipeline from scratch.
While Large Language Models (LLMs) have demonstrated striking capabilities on frontier mathematical problems, it remains unclear whether they possess the structural mathematical understanding underlying their solutions. In this paper, we take a first step toward systematically studying mathematical understanding in LLMs, from diagnosing its distinct capabilities to leveraging these findings to improve post-training. First, we introduce the notion of Mathematical Primitive to probe structural mathematical understanding and propose \hlei{}, a novel benchmark that evaluates mathematical reasoning along four distinct dimensions: Discovery, Generation, Digestion, and Execution. Second, our systematic diagnosis shows that solution accuracy masks distinct capability profiles, primitives unlock substantial latent execution capacity, and Discovery is the dominant bottleneck in mathematical reasoning. Our post-training analysis further shows that discovery-limited failures are particularly amenable to repair. Finally, building on these findings, we introduce \abs{}, a primitive-privileged self-distillation framework that selectively transfers primitive-guided reasoning into the student model. Extensive experiments demonstrate that \abs{} consistently improves mathematical reasoning over baselines across model scales and challenging benchmarks.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.