Evaluating Agentic Capabilities Before Post-Training: An End-to-End Approach
Base models are hard to evaluate on agentic tasks because standard benchmarks assume instruction-following chat interfaces. ASAP adapts existing agent scaffolds to pretrained base models by treating execution as text continuation, using a Markdown+XML format, a short reference trajectory prefix, and minimal failure recovery. The framework yields discriminative end-to-end scores across models and checkpoints, and exposes execution trajectories that enable fine-grained behavioral diagnosis before post-training.
ASAP is a practical and novel contribution that fills a real gap in pretraining evaluation. The three-component design is lightweight and well-motivated, and the trajectory-level behavioral decomposition across checkpoints is genuinely useful. However, the work is partly undermined by a small convenience sample of models, limited causal claims about pretraining dynamics—notably conflating trajectory data with context-length scaling in the critical stage2$\to$stage3 transition—and a descriptive rather than causal base-vs-post-trained comparison that uses different scaffolds by design.
The core design insight is strong: treating base models as continuation engines rather than broken chat agents is the right framing. The ablations empirically support the Markdown+XML format and prefix bootstrapping, while the trajectory-level behavioral analysis demonstrates value that static proxies cannot provide. As the authors note, "Markdown offers a natural continuation space for free-form reasoning and textual context, while XML provides explicit local structure for tool actions and observations" (Appendix A.4.1). Another key result is that "ASAP consistently separates the midpoint and final checkpoint within each stage... whereas the conventional benchmarks scores changes by only +0.25, +0.09, and $-$0.13 points" (Section 5.1.1).
The reliance on reference prefixes from instruction-tuned models introduces a contamination and elicitation bias that the paper downplays. While prefixes are verified to avoid golden patches, they still provide task navigation and repository structure from successful trajectories, so the evaluated model is not solving cold-start agent execution. The stage-wise analysis of the internal MM model conflates the introduction of trajectory-like data with a simultaneous jump from $8$K to $32$K context length, making causal attribution to agentic behavior unclear. Moreover, the comparison of base and post-trained models uses different scaffolds, which the authors admit is "descriptive rather than a scaffold-controlled estimate of the effect of post-training" (Section 5.2). Similarly, the largest capability transition is attributed to stage3, where "trajectory-like data is introduced together with native $32$K long-context training" (Section 5.1.1), yet the design cannot disentangle these two changes.
The evidence supports the central claim that ASAP discriminates base-model agentic capability. The six-model cross-section shows consistent separation across benchmarks and scaffolds, and format/prefix ablations are thorough. Comparisons to related work are fair: APTBench is accurately characterized as static proxy tasks rather than end-to-end execution, which the fetched source confirms—"converts real-world agent tasks and successful trajectories into multiple-choice or text completion questions tailored for base models" (Qin et al., 2025, Abstract). However, the study lacks head-to-head comparisons against likelihood-based proxies on identical checkpoints and tasks, so the marginal utility of end-to-end evaluation over cheaper signals remains asserted rather than quantified. As the authors state, existing proxies "evaluate fixed examples or reference trajectories rather than the behavior generated by the evaluated model itself" (Section 2), but they do not quantify how much this theoretical distinction matters in practice.
Reproduction would be difficult. While Table 4 lists decoding hyperparameters, the paper does not clearly commit to releasing code or scaffold patches, and the reference prefixes are drawn from trajectories generated by proprietary or unreleased models including Kimi K2.5, GLM-5.2, and MiniMax M2.7. The internal MM checkpoints are proprietary, so the key pretraining-stage findings cannot be independently verified. Exact recovery-hint wording is omitted, and the scaffold-patching descriptions remain high-level, leaving ambiguity about what was changed. The authors note they "implement our method as lightweight patches to existing agent scaffolds" (Section 3.4) and that prefixes come from "a reference trajectory produced by an instruction-tuned model" (Section 3.2), but the precise diffs, prompts, and trajectory sources are not provided.
Agentic capabilities are increasingly important for modern LLMs, yet remain difficult to evaluate during pretraining. Existing approaches typically rely on static trajectory-derived tasks or likelihood-based proxies, which are lightweight but provide limited visibility into how base models behave in interactive environments. We introduce ASAP (Agent Scaffold Adaptation for Pretraining Evaluation), a lightweight framework for eliciting and evaluating agentic behavior from pretrained base models on end-to-end tasks. Rather than converting base models into chat-style agents, ASAP preserves the completion interface and adapts existing agent scaffolds through structured trajectory continuation, short reference prefixes, and minimal recovery mechanisms. Across software-engineering and deep-research benchmarks, ASAP yields discriminative end-to-end performance across base models and pretraining checkpoints. Beyond aggregate scores, ASAP exposes model-generated execution trajectories, enabling analysis of exploration, editing, testing, recovery, and task completion across training stages. These trajectories provide a behavioral view of how agent-relevant capabilities change during pretraining and how execution patterns differ after post-training. Our results suggest that end-to-end agent evaluation can complement conventional pretraining benchmarks by providing a practical diagnostic of interactive model behavior.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.