Do Your Own Research: Learning to Forecast by Learning to Search
The paper tackles the frozen-context bottleneck in outcome-based reinforcement learning for forecasting: prior work either pastes fixed research snippets into the prompt before training or lets a frozen model search only at test time, so the policy is never rewarded for gathering better evidence. The authors close this gap by building a multi-turn agentic harness from 2,100+ resolved Polymarket questions, filtering every search and page-read to pre-cutoff information, and training Qwen3.5-35B-A3B with single-epoch GRPO under the Brier-score reward $r=1-(p-y)^2$. The core advance is that the model learns when and what to search for as a temporally grounded skill, rather than optimizing a frozen context window. This matters because it moves forecasting from prompt engineering to genuinely learned research discipline, and the trained open model reportedly beats several frontier systems at roughly 5% of the inference cost.
The paper makes a conceptually important contribution by integrating live, temporally constrained retrieval into the RL training loop for forecasting, and the environment design is thoughtful. However, the central empirical claim, that the trained policy beats Claude Opus 4.5, rests on a soft-Brier gap of only $0.002$ ($0.254$ vs. $0.256$) on $n=265$ questions, and no confidence intervals, p-values, or variance estimates are provided. The evaluation therefore risks overstating a margin that may well be noise. Additionally, the entire validity guarantee against temporal leakage depends on Claude Haiku as a per-page judge, introducing a heavy proprietary dependency. As the authors note in Section 4: "the trained policy also finishes ahead of every frontier model tested at evidence-based forecasting, including Claude Opus 4.5 (soft-Brier 0.254 vs. 0.256" and in Section 2: "Haiku alone scored perfectly among five judges benchmarked on labeled keep/drop cases."
The strongest part of the work is the carefully constructed agentic environment. The three-layer leak filter (domain blocklist, heuristic date parser, and LLM judge) is validated on labeled cases, and the authors release the full harness, dataset, and per-rollout traces. Behaviorally, the training signal produces genuine discipline rather than mere overfitting: expected calibration error drops $31\%$ ($0.185 \to 0.128$) and search attempts fall from $3.8$ to $2.25$ per rollout, indicating the policy learns to stop querying after unproductive results and reason from surviving evidence. The paper also correctly benchmarks in an identical harness and withholds the market price, preventing the trivial price-copying baseline that collapses all models to the crowd. In the authors' words: "Search attempts also fall from 3.8 to 2.25 per rollout: the trained policy re-queries less after empty or filtered results and reasons more from the surviving evidence" and "With market-price tools available, every policy converges toward the crowd (soft-Brier band 0.186--0.215 vs. crowd 0.189) and none reliably beats it."
First, statistical rigor is lacking for the headline comparison. A $0.002$ soft-Brier difference on $n=265$ questions is negligible without error bars; on the pre-declared uncertain subset ($n=104$) the gap widens to $0.007$, but this is still small and could easily be sampling variation. Second, the temporal-leak safety guarantee rests almost entirely on a proprietary model (Claude Haiku via Amazon Bedrock) whose behavior is not reproducible across versions or providers. The filter benchmark uses only five alternative judges on a labeled set, which is thin evidence for "perfect" generalization. Third, the paper relies on future-dated models (Claude Opus 4.5, Gemini 3.6 Flash) and a 2026 publication date, making the frontier-model comparisons impossible to verify today. Finally, the training dataset is modest ($\sim$2,100 questions), and the paper does not explore whether gains transfer to out-of-domain topics beyond politics and crypto/finance. The authors state: "The trained policy also finishes ahead of every frontier model tested at evidence-based forecasting, including Claude Opus 4.5 (soft-Brier 0.254 vs. 0.256" and "Haiku alone scored perfectly among five judges benchmarked on labeled keep/drop cases."
The internal evidence for calibration gains is robust across multiple ablations (market-visible, search-off, evidence-based), and the behavioral traces in Appendix B show the trained policy reasoning about source reliability. Comparisons to prior work are fair: Turtel et al. (2025) and Jeen et al. (2026) are accurately described as freezing context before training, and Murphy (2026) as test-time-only retrieval. However, the identical-harness claim is weakened by the absence of statistical testing; the authors compare multiple models on a small held-out set and declare ranking without variance estimates. The cost comparison ($\sim$5% of inference cost) is a back-of-the-envelope calculation based on token counts and August 2026 list prices, which is illustrative but fragile to serving optimizations and API discounts. The authors correctly note that giving models access to the market price erases differentiation: "Price access makes every model, from the untrained 35B to Opus 4.5, converge on the crowd without reliably beating it, so a market-visible comparison measures price-copying, not forecasting." As confirmed in the Turtel et al. source, their pipeline uses "the Exa.ai API to retrieve news headlines from before the sampled prediction date," matching the paper's characterization of frozen context.
Partially strong but with notable barriers. On the positive side, the authors open-source the environment on Prime Intellect's hub, publish per-rollout records, and use open training libraries (verifiers, prime-rl). On the negative side, (1) the leak filter requires Claude Haiku via Amazon Bedrock, a proprietary API dependency; (2) web search is served through an Amazon Bedrock AgentCore gateway, which is not standard or easily replicated; (3) training hyperparameters are sparse beyond "$33$ steps $\times$ $64$ questions $\times$ $8$ rollouts"; (4) the Qwen3.5-35B-A3B model may not be widely accessible; and (5) the future-dated frontier models (Claude Opus 4.5, Gemini 3.6 Flash) make the central benchmark unverifiable for contemporary readers. The authors acknowledge in Appendix C: "The full training stack is open source (verifiers, prime-rl) and runs on self-managed GPUs; we used the hosted service as a cost and convenience choice," yet the critical search and leak-filter infrastructure remains cloud-locked.
Outcome-based reinforcement learning can train language models to forecast real-world events, but prior forecasting work either freezes research context before training or deploys agentic research only at test time, so the skill of gathering evidence is never shaped by the reward. We introduce an agentic forecasting environment, dataset, and harness built from 2,100+ resolved Polymarket questions; the agent acquires its own context at rollout time (web search, page reading, and financial time series, all restricted by layered leak filtering to information published before each question's cutoff), and we train Qwen3.5-35B-A3B (3B active parameters) on it with single-epoch GRPO under a Brier-score reward. Training changes how the agent interacts with information: calibration improves 30-40%, and search attempts fall from 3.8 to 2.25 per rollout as evidence discipline is learned. Evaluated in an identical harness against four frontier models, the trained policy also finishes ahead of every frontier model tested at evidence-based forecasting, including Claude Opus 4.5 (soft-Brier 0.254 vs. 0.256, n=265), at about 5% of the inference cost, and its margin is widest on the hardest questions, the ones the crowd itself had not decided. We release the environment, dataset, and per-rollout records as a reusable harness for temporal forecasting agents.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.