RISED: RubrIcs for agentic multi-environment Selection and sElf-Distillation
Joint training of a single LLM agent across diverse interactive environments is difficult because coarse scalar rewards hide cross-environment behavioral relationships and collapse to zero within-group contrast when all rollouts succeed or fail. RISED tackles this by having an LLM judge tag rollouts with predefined textual rubrics, producing behavioral profiles that are reused for both online prompt-group selection and on-policy self-distillation. The premise is that rubrics supply rich, comparable supervision even when reward signals are uninformative.
RISED is a sensible marriage of rubric extraction, online batch selection, and self-distillation for multi-environment agentic RL. The experiments show consistent but modest gains—typically 0.6–1.2 percentage points over the strongest variance-based baseline—and the rubric vocabulary demands heavy external model support (Gemini-2.5-Pro, Claude-Opus-4.6, Qwen3-32B). The method adds meaningful compute and engineering overhead, which the authors candidly flag.
The core empirical finding is compelling: rubric-profile cosine similarity is a better predictor of cross-environment transfer than gradient similarity ($\rho=+0.81$ vs $+0.28$ on all environments in Table 1). The analysis also shows that "Rubric similarity also consistently identifies which environment benefits most from training across all four environments: it identifies the most-improved environment at all 19 evaluated checkpoints, compared with 8 checkpoints for gradient similarity." Reusing the same tags for selection, teacher conditioning, and negative steering is an elegant design, and the component ablation confirms that "Combining selection and distillation yields the strongest overall performance, while their individual contributions vary across environments."
The absolute gains are slim: on Qwen2.5-3B, RISED improves overall pass rate by only 1.2 points over the best baseline (0.542 vs 0.530), and on Qwen3-4B by 0.6 points (0.585 vs 0.579). The authors note that "Relative to the strongest baseline on each backbone, the gains are 1.2 and 0.6 points." Moreover, the rubric pipeline is expensive and brittle. Vocabulary construction relies on frontier models (Gemini-2.5-Pro and Claude-Opus-4.6), and online tagging uses a dedicated 8-GPU Qwen3-32B server for "one judge inference call per rollout." The curation involves clustering, merging, and skill discovery steps that are hard to replicate exactly. The selection objective ($\operatorname{score}(g\mid\mathcal{S})$ in Eq. 1) is motivated by batch coverage, but no ablation isolates relevance vs. redundancy, leaving it unclear which term drives the benefit.
The evidence supports the key claims, though comparisons are somewhat narrow. Variance-based baselines are fair representatives of online selection, and the authors explain why recent alternatives such as CROPI are excluded: "Its reliance on validation data for selection places it outside our comparison protocol, which uses neither validation-guided scoring nor per-group gradient computation." The All-Rubric SD baseline shows that naively conditioning on all rubrics underperforms RISED's selective use of positive rubrics, and the component ablation validates that selection and distillation both contribute. Still, the strongest baseline on each backbone is competitive, as highlighted by the fact that "RISED achieves the highest mean overall pass rate on both Qwen2.5-3B and Qwen3-4B, improving over GRPO-64 by 5.8 and 6.9 percentage points, respectively"—a comparison anchored partly on a weaker straw-man rather than the strongest rival.
Reproduction is theoretically possible but practically taxing. The paper provides prompts (Appendix A) and hyperparameters (Appendix B), yet the 82-rubric vocabulary emerges from a multi-stage pipeline—warm-up rollouts, free-form extraction by Gemini-2.5-Pro, clustering with all-MiniLM-L6-v2, merging, skill discovery, and augmentation with Claude-Opus-4.6 rollouts—described in Appendix A.1 as requiring substantial curation. An exact quote is: "Because the initial vocabulary is extracted from rollouts of a weak warm-up policy, it provides substantially better coverage of failure modes than of successful behaviours." Additionally, "The vocabulary is constructed from warm-up and teacher rollouts across four environments." An independent group would need to replay this semi-automatic pipeline, host a Qwen3-32B judge, and synchronize asynchronous tagging with rollout generation. The authors note tagging overhead is "reported in App. C.6," but the need for a second 8-GPU server during training is itself a significant barrier.
Training a single LLM agent jointly across diverse interactive environments has attracted increasing attention as a route to generalist agents. Existing curriculum and data-selection strategies often allocate training at the environment level or prioritize local reward-based signals, without explicitly considering relationships between current rollouts across environments for prompt-group selection. Meanwhile, as environments are learned at different rates, all-failure and all-success rollout groups can coexist within a batch, leaving those data without group-relative reward signals. Both challenges highlight limitations of relying solely on scalar rewards in multi-environment RL: they provide limited information about cross-environment relationships and no within-group reward contrast when rewards are identical. This motivates richer textual feedback, such as rubrics describing rollout behaviours, to guide learning. Beyond rubrics' usage as reward, we repurpose rubrics to guide both online data selection and policy supervision. An LLM judge tags each rollout using a predefined rubric vocabulary shared across environments. The resulting profiles guide the selection of data that aligns with the overall behavioural composition of the mixed-environment batch while limiting overlap with already-selected data. Available positive rubrics (describing desired behaviours) provide privileged context for an on-policy self-distillation teacher, supplying additional token-level supervision, while negative rubrics (describing undesired behaviours) guide subsequent rollout generation away from recurring failure modes. Together, these components form RISED. Across model backbones, RISED achieves the highest mean pass rate across environments and ranks first or second in every individual environment. Rubric-based analysis of RISED can further characterize the behavioural changes accompanying these gains.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.