RuleTrek: Benchmarking Agentic Rule Reasoning in Complex Scenarios
Most rule-reasoning benchmarks present scenarios as a few flat facts, making it easy to spot relevant rules. RuleTrek instead distributes scenario evidence across heterogeneous files and decomposes agentic reasoning into Rule Identification (RI), Rule Deduction (RD), and End-to-End (E2E) evaluation. Experiments on 11 frontier models show that even the best agent achieves only 62.47% RI and 60.43% RD, while E2E caps at 55.29%, suggesting that distributed, multi-file scenarios pose severe challenges for current agents.
The benchmark offers a valuable conceptual contribution by isolating rule identification from rule application in complex multi-file scenarios, but its small scale, opaque composition semantics, and heavy reliance on opaque LLM judges and constructors temper the strength and generalizability of its conclusions.
The three-way decomposition into RI, RD, and E2E is the paper's most valuable advance, letting the authors cleanly separate whether agents fail at finding rules or at using them. The human verification pipeline is extensive: initial pass rates averaged 86.1% across five quality criteria, and multi-round revision achieved 100% final pass. Cross-family judge validation further supports scoring stability, with submission-level Spearman correlations of 0.903–0.934 and model-ranking correlations of 0.955–0.991.
The benchmark comprises only 240 instances, which is small for robust comparison of 11 models across six domains. The construction pipeline relies entirely on GPT-6-Astra for extraction, sub-scenario generation, and composition, yet the equation $S_{t+1} = \mathcal{C}_{\Gamma_d}(S_t, s_{i_{t+1}})$ merely labels an opaque LLM-assisted operation rather than defining a reproducible algorithm. Automated scoring uses gpt-5.6-luna, and while human–judge correlations are reasonable ($\rho = 0.822$–$0.857$), they lag human–human agreement ($\rho = 0.933$–$0.951$) by a noticeable margin, suggesting residual judge bias.
The central claim is that distributed multi-file scenarios make rule reasoning harder than simplified factual statements. Table 2 shows low scores, but the authors never ablate against a single-file version of the same instances to isolate the effect of distribution. The comparison with prior work in Table 1 marks RuleTaker as using only 'a few factual statements,' which is accurate: RuleTaker inputs are simple synthetic sentences. However, the binary/partial ratings for 'Native Rules' and 'Dist. Scenario' are coarse and not individually substantiated with evidence from each benchmark paper. Trajectory case studies and the finding that rule-selection errors account for 54.9% of E2E failures strengthen the analysis, yet without a simplified-scenario control, the performance drop cannot be cleanly attributed to distributed information alone.
Evaluation uses the open-source pi CLI in isolated workspaces, with a fixed 30-minute timeout, a 200-step limit, and temperature 1, all clearly stated. However, full reproduction is blocked by heavy dependence on a specific LLM judge (gpt-5.6-luna) and constructor (GPT-6-Astra) whose exact weights and version history are not guaranteed to remain available. Appendix D shares detailed prompts, but the LLM-driven composition and judging steps are not deterministic, and no per-instance latency or cost figures are reported.
Large language models (LLMs) increasingly tackle complex real-world tasks in specialized domains. However, identifying and correctly applying domain-specific rules remains challenging when scenario information is distributed across multiple files. Existing rule-reasoning benchmarks typically represent scenarios using only a few factual statements, simplifying the identification and application of relevant rules. To address this gap, this paper introduces \textbf{RuleTrek}, a multi-domain benchmark for agentic rule reasoning in complex scenarios. Its rule-grounded construction pipeline extracts structured rule graphs, generates and incrementally composes rule-specific sub-scenarios while ensuring consistency and solvability. To systematically evaluate agentic rule reasoning in complex scenarios, RuleTrek assesses three capabilities in interactive workspaces: Rule Identification (RI), Rule Deduction (RD), and End-to-End Rule Reasoning (E2E). Experiments on 11 frontier LLMs show that the highest RI and RD scores are only 62.47\% and 60.43\%, respectively, while E2E scores range from 26.97\% to 55.29\%. These results demonstrate that current LLM agents face substantial challenges in rule reasoning over complex scenarios. We make our code and dataset publicly available at \href{https://github.com/yuan247/RuleTrek}{RuleTrek}.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.