Beyond Retrieval: Evidence Access and Event Aggregation in Agent Memory
This paper tackles questions about an agent's history that require aggregating multiple events, a task that pure retrieval cannot solve. It proposes Trajectory Memory (TM) for deferred, source-preserving evidence selection, and TallyMem for executable event aggregation via a typed query language over a reusable ledger. The core insight is that aggregation load—the number of events an answer must combine—predicts failure better than raw history length, with strong controlled evidence (+11.0 to +13.8 points) supporting the mechanism, though full-benchmark gains are more modest.
The paper makes a valuable conceptual distinction between evidence access and event aggregation, backing it with rigorous controlled experiments and honest reporting of limitations. The controlled probe is methodologically strong, but the full AMA-Bench gains are small and the main bottleneck shifts from miscounting to misspecification. The work is solid and well-executed, if somewhat incremental in its ultimate practical impact.
The controlled aggregation probe is the paper's strongest contribution: questions and event sets $S^\star$ are generated deterministically from raw logs before any model sees them, and execution is exact and linear-time. The diagnosis that aggregation load $|S^\star(q)|$, not just trajectory length, determines difficulty is well-supported across five readers and two domains. The separation of access (TM) from aggregation (TallyMem) is conceptually clean, and the cross-dataset MINTEval evaluation on 99 held-out repositories provides genuine out-of-sample evidence.
Full-benchmark gains on AMA-Bench are modest: +1.3 points with routing (69.8% to 71.1%) and not significant without it (p=0.21). The router itself is a coarse threshold on estimated load, and gains are concentrated on the 212 high-load questions where it applies. More troublingly, TM alone trails long-context reading on MINTEval (35.2% vs. 39.2%), contradicting the claim that deferred selection preserves necessary evidence and requiring a domain-specific ledger adaptation to recover. Event specification errors are the largest residual category (51 of 148 analysed errors), meaning the deterministic executor is only as good as the model-generated plan. The paper also does not state whether code or frozen evidence will be publicly released.
The controlled probe strongly validates the aggregation mechanism when event sets are known, and comparisons to AMA-Agent and external memory baselines on AMA-Bench are fair and within matched deployments. However, the MINTEval evaluation required a commit-specific ledger adaptation, which limits claims of easy cross-domain transfer. The paper honestly notes that TM outperforms previously reported AMA-Bench methods but underperforms the long-context baseline on MINTEval without TallyMem's computation path, suggesting the access mechanism is not universally sufficient.
The authors pre-registered the MINTEval test, froze the principal configuration before final evaluation, and provide extensive appendices with hyperparameters, prompts, and deterministic execution semantics. An independently written matcher reproduces all stored execution results. However, there is no explicit statement releasing code, data, or frozen evidence artifacts. The router uses different rules on AMA-Bench ($\gamma_{\mathrm{load}}$, load > 15) and MINTEval ($\gamma_{\mathrm{cov}}$, coverage-based), which was fixed on development repositories, complicating exact reproduction of the routed full-benchmark numbers. Multiple deployment changes (diagnostic fp8 vs. full bf16) also mean absolute accuracies are not interchangeable across tables.
Questions about an agent's history often require aggregating events, which retrieval alone does not resolve. We combine source-preserving evidence access with executable aggregation. Trajectory Memory (TM) retains the original trajectory and constructs a source-addressable reading view at question time. TallyMem augments it with a reusable event ledger, model-generated event specifications and deterministic execution. With a commit-specific adaptation on 776 questions from 99 held-out MINTEval repositories, TallyMem raises Qwen3-32B exact match from 35.2% to 49.0%, a 13.8-point gain over TM; gains span 12.8-13.9 points across three readers. A controlled probe with event sets parsed independently from raw logs shows an 11.0-point gain, from 82.8% to 93.8%, without routing. On full AMA-Bench, the routed system yields a smaller 1.3-point gain, from 69.8% to 71.1%. These results support executable aggregation on tasks that require combining events, with benefits dependent on the task and event specification, the largest residual system-error category in our diagnostic analysis.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.