ColaBench: Evaluating Memory-Guided Knowledge Acquisition in Personal Agents
ColaBench tackles the problem of evaluating how personal agents use long-term interaction history to guide the acquisition and application of external knowledge. The core idea is a traceable benchmark built on an Evidence Graph $\mathcal{G}=(V,E)$ that links personal facts to their disclosures and to supporting documents, paired with MemKB, a retrieval method that connects state summaries and memory-document links. This matters because current benchmarks test memory recall and retrieval in isolation, leaving open whether agents can jointly recover historical conditions and apply newly retrieved rules to a specific user's situation.
The paper presents a well-motivated, methodologically transparent benchmark and a competent retrieval method, though its empirical scope is narrow and its reliance on synthetic generation and automated judge models leaves room for skepticism about real-world validity. The staged, multi-agent construction with an Evidence Graph is a genuine contribution, as is the ablation showing that memory-document links alone improve Accuracy by 10.81 percentage points. As stated in Section 1, "A remaining gap concerns the joint evaluation of how disclosed personal conditions guide the acquisition and application of knowledge not yet encountered in the interaction," and the authors address this by building a benchmark whose "Evidence Graph connects personal Facts to the public messages that disclose them, records document exposure during interaction, and links subsequent answer requirements to their personal and knowledge sources" (Section 3). MemKB's 86.25% Accuracy against Mem0+Hybrid's 79.94% demonstrates a meaningful gain, but the small profile count and lack of human evaluation calibration temper the strength of the claims.
The Evidence Graph construction protocol is the strongest element. By recording Fact disclosures, document exposure, and answer requirements across stages, the authors create a structurally explicit audit trail expressed through $\mathcal{G}=(V,E)$. The ablation study is also convincing: "Removing state summaries and document links reduces Accuracy by 6.31 and 10.81 percentage points, respectively" (Section 6.5, Table 6), which isolates the value of organized memory beyond generic retrieval. The task-level gain pattern---"The Accuracy gain is 4.96 percentage points on single-Fact tasks and 9.35 points on multi-Fact tasks" (Section 6.2)---further supports the thesis that combining dispersed personal conditions is the hard problem this benchmark captures.
The benchmark's ecological validity is limited by its small, synthetic footprint. Only ten synthetic Profiles and 349 Items are produced, with ablations run on just three Profiles. The authors acknowledge this: "Generalization to new users and the reliability of automated evaluation require independent data and human calibration" (Section 7).
Furthermore, Read Recall for MemKB is only 45.56% (Table 3), meaning the correct reference document was read on fewer than half the tasks, yet Accuracy exceeds 86%. This discrepancy suggests many correct answers may rest on search snippets or parametric knowledge rather than verified document reading. Error analysis also reveals memory-level hallucinations: "The user confirms taking a prescription pain medicine, but the derived state assigns it to the opioid class. The answer then treats opioid-related guidance as personally applicable" (Section 6.6), which raises questions about whether the state extraction pipeline is reliable enough to support the benchmark's claims.
The comparison to related work is generally fair. The authors correctly distinguish LoCoMo and LongMemEval, which evaluate "memory through cross-session question answering, temporal reasoning, and state updates," from ColaBench's focus on "history-guided knowledge retrieval and rule application" (Section 2). Baseline comparisons are well-controlled: Mem0+Hybrid uses the same underlying Mem0 memories and Hybrid retrieval as MemKB, so gains are attributable to the added state summaries and document links. However, the No Memory+Hybrid baseline at 6.59% Accuracy is artificially low because "the history-removal test checks whether the task can be completed without the public interaction history" (Section 3.5) and only history-dependent items are retained, making it a sanity check rather than a competitive baseline.
Appendix B.2 provides detailed hyperparameters, model versions, and retrieval budgets, and the frozen 13,033-document corpus with ten fixed Profiles makes the benchmark reproducible in principle. However, the paper states that "Full construction, memory, and evaluation prompts and input templates will be provided on GitHub" (Appendix D.1), implying that code and templates were not yet released at submission. All experiments rely on API-dependent models---DeepSeek v4.1 Flash for answering and GLM-5.3 Flash for evaluation---so exact reproduction depends on API stability and opaque system-prompt behavior that future model updates may alter.
Personal agents need to draw on long-term interaction history to retrieve and apply external knowledge suited to a user's current request. Even an answer grounded in relevant documents can be unsuitable when it overlooks the user's circumstances. This creates two connected challenges: using historical conditions to identify the knowledge needed for a new request, and applying that knowledge to the user's situation. Existing benchmarks advance memory recall, personalized retrieval, and memory-guided action, while the joint evaluation of these two challenges remains underexplored. We introduce ColaBench, a synthetic benchmark in health and daily living, with 349 tasks across 10 Profiles and a knowledge base of 13,033 documents. Its Evidence Graph connects personal conditions to their disclosures across public interaction sessions, records prior document exposure, and links answer requirements to personal and newly acquired document evidence. We also introduce MemKB, which uses state summaries to inform queries and rule application, and memory-document links to retrieve knowledge associated with the user's history. MemKB achieves 86.25% Accuracy with an average of 4.07 interaction rounds per task. Task-level analysis finds larger gains on tasks integrating information across multiple historical Facts; error analysis further identifies unsupported personal assumptions and rule misapplication as obstacles to using retrieved evidence. ColaBench provides traceable tasks and an evaluation framework for studying how memory guides knowledge acquisition and rule application.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.