From Knowledge Access to Source Learning: Developing Source-Specific Competence
Most LLM agents treat repeated use of the same external source as a sequence of independent retrieval events. This paper reframes the deeper problem as source learning: progressively building a persistent, reusable source model that captures how a source’s knowledge is structured, interpreted, and applied. The proposed SourceLearn system combines self-directed study with task-guided refinement, and the authors report large gains across document, code, API, and interactive benchmarks.
SourceLearn is a conceptually clean and empirically strong contribution that reframes repeated source use as cumulative learning rather than repeated access. The paper articulates a crisp distinction between source-level competence and both per-task retrieval and generic agent memory, backs it with a multi-benchmark, multi-backend evaluation, and provides probing analyses of what the learned source model actually encodes. However, the heavy reliance on opaque, future-dated LLM backends, the lack of runtime and cost reporting, and a task-time context budget that favors the proposed method weaken the practical immediacy of the results. As a research direction, source learning is well motivated; as a reproducible system, it currently depends on resources and model versions that most readers cannot verify.
The conceptual framing is the paper’s strongest asset. The authors clearly separate source access, source representation, and task experience, and they design two complementary learning mechanisms that are well supported by ablations: "Omitting either Self-Directed or Task-Guided Source Learning lowers accuracy, and the initial source model $M_0$ is the lowest, showing that the two learning mechanisms are complementary" (Section 5.3). The empirical coverage is also notable: five benchmarks spanning documents, code, APIs, and interactive environments, evaluated with three different backends, and accompanied by analyses showing that the model shifts toward explicit rules and higher task-relevant coverage (Section 5.4). The grounding audit in Appendix B.3 adds rigor by explicitly measuring supported vs. unsupported units, contradicting the concern that the source model is just a hallucinated summary.
A major practical concern is reproducibility and transience: the experiments rely on future commercial models—GPT-5.6-Luna, DeepSeek-V4.1-Flash—whose weights and behavior are unavailable to the community, and on datasets dated 2026 (SWE-QA at ACL 2026). This makes independent verification impossible and raises questions about whether the reported gains will transfer to currently accessible models. Methodologically, SourceLearn receives both standard retrieved evidence and its own activated source model at task time, giving it a strict superset of the context available to retrieval-only baselines such as Hybrid RAG. While Appendix B.2 attempts a matched-budget comparison on four sources, the main claims in Table 1 are not controlled for context length, and the ablation study in Section 5.3 covers only two sources per benchmark with a single backend. Finally, the construction and learning pipelines involve dozens of unprompted LLM calls (e.g., $K=20$ self-study actions plus task-guided refinement) without any reported latency or cost; scalability to large repositories or streaming sources remains unexplored. The authors acknowledge scope limitations—"we focus on persistent, authoritative, and relatively stable sources with repeated same-source tasks" (Section 6)—but the method's dependence on expensive, iterative LLM rewriting is an unsized barrier.
The evidence broadly supports the core claim that jointly modeling the source and accessing it at task time outperforms access-only or memory-only alternatives. The comparisons to RAPTOR and HippoRAG 2 are fair insofar as those methods are static source representations without a learning stage; our own fetch of the RAPTOR paper confirms its recursive summarization tree and HippoRAG’s knowledge-graph architecture, matching the authors' descriptions. However, SourceLearn’s "best in 13 of 15 settings" headline conflates benchmark-level macro-averages across highly heterogeneous sources (e.g., 12 stories in NarrativeQA, 6 code repositories in SWE-QA), potentially masking source-level variance. Moreover, the QA evaluation relies on an LLM judge, and on AppWorld all methods share the same underlying agent scaffold, which may homogenize differences attributable to source access alone. The failure-guided and cross-task pathways are carefully ablated, but the ablation scale is small, and the matched-context-budget control is only reported for four sources in Appendix B.2.
The authors release code and prompts (GitHub: https://github.com/luchengfu6/SourceLearn), and Appendix C reproduces the full construction, inspection, and consolidation prompts, which is exemplary. Key hyperparameters are also stated: $K=20$ study actions, $B=24$k tokens for source-model activation, and three fixed 30/70 splits. Yet independent reproduction is blocked by the exclusive use of commercial, future-dated API models—GPT-5.6-Luna at "medium reasoning effort," gpt-oss-120b served locally on six A100s, and DeepSeek-V4.1-Flash—none of which are publicly available as of this review. No wall-clock time, API cost, or total token consumption is reported, so it is impossible to assess whether source learning is practical for large or frequently updated sources. The reliance on a single embedding model (text-embedding-3-large) and undisclosed backend-specific reasoning-effort tiers further narrows reproducibility to exact API snapshots that may not persist.
Large language model (LLM) agents increasingly rely on persistent external sources to solve sequences of knowledge-intensive tasks. Existing methods improve how source content is accessed and organized, while agent-memory systems preserve reusable knowledge from prior interactions, but repeated use of the same source is still largely treated as repeated access rather than an opportunity to progressively improve understanding of that source. We study source learning: developing reusable source-specific competence over a persistent authoritative source. We represent this competence with a persistent source model that captures reusable understanding of the source, including how its knowledge is structured, interpreted, and applied. To construct and progressively refine such models, we propose SourceLearn, which combines two complementary learning mechanisms. Self-Directed Source Learning identifies what remains incompletely understood and adaptively revisits the source, while Task-Guided Source Learning uses downstream experience to reveal local representational gaps and recurring needs in how source knowledge should be organized. In both cases, learning signals determine what should be reconsidered, while persistent updates are reconstructed from the authoritative source. Across five benchmarks and three LLM backends, SourceLearn achieves the best performance in 13 of 15 settings, with gains of up to 22.6 points over Hybrid RAG and substantial overall improvements over static source representations and experience-based memory baselines.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.