Mostly Reading: What Open-Book Scores Measure in Document-Level LoRA Adapters
Document-level LoRA adapters are pitched as modular knowledge units for retrieval and merging, yet standard open-book evaluation cannot tell whether an adapter has memorized a document or merely learned to read it well. This paper introduces a cloze probe that withholds the source text and finds that open-book scores are almost entirely generic reading behavior: a random adapter recovers 82% of a specialist's open-book advantage but only 2% of its cloze gain. The findings argue that adapter-bank evaluation needs a closed-book probe alongside open-book scores, and the authors' candid disclosure of four silent errors is a model of experimental hygiene.
The paper presents a clean, important methodological critique: open-book QA systematically overestimates what document-level LoRA adapters actually store in their weights. The evidence is well-structured across training-text ablations, transfer tests, random-adapter baselines, and merge evaluations, all pointing to the same conclusion. However, every main result is tied to a single model (Llama-3.1-8B-Instruct), one LoRA rank, and a narrow corpus of 150 Wikipedia articles, and the per-document evaluation is sparse enough that some key contrasts lack statistical power.
The strongest evidence comes from the training-text conditions in Table 1, where the no-context adapter 'comes second of the five conditions at 0.773' in open-book $F_1$ but falls 0.132 below the base model on cloze, while only adapters that saw the full text show a cloze gain. This cleanly separates format adaptation from knowledge storage. The random-adapter decomposition and merge grid replicate the pattern consistently, and the weight-geometry analysis corroborates that update distances track the text an adapter read rather than the questions it answered. The four silent errors section is unusually honest and valuable for the community.
Generalization is severely limited: all primary results rest on one model, one rank, and 150 encyclopedic articles that may have appeared in pre-training, where 'our base model already reaches 0.81 of the specialists' cloze recall.' The per-document evaluation is tiny—three factual items plus one true/false item—so the 30-query retrieval test is underpowered to the point that 'over these 30 documents even the own adapter is not distinguishable from the base model on cloze (+0.085).' The true/false item also behaves anomalously: specialists score below the base model open-book, and because it is included in open-book but excluded from cloze, the two probes are not perfectly comparable. The authors acknowledge these limits, but the restricted scope still weakens broad claims about adapter banking.
The cloze probe is a strong methodological contribution: by rewriting held-out facts into a declarative surface form no adapter was trained on, it isolates parametric knowledge in a way that standard QA does not. The divergence between open-book and cloze scores is large, consistent, and replicated across conditions. Prior work is cited fairly for parametric RAG. However, the two probes are not strictly comparable because 'the true/false item does not behave like the others' and is excluded from cloze while included in open-book $F_1$, and the claim about cross terms costing 'about ten points' on vision tasks is hard to verify from the abstract of Zheng et al. (2025) alone.
Training hyperparameters, software versions, and merge formulas are documented with unusual care, and the four silent errors section is exemplary: 'Each of these errors produced a clean result we were prepared to report, and none stopped a run or raised an error.' Still, no code release or trained adapter weights are mentioned, and the paper shows that small changes in truncation window, repetition penalty, merge scaling, or evaluation precision materially alter conclusions. An independent reproduction would therefore require exact duplication of these fragile settings, and the authors' own experience suggests that偏离 from their exact configuration risks silent, wrong results.
Single-document LoRA adapters have been proposed as units of knowledge to be merged and retrieved; yet open-book scoring, with the document in the prompt, can credit a document's own adapter (its specialist) for reading that document as readily as for holding it. Training one adapter for each of 150 Wikipedia articles on Llama-3.1-8B-Instruct, we find that open-book scores mostly measure a reading behaviour every adapter shares: a random adapter trained on another document recovers 82 % of the specialist's open-book advantage over the base model but only 2 % of its advantage on a cloze probe that asks for the same document's facts without providing the source text, a share we cannot distinguish from zero. Content does reach the weights, since the specialists beat the base model in cloze recall by 0.094, and the geometry of their updates follows the documents they read; yet, in a test of 30 queries, adapters chosen by weight-space proximity or text embedding are not distinguishable from random ones on either probe, although that test cannot exclude an advantage of about four fifths of what a query's own adapter gains over a random one. At 16 documents or more, correctly scaled merges recover 87 % to 89 % of the specialists' open-book advantage but only 12 % to 16 % of their cloze advantage, with intervals that run from below zero to at most 32 %, against 94 % for joint training.We also report four silent errors, each of which produced a clean, wrong result.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.