Maximising Transparency in Context Compaction Improves Retention of Facts and Commitments for Lightweight Coding Agents
Coding agents outgrow their context windows and compact older history into summaries, silently dropping commitments and facts that lightweight models cannot recover. This paper introduces transparent-compact, a deterministic compaction method that keeps all user and assistant prose verbatim, explicitly flags removed tool outputs and reasoning, and delegates recovery in a single lookup step. The core claim is that sparing lightweight models from guessing around lossy summaries improves retention of rules and facts without task-specific engineering. The evidence is strongest for local models like Qwen3.8-27B and weaker for frontier models, with a clear ceiling on how many verbatim rounds are possible before truncation.
The paper makes a crisp, well-motivated contribution: a deterministic compaction scheme with verbatim retention and explicit recovery, backed by two probe benchmarks and a detailed mechanism analysis. The main empirical argument—that transparent-compact avoids the blind spots of selection-based alternatives—holds up for lightweight models, though the statistical margin over pi-default on commitments is not significant ($p=0.15$, $\eta_p^2=0.08$) and the frontier-model advantage is reversed. The thoughtful failure-mode analysis tracing pi-default’s rewording errors and pi-vcc’s truncation of qualifying clauses is a genuine strength.
The method is not a panacea: it has a hard budget cap, higher token usage, and its benefits are smaller on frontier models that can better compensate for ambiguous summaries.
The experimental design is careful and transparent. Probes for commitments use real sessions with blind Claude-judge scoring ($\kappa=0.67$–1.0), while planted-fact probes stress-test repeated compaction with look-alike distractors. The mechanism discussion in Section 5.1 is exemplary: it shows that pi-default rewords rules into template categories, causing confident misapplication, while pi-vcc truncates the qualifying clauses that carry constraints. The claim that verbatim traces prevent schema-driven reconstruction is grounded in both the empirical results and an apt analogy to fuzzy-trace theory.
The method has a hard scalability ceiling: verbatim retention is capped at roughly ten compactions before conservative truncation applies, and testing beyond that is explicitly out of scope. Token costs are meaningfully higher—transparent-compact processes about $2.7\times$ more prompt tokens per probe turn than pi-vcc on Qwen—and the planted-fact sessions are intentionally retrieval-heavy, so real-world latency may differ. Statistical power is uneven: the commitments advantage over pi-default is not significant overall ($p=0.15$, $\eta_p^2=0.08$), and the Sol planted-fact results are an exploratory update excluded from the ANOVA. Finally, treating bounded pass rates as continuous outcomes makes the $F$ tests approximate, and Gemma probe timeouts are coded as failures.
For lightweight models the evidence clearly supports the claims: transparent-compact leads pi-default by 22–42 points on planted facts and outperforms pi-vcc on commitments across Qwen, Luna and Gemma. The comparison to Chen et al.’s governance-decay result is accurate—we verified that Chen reports violations rising from 0% to 30% after compaction and 38% when the constraint is dropped. However, the advantage shrinks on the frontier model, where Pi’s own summary scored 74.1% versus 69.1% for transparent-compact on commitments, and the Sol planted-fact comparison is exploratory and under-tested. The Lindenbauer et al. claim that masking matches summarisation on SWE-bench is directionally supported by their abstract and figure caption, though the exact solve rates quoted in Woolford (54.8% vs 53.8%) are from a table not fully reproduced in the fetched source.
The replication package, code and extension are publicly hosted, and the paper reports model versions, quantization levels, sampling parameters and Pi settings with unusual thoroughness. Scoring was blind to condition, with dual-judge agreement reported. The main barriers to reproduction are external dependencies: the judge and probe writers are Claude models, making the benchmark partially a black-box LLM-evaluation pipeline, and the API models (Luna, Sol) expose no temperature or seed control. In addition, pi-default summaries failed outright at stock settings on Qwen, requiring a $4\times$ larger generation budget than Pi’s default; this necessary deviation should be noted when comparing runs.
Long coding agent sessions outgrow the model's context window, so agents compact: older history is replaced by something shorter, usually a model-written summary. A summary selects and rewords, and the agent cannot see what was lost. Existing work has explored a range of compaction techniques, but comparatively little attention has been paid to their effectiveness with lightweight models, including those deployed locally or used for high-volume workloads. Here, we present transparent-compact, a simple deterministic method for the open-source Pi coding agent, designed specifically for use with such models and with the purpose of maximising the transparency of the agent's view of its own compacted context. It guarantees that user and assistant prose in its context is unchanged by a model or algorithm, and explicitly flags where compaction has stripped information. We compare it with Pi's default model-written summary compaction and with pi-vcc, a popular deterministic method with similar methods but without verbatim retention of prose. Our benchmarks run on four models of varying performance levels, with emphasis on Qwen3.8-27B: a lightweight model at the top end of what consumer hardware can serve locally. On sets probing commitments (rules the user set which exist only in the conversation) and recall of facts, transparent-compact matched or outperformed both alternatives. In particular, it showed an advantage in avoiding confident blunders, i.e. assertion of false information as opposed to recognizing that something was not recalled. Even with a frontier model, its advantage was present but smaller. We argue that transparent-compact spares lightweight models from reasoning or searching their way around what compaction lost, and results in improved recall and application of information as a result, without the need for task or context-specific structuring of the session history.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.