Padding to k: A Minimum Count in the Schema Makes Open-Weight Models Pad Their Extraction Lists
Extraction pipelines often ask language models to return JSON matching a schema that sets a minimum list length. This paper shows that when real documents contain fewer items than that minimum, open-weight models pad their lists---mostly with real text from the document that evades simple source checks. The authors preregister a design with 558 documents, eight GPU models, and paired contrasts, finding that an unenforced minimum raises over-counting from 34.5% to 55.4% and that common fixes fail or trade recall for precision. The work matters because schema-driven extraction is ubiquitous, and the padding is hard to catch without ground-truth answer lists.
The paper is a carefully preregistered, well-controlled empirical study that isolates a genuine and underappreciated failure mode: schema minimums act as targets rather than lower bounds. The main claim is strongly supported by paired contrasts and controls, and the release of code and run records is exemplary. However, some headline numbers rest on an item matcher whose invented-label precision is only 69%, and the two smallest models either fail to produce readable output or have wide uncertainty intervals, making the pooled conservative estimate somewhat misleading.\nThe practical guidance is useful but should be tempered by the admission that no tested fix fully eliminates padding.
The experimental design is the paper's greatest strength. The authors preregistered five confirmatory contrasts, used paired differences on the same document and model to cancel baseline extraction errors, and ran registered controls---maxItems, documents with $n \geq k$, and empty documents---that cleanly attribute the effect to the minimum itself rather than schema formatting in general. The finding that most padding is grounded in real document text, not outright invention, is well-supported by matcher breakdowns and a post-hoc source check.
The item matcher is a critical weak link: against 400 independently labelled entries, it is right on only 69% of its invented calls and 85% of its wrong-kind calls (Appendix B, Table 6), so nearly a third of entries labelled invented may be misclassified. The two smallest models behave as exceptions not because they resist padding, but because Qwen3.5-0.8B is often unreadable (36.6% unreadable outputs); post-hoc exclusion of this model raises the main effect from +0.62 to +0.80 entries, suggesting the registered pooled number understates the problem for functional models.\nSeveral exploratory tests---thinking, re-asking, and prose wording---are interesting but underpowered or confounded, and the prose conditions conflate count wording with instruction wording, weakening causal interpretation.
The evidence strongly supports the core claim. The shown-minimum versus no-count contrast (C2) is +0.62 entries with a tight 95% bootstrap interval [0.46, 0.78], and the maxItems control shows no equivalent lengthening ($-0.49$ entries), isolating the minimum as the cause. The comparison to prior work is fair: the authors correctly note that PhantomFill uses no-true-item inputs by construction, so every filled slot is invented, whereas their own design varies $k$ against $n$ on real documents.\nThe absence of closed API tests is an acknowledged limitation that tempers generalizability to production stacks.
Reproducibility is unusually good. The authors release code, answer lists, and run records, and the preregistration is on OSF. Models are pinned to Hugging Face revisions, served by vLLM v0.31.0 with XGrammar 0.2.7, and decoding is greedy (temperature 0, seed 0).\nWhat would block reproduction: the Bedrock hosted runs lack pinned revisions; two chat templates inject system prompts; and the item matcher, though frozen, was validated only by the authors' own 400-entry check. A replicator would also need the exact document windowing logic and the five counting-rule extensions that determined the answer lists.
Extraction pipelines ask language models for JSON that follows a schema, and the schema can set a minimum number of list entries. We ask what models do when the document holds fewer items than that minimum. Under a preregistered plan, we test eight open-weight models on 558 real documents from nine annotated datasets, with six more models on hosted services as a reference. Three findings stand out. First, models treat the count as a target: a minimum shown in the schema, but not enforced, raises the share of lists longer than the true count from 34.5% to 55.4%. Second, the padding is hard to catch, because most added entries are real text taken from the document; about half of them pass a check that each entry occurs in the source. Third, common fixes help only partly: a line saying fewer items are fine removes the effect on average but misses more true items, a count in prose ("up to k") also raises over-counting, an "N/A" option under an enforced minimum still leaves half the lists padded, and re-asking with the validation error makes padding far more common. We release the code, answer lists and run records at https://github.com/AbhishekBarali/padding-to-k
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.