Can an Agent Debug a Web Corpus? Agentic Auditing of Slovak FineWeb2
Multilingual web-corpus pipelines such as FineWeb2 apply one recipe across thousands of languages, leaving individual languages’ failure modes unexamined. This paper proposes treating an LLM as a one-time debugger rather than a classifier: a budgeted, tool-using agent explores the Slovak slice of FineWeb2 through a read-only API and compiles each finding into a cheap executable test validated by a native speaker. The work matters because it replaces a linear, manual audit with a reusable test suite that scans 26.5M documents in under a second without a GPU.
The paper is admirably self-critical. It shows that an LLM agent can discover severe, quantified defects—most dramatically a 5.32% machine-generated content farm of 1,408,437 documents—in a heavily filtered 26.5M-document corpus. Yet the authors directly acknowledge that the best static baseline validates more selectors per dollar, so the agent’s real contribution is exact prevalence measurement, explanatory characterization, and reusable tests rather than raw cost-efficient detection. The framing as a 'debugger' succeeds as a workflow contribution even if the core detection task is within reach of simple heuristics.
The experimental design is transparent and the release unusually complete: code, prompts, trajectories, blinded validation data, and the test suite are all shipped. The three-stage workflow cleanly separates expensive LLM discovery from cheap deterministic deployment, yielding what the authors call 'pytest for web corpora: a community contributes tests, and any future crawl or pipeline revision is checked against them for the cost of a scan over metadata the pipeline already computed.' The blinded native-speaker validation is also well executed: 'We drew 358 documents for blinded annotation by a native speaker ... shuffled and shown without provenance, URL or metadata.'
The agentic approach is not cost-competitive on a per-finding basis. The best baseline, LowConf, validated four selectors across three groups for $1.03, whereas each agent run cost $2.21–$2.84 and validated at most three. 'The best static sampling baseline validates more selectors than any agent run, at under half the cost.' More importantly, every validated phenomenon was also reachable by static strategies: the Czech slices sit entirely inside the LID-disagreement slice, the adult-spam family lies entirely inside the profanity count $n\_bad\_words \geq 3$, and the content farm is expected in a 300-document uniform sample. Validation is also thin—only 255 documents labeled by a single native speaker with wide intervals—and the adjudication process relied on model-proposed verdicts that were later corrected. Scope is limited to Slovak and three runs of one model, leaving cross-lingual robustness unproven.
The evidence solidly supports the existence of the reported defects, but the comparison to static baselines weakens any claim that the agent is a more effective detector. The authors candidly note that their matched-budget comparison changes two things at once: 'The agent chooses what to read, and it also has tools the baselines lack ... It therefore measures the whole agentic setup rather than adaptivity alone.' The value of the agent lies in what it ships with each finding: 'every agent selector also arrives with an exact prevalence measured on the whole corpus. That characterisation is work a curator would otherwise do by hand.' The related-work discussion is fair, correctly positioning the paper downstream of manual audits and upstream of pure LLM classification.
Reproducibility is strong on the artefact side: the repository includes the adapter, eleven tools, prompts, trajectories, annotation packages, adjudication logs, and the test suite. The authors specify the model (claude-sonnet-5), budget (300 inspections, 250 turns), and exact costs. However, exact trajectory replication is limited because the runs use 'default sampling settings, and we set no temperature,' making the agent non-deterministic. The validation protocol also had post-hoc corrections: 'Two of those verdicts were later corrected by the stated rule, at the author’s direction and without a new deliberation, after labels lost in our tabulation were restored.' Independent reproduction is therefore gated mainly by LLM variance and the requirement for native-speaker annotation.
Multilingual pipelines such as FineWeb2 apply one recipe across a thousand languages, leaving the failure modes of individual languages unanticipated. We use an LLM once, as a corpus debugger rather than a classifier: a tool-using agent explores the corpus through a read-only, budgeted API and compiles each finding into an executable test, which a native speaker then validates. On the 26.5M-document Slovak portion of FineWeb2, three runs costing \$2.21-\$2.84 each found seven findings over four phenomena. The largest is a machine-generated content farm of 1,408,437 documents (5.32%) that passed the pipeline's filters, 86.6% of it passing even stricter thresholds. Blinded labels support 6 of the 7 findings (defect rates .53-1.00 against .08 in a random control) and do not support one. The compiled suite scans the full corpus in 0.7 s of wall-clock time without a GPU, against \$12k-\$90k per LLM classification pass. The best static sampling baseline validates more selectors than any agent run, at under half the cost. The agent contributes exact prevalence, a precision estimate checked against sampled documents, an account of why the pipeline missed each slice, and a reusable test. We release the framework, Slovak test suite, validation data and agent trajectories.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.