AFCS: Atomic-Fact-Conditioned Synthesis for Factual Knowledge Pretraining
Most synthetic pretraining pipelines generate samples by conditioning on an entire source passage or paragraph and asking a model to rewrite or paraphrase it. The paper identifies a systematic failure mode of this approach: fine-grained facts—such as secondary entities, list and table entries, and precise temporal attributes—are frequently omitted because the generator implicitly selects which facts to preserve. Atomic-Fact-Conditioned Synthesis (AFCS) fixes this by first extracting source-supported atomic facts, then conditioning each synthetic training sample on an explicit retained fact and its local context. Continued-pretraining experiments on 7B and 32B models show consistent gains on factual QA benchmarks, especially for middle- and long-tail knowledge, and granularity ablations isolate synthesis granularity as a causal lever for factual recall beyond surface-form diversity.
AFCS is a carefully motivated and empirically well-supported data-construction pipeline. The core claim—that explicit, fact-level conditioning improves coverage of fine-grained knowledge—is backed by corpus-level diagnostics (85.1% source-fact coverage versus 73.0% for WRAP and 77.0% for Nemotron-CC) and by downstream model performance after matched-budget continued pretraining. The 32B scale replication and detailed ablations over conditioning granularity strengthen the causal argument. However, the paper does not bridge the gap between corpus coverage and parametric acquisition, and the mid-training integration results are mixed, with five of twelve general tasks declining when AFCS is substituted into a 100B-token mix.
The granularity ablation is the strongest element of the paper. Under an identical 20B-token continued-pretraining budget, fact-level AFCS consistently outperforms chunk-, paragraph-, and entity-level conditioning, improving over chunk-level synthesis by +1.42 on SimpleQA and +2.37 on PopQA. The structure-aware extractor also demonstrates high precision (94.1%) and recall (87.6%), and it recovers list- and table-embedded facts far better than a generic extractor. A popularity-stratified PopQA analysis provides a compelling signal that AFCS delivers its largest gains on rare and middle-frequency entities.
Despite its strengths, the paper conflates corpus coverage with learned knowledge more than is justified. The authors explicitly acknowledge in the Limitations that covered facts are not automatically stored or retrieved by the trained model, yet the main narrative often treats higher coverage and higher downstream accuracy as mutually reinforcing evidence for the same mechanism. Second, the 100B-token mid-training integration (Section 5.7) highlights a distributional trade-off: while factual QA benchmarks improve, five of the twelve general benchmarks decrease, including OpenBookQA and GSM8K, raising the possibility that AFCS skews the pretraining mixture toward factual surface forms at some cost to reasoning and general knowledge. Third, the evaluation relies heavily on LLM judges—Qwen2.5-7B-Instruct for benchmark scoring and gpt-oss-120b for coverage judging—introducing unquantified model-dependent bias into both corpus and model evaluation.
The comparisons to WRAP and Nemotron-CC are fair and tightly controlled: all methods use the same FineWiki source corpus, the same base model, and the same 20B-token continued-pretraining budget. The authors accurately characterize these baselines as source-conditioned pipelines that leave fact selection implicit. Adapted versions of EntiGraph and Mecklenburg et al.'s fact-to-QA pipeline are evaluated under the same budget, providing useful context, though the EntiGraph adaptation underperforms, likely because it targets a different setting. The evidence supports the core claim that finer-grained conditioning improves factual recall, but the strong dependence on LLM judges and the absence of a human-evaluation arm for downstream accuracy weaken the evidentiary chain.
Experimental details are unusually thorough. The authors report exact optimizer hyperparameters ($\beta_1=0.9, \beta_2=0.95$, peak learning rate $2.07 \times 10^{-5}$ for 7B), batch sizes, sequence lengths, the generator checkpoint (Qwen3-30B-A3B served with vLLM), decoding temperatures, and deduplication rules. The extraction, verification, and synthesis prompts are reproduced verbatim in Appendix D, and the coverage-judge prompt is reproduced in Appendix E, which significantly aids replication. The main barrier to independent reproduction is the lack of released code or the final 142M-sample corpus. Replicating the structure-aware extraction pipeline over 6.7M articles would require rebuilding a complex parser for tables, lists, infoboxes, and prose, and subtle filtering choices—such as removing degenerate or condition-leaking generations—are described but not extensively validated.
Synthetic data is increasingly used in language-model pretraining. Existing knowledge-corpus synthesis pipelines, including WRAP and Nemotron-CC, usually take a source passage or document segment as input and generate rewritten or QA-style samples from that text. This can improve surface form and prompt diversity, but it does not explicitly require fine-grained source facts -- especially secondary entities, list/table entries, and precise temporal attributes -- to appear in the synthesized corpus. We propose AFCS (Atomic-Fact-Conditioned Synthesis), a pipeline that first extracts source-supported atomic facts and then uses each retained fact to drive generation. AFCS uses structure-aware extraction over the document tree, source-grounded verification, local context expansion, and multi-format generation. In continued-pretraining experiments with OLMo-3-7B and a 32B base on the same FineWiki source corpus, AFCS outperforms WRAP and Nemotron-CC on SimpleQA, SimpleQA Verified, PopQA, and HotpotQA. Granularity ablations show that fact-level synthesis improves over chunk-level, paragraph-level, and entity-level synthesis; and a 100B-token mid-training integration improves factual benchmarks and raises the 12-task general benchmark average by +1.09.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.