Distilling Directional Verification
Knowledge distillation risks propagating the reversal curse when a teacher can recall a relation forward but not backward. This paper proposes directional label distillation: frozen teachers score candidate answers in the direction they know, and the highest-scoring candidate becomes the student's training target. On parent-child facts from Wikidata, this yields 13-15 point gains in student open-ended accuracy and shows that verification can provide accurate supervision where generation fails.
The paper tackles a well-documented failure mode: 'If a model is trained on a sentence of the form A is B, it will not automatically generalize to the reverse direction B is A' (Berglund et al., 2024). Its core insight is that 'A teacher that cannot generate an answer may nevertheless recognize it' (Section 1), and that distilling this verification signal produces more accurate pseudo-labels than generation-based baselines. The evidence is rigorous within a deliberately narrow domain.
The conceptual contribution is clean and the internal controls are thorough. The authors isolate label quality from student architecture by showing that, after inventory matching, students reproduce their selected labels for "more than 97% of queries on average over seeds, despite large differences in label accuracy" (Section 4.1). The direction effect is also large and stable: "Known-direction labels exceed summed DC labels by 12.47 to 16.73 points under all six rules, whereas the rules differ by at most 2.33 points within a score form" (Section 4.2). Controls for name priors, surname cues, and forward-edge withholding are carefully executed.
The experiments are restricted to a single, highly structured relation. The authors candidly admit that "We study one relation over a fixed name inventory, which lets us control candidate sets, forward exposure, and evaluation exactly; relations with open-ended answers will need other ways to propose candidates" (Section 7). This means the practical recipe—enumerate candidates, score each in the known direction, pick the best—does not obviously transfer to open-ended QA.
A second limitation is that most reported student gains are measured on the exact queries used to produce the training labels. The authors state that "Evaluating students on the queries whose labels they learn isolates the contribution of label quality to the answers they give" (Section 7), but this design leaves open whether the student generalizes to unseen parent queries or benefits from memorization. The notability-based rule for choosing the scoring direction is also post-hoc and not yet usable without access to ground-truth facts.
The quantitative evidence supports the central claims. The main student comparison finds that "On the screened cohort’s exposure-free queries, known-direction labels improve open accuracy by 14.77 points over summed-token DC labels, the most accurate reverse labels without a tuned coefficient" (Section 4.1), and the authors add that "These gains hold in every paired seed and under whole-answer evaluation" (Section 4.1). Baselines include prior-corrected reverse scores, tuned reverse transfer, teacher generation, and teacherless heuristics, all of which trail the proposed method. The positioning against prior work is fair: sequence-level and best-of-N distillation are discussed, and the authors correctly note that any-order masked diffusion training alone leaves reverse accuracy near 2% without reverse labels (Table 3b), undercutting claims that architecture alone cures reversal failures.
Reproducibility is relatively strong. The code repository is public, the corpus is drawn from Wikidata, and training details are extensive: "AdamW uses learning rate $10^{-4}$, batch size 16, gradient clipping 1.0, and no schedule or warmup" (Appendix A). That said, the exact training cohort depends on a screen performed by Qwen3-8B that retains only 56.3% of screened facts, and replicating the precise candidate pools requires specific model versions and prompt templates. GPU type varies across seeds (A100 vs RTX A6000), though comparisons are matched within runs. The main barrier to independent reproduction is the multi-teacher scoring pipeline and the bespoke screen, not missing hyperparameters.
Knowledge distillation aims to transfer the factual knowledge of large language models to smaller models for efficient deployment. Yet a teacher may recall a relation in one direction while failing to generate the answer in the reverse direction. Distillation from its generated answers can therefore propagate this directional limitation to the student. The same teacher can nevertheless recognize such an answer by scoring the relation in the direction it knows. We introduce directional label distillation, in which frozen teachers score candidate answers in that known direction and the best-scoring candidate becomes the student's training target. On facts about parents and their children, known-direction scoring yields more accurate labels than scoring the requested direction, even after tuned corrections for name priors. With prior-corrected scores, the better direction depends on the facts rather than the template, and reverses on mined facts whose notable entity is the parent rather than the child. With the evaluated children's forward facts withheld, students trained on known-direction labels improve open-ended accuracy on their trained queries by 13 to 15 points over students trained on prior-corrected reverse labels. After generated answers are matched to a fixed name list by lexical similarity, students reproduce nearly all selected labels. Their accuracy largely follows label quality. The label advantage holds on unscreened queries and when candidates are retrieved without inserting correct answers. Our findings show that directional verification mitigates the transfer of errors from teacher-generated answers to students by providing more accurate training targets. Code is available at https://github.com/js-lee-AI/directional-verification.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.