OP-Print: Tracing Membership Imprints in On-Policy Self-Distillation

The strongest part of the paper is its careful characterization of where the membership signal lives. Figure 2 demonstrates that teacher-student divergence is organized by token difficulty rather than absolute position: divergence rises substantially in higher-uncertainty regions and occupies only a restricted region of the reliability map. This observation is well exploited methodologically—the gray-box score $S_{\mathrm{gray}}(x)$ directly targets the KL gap between privileged and standard conditioning, and the black-box score $S_{\mathrm{black}}(x)$ uses restart-text stability as a principled surrogate. Table 1 shows consistent gains over likelihood-based baselines, with OP-Print-G-Max reaching AUCs up to 0.611 on Qwen3-4B where the best baseline stagnates near 0.52.

The experimental scope is worryingly narrow for general claims about on-policy self-distillation privacy. All results derive from just 500 member and 500 non-member math problems drawn from OpenThoughts, trained for only three epochs on models no larger than 8B parameters. This small scale raises doubts about whether the leakage pattern persists in larger, more realistic post-training regimes. The authors also adopt a fixed-teacher setup—which diverges from many on-policy self-distillation methods that use an online or evolving teacher—meaning the leakage surface may differ in practice. A second major concern is query cost: the black-box attack requires sampling $M$ full generations, truncating at $|P|$ prefix ratios, and restarting $R$ times each, making it far more expensive than simple loss-based MIAs. The ablation in Figure 5(c) confirms diminishing returns as restarts increase, suggesting the attack is inherently costly.

The paper's central empirical claim—that generic MIAs remain near random guessing while OP-Print succeeds—is supported by the data in Table 1, where Min-K% Prob and R\'enyi variants hover around AUC 0.49-0.52. This aligns with broader findings in the literature: Duan et al. showed that for LLM pre-training, "the performance across most MIAs and target domains is near-random." However, the baseline selection is weak. The authors pit OP-Print against pre-training detection heuristics like Min-K% Prob rather than stronger reference-based or calibrated attacks that exploit the exact reference solutions available in their threat model. A fairer comparison would include a reference-model-calibrated loss or a direct likelihood ratio using the initial teacher checkpoint, which is already available in their gray-box setting. Without this, the gap may be inflated by baseline mismatch rather than a fundamentally new attack surface.

Training hyperparameters are documented in Appendix A (Table 3), including learning rate $5 \times 10^{-6}$, bfloat16 precision, and vLLM-based generation settings, which will help reproduction. The dataset construction protocol is also clearly described: the authors split the data with a fixed seed and sample 500 examples per split. Yet no code or checkpoint release is mentioned, and the black-box attack's query budget is substantial, requiring many generations and restarts per query. Independent reproduction is therefore feasible in principle but blocked by the absence of published code and by the high inference cost of the black-box variant, which the authors acknowledge is left to the adversary to configure.