Why Does Train-Validation Separation Emerge? Update-Pressure Density Dynamics in Pretrained Backbones

The constructed five-level ResMLP hierarchy is the strongest evidence, directly intervening on private-feature share $p \in \{.3,.5,.7\}$ and showing monotonically increasing final accuracy gaps (from $.185$ to $.331$ to $.527$) alongside rising validation-side private masked-loss burden. The use of a training-probe-weighted within-class dispersion $D_{\mathrm{within},t}^{\mathrm{wt}}$ is a smart control: it demonstrates that gradient-allocation heterogeneity is not merely a between-class artifact but persists, as the authors state, "among examples sharing the same label across these 90 trajectories."\n\nAdditionally, temporal checks—linear-time partial correlations and approximately one-epoch paired differences that remain positive in 86/90 and 87/90 NLP runs—go beyond simple common-trend confounding and lend credibility to the readout's trajectory-level information.

The central weakness is the leap from controlled intervention to real-data mechanism identification. In NLP and vision, the paper shows only that $D_{\mathrm{den},t}$ correlates positively with the accuracy gap; this is equally consistent with any overfitting process that increases per-example gradient specialization, without requiring a semantic shift from shared to private support. The local dynamical theory relies on strong, effectively untestable assumptions—fixed consumption rates $\kappa_u > \kappa_v$, a noiseless two-component mixture, and constant transfer efficiencies $\tau_u > \tau_v$—that are acknowledged in Section 7 as bounding the scope of the mechanism.\n\nA further tension exists between theory and measurement: the differentiable surrogate-risk analysis (Proposition 2) is built on cross-entropy dynamics, yet the empirical endpoint is the nondifferentiable accuracy gap; as the authors note, "cross-entropy can move through confidence, calibration, and logit scale and may therefore change substantially even when argmax class labels, and hence accuracy, are unchanged."

Comparison to related work is fair in scope but not empirically discriminating. The authors correctly note that memorization studies operate at the output level and that stiffness/coherence analyses focus on signed-gradient alignment rather than magnitude-allocation dispersion. However, the paper does not empirically disentangle its predictions from competing explanations such as fine-tuning distortion or gradient starvation, both of which could increase per-example gradient heterogeneity without invoking the paper's specific shared-to-private support transition. As stated in Section 2, "The closest works explain important parts of the phenomenon, but they do not jointly provide a mechanism account of train–validation separation, an update-pressure density readout, and empirical tests across mainstream pretrained language backbones"—a positioning claim that highlights novelty but does not constitute independent evidence against those alternatives.

Reproducibility is moderate but incomplete. The authors report backbone-specific monitoring protocols in Appendices E and F, provide pseudocode in Algorithm 1, and include sensitivity grids for monitor depth and probe count in Appendix B. However, the exact tensor-family rule for selecting interface coordinates $\mathcal{C}$, the random-seeding protocol, and the custom ResMLP architecture details are only partially exposed in the main text. The paper does not mention public release of code or pretrained checkpoints, and because the readout magnitudes are sensitive to monitor depth—as the authors warn, "depth sensitivity results therefore call for interpreting readout magnitudes and association strength within each specified setting"—an independent reproduction would require precise replication of these engineering choices.