Counterfactual Gate-Support Decomposition: Separating Selection from Redistribution in Training-Data Filtering
Data-quality gates conflate within-region example selection with across-region mass redistribution, so a gate-versus-random comparison cannot isolate what improved downstream performance. This paper introduces Counterfactual Gate–Support Decomposition (CGSD), a four-arm, budget-matched design that decomposes the gate effect into within-support selection $E$, between-support redistribution $S$, and their interaction $I$ by crossing eligibility and support-share factors. The core contribution is not the factorial algebra itself, but operationalizing it for finite pools while characterizing when the required arms are unrealizable.
The paper offers a sharp conceptual reframing and a genuinely useful feasibility characterization, but its empirical scope is narrow and two of the three decomposition terms are explicitly underpowered. The abstract notes that 'redistribution and interaction remain unresolved at the available power,' which limits the strength of any policy recommendation derived from the decomposition.
The identification of the conflation problem is incisive, and the $2\times 2$ arm construction yields exact estimands without modeling assumptions. The authors correctly note that their 'contribution is making that contrast operational for finite training pools under fixed exposure, and characterizing when the arms the identity requires cannot be realized at all.' The insistence that $E$ is not 'quality' but merely the eligible effect at fixed support mitigates overclaiming.
The empirical evidence is restricted to retrieval in two language pairs, and the abstract explicitly states that 'redistribution and interaction remain unresolved at the available power'—so the claim that redistribution is negligible is unsupported. The summary statistics are reported without confidence intervals or standard errors, making precision impossible to assess. Moreover, the provided text truncates before Sections 5–7, so experimental details, hyperparameters, and the promised limitations discussion—including the resolution of interference concerns noted in Section 2—cannot be verified.
The evidence supports the narrow claim that $E$ can exceed its null band, but not the broader claim that redistribution is generally negligible. The authors fairly position CGSD against prescriptive filtering ('CGSD is diagnostic and takes the rule as given') and mixture reweighting ('CGSD is the measurement that separates the two levers for a filter that was not designed to move either one deliberately'), and they appropriately cite Oaxaca–Blinder and propensity methods as conceptual precedents.
Independent reproduction is blocked by the lack of visible code, data, and training details in the provided excerpt: model architecture, optimizer settings, token budgets, and the packer algorithm are not shown. The validation protocol mentions 'a null band that is bound to the exact configuration it was measured on,' but without the exact construction thresholds and the truncated results tables, the band cannot be recomputed.
A training-data quality gate changes both which examples survive within regions of the input space and how training mass is distributed across those regions, so a gate-versus-random comparison does not isolate these mechanisms. We introduce Counterfactual Gate-Support Decomposition (CGSD), a four-arm, budget-matched design that crosses gate eligibility with support distribution and decomposes the observed gate effect into within-support selection, between-support redistribution, and their interaction. The practical constraint is finite-pool realizability: we characterize mass and token-assembly conditions under which the required arms can be constructed, together with a support-extinction case in which the ordinary design is undefined. On Korean-English and Korean-Japanese retrieval, the mean within-support selection component is the only one that exceeds its calibrated null band (+29.7 and +5.6 holdout items; 30/30 positive replicates), decisively in Korean-English and marginally in Korean-Japanese; redistribution and interaction remain unresolved at the available power. The same pattern holds for a second, unrelated gate family -- a model-confidence threshold rather than a deterministic verifier -- judged against its own null (+27.3 items, 9/9 positive), so the result is not a property of one gate's mechanism. On a mathematics gate, zero-eligibility supports leave the ordinary design undefined at every budget, while the extended design is constructible only inside narrow budget bands that an exhaustive scan locates and that the natural operating point misses -- the assembly condition separating a property of the pool from a limitation of the packer. CGSD therefore returns a decomposition, a constructible-but-underpowered design, or an explicit non-constructibility result, and says which; it prevents support redistribution from being silently interpreted as sample-level gate quality.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.