Tolerance-Based Fairness Auditing: Violation Certification and Sensitivity Screening

The problem formulation is sharp: the authors clearly distinguish violation certification, where 'falsely declaring a disparity within the tolerance region to be a violation may incur substantial regulatory, organizational, or decision costs,' from sensitivity screening, where 'missing a substantively meaningful disparity may be more costly than issuing an additional warning' (Section 1). The asymptotic theory is carefully executed. Theorems 1 and 2 establish that CEL achieves the required least-favorable-point behavior—exact level $\alpha$ at the boundary $\epsilon_G = \epsilon_0$ and vanishing Type I error in the interior—while Corollary 1 verifies consistency against fixed alternatives. Theorem 3 further shows that CEL combined with BH asymptotically controls the false flagging rate for disjoint groups. The COMPAS application is well-executed, contextualizing the results against Cherian and Cand\`es (2024) and yielding interpretable subgroup bounds.

The main limitation is the deliberate departure from uniform level-$\alpha$ control in SEL and ASEL. As the authors note, 'the tolerance boundary belongs to the composite null, this increase is formally an increase in the boundary Type I rejection probability. Under the screening objective, however, it is interpreted as an explicit operating trade-off rather than as a violation-certification guarantee' (Section 3.2.1). In practice, users may conflate screening alerts with certified violations, especially because ASEL produces the highest power among the procedures tested. When SEL and ASEL are combined with BH for simultaneous auditing, the empirical false flagging rate inflates substantially—Section 4.2 reports that 'at the group sample sizes considered, both SEL-BH and ASEL-BH exhibit substantial FFR inflation'—Table 1 shows values as high as 0.37 and 0.90 under the global null, forcing the authors to retreat to an 'exploratory benchmark' framing. Additionally, Condition (3) requires $\sqrt{n}|\hat{\theta} - \theta_P| = o_P(1)$, which is stronger than typical $\sqrt{n}$-consistency and is not discussed for high-dimensional or complex nuisance estimators. Finally, the simulation studies are limited to symmetric and skewed location families with 2,000 replications, while simultaneous auditing uses only 100 replications, giving relatively noisy FFR estimates.

The theoretical evidence is well aligned with the claims. Theorems 1–6 and Corollaries 1–2 explicitly characterize behavior at the boundary, interior, and local alternatives, and the Supplementary Material contains full proofs. The simulation evidence supports the predicted trade-offs: 'CEL provides conservative protection against false violation declarations, whereas SEL and ASEL maintain nondegenerate interior rejection probabilities and exhibit greater detection sensitivity' (Section 4.1). Comparisons to the one-sided Student $t$ test show CEL is more robust in small samples and under skewness. The COMPAS analysis compares favorably to Cherian and Cand\`es (2024), reporting that 'CEL yields a larger lower bound of 2.51%' versus their 1.87% at the 90% level (Section 5). However, the scope of empirical validation is narrow: only a single real dataset, one predictive parity metric, and unidirectional disparities are examined, leaving open how the methods perform under intersectional definitions with many small subgroups or under model misspecification.

The authors provide code at https://github.com/Tang-Jay/TFA and specify simulation settings clearly—sample sizes $n \in \{30, 100, 300, 1000\}$, tolerance $\epsilon_0 = 0.25$, nominal level $\alpha = 0.05$, and $B = 100$ replications for simultaneous auditing (Section 4). The COMPAS filtering criteria are explicit: 'We restrict our analysis to the subset ($n = 2174$) of the holdout data with $f(X)=1$ and $X_{\text{race}} \in \{\text{African-American}, \text{Caucasian}\}$' (Section 5). That said, details needed to reproduce the empirical-likelihood optimization—such as the numerical solver, convergence tolerances, and handling of empty convex hulls—are not stated. The Supplementary Material contains proofs but no experimental log or runtime information. Because the repo was not verified in the paper, an independent reproduction would require trusting that the provided code matches the reported figures exactly.