Noisy-Label Calibration with Multiple Unreliable Labels
In medical and crowdsourced annotation, the absence of clean ground truth biases standard expected calibration error (ECE) evaluation because annotator errors confound predictor miscalibration. This paper proposes Noise Corrected ECE (NC-ECE), which inverts annotator-specific class-conditional noise channels to recover clean-label ECE, and provides methods to estimate those channels from repeated annotations alone. It then derives post-hoc recalibration algorithms with either clean-risk or calibration guarantees. The work is directly motivated by settings such as medical imaging, where expert disagreement is common and reliable uncertainty assessment is essential.
The paper delivers a principled statistical treatment of calibration evaluation under the Dawid–Skene class-conditional noise model, with rigorous finite-sample bounds and sensible identification strategies for unknown channels. Its main limitation is that the theoretical guarantees rest on strong assumptions—score-independent error rates, conditional annotator independence, and a known label orientation—that are unlikely to be fully satisfied in practice. The authors partially acknowledge this gap in the limitations section and the CIFAR-10N stress test, but the practical reliability of the method outside controlled settings remains an open question.
The corrected residual construction is clean and exact: Lemma 1 shows that $\mathbb{E}[(A_j-a_j)/\delta_j \mid Y,Z] = Y$, so bin averages of corrected residuals yield unbiased estimates of clean-label moments. Theorem 1 provides an interpretable error decomposition into binning approximation $L_r h_{\mathcal{B}}$, sampling fluctuation $\widehat{B}\sqrt{2(M\log 2 + \log(2/\eta))/n_{\mathrm{ev}}}$, and channel estimation $d_{\mathrm{ch}}$, matching the clean-label rate when $n_{\mathrm{ch}} \asymp n_{\mathrm{ev}}$. The identification results—Theorem 2 for two annotators with score groups and the Dawid–Skene product-mixture analysis for three or more—are classical but carefully adapted, and Proposition 2 gives a honest calibration guarantee for projected histogram recalibration that explicitly includes the channel-estimation error term $d_{\mathrm{ch}}$.
The class-conditional noise model (Assumption 1) excludes score-dependent annotator behavior, yet real annotators are often more accurate on high-confidence predictions; the CIFAR-10N experiment explicitly confronts this mismatch because the underlying noise is feature-dependent. Conditional independence (Assumption 2) is also strong, as shared expertise or common visual features can induce correlated errors across annotators. When annotator channel gaps $\delta_j$ are small, the inverse-variance weights require division by near-zero values, amplifying variance; the paper warns of this but does not relax the $\delta_j \neq 0$ requirement. On the algorithmic side, finite EM runs need not find a global maximizer, and the corrected-loss recalibration guarantee is only for log-risk, not ECE—indeed, corrected temperature scaling worsens ECE on the MEPS dataset in Table 2.
The semi-synthetic tabular and toy experiments provide strong support under assumption-satisfying conditions: NC-ECE closely tracks the Oracle, and corrected fixed-bin recalibration consistently beats naive averaging. Comparisons to related work are careful and accurate, distinguishing joint predictor-and-channel estimation (Tanno et al., 2019) from the fixed-predictor post-hoc setting studied here. The CIFAR-10N stress test is valuable but limited: while it shows directional agreement between NC-ECE and Oracle ECE, the authors rightly note that the core noise assumptions are unverified in this benchmark, so the result is suggestive rather than conclusive.
The paper is commendably thorough: appendices contain complete notation, proofs, algorithmic pseudocode (Appendix F), and extended experimental details (Appendix G), which greatly aids reproducibility. However, no code repository or data release is mentioned in the main text, and exact hyperparameters for EM initialization, convergence thresholds, and GMM constraints are buried in appendices. Reproducing the channel estimation would require careful reimplementation of constrained GMM or EM from scratch, and independent verification would be significantly easier with released code and scripts for the semi-synthetic noise generation.
In medical annotation and crowdsourcing, multiple annotators assign labels to each example, but these labels may be incorrect. Treating such noisy labels as ground truth can bias calibration evaluation. To address this problem, we propose methods for evaluating and recalibrating a fixed binary predictor under the underlying clean-label distribution, using only noisy annotations. First, given annotator-specific noise probabilities, we propose Noise Corrected ECE (NC-ECE), a binned estimator of expected calibration error (ECE) under class-conditional noise. We derive non-asymptotic bounds on the estimator's bias. Then, we propose practical algorithms for estimating unknown noise probabilities under conditional independence and label-orientation assumptions. We also develop loss-based and histogram post-hoc recalibration with clean-risk and calibration guarantees, respectively. Experiments on semi-synthetic benchmarks show that the proposed ECE estimator closely tracks its clean-label oracle counterpart and that the recalibration methods substantially improve latent-label calibration, whereas uncorrected recalibration against noisy annotations can degrade it.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.