ECI: Effective Contrastive Information to Evaluate Hard-Negatives

The theoretical foundation is sound. The InfoNCE lower bound $I(Q;P) \geq \log(N+1) - \mathcal{L}_{\text{InfoNCE}}$ from van den Oord et al. is correctly applied, and the use of harmonic mean to combine Signal ($S_n$) and Safety ($\Delta_{max}$) is mathematically appropriate for penalizing false positives. The ablation comparing harmonic vs. arithmetic mean (Table 5) is particularly compelling: "the Harmonic Mean correctly identifies the degradation in quality when LLM-generated negatives are added to the optimal set," while the arithmetic mean fails by assigning identical scores to contaminated and clean sets. The qualitative analysis in Section 7.5 also effectively illustrates why LLM-generated negatives fail—creating deceptive lexical overlap that collapses the margin.

The primary limitation is scale: the authors admit they use only "10,000 passages from MS-MARCO out of the 500,000+ that are present" due to computational constraints. This severely limits generalization claims about the metric's behavior on the full corpus distribution. Second, the downstream fine-tuning uses a single seed with early stopping=3 and only 1 epoch, raising questions about whether the reported correlations are stable or subject to optimization variance. Third, the metric's utility depends on the choice of embedding model used to compute similarities—a point noted in Section 7.3 where larger models like mxbai-embed-large-v1 achieve higher Signal but systematically compress Safety margins, yet the paper does not prescribe which embedding model practitioners should use for ECI evaluation. Finally, the LLM baseline uses only 3 negatives per query vs. 50 for BM25 due to API costs, creating an unfair comparison that the logarithmic term only partially addresses.

The evidence supports the core claim that ECI predicts downstream performance. The BM25+Cross-Encoder hybrid achieves the highest ECI (1.25) and the best average nDCG@10 (0.337) across 12 BEIR datasets, outperforming pure BM25 (0.321) and Cross-Encoder (0.321) baselines. However, the comparison is incomplete: the paper does not compare ECI against established heuristics like average cross-encoder scores or lexical overlap metrics in the same correlation analysis. While Section 7.4 shows that gradient-based heuristics correlate negatively with performance, it remains unclear whether a simpler composite of Signal+Safety without the logarithmic capacity term would achieve comparable predictive power. The claim that ECI "reduces both experimental turnaround time and computational expenditure" is supported but not quantified with actual compute-hour savings.

Reproducibility is limited. The paper uses public datasets (MS MARCO, BEIR) and standard models (DistilBERT, all-MiniLM-L6-v2, mxbai-embed-large-v1), but no code, data generation scripts, or hyperparameter configurations are publicly released. The LLM generation uses GPT-4o-mini via API with a detailed prompt template provided, but the exact temperature, sampling parameters, and parsing logic are not specified. Crucially, the random seed and optimizer details for fine-tuning are absent. The reliance on commercial API calls (OpenAI) for the LLM baseline creates a reproducibility barrier, and the limited 10k sample size means independent reproductions on the full MS MARCO may yield different ECI score distributions. Full reproducibility would require: (1) release of generated negative sets, (2) exact training configurations for DistilBERT fine-tuning, and (3) embedding model checkpoint versions used for computing ECI scores.