Low-Budget Active Learning through Entropic Optimal Transport
Low-budget active learning seeks a tiny labeled subset—a coreset—that still trains an accurate model, which is crucial in medical domains where labeling is expensive. The authors propose using the Sinkhorn divergence, an entropic optimal transport distance, as a coreset quality criterion, and develop a Frank–Wolfe warm-started swap heuristic (FW-Swap) to optimize it. Their framework yields dimension-free generalization guarantees and shows competitive or superior accuracy on vision and EEG benchmarks with very small budgets.
The paper makes a solid conceptual contribution by grounding low-budget coreset selection in entropic optimal transport and by rigorously motivating the Sinkhorn divergence over the biased entropic cost. However, its algorithmic guarantees are weak—the suboptimality bound in Theorem 2 is loose and leans on an unverified structural assumption—while the empirical evaluation, though promising, is limited to five datasets and modest budgets. The result is a theoretically elegant framework whose practical edge over simpler heuristics is suggestive but not conclusively demonstrated.
The bias argument for preferring the Sinkhorn divergence over the raw entropic cost is crisp and well illustrated: on the CCO$(\Delta,m)$ instances the divergence recovers the Wasserstein-optimal extreme configuration, whereas the entropic cost collapses to the central outlier for large $\varepsilon$. The dimension-free generalization bound also provides a principled justification for the criterion, at least when the final classifier is Lipschitz.
The suboptimality guarantee for FW-Swap (Theorem 2) is largely qualitative because the constant $L^{\mathrm{swap}}_{\varepsilon,\lambda}$ is deferred to the appendix and $k^{*}$ can be as large as the budget $B$, leaving the bound potentially vacuous.\nMore critically, the runtime analysis relies on Assumption 1, a strong structural assumption about a stable core $S_0$ that is never proven and is only supported by a single-dataset empirical trace in Appendix B; without it there is no formal iteration-complexity guarantee. Finally, Theorem 1 requires the final classifier to be Lipschitz and its quantitative value depends on $\kappa(\varepsilon)$, which Lemma 1 shows scales exponentially in $1/\varepsilon$, severely limiting its predictive power for realistic regularizations.
The authors narrowly distinguish their work from concurrent approaches: they correctly note that Kokot and Luedtke (2025) address asymptotic compression rather than low-budget active learning, and that Yin et al. (2025) do not tackle the combinatorial discrete selection. These are fair and verified characterizations. Yet the experimental validation is narrow—only five datasets, all with very small budgets—and the paper does not report statistical significance tests or full budget sweeps beyond the appendix. The strong claim that FW-Swap outperforms state-of-the-art heuristics in low-budget settings is therefore supported but not thoroughly stress-tested.
No code repository or open-source release is mentioned, which immediately blocks independent reproduction. The main text reports key hyperparameters—$\varepsilon=1.5$ for vision and $\varepsilon=0.5$ for EEG, a learning rate of $0.1$, momentum $0.9$, $30$ epochs, and batch size $32$—but omits other implementation details such as the mixture parameter $\lambda$, the number of Frank–Wolfe iterations $T$, and the exact stopping criteria for the swap phase. Since the algorithm relies on Julia packages and a specific GPU/CPU cluster, reproducing the results would require substantial guesswork.
We consider low-budget active learning, which consists of selecting a limited number of points, the coreset, such that a model can be trained to high accuracy on the selection only. This problem is particularly relevant in contexts where labeling requires costly expert intervention, as in medical applications. We leverage features extracted from a pretrained self-supervised model to represent the data, and perform coreset selection directly in this feature space. In this paper, we use entropic optimal transport, specifically the Sinkhorn divergence, as the coreset selection criterion, which first allows us to get dimension-free sample complexity results, and second admits computationally efficient gradient evaluations. This opens the way to using gradient-based algorithms to rapidly compute solution candidates, further improved by a swap-based local search, with guarantees on the solution quality. Experiments on image benchmarks and medical datasets show that our method outperforms state-of-the-art heuristics in low-budget settings.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.