Distillation of Tabular Foundation Models into Efficient Predictors
Tabular foundation models (TFMs) achieve strong zero-shot performance through in-context learning, but repeatedly encoding the labeled context at inference time is expensive. This paper studies how to distill TFMs into lightweight, dataset-specific students. It systematically examines two design choices—how to construct teacher supervision (full-context versus out-of-fold, mixed with labels or not) and whether cheap synthetic queries improve distillation—and derives a simple recipe that yields median inference speedups of $3.0\times$–$21.6\times$ while recovering roughly two-thirds of the teacher's accuracy advantage over a standard supervised student.
The paper presents a clean, large-scale empirical study that isolates two key distillation design questions and yields a simple but effective recipe: use the full training set as teacher context, train the student solely on teacher predictions ($\alpha = 1$), and augment with synthetic CutMix queries at a $2{:}1$ ratio. This fixed recipe outperforms heavily tuned and ensembled supervised baselines on TabArena and generalizes to hundreds of TALENT datasets. However, the study covers only two teacher models from the same transformer-based PFN family, does not benchmark against concurrent distillation methods such as GEAR, and shows modest speedups and high break-even volumes when teacher contexts are small.
The experimental coverage is extensive and the ablations are well-controlled. TabArena provides 51 datasets under an official benchmark protocol, and TALENT adds 300 datasets for external validation. The design studies cleanly isolate supervision construction and query coverage, and the same pattern holds across both neural (TabM) and tree-based (XGBoost) students. Importantly, the authors retain default student configurations, strengthening the claim that distillation itself drives the gains. The appendix also confirms the main evaluation results hold after excluding the design splits, reducing concerns about split reuse.
First, teacher diversity is narrow: both teachers (TabICLv2 and TabPFN v3) are transformer-based PFN-style models, so it is unclear whether the recipe generalizes to other tabular foundation model families such as hypernetworks or diffusion-based models. Second, the synthetic generators are extremely simple (CutMix, Mixup, FullMix, and Gaussian perturbation). The authors acknowledge concurrent work such as GEAR that uses more sophisticated generators, but no empirical comparison is provided, leaving open how much gain is uniquely attributable to expanding query coverage versus using better generative models. Third, the speedup analysis reveals less compelling economics for small-data settings: with fewer than 1K teacher context rows, median speedups drop to $1.9\times$–$13.1\times$, and for TabICLv2 distilled into TabM the median break-even volume exceeds 3,200 requests, making distillation costly for light serving loads.
The evidence broadly supports the central claims, though comparisons to related distillation work are restricted to the Related Work section. Against supervised competitors, the comparisons are fair and benchmark-aligned: the distilled students beat architecture-matched tuned-and-ensembled versions and outperform the strongest configuration of 14 of 15 conventional supervised families on TabArena. The authors are also careful to exclude the design splits in Appendix C and show that the same trend holds, which strengthens the evaluation. However, there is no head-to-head experiment with Pocket Foundation Models or GEAR on the same datasets, leaving open the question of how much performance is left on the table by avoiding direct label supervision or by using simple synthetic generators.
Reproducibility is reasonably strong. The authors state that code is available, and the experimental protocol is spelled out in detail across the main text and appendices. They rely on benchmark defaults (TabArena pinned to a specific commit, TALENT adapters) and explicitly note that they use default student configurations with no additional temperature scaling. Hardware and measurement specifics are disclosed: throughput is measured on an H200 with 32 pinned CPU cores using 32-row requests after warm-up. The reported GitHub link in the abstract appears partially escaped, so readers should verify the repository is public and contains runnable scripts. Exact reproduction will also depend on pinned teacher model versions and API access, which are not explicitly versioned beyond the paper citations.
Tabular foundation models (TFMs) achieve strong predictive performance through in-context learning, yet repeatedly conditioning on labeled data makes inference expensive. Knowledge distillation can reduce this cost by transferring their predictive ability to lightweight, dataset-specific students. However, the dependence of TFM predictions on both a labeled context and a query introduces two design questions: how to construct teacher supervision and whether expanding query coverage improves distillation. We examine these questions across two TFMs and both neural and tree-based students, and derive an effective distillation recipe. The recipe uses the full labeled training set as teacher context and trains students solely on teacher predictions for observed and synthetic queries. On TabArena, the resulting students outperform their supervised trained tuned-and-ensembled counterparts by 57-98 Elo points. Applied unchanged to TALENT, the same recipe improves matched default students on 236-258 of 300 datasets and reduces median primary error by 4.0-6.4%. The distilled students also achieve median inference speedups of 3.0-21.6 times over their teachers, offering a practical trade-off between predictive performance and repeated inference cost. Code is available at https://github.com/nums-ai/TFM_Distillation .
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.