In-Context Learning of Soft Nearest Neighbor Classifiers for Intelligible Tabular Machine Learning

MCML Authors

Matthias Feurer

Prof. Dr.

Thomas Bayes Fellow

* Former Thomas Bayes Fellow

Abstract

With in-context learning foundation models like TabPFN excelling on small supervised tabular learning tasks, it has been argued that 'boosted trees are not the best default choice when working with data in tables'. However, such foundation models are inherently black-box models that do not provide interpretable predictions. We introduce a novel learning task to train ICL models to act as a nearest neighbor algorithm, which enables intelligible inference and does not decrease performance empirically.

inproceedings KFE25

---

TRL @ACL 2025

4th Table Representation Learning Workshop at the 63rd Annual Meeting of the Association for Computational Linguistics. Vienna, Austria, Jul 27-Aug 01, 2025.

---

Authors

M. Koshil • M. Feurer • K. Eggensperger

---

Links

DOI

---

Research Area

 A1 | Statistical Foundations & Explainability