Home | Publications | HDL+26

Probly: Uncertainty-Aware Machine Learning

MCML Authors

Link to Profile Eyke Hüllermeier

Eyke Hüllermeier

Prof. Dr.

Core PI

Artificial Intelligence and Machine Learning

Abstract

As machine learning systems are increasingly deployed in real-world applications, the question of how to represent and quantify uncertainty has moved from a methodological side issue to a central concern. In practice, however, making a model uncertainty-aware is still surprisingly difficult: relevant tools tend to be scattered across libraries, each tied to a particular framework and a particular approach to modeling uncertainty, and the choice of representation, quantification, and evaluation is typically left to the user without much guidance. In this paper, we present probly, a Python package that addresses these issues in a single, modular framework. probly offers (i) lightweight transformations that turn existing models into uncertainty-aware ones, currently supporting PyTorch, scikit-learn, and Flax, (ii) several representations, including second-order distributions, credal sets, and conformal prediction, (iii) the corresponding quantification measures, and (iv) a number of evaluation protocols, which can be combined more or less arbitrarily. Using probly, we conduct a benchmark study on three standard tasks — selective prediction, out-of-distribution detection, and active learning. In addition, and unlike existing benchmarks, we evaluate suitable methods on first-order data, i.e., datasets for which the target itself is a distribution over outcomes. We further illustrate the flexibility of the package on a number of less standard case studies, including large language models, graph neural networks, and data streams.

inproceedings HDL+26

NeurIPS 2026

40th Conference on Neural Information Processing Systems. Sydney, Australia, Dec 06-12, 2026. Oral Presentation. To be published.
Conference logo
A* Conference

Authors

P. Hofman • C. Damke • T. Löhr • S. M. A. R. Thies • A. Javanmardi • V. Margraf • J. Paplhám • Y. Sale • E. Hüllermeier • M. Muschalik

Links

GitHub

Research Area

 A3 | Computational Models

BibTeXKey: HDL+26

#p-huellermeier
Back to Top