One Toolbox for Many Kinds of Uncertainty

Most machine learning models return a prediction but say little about what they don’t know. The Python package probly turns existing models into uncertainty-aware ones and brings many ways of representing, measuring, and evaluating uncertainty into one toolbox.

Understanding the Landscape

A machine learning model always produces a prediction. Whether that prediction deserves trust is a different question. It matters most when models leave controlled benchmarks and meet ambiguous, unfamiliar, or changing data.

Researchers have developed many ways to represent and measure this uncertainty. In practice, however, using them often means switching libraries, rebuilding models for a specific framework, and reconciling incompatible evaluation protocols.

MCML Junior Members Paul Hofman, Clemens Damke, Timo Löhr, Santo Thies, Alireza Javanmardi, Valentin Margraf, and Yusuf Sale, MCML Postdoc Maximilian Muschalik, MCML PI Eyke Hüllermeier, and collaborator Jakub Paplhám (Czech Technical University in Prague) tackle this problem with probly: Uncertainty-Aware Machine Learning. The paper introduces an open-source Python package that unifies these approaches in one modular framework and uses it to benchmark around 20 uncertainty methods side by side. It has been accepted as an oral presentation at NeurIPS 2026.

Why It Matters

Imagine an image classifier shown a photo of a dog. One model answers “dog” with 95% confidence. Another hesitates between “dog” (62%) and “fox” (29%), as in Figure 1. Both return the same label, but they carry very different information. Uncertainty tells us whether to trust a prediction, collect more data, or hand the decision to a human.

Uncertainty also comes in different kinds. Some of it lies in the data itself: a blurry photo can be genuinely ambiguous, and no amount of training will resolve that (aleatoric uncertainty). Some of it reflects the model’s limited knowledge, for example when an input looks nothing like the training data (epistemic uncertainty).

©Hofman et al.

Figure 1: A probabilistic model does not just output a label; it predicts a distribution over possible outcomes.

Importantly, methods also differ in how they represent uncertainty, and the representation decides what a method can express. A single predicted label carries no uncertainty at all. A set of plausible labels, as in conformal prediction, or a single probability distribution can express ambiguity. Only second-order representations can also express what the model does not know, and so separate aleatoric from epistemic uncertainty. These include samples of distributions, a distribution over distributions, and a set of plausible distributions (a credal set). Figure 2 shows these options side by side.

©Hofman et al.

Figure 2: Figure 2: Uncertainty can be represented in different ways, from a single predicted outcome to sets of probability distributions.

Representations are therefore more than an implementation detail. They are a precise mathematical language for what a method can and cannot do, and probly is organized around them.

Until now, these approaches were scattered across separate code bases, each tied to one framework and usually to one representation. No existing library supported credal sets at all. Evaluation practices differed too, so results were hard to compare across papers.

One Framework, Many Approaches

probly is organized around a four-stage workflow (Figure 3):

1. Transform: Turn any existing model into an uncertainty-aware one, for example by adding dropout, building an ensemble, or adding post-hoc calibration. The model stays in its own framework.
2. Represent: Extract the model’s uncertainty in a suitable form, such as a distribution, a credal set, or a prediction set.
3. Quantify and decide: Measure total uncertainty, split it into aleatoric and epistemic parts where possible, and turn it into a decision such as a label, an abstention, or a set of candidate labels.
4. Evaluate: Test whether the uncertainty is actually useful for a given task.

©Hofman et al.

Figure 3: probly enables a composable four-stage workflow.

Representations are the backbone. Every method in probly produces one of a small set of well-defined representation types: categorical or Gaussian distributions, conformal prediction sets, samples, Dirichlet distributions, or one of several kinds of credal sets. Every later stage works on these types. A method’s representation therefore tells you directly what it supports: whether its uncertainty can be split into aleatoric and epistemic parts, which measures and decision rules apply, and which evaluations make sense. Methods that produce the same representation can be swapped and compared one-to-one. The representations also behave like ordinary NumPy or PyTorch arrays, so whole batches of distributions or credal sets are handled like any other array.

Your model stays your model. A transformation does not rewrite the model in a new modeling language. A PyTorch model stays a PyTorch model, and a scikit-learn model stays a scikit-learn model. probly records the added capabilities without wrapping the model in a class of its own, so custom methods keep working and the model still trains in your own training loop. Practitioners keep the tools and pipelines of their own domain; probly does not dictate how models are built or trained. Most other uncertainty libraries instead require expressing the model inside their own framework, for example as a custom training module.

Putting Uncertainty Methods to the Test

Using probly, the researchers compared around 20 uncertainty estimation methods across several machine learning tasks. The results challenge some common assumptions: ensemble-based approaches performed particularly well in identifying unreliable predictions and unfamiliar data, while popular evidential methods showed weaker results. In active learning, where models select which data should be labeled next, even sophisticated uncertainty methods often failed to outperform simple baselines.

The team also tested how well these methods capture ambiguity in data, using datasets where multiple people labeled the same examples. Such evaluations offer a more direct way to assess whether a model’s uncertainty estimates reflect genuine disagreement or ambiguity. probly includes eleven datasets designed for this purpose.

Beyond Standard Classification

The paper also demonstrates probly in four less standard settings: For large language models, it measures uncertainty by comparing multiple answers while significantly reducing computation time. For graph neural networks, it helps identify uncertain regions where errors are more likely. In streaming data, it tracks how uncertainty changes as models adapt to new information. And by integrating with libraries such as river and torch-uncertainty, probly supports a wide range of existing workflows.

These case studies show the framework’s flexibility rather than treating each application exhaustively. The authors see probly as a living code base and invite contributions of new methods, representations, and evaluation protocols.

Further Reading & Reference

Want to try it? probly is open source on GitHub. The paper, which describes the package design, the full benchmark, and all case studies, has its own NeurIPS 2026 page with the oral and poster sessions.

P. Hofman • C. Damke • T. Löhr • S. M. A. R. Thies • A. Javanmardi • V. Margraf • J. Paplhám • Y. Sale • E. Hüllermeier • M. Muschalik
probly: Uncertainty-Aware Machine Learning.
NeurIPS 2026 - 40th Conference on Neural Information Processing Systems. Sydney, Australia, Dec 06-12, 2026. Oral Presentation. To be published. URL GitHub