Reliable Conformal Prediction for Ordinal Classification Using the Ranked Probability Score

MCML Authors

Alireza Javanmardi

→ Group Eyke Hüllermeier
 Artificial Intelligence and Machine Learning

Eyke Hüllermeier

Prof. Dr.

Core PI

Artificial Intelligence and Machine Learning

Abstract

Ordinal classification (OC) arises in high-stakes domains such as medicine and finance, where uncertainty quantification must account for the severity of ordinal errors. Conformal prediction (CP) provides distribution-free prediction sets with marginal coverage guarantees; however, its practical effectiveness depends critically on the choice of nonconformity function. We introduce a CP method for ordinal classification based on the ranked probability score (RPS), a proper scoring rule defined over cumulative predictive distributions. Although it reflects ordinal risk quite naturally, it has largely been neglected in conformal ordinal prediction (COP). When used as a measure of nonconformity, RPS yields median-centered contiguous prediction sets by construction. The method is model-agnostic, supports both assessed and grouped ordered categorical outcomes, and permits efficient implementation compared to greedy interval selection procedures. Across multiple ordinal image and tabular datasets, RPS-based CP produces contiguous prediction sets and strikes a favorable balance between prediction set width and the magnitude of ordinal miscoverage relative to existing CP methods.

inproceedings HKJ+26

---

UAI 2026

42nd Conference on Uncertainty in Artificial Intelligence. Amsterdam, The Netherlands, Aug 17-21, 2026. To be published. Preprint available.

---

Authors

S. Haas • L. Killmaier • A. Javanmardi • E. Hüllermeier

---

Links

arXiv

---

Research Area

 A3 | Computational Models