Incremental Uncertainty-Aware Performance Monitoring With Labeling Intervention

MCML Authors

Thomas Decker

→ Group Volker Tresp
 Database Systems, Data Mining and AI

Volker Tresp

Prof. Dr.

Principal Investigator

Database Systems, Data Mining and AI

Abstract

We study the problem of monitoring machine learning models under temporal distribution shifts, where circumstances change gradually over time, often leading to unnoticed yet significant declines in accuracy. We propose Incremental Uncertainty-aware Performance Monitoring (IUPM), a novel label-free method that estimates model performance by modeling time-dependent shifts using optimal transport. IUPM also quantifies uncertainty in performance estimates and introduces an active labeling strategy to reduce this uncertainty. We further showcase the benefits of IUPM on different datasets and simulated temporal shifts over existing baselines.

inproceedings

---

BDU @NeurIPS 2024

Workshop Bayesian Decision-making and Uncertainty: from probabilistic and spatiotemporal modeling to sequential experiment design at the 38th Conference on Neural Information Processing Systems. Vancouver, Canada, Dec 10-15, 2024.

---

Authors

A. Koebler • T. Decker • I. Thon • V. Tresp • F. Buettner

---

Links

URL

---

Research Area

 A3 | Computational Models