PrivBench: A Holistic and Modular Benchmarking Platform for Evaluating Text-to-Text Privatization

MCML Authors

Stephen Meisenbacher

→ Group Jana Diesner
 Human-Centered Computing

Jana Diesner

Prof. Dr.

Core PI

Human-Centered Computing

Abstract

Natural Language Processing methods have enabled novel solutions and advances in the field of privacy, particularly in the sub-domain of *text-to-text privatization*, where the goal is to transform a sensitive input text into a privatized output by ideally masking (in)directly identifiable or otherwise private information. The evaluation of text-to-text privatization, however, is not straightforward, and the extant literature has utilized a myriad of techniques and metrics to quantify the privacy-preserving capabilities of privatization methods. Seeking to unify the evaluation of text-to-text privatization, we introduce PrivBench, a holistic and modular benchmarking platform for researchers and practitioners working on text privatization. PrivBench is holistic in that it evaluates privatization on a series of defined desiderata, which are structured into *modules*. PrivBench is not only modular but also extensible, allowing for future updates and benchmark versions. PrivBench is user-centered and promotes competition via real-time evaluation and a live public leaderboard. The platform is free to use and openly accessible at https://privbench.com/.

inproceedings MBA+26

---

EMNLP 2026

Conference on Empirical Methods in Natural Language Processing. Budapest, Hungary, Oct 24-29, 2026. System demonstrations. To be published. Preprint available.

---

Authors

S. Meisenbacher • A.-E. Bodea • A. B. Akin • A. Klymenko • J. Diesner • F. Matthes

---

Links

URL URL

---

Research Area

 C4 | Computational Social Sciences