Home | Publications | SWL+26

PRSM: A Measure to Evaluate CLIP's Robustness Against Paraphrases

MCML Authors

Link to Profile Thomas Seidl

Thomas Seidl

Prof. Dr.

Director

Database Systems, Data Mining and AI

Abstract

Contrastive Language-Image Pre-training (CLIP) is a widely used multimodal model that aligns text and image representations through large-scale training. While it performs strongly on zero-shot and few-shot tasks, its robustness to linguistic variation, particularly paraphrasing, remains underexplored. Paraphrase robustness is essential for reliable deployment, especially in socially sensitive contexts where inconsistent representations can amplify demographic biases. In this paper, we introduce the Paraphrase Ranking Stability Metric (PRSM), a novel measure for quantifying CLIP’s sensitivity to paraphrased queries. Using the Social Counterfactuals dataset, a benchmark designed to reveal social and demographic biases, we empirically assess CLIP’s stability under paraphrastic variation, examine the interaction between paraphrase robustness and gender, and discuss implications for fairness and equitable deployment of multimodal systems. Our analysis reveals that robustness varies across paraphrasing strategies, with subtle yet consistent differences observed between male- and female-associated queries.

inproceedings SWL+26

MMM 2026

32nd International Conference on Multimedia Modeling. Prague, Czech Republic, Jan 29-31, 2026.

Authors

U. Schlegel • F. Weeber • J. Lan • T. Seidl

Links

DOI

Research Area

 A3 | Computational Models

BibTeXKey: SWL+26

#p-seidl
Back to Top