ProtoCLAP – Prototypical Contrastive Language-Audio Pretraining

MCML Authors

Adria Mallol-Ragolta

* Former Member

→ Group Björn Schuller
 Health Informatics

Björn Schuller

Prof. Dr.

Core PI

Health Informatics

Abstract

We propose ProtoCLAP, a framework that integrates prototypical representations of the targeted classes in the languageaudio contrastive learning paradigm. Projecting the audio and the language representations in a shared embeddings space – where the prototypical representations are computed –, ProtoCLAP aims to maximise the similarity of the audio embeddings and their corresponding audio and language prototypes, while enforcing the similarity between both prototypical representations. We conduct our experiments on the MASCFLICHT Corpus and the Second DiCOVA Challenge Dataset. ProtoCLAP achieves the best results in three out of the six scenarios investigated. For face mask type and face mask coverage area recognition, ProtoCLAP scores the best Unweighted Average Recall on the test set, 62.8% and 56.7%, respectively. For COVID-19 detection, ProtoCLAP obtains the highest Area Under the Curve on the test set when exploiting the breathing sounds, 84.77%.

inproceedings MS25

---

ASRU 2025

IEEE Automatic Speech Recognition and Understanding Workshop. Honolulu, HI, USA, Dec 06-10, 2025.

---

Authors

A. Mallol-Ragolta • B. W. Schuller

---

Links

DOI

---

Research Area

 B3 | Multimodal Perception