PERSEID - Perspectivist Irony Detection: A CALAMITA Challenge

MCML Authors

Silvia Casola

Dr.

→ Group Barbara Plank
 AI and Computational Linguistics

Abstract

Works in perspectivism and human label variation have emphasized the need to collect and leverage various voices and points of view in the whole Natural Language Processing pipeline. PERSEID places itself in this line of work. We consider the task of irony detection from short social media conversations in Italian collected from Twitter (X) and Reddit. To do so, we leverage data from MultiPICO, a recent multilingual dataset with disaggregated annotations and annotators’ metadata, containing 1000 Post, Reply pairs with five annotations each on average. We aim to evaluate whether prompting LLMs with additional annotators’ demographic information (namely gender only, age only, and the combination of the two) results in improved performance compared to a baseline in which only the input text is provided. The evaluation is zero-shot; and we evaluate the results on the disaggregated annotations using f1.

inproceedings

---

CLiC-it 2024

10th Italian Conference on Computational Linguistics. Pisa, Italy, Dec 04-06, 2024.

---

Authors

V. Basile • S. Casola • S. Frenda • S. M. Lo

---

Links

URL

---

Research Area

 B2 | Natural Language Processing