16.08.2024

Tiny logo
Teaser image to Success at ACL 2024: Area Chair Award for Barbara Plank and her team

Success at ACL 2024: Area Chair Award for Barbara Plank and Her Team

Recognized for Exceptional Contribution to NLP at ACL 2024

At ACL 2024, the premier conference in computational linguistics, the Senior Area Chairs of each track selected exceptional papers for the Area Chair Awards.

We are proud to announce that our PI Barbara Plank, together with Junior Members Leon Weber‑Genzel and Siyao Peng, and collaborator Marie-Catherine de Marneffe, received an ACL 2024 Area Chair Award for their paper: “VariErr NLI: Separating Annotation Error from Human Label Variation.”

Congratulations from us!

Check out the full paper:

L. Weber-Genzel, S. Peng, M.-C. De Marneffe and B. Plank.
VariErr NLI: Separating Annotation Error from Human Label Variation.
ACL 2024 - 62nd Annual Meeting of the Association for Computational Linguistics. Bangkok, Thailand, Aug 11-16, 2024. DOI
Abstract

Human label variation arises when annotators assign different labels to the same item for valid reasons, while annotation errors occur when labels are assigned for invalid reasons. These two issues are prevalent in NLP benchmarks, yet existing research has studied them in isolation. To the best of our knowledge, there exists no prior work that focuses on teasing apart error from signal, especially in cases where signal is beyond black-and-white.To fill this gap, we introduce a systematic methodology and a new dataset, VariErr (variation versus error), focusing on the NLI task in English. We propose a 2-round annotation procedure with annotators explaining each label and subsequently judging the validity of label-explanation pairs.VariErr contains 7,732 validity judgments on 1,933 explanations for 500 re-annotated MNLI items. We assess the effectiveness of various automatic error detection (AED) methods and GPTs in uncovering errors versus human label variation. We find that state-of-the-art AED methods significantly underperform GPTs and humans. While GPT-4 is the best system, it still falls short of human performance. Our methodology is applicable beyond NLI, offering fertile ground for future research on error versus plausible variation, which in turn can yield better and more trustworthy NLP systems.

MCML Authors
Leon Weber-Genzel

Leon Weber-Genzel

Dr.

* Former Member

Link to website

Siyao Peng

Dr.

AI and Computational Linguistics

Link to Profile Barbara Plank

Barbara Plank

Prof. Dr.

AI and Computational Linguistics

16.08.2024


Subscribe to RSS News feed

Related

Link to Fabian Theis receives 2025 ISCB Innovator Award

01.08.2025

Fabian Theis Receives 2025 ISCB Innovator Award

Fabian Theis receives 2025 ISCB Innovator Award for advancing AI in biology and mentoring the next generation of scientists.

Link to Yusuf Sale receives IJAR Young Researcher Award

29.07.2025

Yusuf Sale Receives IJAR Young Researcher Award

MCML Junior Member Yusuf Sale received an IJAR Young Researcher Award at ISIPTA 2025 for his work.

Link to Barbara Plank awarded 2025 Imminent Research Grant for work on language data

29.07.2025

Barbara Plank Awarded 2025 Imminent Research Grant for Work on Language Data

Barbara Plank’s MaiNLP lab wins 2025 Imminent Research Grant for a project on language data with Peng and de Marneffe.

Link to Eyke Hüllermeier to Lead New DFG-Funded Research Training Group METEOR

22.07.2025

Eyke Hüllermeier to Lead New DFG-Funded Research Training Group METEOR

MCML PI Eyke Hüllermeier to lead new DFG-funded RTG METEOR, uniting ML and control theory to build robust, explainable AI systems.

Link to Outstanding Paper Award at ICML 2025 for MCML Researchers

18.07.2025

Outstanding Paper Award at ICML 2025 for MCML Researchers

MCML researchers win ICML 2025 Outstanding Paper Award for work on prediction and identifying the worst-off.