Revealing and Reducing Gender Biases in Vision and Language Assistants (VLAs)

MCML Authors

Leander Girrbach

→ Group Zeynep Akata
 Interpretable and Reliable Machine Learning

Yiran Huang

→ Group Zeynep Akata
 Interpretable and Reliable Machine Learning

Stephan Alaniz

Dr.

→ Group Zeynep Akata
 Interpretable and Reliable Machine Learning

Zeynep Akata

Prof. Dr.

Principal Investigator

Interpretable and Reliable Machine Learning

Abstract

Pre-trained large language models (LLMs) have been reliably integrated with visual input for multimodal tasks. The widespread adoption of instruction-tuned image-to-text vision-language assistants (VLAs) like LLaVA and InternVL necessitates evaluating gender biases. We study gender bias in 22 popular open-source VLAs with respect to personality traits, skills, and occupations. Our results show that VLAs replicate human biases likely present in the data, such as real-world occupational imbalances. Similarly, they tend to attribute more skills and positive personality traits to women than to men, and we see a consistent tendency to associate negative personality traits with men. To eliminate the gender bias in these models, we find that finetuning-based debiasing methods achieve the best tradeoff between debiasing and retaining performance on downstream tasks. We argue for pre-deploying gender bias assessment in VLAs and motivate further development of debiasing strategies to ensure equitable societal outcomes.

inproceedings

---

ICLR 2025

13th International Conference on Learning Representations. Singapore, Apr 24-28, 2025.

---

Authors

L. Girrbach • Y. Huang • S. Alaniz • T. Darrell • Z. Akata

---

Links

URL

---

Research Area

 B1 | Computer Vision