Can Vision-Language Models Be a Good Guesser? Exploring VLMs for Times and Location Reasoning

MCML Authors

Gengyuan Zhang

→ Group Volker Tresp
 Database Systems, Data Mining and AI

Volker Tresp

Prof. Dr.

Principal Investigator

Database Systems, Data Mining and AI

Abstract

Vision-Language Models (VLMs) are expected to be capable of reasoning with commonsense knowledge as human beings. One example is that humans can reason where and when an image is taken based on their knowledge. This makes us wonder if, based on visual cues, Vision-Language Models that are pre-trained with large-scale image-text resources can achieve and even surpass human capability in reasoning times and location. To address this question, we propose a two-stage Recognition & Reasoning probing task applied to discriminative and generative VLMs to uncover whether VLMs can recognize times and location-relevant features and further reason about it. To facilitate the studies, we introduce WikiTiLo, a well-curated image dataset compromising images with rich socio-cultural cues. In extensive evaluation experiments, we find that although VLMs can effectively retain times and location-relevant features in visual encoders, they still fail to make perfect reasoning with context-conditioned visual features.

inproceedings

---

WACV 2024

IEEE/CVF Winter Conference on Applications of Computer Vision. Waikoloa, Hawaii, Jan 04-08, 2024.

---

Authors

G. Zhang • Y. Zhang • K. Zhang • V. Tresp

---

Links

DOI GitHub

---

Research Area

 A3 | Computational Models