NICE FACT: Diagnosing and Calibrating VLMs in Quantitative Reasoning for Kinematic Physics
MCML Authors
Haokun Chen
Dr.
* Former Member
Barbara Plank
Prof. Dr.
Core PI
Thomas Seidl
Prof. Dr.
Director
Abstract
Haokun Chen
Dr.
* Former Member
Barbara Plank
Prof. Dr.
Core PI
Thomas Seidl
Prof. Dr.
Director
Abstract
The ability to derive precise spatial and physical insights is a cornerstone of vision-language models (VLMs), yet their poor performances in related spatial intelligence tasks such as physical reasoning remain a fundamental barrier. The community critically lacks a scientific analysis revealing whether VLMs faithfully reach answers or plausibly make guesses. This work aims to provide a fundamental understanding of how VLMs perceive the physical world, and utilize physical laws, while assessing the reliability of model confidence. We propose NICE and FACT, a dual-diagnostic paradigm that explicitly decomposes quantitative reasoning for kinematic physics: FACT diagnoses visual fidelity, physical law comprehension, and temporal grounding. NICE studies our novel neighborhood-informed calibration method and novel metrics to evaluate and calibrate confidence reliability. Evaluated across 6 latest state-of-the-art VLMs, we uncover that models fail to identify visual preconditions or utilize necessary physical laws to reach answers. This work highlights and establishes a standardized diagnostic paradigm to guide the development of faithful, physically-grounded VLMs.
misc LLW+26a
Preprint
May. 2026Authors
J. Lan • Z. Liu • X. Wang • Y. Zhou • H. Chen • J. Lv • B. Plank • T. SeidlLinks
arXivResearch Areas
BibTeXKey: LLW+26a