PRISM: Self-Pruning Intrinsic Selection Method for Training-Free Multimodal Data Selection
MCML Authors
Hinrich Schütze
Prof. Dr.
Core PI
Volker Tresp
Prof. Dr.
Core PI
Abstract
Hinrich Schütze
Prof. Dr.
Core PI
Volker Tresp
Prof. Dr.
Core PI
Abstract
Visual instruction tuning adapts pre-trained Multimodal Large Language Models (MLLMs) to follow human instructions for real-world applications. However, rapidly growing instruction datasets introduce substantial redundancy and computational costs. Existing data selection methods aim to prune this redundancy, but predominantly rely on computationally demanding proxy-based inference or training-based metrics, often exacerbating the efficiency bottlenecks they are intended to resolve. To address this challenge, we identify a critical yet overlooked factor: anisotropy in visual feature distributions. We find that this anisotropy induces a Global Semantic Drift that distorts similarity estimates and limits efficient data selection. Motivated by this insight, we devise PRISM, a training-free framework for efficient visual instruction selection. PRISM mitigates the dominant global drift through implicit re-centering, allowing intrinsic visual semantics to provide a reliable redundancy signal without proxy models or additional training. Empirically, PRISM reduces end-to-end GPU time by about 70% while simultaneously enhancing performance, surpassing models fine-tuned on the full dataset across multimodal and language understanding benchmarks and reaching 101.7% relative performance against the full-data baseline.
inproceedings BAJ+26
EMNLP 2026
Conference on Empirical Methods in Natural Language Processing. Budapest, Hungary, Oct 24-29, 2026. Oral Presentation. To be published. Preprint available.
Authors
J. Bi • Aniri • Z. Wang • D. Yan • W. Huang • X. Ma • S. Yan • A. Hecker • M. Ye • X. Xiao • H. Schütze • V. Tresp • Y. MaLinks
URL GitHubIn Collaboration
Huawei
Research Areas
BibTeXKey: BAJ+26