PyraVid: Hierarchical Multimodal Memory for Long-Horizon Video Reasoning

MCML Authors

Sikuan Yan

→ Group Volker Tresp
 Database Systems, Data Mining and AI

Jinhe Bi

→ Group Volker Tresp
 Database Systems, Data Mining and AI

Volker Tresp

Prof. Dr.

Core PI

Database Systems, Data Mining and AI

Yunpu Ma

Dr.

→ Group Volker Tresp
 Database Systems, Data Mining and AI
→ Co-Group Hinrich Schütze

Abstract

Memory has become an increasingly important component of agentic systems, as these systems are expected to reason over long-term experience. However, prior work has largely focused on unimodal memory, leaving multimodal memory relatively underexplored despite its central role in real-world applications. Compared with unimodal settings, multimodal memory introduces additional challenges, including heterogeneous input integration, person-centric information alignment, and evidence aggregation across different granularities. We present PyraVid, a hierarchical multimodal memory framework inspired by Event Segmentation Theory from cognitive science. PyraVid organizes long videos into a coarse-to-fine pyramid structure, enabling structured memory access and effective evidence aggregation. It further supports structure-guided memory expansion with pruning, allowing the retrieval of related events with strong causal connectivity but low semantic similarity while reducing noise. Experiments on multiple long-video understanding benchmarks show that PyraVid consistently improves performance across datasets, model scales, and question types, highlighting the effectiveness of hierarchical multimodal memory for long-horizon reasoning.

misc YDW+26

---

Preprint

May. 2026

---

Authors

S. Yan • S. Dong • H. Wang • E. Nie • Y. Liu • J. Bi • Y. Xu • S. Schwarzmann • R. Trivisonno • V. Tresp • Y. Ma

---

Links

arXiv

---

Research Area

 A3 | Computational Models