Contrastive Energy Fields for Inference-Time Procedure Planning in Instructional Videos

MCML Authors

Christoph Reich

→ Group Daniel Cremers
 Computer Vision & Artificial Intelligence

Daniel Cremers

Prof. Dr.

Director

Computer Vision & Artificial Intelligence

Abstract

Procedure planning seeks to estimate a sequence of actions to transition from an observed initial state to a given goal state. Current procedure planning approaches directly predict action sequences from latent representations using feed-forward neural networks or diffusion-based inference. These paradigms treat every action as plausible, lacking the ability to enforce task-specific logical constraints that render certain actions irrelevant or not plausible. We propose CEFITO, a procedure planning approach that learns a predictor to express an action-conditioned representation space. Based on this representation space, we formulate procedure planning as a task-constrained optimization problem. Unlike prior methods, CEFITO explicitly reasons over the action space by omitting irrelevant actions during inference-time planning. This reformulation enables effective procedure planning and achieves state-of-the-art accuracy on two established procedure planning benchmarks.

inproceedings ARH+26

---

GCPR 2026

German Conference on Pattern Recognition. Siegen, Germany, Sep 22-25, 2026. To be published. Preprint available.

---

Authors

M. Afham • C. Reich • O. Hahn • D. Cremers • S. Roth

---

Links

arXiv GitHub

---

Research Area

 B1 | Computer Vision