Home  | Publications | ZTL+26

WebArbiter: A Principle-Guided Reasoning Process Reward Model for Web Agents

MCML Authors

Link to Profile Volker Tresp

Volker Tresp

Prof. Dr.

Principal Investigator

Database Systems, Data Mining and AI

Abstract

Web agents hold great potential for automating complex computer tasks, yet their interactions involve long horizons, multi-step decisions, and actions that can be irreversible. In such settings, outcome-based supervision is sparse and delayed, often rewarding incorrect trajectories and failing to support inference-time scaling. This motivates the use of Process Reward Models (WebPRMs) for web navigation, but existing approaches remain limited: scalar WebPRMs collapse progress into coarse, weakly grounded signals, while checklist-based WebPRMs rely on brittle template matching that fails under layout or semantic changes and often mislabels superficially correct actions as successful, providing little insight or interpretability. To address these challenges, we introduce WebArbiter, a reasoning-first, principle-inducing WebPRM that formulates reward modeling as text generation, producing structured justifications that conclude with a preference verdict and identify the action most conducive to task completion under the current context. Training follows a two-stage pipeline: reasoning distillation equips the model with coherent principle-guided reasoning, and reinforcement learning corrects teacher biases by directly aligning verdicts with correctness, enabling stronger generalization. To support systematic evaluation, we release WEBPRMBENCH, a comprehensive benchmark spanning four diverse web environments with rich tasks and high-quality preference annotations. On WEBPRM-BENCH, WebArbiter-7B outperforms the strongest baseline, Gemini Flash, by 10.9%. In reward-guided trajectory search on WebArena-Lite, it surpasses the best prior WebPRM by up to 7.2%, underscoring its robustness and practical value in real-world complex web tasks.

inproceedings ZTL+26

LaMAS @AAAI 2026

Workshop on LLM-based Multi-Agent Systems: Towards Responsible, Reliable, and Scalable Agentic Systems at the 40th Conference on Artificial Intelligence. Singapore, Jan 20-27, 2026. To be published.

Authors

Y. Zhang • S. Tang • Z. Li • Z. Han • V. Tresp

Research Area

 A3 | Computational Models

BibTeXKey: ZTL+26

#p-tresp
Back to Top