Home | Publications | MHR+26

Deferred Aggregation in Hierarchical Bayesian Optimization

MCML Authors

Valentin Margraf

→ Group Eyke Hüllermeier
Artificial Intelligence and Machine Learning

Jonas Hanselle

→ Group Eyke Hüllermeier
Artificial Intelligence and Machine Learning

Eyke Hüllermeier

Prof. Dr.

Core PI

Artificial Intelligence and Machine Learning

Abstract

To account for uncertainty, hierarchical Bayesian optimization marginalizes over Gaussian process hyperparameters, usually averaging models or acquisition functions. However, this early aggregation discards information about model disagreement and is sensitive to outliers. To address these limitations, we propose to defer aggregation to the decision level: each model independently proposes a candidate, and we aggregate over these candidates via the medoid. We study a misspecified hierarchical Gaussian process setting in which most models have low misspecification error, while a minority are severely misspecified. Theoretically, we derive regret bounds showing that the misspecification penalty of early aggregation depends on the average error across all models, whereas decision-level aggregation depends only on the low-misspecification majority. Empirically, we validate these findings on toy examples and further demonstrate that, across a wide range of low- to high-dimensional synthetic and real-world benchmarks, decision-level aggregation is competitive with standard Bayesian optimization baselines and provides particular strong gains on high-dimensional synthetic problems.

inproceedings MHR+26