SynthACticBench: A Capability-Based Synthetic Benchmark for Algorithm Configuration

MCML Authors

Valentin Margraf

→ Group Eyke Hüllermeier
 Artificial Intelligence and Machine Learning

Eyke Hüllermeier

Prof. Dr.

Principal Investigator

Artificial Intelligence and Machine Learning

Abstract

Algorithm configuration deals with the automatic optimization of an algorithm's parameters to maximize its performance on a distribution of problem instances, such as Boolean satisfiability or the traveling salesperson problem. While significant progress has been made in developing optimizers for algorithm configuration - so-called algorithm configurators - their evaluation remains computationally expensive and often relies on real-world scenarios with hard-to-control characteristics. This makes it challenging to analyze their strengths and weaknesses systematically. To address this, we introduce SynthACticBench, a synthetic benchmark specifically designed to isolate and investigate key properties of algorithm configuration problems. Our benchmark distinguishes between properties related to the configuration space and those associated with the objective function. We define a configurator's ability to handle a particular property as its capability -for example, the capability to manage hierarchical configuration spaces. Using SynthACticBench, we evaluate two state-of-the-art algorithm configurators, SMAC and irace, examining their complementary capabilities and analyzing their performances across diverse benchmark functions. By providing a controlled, scalable, and capability-based evaluation environment, SynthACticBench facilitates a more targeted analysis of algorithm configurators, helping to advance research in the field.

inproceedings

---

GECCO 2025

Genetic and Evolutionary Computation Conference. Málaga, Spain, Jul 14-18, 2025.

---

Authors

V. Margraf • A. Lappe • M. Wever • C. Benjamins • E. Hüllermeier • M. Lindauer

---

Links

DOI GitHub

---

Research Area

 A3 | Computational Models