26.09.2025

 

Making Machine Learning More Accessible with AutoML

Researcher in Focus: Matthias Feurer

Tell us a little bit about your research. What are you focusing on at the moment?

My current focus is on improving hyperparameter optimization, an important ingredient in the machine learning workflow. Every machine learning method has hyperparameters, and our goal is to make tuning them as easy and robust as possible. While we as a community have recently made a lot of progress in improving the hyperparameter optimization process itself, we have also realized several ways it can fail, and we are now working on overcoming these edge cases.

Your work centers around Automated Machine Learning (AutoML). Can you explain what AutoML is, and how hyperparameter optimization relates to AutoML?

Let me first explain what AutoML is, and then explain how hyperparameter optimization fits in there. AutoML supports machine learning users in building machine learning models. This can, on the one hand, be via completely hands-free AutoML systems where there is no human intervention, and where the AutoML system creates a fully functional machine learning pipeline for the user. On the other hand, this can be human-in-the-loop systems where only certain parts of a larger machine learning system are optimized. The first case is interesting in settings that are fairly standardized, such as supervised classification and regression. The second case would happen more often when one looks at novel workflows and implements novel machine learning tasks, such as adapting a large language model to a new downstream task. Hyperparameter optimization is one important aspect of AutoML, namely tuning the hyperparameters of the machine learning pipeline to the data at hand.

Why is this research important?

Machine learning is everywhere. Many organizations have access to large amounts of data and could potentially use them to make important decisions. Often, it is more important to thoroughly understand the problem and improve the data collection process than to tinker with the model(s) to improve the performance of the ML solution, something where domain experts shine. Helping them with the modeling will make it easier to unlock the potential of machine learning. Also, in many cases, it is important to not only target a single objective function, but rather multiple ones, such as power consumption or fairness in the predictions, a use-case where humans are not the best in finding solutions with optimal tradeoffs..

How can one handle such cases?

We use specialized multi-objective optimization algorithms in hyperparameter optimization. Multi-objective optimization finds solutions with different tradeoffs between such objectives, and where it cannot further decide which solution is better. At this point, a human user needs to decide this based on experience or policies. It is important to realize one’s own specialization and limitations. I know how to tune hyperparameters, but I do not claim to be an expert in topics such as fairness. Therefore, it is important to collaborate with domain experts. In a recent project, we thought about how to integrate fairness-related aspects into our AutoML methods, but in the course of the project, we realized that this is a much more complex task. We then joined forces with experts in the fairness domain and published our findings in a paper titled “Can Fairness be Automated”, in which we highlight the pitfalls when trying to “solve” fairness with automated machine learning.

What are currently the most important topics in Automated Machine Learning?

The most important topic is the integration of AutoML and foundation models. The most obvious question is how we can use AutoML methods to improve the training of foundation models. However, this is an extremely costly endeavor as training a single large language model costs millions of dollars. Therefore, AutoML researchers investigate topics using smaller models or adapting foundation models, for example, via fine-tuning or prompt optimization, a topic on which we just published a paper at the AutoML conference. But the integration can also happen the other way around, and foundation models, mostly large language models, can be used to improve the AutoML process, and for example, provide a no-code user interface to machine learning workflows. Other, hot research directions are agentic workflows, in which several so-called agents work together to solve machine learning tasks autonomously, or incorporate contextual knowledge that would be hard to integrate into the machine learning workflow using more traditional AutoML methods.

Are there topics besides AutoML that you work on?

Yes! One problem every machine learning researcher faces is the proper empirical evaluation of their methods. Since the beginning of my PhD, I’ve been contributing to the OpenML (Open Machine Learning) project and, over time, became a core contributor and also a founding member of the Open Machine Learning Foundation. OpenML facilitates the sharing of machine learning experimentation artifacts, such as datasets, tasks, and the outcome of running models on tasks. My main contribution was the Python API, which got quite some traction and is now used in many successful projects in AutoML, but also in more general machine learning research, to share datasets and tasks in a reproducible manner. We have recently published an article in the Patterns journal reflecting on the last ten years of running OpenML.org, where we also give an overview of the current state of the platform. And of course, we always look for new contributors who want to improve machine learning from the evaluation side.

That sounds very interesting, can you give us an example of how OpenML is used?

Of course. OpenML allows sharing of datasets, and a very successful stream of recent works on deep learning for tabular data have been using OpenML as their primary data source. For example, we have reviewed the usage of datasets in tabular machine learning and uploaded all datasets used in previous studies to OpenML so that future researchers have easier access to the data (available at OpenReview, note from the editors). Also, recently, colleagues from the Universities of Freiburg and Mannheim as well as Amazon started a project called TabArena (available at tabarena.ai, note from the editors) with the goal of having an open community-based leaderboard for tabular machine learning models, and this is also based on OpenML.

You are going to become a junior professor for automated machine learning and optimization at TU Dortmund in the upcoming semester. First of all, congratulations. Can you tell us a bit how the Thomas Bayes Fellowship helped you secure this position?

The Thomas Bayes Fellowship allowed me to experience an exciting new research environment at LMU Munich with access to an incredible network of fellow researchers through the MCML, which enabled me to continue my research on AutoML. It gave me the opportunity to hire a PhD student via the MCML matchmaking, gain experience leading PhD students, and conduct research independently. Since the fellowship is awarded as a temporary professorship, I also gained plenty of teaching experience. So it really helped me gain the necessary skills for my next step.

Can you also give some advice to junior scientists?

Always release the code to reproduce the results of your publications, and ideally, also make it easy for others to use the proposed methods themselves. It doesn’t have to grow into a large project, as the AutoML toolkit Auto-sklearn for which I was the main developer during my PhD, but also smaller projects can have lots of impact. For example, together with three extremely motivated Master’s students, I recently published a paper on prompt optimization, and the students went out of their way and created nicely reusable code that other researchers used to optimize prompts in their own paper, so the method now has an impact beyond the publication.