23.07.2025
How Reliable Are Machine Learning Methods? With Anne-Laure Boulesteix and Milena Wünsch
Research Film
Often a new machine learning method claims to outperform the last. Whether it’s in bioinformatics, finance, or image recognition, the message is the same: this algorithm is faster, more accurate, more powerful. But can we trust those claims?
«It’s not just about the algorithms. It’s about how we compare them—and what we choose to report or ignore.»
Milena Wünsch
MCML Junior Member
Beneath the surface of many benchmarking studies lies a quiet problem: subtle biases that skew comparisons and inflate performance. These issues often go unnoticed — but they can have real consequences, especially when such models are used to inform research or high-stakes decisions.
«It doesn’t matter whether the bias is deliberate or not. It still shapes how methods are judged and used.»
Anne-Laure Boulesteix
MCML PI
Anne-Laure Boulesteix, Professor of Biometry at LMU and MCML PI, and Milena Wünsch, PhD student at LMU and MCML, study how seemingly harmless methodological choices can lead to misleading results.
One common issue: when a method fails on a dataset, researchers may simply drop it from the analysis. While convenient, this can introduce bias and overstate performance.
Bias can also arise from less obvious sources — like spending more time tuning one method, being more familiar with a tool, or unconsciously interpreting results in its favor.
With so many studies promoting the “next best” algorithm, it’s hard to know which results to trust. Researchers may end up using a method that only looked good due to biased comparisons. Still, the researchers are hopeful. In recent years, the methodological machine learning community has made real progress — pushing for better standards, more transparency, and more careful benchmarking.
©MCML
The film was produced and edited by Nicole Huminski and Nikolai Huber.
23.07.2025
Related
©LMU/Stephan Höck
23.07.2025
Philipp Wicke Explores Embodied AI
MCML Junior Member Philipp Wicke explores how embodiment shapes intelligence in a new LMU research film on the role of the body in AI.
22.07.2025
Eyke Hüllermeier to Lead New DFG-Funded Research Training Group METEOR
MCML PI Eyke Hüllermeier to lead new DFG-funded RTG METEOR, uniting ML and control theory to build robust, explainable AI systems.
16.07.2025
AI-Powered Cortical Mapping for Neurodegenerative Disease Diagnoses - With Christian Wachinger
Research film with Christian Wachinger shows how AI maps the brain’s cortex to support diagnoses of neurodegenerative diseases.
©TUM
15.07.2025
ERC Proof of Concept Grants for Fabian Theis
Fabian Theis receives ERC Proof of Concept Grants for his project on deep learning methods for dynamic single-cell data analysis.
