Home  | Publications | AKS+24

Introducing Wwm-German-18k - Can LLMs Crack the Million? (Or Win at Least 500 Euros?)

MCML Authors

Abstract

Language-specific evaluation of large language models (LLMs) for multiple-choice question answering (MCQA) is an important means to test their abilities for a multitude of different dimensions. With a data set assembled from questions from the German variant of 'Who Wants to Be a Millionaire?' we evaluate a set of German models and ChatGPT concerning factual/commonsense knowledge, syntactic abilities, and logical reasoning, amongst others. We contribute this new MCQA data set, extracted from the show’s episodes and designed to evaluate the ability of models to answer this diverse range of questions. To ensure data quality, we describe our preprocessing, encompassing data cleaning, deduplication, and the creation of stratified splits. Furthermore, we fine-tune a set of German LLMs and prompt ChatGPT to provide baseline results. Our findings reveal that these models achieve (partly) satisfactory performance on questions of lower difficulty levels (≤ 1000 euros). As the difficulty increases, performance steadily declines, highlighting the challenging nature of the later stages of the game. We contribute to the ongoing efforts to advance the capabilities of LLMs in comprehending and answering questions by providing a valuable resource for German MCQA research as well as further insights into the limitations of current LLMs.

inproceedings

ICNLSP 2024

7th International Conference on Natural Language and Speech Processing. Trento, Italy, Oct 19-20, 2024.

Authors

M. Aßenmacher • L. Karrlein • P. Schiele • C. Heumann

Links

URL

Research Area

 A1 | Statistical Foundations & Explainability

BibTeXKey: AKS+24

Back to Top