Tokenization in Large Language Models

MCML Authors

Valentin Hofmann

Prof. Dr. Dr.

Core PI

Information and Language Processing Using AI Methods

Abstract

Tokenization defines how data is represented as input and output for machine learning systems, including large language models (LLMs). While tokenization has long been a foundational component of NLP systems, it has historically remained a niche research area, often considered solved. The recent proliferation of LLMs has changed this: as these models are deployed across diverse languages and domains, tokenization-related issues have become increasingly apparent—from cross-lingual representation disparities to efficiency bottlenecks. This tutorial will provide a comprehensive overview of tokenization methods in text, starting from classical word-based and morphological approaches, to the current standard of statistical subword tokenizers, and emerging methods on end-to-end learnable tokenizers. We will also discuss the implications of tokenization choices on model performance, fairness, and efficiency, and present best practices for evaluating and designing tokenizers in multilingual and domain-specific settings.

inproceedings HKK+26a

---

Tutorials @EMNLP 2026

3rd Workshop of Natural Language Processing meets Climate Change at the Conference on Empirical Methods in Natural Language Processing. Budapest, Hungary, Oct 24-29, 2026. To be published. Preprint available.

---

Authors

V. Hofmann • J. Kallini • S. Kumar • J. Libovický • T. Limisiewicz • E. Salesky

---

Links

URL

---

Research Area

 B2 | Natural Language Processing