LogicSkills: A Structured Benchmark for Formal Reasoning in Large Language Models

MCML Authors

Philipp Mondorf

→ Group Barbara Plank
 AI and Computational Linguistics

Barbara Plank

Prof. Dr.

Core PI

AI and Computational Linguistics

Abstract

Large language models perform well on many logical reasoning benchmarks, but it remains unclear which core logical skills they truly master. To address this, we introduce LogicSkills, a benchmark that isolates three fundamental logical skills: (i) formal symbolizationunicode{x2013}translating premises into first-order logic; (ii) countermodel constructionunicode{x2013}showing that an argument is logically invalid by constructing a finite countermodel; and (iii) validity assessmentunicode{x2013}determining whether a conclusion follows from a set of premises. Items are drawn from the two-variable fragment of first-order logic without identity and are presented in both English and a Carrollian nonce-word language. All instances are solver-verified with Z3 for correctness and non-triviality. Across conventional instruction-tuned LLMs, performance is high on validity assessment but substantially lower on formal symbolization and countermodel construction, highlighting that high task-level accuracy can mask weaknesses in core logical skills. In contrast, recent reasoning-tuned models perform strongly across all three tasks, suggesting a more systematic logical skill profile.

inproceedings RMP26

---

Findings @EMNLP 2026

Findings of the Conference on Empirical Methods in Natural Language Processing. Budapest, Hungary, Oct 24-29, 2026.

---

Authors

B. Rabern • P. Mondorf • B. Plank

---

Links

DOI

---

Research Area

 B2 | Natural Language Processing