CRAFT Your Dataset: Task-Specific Synthetic Dataset Generation Through Corpus Retrieval and Augmentation

MCML Authors

Abdullatif Köksal

* Former Member

→ Group Hinrich Schütze
 Computational Linguistics

Hinrich Schütze

Prof. Dr.

Principal Investigator

Computational Linguistics

Abstract

Building high-quality datasets for specialized tasks is a time-consuming and resource-intensive process that often requires specialized domain knowledge. We propose Corpus Retrieval and Augmentation for Fine-Tuning (CRAFT), a method for generating synthetic datasets, given a small number of user-written few-shots that demonstrate the task to be performed. Given the few-shot examples, we use large-scale public web-crawled corpora and similarity-based document retrieval to find other relevant human-written documents. Lastly, instruction-tuned large language models (LLMs) augment the retrieved documents into custom-formatted task samples, which then can be used for fine-tuning. We demonstrate that CRAFT can efficiently generate large-scale task-specific training datasets for four diverse tasks: biology question-answering (QA), medicine QA and commonsense QA as well as summarization. Our experiments show that CRAFT-based models outperform or achieve comparable performance to general LLMs for QA tasks, while CRAFT-based summarization models outperform models trained on human-curated data by 46 preference points.

article

---

Transactions of the Association for Computational Linguistics

Nov. 2025. To be published. Preprint available.

---

Authors

I. Ziegler • A. Köksal • D. Elliott • H. Schütze

---

Links

---

Research Area

 B2 | Natural Language Processing