Home | Publications | SFS26

BoundKD: Distillation as Constraint Satisfaction Under Sampled Supervision

MCML Authors

Abstract

We study black-box distillation: training a compact student from a teacher that returns only samples, never logits. From N samples per context, given the teacher's candidate tokens, we compile certified lower bounds on the teacher's pairwise token preferences, train the student to satisfy each bound as a constraint (bound-constrained knowledge distillation, BoundKD), and reuse the same constraints after training as an audit. Across six teacher-student pairs at 3× to 98× compression, BoundKD improves every ranking metric in every run on a 98× smaller student (Pythia-6.9B to 70M), ahead of sampled-KL throughout, and never raises held-out perplexity more than 2% above baseline at the smallest budget, while regressing onto the same bounds with Margin-MSE inflates perplexity up to 443×. A certification-ceiling lemma explains why: every sample budget caps the margin it can certify, so the bounds systematically understate the teacher's confident preferences, and regression trains the student to reproduce that error.

inproceedings SFS26


LIGHT @NeurIPS 2026

Workshop on Deployable Small Foundation Models at the 40th Conference on Neural Information Processing Systems. Sydney, Australia, Dec 06-12, 2026. To be published. Preprint available.

Authors

R. Srivastava • P. Fatemi • S. Sra

Links

arXiv

Research Area

 A2 | Mathematical Foundations

BibTeXKey: SFS26

Back to Top