knowledge distillation: Quick Facts

Written and fact-checked by

The Information Architects of Encyclopaedia Britannica

Facts

Also Known As	model distillation • KD

Photos

Diagram illustrating knowledge transfer in machine learning, showing a complex teacher neural network model passing distilled knowledge to a simpler student model, both using the same data source.

Diagram showing three knowledge distillation methods: offline distillation with a pre-trained teacher model transferring knowledge to a student, online distillation with both models trained together, and self-distillation where a model teaches itself. Color coding distinguishes pre-trained and to-be-trained models.

See All Images →

Topics

large language model neural network machine learning cross-validation What is machine learning?