knowledge distillation

References & Edit History Quick Facts & Related Topics

Images

Diagram illustrating knowledge transfer in machine learning, showing a complex teacher neural network model passing distilled knowledge to a simpler student model, both using the same data source.

Diagram showing three knowledge distillation methods: offline distillation with a pre-trained teacher model transferring knowledge to a student, online distillation with both models trained together, and self-distillation where a model teaches itself. Color coding distinguishes pre-trained and to-be-trained models.

knowledge distillation: Media

computer science

Images

Diagram illustrating knowledge transfer in machine learning, showing a complex teacher neural network model passing distilled knowledge to a simpler student model, both using the same data source.

Knowledge transfer from a teacher model to a student model

Knowledge distillation helps replicate the performance of a large model on a smaller...

Encyclopædia Britannica, Inc.

Diagram showing three knowledge distillation methods: offline distillation with a pre-trained teacher model transferring knowledge to a student, online distillation with both models trained together, and self-distillation where a model teaches itself. Color coding distinguishes pre-trained and to-be-trained models.

Methods of knowledge distillation

The three KD methods are offline distillation, online distillation, and self-distillation.

Encyclopædia Britannica, Inc.