knowledge distillation

knowledge distillation (KD), process in machine learning and deep learning for replicating the performance of a large model or set of models on a smaller model. The process is especially useful in the context of transferring learning techniques to smaller, more efficient models from large language models (LLMs), such as ChatGPT and Google Gemini.

Background

Knowledge distillation (KD) techniques are used to improve complex neural network models, which may have multiple layers and learnable parameters. Such models have been used since the early 2010s across a range of applications, including image recognition, natural language processing (NLP), and speech recognition. KD gained increased attention for its importance in the development of LLMs, which entered widespread public use in the early 2020s.

The sheer size and complexity of an LLM complicates its use. Compared with a smaller model, an LLM needs more computational resources to run, has greater inference times, and is harder to manage. Indeed, the money, time, memory, and other assets that an LLM requires make its use impractical in many applications. Training a smaller, specialist model to match the abilities of a generalist LLM effectively solves such issues.

Discover how ordinary objects work, and expand your understanding of the familiar.

SIGN UP

" height="20" alt="HEW Logo">

Thank you for signing up!"

Back to homepage

By clicking "Sign Up", you consent to the Britannica Privacy Policy and Terms of Use.

Initial steps

A neural network conventionally “learns” by using a loss function, which produces a vector that represents the divergence (loss) between a model’s outputs and the correct outputs across various inputs. Correct outputs are typically referred to as “ground truth.” The model is trained by adjusting its parameters, or the weights or biases that are applied to its mathematical operations, to minimize the gradient (or change over time) of this vector, thereby bringing the model’s outputs closer to the ground truth. A network’s learned parameters are thought of as its “knowledge.” KD is different: “knowledge” is redefined as the steps by which a network reasons its way from its input vectors to its output vectors—i.e., as how the network generalizes to new data.

KD transfers these reasoning steps from larger (“teacher”) models to smaller (“student”) models by introducing a new function that measures distillation loss. To understand distillation loss, consider the following. A neural network’s outputs may be understood as predictions. For instance, an image classification model will predict the category of a specified image, such as that of a horse. The model’s final prediction—its answer to the question it has been posed—is called a “hard target.” Prior to arriving at its final prediction, the model will make multiple preliminary predictions, assigning to each a probability of being correct. These intermediate predictions from which the final prediction is chosen are sometimes called “soft targets” but are more commonly called “logits.”

Perhaps counterintuitively, how the model arrives at its logits is more-valuable information than how the model decides between logits when choosing its hard target. This is because the former process provides a great deal more insight into how the teacher model generalizes. For example, an image classification model is more likely to incorrectly categorize a horse as a cow than as a building. A student model can train on fewer examples of how its teacher model reaches logits than examples of how it reaches hard targets.

Measuring ability to generalize

A student model measures its ability to generalize in comparison with the teacher model via two functions. The first function operates on “hard loss,” or the divergence between the student model’s hard target and the ground truth. The second function operates on aforementioned distillation loss, also known as “soft loss” because it measures the divergence in logits between the student model and the teacher model. Specifically, it measures the difference between the probability distribution of the student model’s logits and that of its teacher’s logits.

Explore Britannica Premium!

The trusted destination for professionals, college students, and lifelong learners.

SUBSCRIBE

Although logits are the primary focus of knowledge transfer between teacher and student models, there are other forms of knowledge that require consideration. All forms of knowledge in neural networks can generally be placed in one of three categories: response-based knowledge, feature-based knowledge, or relation-based knowledge. Logits constitute response-based knowledge, since those generated by teacher and student models are compared for training purposes. Feature-based knowledge relates to a model’s discrimination of features at each reasoning step (or layer) of its logit generation. One might describe logit-based knowledge as concerned with the “what” and feature-based knowledge as concerned with the “how.” Finally, relation-based knowledge involves the relationships or correlations between the model’s layers and feature activations. This can describe how the student copies the teacher’s “thought process.”

Methods of knowledge distillation

KD can also be classified according to the method of teaching a student model. There are three distinct KD methods: offline distillation, online distillation, and self-distillation. Each method differs from the others on the basis of whether the teacher model is modified at the same time as the student model. The most common KD technique is offline distillation, wherein the teacher model is pretrained on a dataset and then has its knowledge distilled into the student model.

Less common is online distillation, which simultaneously trains both the teacher and the student models via a single end-to-end training process. Finally, self-distillation involves one model acting as both teacher and student. The model essentially trains its “shallow” layers (those that are closer to the input layer and hence learn only basic features) with its “deeper” layers (those that are closer to the output layer and hence deal with representations that are more complex and abstract).