Less common is online distillation, which simultaneously trains both the teacher and the student models via a single end-to-end training process. Finally, self-distillation involves one model acting as both teacher and student. The model essentially trains its “shallow” layers (those that are closer to the input layer and…
Read More
