Why are frequently used words often shorter than less common words?
Consider a simple telephone conversation: A person (message source) speaks into a telephone receiver (encoder), which converts the sound of the spoken word into an electrical signal. This electrical signal is then transmitted over telephone lines (channel) subject to interference (noise). When the signal reaches the telephone receiver (decoder) at the other end of the line it is converted back into vocal sounds. Finally, the recipient (message receiver) hears the original message.
In the context of information theory, the tendency for frequently used words to be shorter than less common ones is a strategy that promotes greater communication efficiency.1
By assigning the shortest sequences to the most common words, a language minimizes the average length of its messages, which is a fundamental principle in data compression. While it is uncertain to what extent this phenomenon is driven by a "principle of least effort," it aligns with the goal of optimizing encoding efficiency.12
This observation is closely related to Zipf’s law, which states that the frequency of a word is inversely proportional to its rank. Because the most frequent words occur much more often than others, using short representations for them significantly reduces the overall resources required to transmit or store information.31