How does information theory help determine the maximum possible compression of text?
Consider a simple telephone conversation: A person (message source) speaks into a telephone receiver (encoder), which converts the sound of the spoken word into an electrical signal. This electrical signal is then transmitted over telephone lines (channel) subject to interference (noise). When the signal reaches the telephone receiver (decoder) at the other end of the line it is converted back into vocal sounds. Finally, the recipient (message receiver) hears the original message.
In information theory, the maximum theoretical compression of text is determined by the concept of entropy. Entropy serves as a mathematical measure of the efficiency of an encoding scheme, representing the lower bound on the average number of bits required to represent symbols from a given source.1
If the entropy of a message alphabet is lower than the average length of its current encoding, further compression is mathematically possible. For instance, while standard coding often uses 8 bits per character, the entropy of large samples of English text is approximately 4 bits per character, suggesting that an encoding twice as efficient is theoretically achievable.1
In practice, compression algorithms approach these limits by modeling the probability distribution of characters or sequences. While static models like Huffman codes rely on known frequencies, adaptive methodsâsuch as the Lempel-Ziv algorithmâdynamically build dictionaries based on the text itself to identify and replace repetitive patterns with shorter codes.123