Shannon’s entropy

Shannon's entropy is a fundamental concept in information theory, a field pioneered by Claude Shannon. It quantifies the amount of uncertainty or randomness in a message or data source. Analogous to the concept of entropy in physics, which measures disorder, Shannon's entropy provides a measure of the efficiency of a communication system.123

This measure is calculated based on the statistical properties of the message source. Shannon's work established that this concept could be used to determine the most efficient way to encode messages and to understand the steps needed to transmit information reliably, even in the presence of noise. Information theory, and Shannon's entropy within it, has applications in various fields, including cryptography and physics.143