Shannon Entropy

Noun · Development

Definitions

  1. A measure of the average information content (or unpredictability) in a message, measured in bits. Higher entropy means more randomness and less compressibility. Used in data compression (approaching entropy = optimal), cryptography (key strength), and machine learning (cross-entropy loss).

    In plain English: A measure of how much surprise or randomness is in a message — more entropy means it's harder to predict and compress.

    Example: "The English language has about 1.5 bits of entropy per character — that's why text compresses so well compared to random data."

Etymology

1948
Claude Shannon published 'A Mathematical Theory of Communication,' founding information theory and defining entropy.

Related Terms