Problem Statement

Calculate the Shannon entropy   H   of a given input string. Given the discrete random variable

X

{\displaystyle X}

that is a string of

N

{\displaystyle N}

"symbols" (total characters) consisting of

n

{\displaystyle n}

different characters (n=2 for binary), the Shannon entropy of X in bits/symbol is : where

c o u n

t

i

{\displaystyle count_{i}}

is the count of character

n

i

{\displaystyle n_{i}}

. For this task, use X="1223334444" as an example. The result should be 1.84644... bits/symbol. This assumes X was a random variable, which may not be the case, or it may depend on the observer. This coding problem calculates the "specific" or "intensive" entropy that finds its parallel in physics with "specific entropy" S0 which is entropy per kg or per mole, not like physical entropy S and therefore not the "information" content of a file. It comes from Boltzmann's H-theorem where

S

k

B

N H

{\displaystyle S=k_{B}NH}

where N=number of molecules. Boltzmann's H is the same equation as Shannon's H, and it gives the specific entropy H on a "per molecule" basis. The "total", "absolute", or "extensive" information entropy is This is not the entropy being coded here, but it is the closest to physical entropy and a measure of the information content of a string. But it does not look for any patterns that might be available for compression, so it is a very restricted, basic, and certain measure of "information". Every binary file with an equal number of 1's and 0's will have S=N bits. All hex files with equal symbol frequencies will have

S

N

log

2

⁡ ( 16 )

{\displaystyle S=N\log _{2}(16)}

bits of entropy. The total entropy in bits of the example above is S= 10*18.4644 = 18.4644 bits. The H function does not look for any patterns in data or check if X was a random variable. For example, X=000000111111 gives the same calculated entropy in all senses as Y=010011100101. For most purposes it is usually more relevant to divide the gzip length by the length of the original data to get an informal measure of how much "order" was in the data. Two other "entropies" are useful: Normalized specific entropy: which varies from 0 to 1 and it has units of "entropy/symbol" or just 1/symbol. For this example, Hn<\sub>= 0.923. Normalized total (extensive) entropy: which varies from 0 to N and does not have units. It is simply the "entropy", but it needs to be called "total normalized extensive entropy" so that it is not confused with Shannon's (specific) entropy or physical entropy. For this example, Sn<\sub>= 9.23. Shannon himself is the reason his "entropy/symbol" H function is very confusingly called "entropy". That's like calling a function that returns a speed a "meter". See section 1.7 of his classic A Mathematical Theory of Communication and search on "per symbol" and "units" to see he always stated his entropy H has units of "bits/symbol" or "entropy/symbol" or "information/symbol". So it is legitimate to say entropy NH is "information". In keeping with Landauer's limit, the physics entropy generated from erasing N bits is

S

H

2

N

k

B

ln ⁡ ( 2 )

{\displaystyle S=H_{2}Nk_{B}\ln(2)}

if the bit storage device is perfectly efficient. This can be solved for H2*N to (arguably) get the number of bits of information that a physical entropy represents.

Let's start with the solution:

: flog2 ( f -- f ) fln 2e fln f/ ;

create freq 256 cells allot

: entropy ( str len -- f )
  freq 256 cells erase
  tuck
  bounds do
    i c@ cells freq +
    1 swap +!
  loop
  0e
  256 0 do
    i cells freq + @ ?dup if
      s>f dup s>f f/
      fdup flog2 f* f-
    then
  loop
  drop ;

s" 1223334444" entropy f.     \ 1.84643934467102  ok