Information Theory and Coding > Entropy > What is entropy in Information Theory?

Let X be a discrete random variable with alphabet X and probability mass function

                                                          p(x) = Pr{X = x}, x ∈ X

The entropy is a measure of the average uncertainty in the random variable. It is the number of bits on average required to describe the random variable.

The entropy of a random variable X with a probability mass function p(x) is defined by

If the base of the logarithm is 2 then it is called that the information is measured in bits. If base of the logarithm is e, then it is called measured in nats. If base of the logarithm is 10, then it is called measured in Hartley / decit.

 

Example: In a coin tossing which results 'head' and 'Tail', assume a random variable x is defined as  x=1 for head with probability p and x=0 for tail with probability (1-p). Then entropy H(x) = -p log p – (1-p) log(1-p). We can calculate entropy for different valy of p:

from math import log2
from matplotlib import pyplot
# list of possible probabilities
probs = [0.05, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 0.95]

# calculating entropy in array
entropy = [-log2(p)*p-log2(1-p)*(1-p) for p in probs]

# plot Probability vs Entropy
pyplot.plot(probs, entropy, marker='.')
pyplot.title('Probability vs Entropy')
pyplot.xlabel('Probability')
pyplot.ylabel('Entropy')
pyplot.show()

In particular, H (X) = 1 bit when p = 1/2 . The graph of the function H (p) is shown in Figure. The figure illustrates some of the basic properties of entropy:

• It is a concave function of the distribution and equals 0 when p = 0 or 1. This makes sense, because when p = 0 or 1, the variable is not random and there is no uncertainty.
• The uncertainty is maximum when p = 1/2 , which also corresponds to the maximum value of the entropy.

 

 

 

 

 

 

 

 

 

 

 

 

 

Using python codes :

from math import log2

def sum(arr):
sum = 0
for i in arr:
sum = sum + i
return(sum)

# list of possible probabilities
probs = [1/2, 1/4, 1/8, 1/16, 1/64, 1/64, 1/64,1/64]

# calculating entropy in array
ent_i= [-log2(p)*p for p in probs]
entropy= sum(ent_i)
print('Entropy is =%.2f' %entropy)

Entropy is =2.00

Prev

What Is Information Theory?

Next

Average Information Content of Symbols

Feedback

Submit