Consider the conditional entropy and mutual information for the binary…
2015
Consider the conditional entropy and mutual information for the binary symmetric channel. The input source has alphabet \(𝑋={0,1}\) and associated probabilities \({\dfrac{1}{2}, \dfrac{1}{2}}\). The channel matrix is \(\begin{pmatrix} 1-p & p \\ p & 1-p \end{pmatrix}\) wgere \(p\) is the transition probability. Then the conditional entropy is given by:
- A.
\(1\) - B.
\(-p \log(p)-(1-p) \log(1-p)\) - C.
\(1+p \log(p)+(1-p) \log(1-p)\) - D.
\(0\)
Attempted by 45 students.
Show answer & explanation
Correct answer: B
Compute the conditional entropy H(Y|X):
For X = 0, the output Y is 0 with probability 1-p and 1 with probability p, so H(Y|X=0) = - (1-p) log(1-p) - p log p.
For X = 1, the output distribution is the same, so H(Y|X=1) = - (1-p) log(1-p) - p log p.
With P(X=0)=P(X=1)=1/2, H(Y|X) = sum_x P(X=x) H(Y|X=x) = 1/2 H(p) + 1/2 H(p) = H(p).
Therefore, the conditional entropy is -p log(p) - (1-p) log(1-p)
Note: If logarithms are base 2, this is measured in bits. For an equiprobable input, the mutual information equals 1 - H(p).