I know that Shannon entropy is defined as $-sum_{i=1}^kp_ilog(p_i)$. For the uniform distribution, $p_i=frac{1}{k}$, so this becomes $-sum_{i=1}^kfrac{1}{k}logleft(frac{1}{k}right)$.
Further rearrangement produces the following:
$-sum_{i=1}^kfrac{1}{k}log(k)^{-1}$
$sum_{i=1}^kfrac{1}{k}log(k)$
This is where I am stuck. I need the solution to come to $log(k)$. What is the next step?
Best Answer
Using Lagrange multipliers we have the equation:
$$mathcal{L} = left { -sum_i^k p_i log p_i – lambdaleft ( sum_i^k p_i – 1 right )right }$$
Maximizing with respect to the probability,
$$frac{partial mathcal{L}}{partial p_i} = 0 = -log p_i – 1 – lambda implies $$
$$p_i = e^{-(1+lambda)}tag{1}$$
Maximizing with respect to $lambda$:
$$frac{partial mathcal{L}}{partial lambda} = 0 = – sum_i^k p_i + 1 implies$$
$$ sum_i^k p_i = 1 tag{2}$$
Substituting equation (1) into equation (2):
$$sum_i^k e^{-(1+lambda)} = 1 implies$$
$$k e^{-(1+lambda)} = 1 $$
Since $p_i = e^{-(1+lambda)}$
$$p_i = frac{1}{k}$$
The Shannon Entropy formula now becomes
$$ H = – sum_i^k frac{1}{k}log frac{1}{k}$$
Since $k$ does not depend on the summation,
$$H = frac{k}{k} log k = log k$$
Similar Posts:
- Solved – MLE for normal distribution with restrictive parameters
- Solved – Derivation of M step for Gaussian mixture model
- Solved – Difference between different kinds of entropy
- Solved – Gradients of cross-entropy error in neural network
- Solved – expected value of a score function (the gradient of the log-likelihood function)