I know that Shannon entropy is defined as $-sum_{i=1}^kp_ilog(p_i)$. For the uniform distribution, $p_i=frac{1}{k}$, so this becomes $-sum_{i=1}^kfrac{1}{k}logleft(frac{1}{k}right)$.

Further rearrangement produces the following:

$-sum_{i=1}^kfrac{1}{k}log(k)^{-1}$

$sum_{i=1}^kfrac{1}{k}log(k)$

This is where I am stuck. I need the solution to come to $log(k)$. What is the next step?

**Contents**hide

#### Best Answer

Using Lagrange multipliers we have the equation:

$$mathcal{L} = left { -sum_i^k p_i log p_i – lambdaleft ( sum_i^k p_i – 1 right )right }$$

Maximizing with respect to the probability,

$$frac{partial mathcal{L}}{partial p_i} = 0 = -log p_i – 1 – lambda implies $$

$$p_i = e^{-(1+lambda)}tag{1}$$

Maximizing with respect to $lambda$:

$$frac{partial mathcal{L}}{partial lambda} = 0 = – sum_i^k p_i + 1 implies$$

$$ sum_i^k p_i = 1 tag{2}$$

Substituting equation (1) into equation (2):

$$sum_i^k e^{-(1+lambda)} = 1 implies$$

$$k e^{-(1+lambda)} = 1 $$

Since $p_i = e^{-(1+lambda)}$

$$p_i = frac{1}{k}$$

The Shannon Entropy formula now becomes

$$ H = – sum_i^k frac{1}{k}log frac{1}{k}$$

Since $k$ does not depend on the summation,

$$H = frac{k}{k} log k = log k$$

### Similar Posts:

- Solved – MLE for normal distribution with restrictive parameters
- Solved – Derivation of M step for Gaussian mixture model
- Solved – Difference between different kinds of entropy
- Solved – Gradients of cross-entropy error in neural network
- Solved – expected value of a score function (the gradient of the log-likelihood function)