Solved – Maximum Likelihood Estimation

If $V_1, V_2,ldots V_{n_1}$ and $W_1, W_2,ldots,W_{n_2}$ are independent random samples of size $n_1$ and $n_2$ from normal populations with the means $mu_1$, $mu_2$ and the common variance $sigma^2$, find maximum likelihood estimators for $mu_1, mu_2$ and $sigma^2$.

My idea is to separately find FOC for sample 1 and for sample 2 and then use the 4 equations I'll found to estimate the parameters. Any suggestions? Furthermore: is it a problem the difference in sample size?


According to the question, it is a an assumed fact that both populations have common variance, and not something one wishes to test. Maximum likelihood estimators can be derived as usual either from the two samples separately, or by pooling them, in which case we will have an independent but non-identically distributed sample and corresponding log-likelihood, something that nevertheless creates no special issues. So, more than deriving the MLEs (which is straightforward), I would say that this is a good example in order to examine whether pooling samples ("unite and conquer"?) is more beneficial than keeping the samples separate ("divide and conquer"?). But "more beneficial" according to which criteria?

We will discuss them as we go along.

Note that we need both sample sizes to be larger than unity, $n_1 >1, n_2 > 1$, otherwise the variance estimator will equal zero.

If we keep the samples separate we will obtain

$$hat mu_v = frac 1{n_1}sum_{i=1}^{n_1}v_i,;;; hat sigma^2_1 = frac 1{n_1}sum_{i=1}^{n_1}(v_i-hat mu_v)^2$$

and $$hat mu_w = frac 1{n_2}sum_{i=1}^{n_2}w_i,;;; hat sigma^2_2 = frac 1{n_2}sum_{i=1}^{n_2}(w_i-hat mu_w)^2$$

The MLEs for the means will be unbiased, efficient, consistent and asymptotically normal.

The variance estimators will be biased, consistent and asymptotically normal (see this post, which holds in general, even for normal samples).

Since we have bias here, it is an easy thought to turn to Mean Squared Error. The populations are normal, so we also have a finite-sample result:

$$frac {n_ihat sigma^2_i}{sigma^2} sim chi^2_{n_i-1} Rightarrow hat sigma^2_i sim operatorname{Gamma}(k_i,theta_i),;; k_i = frac {n_i-1}{2},;; theta_i = frac {2sigma^2}{n_i},;;i=1,2$$

Therefore we can calculate the Mean Squared Error (MSE) as

$$MSE(hat sigma^2_i) = text{Var}(hat sigma^2_i)+left[B(hat sigma^2_i)right]^2 = frac{2(n_i-1)}{n_i^2} sigma^4 + frac 1{n_i^2}sigma^4 = frac{2n_i-1}{n_i^2} sigma^4$$

We turn now to the pooled-samples case.
It is easy to verify that the MLE's for the two means will be identical with the separate-samples approach. So as regards these estimators, pooling the two samples or not, makes no difference as regards the functional form of the estimators, or their properties.

But the variance estimator will be different. It is also rather easy to derive that

$$hat sigma^2_p = frac{n_1}{n_1+n_2}hat sigma^2_1+frac{n_2}{n_1+n_2}hat sigma^2_2$$

This is also a biased an consistent estimator, and also asymptotically normal, being the convex combination of two asymptotically normal variables.

Turning to the issue of bias and Mean Squared Error, since the two separate-samples estimators are independent we have that

$$text{Var}(hat sigma^2_p) = frac{n_1^2}{(n_1+n_2)^2}frac{2(n_1-1)}{n_1^2} sigma^4+frac{n_2^2}{(n_1+n_2)^2}frac{2(n_2-1)}{n_2^2}sigma^4 = frac {2n_1+2n_2-4}{(n_1+n_2)^2}sigma^4$$


$$Bleft(hat sigma^2_pright) = frac{n_1}{n_1+n_2}E(hat sigma^2_1)+frac{n_2}{n_1+n_2}E(hat sigma^2_2) – sigma^2 = frac {-2}{n_1+n_2} sigma^2$$

So the MSE here is

$$MSE(hat sigma^2_p) = frac {2n_1+2n_2-4}{(n_1+n_2)^2}sigma^4+frac {4}{(n_1+n_2)^2} sigma^4 = frac {2}{n_1+n_2}sigma^4$$

In order for sample-pooling to be superior in MSE terms we want that

$$MSE(hat sigma^2_p) < MSE(hat sigma^2_i), i=1,2$$

$$Rightarrow frac {2}{n_1+n_2}sigma^4 < frac{2n_i-1}{n_i^2} sigma^4 Rightarrow 2n_i^2 < 2n_in_1 – n_1 + 2n_in_2 – n_2$$

This reduces to the same condition for either $i=1$ or $i=2$, namely $$0 < – n_1 + 2n_1n_2 – n_2 Rightarrow frac {n_1+n_2}{n_1n_2} < 2 Rightarrow frac 1{n_2} + frac {1}{n_1} < 2$$

which holds, since both sample sizes are strictly higher than unity.

Therefore we conclude, that "unite & conquer" is the MSE-efficient approach here.

But we will lose something: if $n_1 neq n_2$ the pooled-sample variance estimator does not give a Gamma finite sample distributional result, because it is the linear combination of two Gamma random variables with different scale parameters (different $theta_i$'s). This does not result into a Gamma, but into a rather complicated infinite sum expression (see this paper). Which means that for conducting tests related to the pooled-sample variance estimator, we will have to resort to the asymptotic normality result.

Alternatively, if the difference between $n_1$ and $n_2$ is not large, and both samples have respectable sizes, we may even consider dropping observations from the larger sample in order to make $n_1 =n_2$ and preserve the Gamma distribution result.

Similar Posts:

Rate this post

Leave a Comment