Solved – Does statistically insignificant difference of means imply equality of means

Suppose I have sets A and B of normally distributed data so that:

 A: mean=250, SD=200, N=25   B: mean=248, SD=200, N=20 

Clearly, there is no statistically significant difference between the means (p=0.9736). But does this mean that the means are equal or that we have no evidence to suggest otherwise? Considering the huge standard deviation it seems intuitively unlikely that the means are equal. But this is what the null hypothesis says and based on the the t-test we didn't reject that.

Having such noisy data, how can one quantify how equal the means are?

This question arose from a discussion between myself and a colleague who was testing the effect of two drug treatments on biological samples. As the test revealed an insignificant difference between treatments, my colleague assumed this allows him to claim that the treatments have the same effect. With the example above I tried to show that this is not necessarily the case. But I didn't know the proper terminology to make things more quantitative. Hence this question.

Formally, no. Your conclusion is only that you "cannot reject" the null that they are equal. They might still be different! and more data could, maybe, show that.

To say more than such formalities, we need to know more about the context. In your applied setting, with your intended use of the results, how large must a difference be to have any practical significance? To help getting into that mindset, it is better for you to calculate a confidence interval (say, 95%) on the difference of the means. If all the values within that interval are small enough for you (and your readers) to conclude "such a difference is to small to have any real import", then, and first then, can you conclude that the means are practically equal.

Empirical measurements can never decide mathematical equality, that is, identity!

As another commenter says, if you really want to prove equality (alternatively, "non-inferiority" could be the relevant question to ask), there is equivalence testing (to prove that your new drug is equivalent to a competitor) or non-inferiority testing (to prove that it is equivalent or better).

Similar Posts:

Rate this post

Leave a Comment