I have two samples ($n approx 70$ in both cases). The means differ by about twice the pooled std. dev. The resulting $T$ value is approximately 10. Whilst it's great to know that I have conclusively shown that the means are not the same, this seems to me to be driven by the large n. Looking at histograms of the data I certainly do not feel that such as small p-value is really representative of the data and to be honest don't really feel comfortable quoting it. I'm probably asking the wrong question. What I'm thinking is: ok, the means are different but does that really matter as the distributions share a significant overlap?

Is this where Bayesian testing is useful? If so where is a good place to start, a bit of googling hasn't yielded anything useful but I may not by asking the right question. If this is the wrong thing does anyone have any suggestions? Or is this simply a point for discussion as opposed to quantitative analysis?

**Contents**hide

#### Best Answer

Let $mu_1$ denote the mean of the first population and $mu_2$ denote the mean of the second population. It seems that you've used a two-sample $t$-test to test whether $mu_1=mu_2$. The significant result implies that $mu_1neqmu_2$, but the difference seems to be to small to matter for your application.

What've you encountered is the fact that *statistically significant* often can be something else than *significant for the application*. While the difference may be statistically significant it may still not be *meaningful*.

Bayesian testing won't solve that problem – you'll still just conclude that a difference exists.

There might however be a way out. For instance, for a one-sided hypothesis you could decide that if $mu_1$ is $Delta$ units greater than $mu_2$ then that would be a meaningful difference that is large enough to matter for your application.

In that case you would test whether $mu_1-mu_2leq Delta$ instead of whether $mu_1-mu_2=0$. The $t$-statistic (assuming equal variances) would in that case be $$ T=frac{bar{x}_1-bar{x}_2-Delta}{s_psqrt{1/n_1+1/n_2}}$$ where $s_p$ is the pooled standard deviation estimate. Under the null hypothesis, this statistic is $t$-distributed with $n_1+n_2-2$ degrees of freedom.

An easy way of carrying out this test is to subtract $Delta$ from your observations from the first population and then carry out a regular one-sided two-sample $t$-test.

### Similar Posts:

- Solved – What to do when the means of two samples are significantly different but the difference seems too small to matter
- Solved – Why do irrelevant regressors become statistically significant in large samples
- Solved – compute a confidence level for a z-test based on the size of the samples
- Solved – confidence interval for 2-sample t test with scipy
- Solved – confidence interval for 2-sample t test with scipy