I have two samples ($n approx 70$ in both cases). The means differ by about twice the pooled std. dev. The resulting $T$ value is approximately 10. Whilst it's great to know that I have conclusively shown that the means are not the same, this seems to me to be driven by the large n. Looking at histograms of the data I certainly do not feel that such as small p-value is really representative of the data and to be honest don't really feel comfortable quoting it. I'm probably asking the wrong question. What I'm thinking is: ok, the means are different but does that really matter as the distributions share a significant overlap?
Is this where Bayesian testing is useful? If so where is a good place to start, a bit of googling hasn't yielded anything useful but I may not by asking the right question. If this is the wrong thing does anyone have any suggestions? Or is this simply a point for discussion as opposed to quantitative analysis?
Best Answer
Let $mu_1$ denote the mean of the first population and $mu_2$ denote the mean of the second population. It seems that you've used a two-sample $t$-test to test whether $mu_1=mu_2$. The significant result implies that $mu_1neqmu_2$, but the difference seems to be to small to matter for your application.
What've you encountered is the fact that statistically significant often can be something else than significant for the application. While the difference may be statistically significant it may still not be meaningful.
Bayesian testing won't solve that problem – you'll still just conclude that a difference exists.
There might however be a way out. For instance, for a one-sided hypothesis you could decide that if $mu_1$ is $Delta$ units greater than $mu_2$ then that would be a meaningful difference that is large enough to matter for your application.
In that case you would test whether $mu_1-mu_2leq Delta$ instead of whether $mu_1-mu_2=0$. The $t$-statistic (assuming equal variances) would in that case be $$ T=frac{bar{x}_1-bar{x}_2-Delta}{s_psqrt{1/n_1+1/n_2}}$$ where $s_p$ is the pooled standard deviation estimate. Under the null hypothesis, this statistic is $t$-distributed with $n_1+n_2-2$ degrees of freedom.
An easy way of carrying out this test is to subtract $Delta$ from your observations from the first population and then carry out a regular one-sided two-sample $t$-test.
Similar Posts:
- Solved – What to do when the means of two samples are significantly different but the difference seems too small to matter
- Solved – Why do irrelevant regressors become statistically significant in large samples
- Solved – compute a confidence level for a z-test based on the size of the samples
- Solved – confidence interval for 2-sample t test with scipy
- Solved – confidence interval for 2-sample t test with scipy