I am trying to better understand statistical significance, effect sizes and the like.
I have a perception (perhaps its wrong) that even irrelevant regressors often become statistically significant in large samples. By irrelevant I mean that there is no subject-matter explanation why the regressor should be related to the dependent variable. Thus irrelevance in this post is a pure subject-matter concept and not a statistical one.
I know that a regressor will be statistically significant given a sufficiently large sample unless the population effect is exactly zero (as discussed here). Hence, an irrelevant regressor that appears statistically significant in a large sample has a non-zero effect size in population.
Questions:
- How come an irrelevant regressor turns out statistically significant?
- Should I look for subject-matter explanation (i.e. try to deny irrelevance) or is this a statistical phenomenon?
This is a continuation of a post where I was trying to clarify how to cure this effect. Meanwhile, here I am asking why it happens in the first place.
Best Answer
Questions:
How come an irrelevant regressor turn out statistically significant?
I think it's helpful to think about what happens as your sample size approaches the population itself. Significance testing is meant to give you an idea of whether not an effect exists in the population. This is the reason why when working with census data (that surveys the population), significance testing is meaningless (because, what are you trying to generalize to?).
With that in mind, what does "an effect in the population" mean? It simply means any relationship between variables in the population, regardless of how small (be it a 1-point or 1-person difference), even if that relationship is due to chance and randomness in the universe.
Thus, as your sample approaches the size of the population, significance tests become less and less meaningful because any difference will be "statistically significant". What you would be more interested in then is effect size – which is analogous to "practically significant".
Should I look for subject-matter explanation (i.e. try to deny irrelevance) or is this a statistical phenomenon?
It's a phenomenon – you should look at effect sizes.
Similar Posts:
- Solved – Can interaction be statistically significant while only one of two main effect are significant
- Solved – If a variable is found with p-value greater than 0.05, why is it also taken for calculation in the regression equation Y=a+b1*X1+b2*X2
- Solved – How to correct for small p-value due to very large sample size
- Solved – Cohen’s d vs. p-value
- Solved – Cohen’s d vs. p-value