I apologize if this question has been done to death, but as a non-statistician I really don't know what is the bottom line takeaway. I am looking at a sample of 30,000 individuals who were the subject of an economic intervention. This intervention produces a mean increase in the annual income of the subjects of $2000. However the result is not statistically significant at usual levels.
From a statistical viewpoint, has the intervention failed, and if not what further can be done?
Best Answer
Statistical insignificance does not mean that the effect being tested for does not exist, but rather, that the data that was observed does not furnish strong evidence for the existence of that effect.
For example, if you have an unloaded six-sided die, but the numbers on its faces are {1,2,3,4,5,5} instead of {1,2,3,4,5,6}, and you roll it only 3 times, it may not be evident through such a small sample size that the die would give you more fives than ones. That doesn't mean the die isn't different than a normal die (after all, we have the benefit of inspecting it and we can clearly see it is different)–it may simply be that we need to collect more data about the die's observed behavior in order to make a statistically significant inference about the intrinsic properties of the die.
Analogously, it may be that even a sample size of 30000 may not be sufficient to detect a difference in the behavior of your population under two treatments, because your statistical test has low power. Or, maybe the truth is that the mean increase you're observing is actually due to random chance and no effect truly exists. Since you have not specified your tolerance for Type I error, I can't really speak to that.
The takeaway here is that failure to detect significance doesn't mean no effect exists–it simply means that, by random chance or by lack of power, the data furnishes insufficient evidence to claim that the hypothesized effect exists with a high degree of confidence.
Similar Posts:
- Solved – What does the p value mean in meta analysis results output
- Solved – Interpreting Mediation Output when Direct Effect is not Stat. Sig but ACME and Total are
- Solved – How to test if predicted value is statistically different from the corresponding observed value, accounting for sample size of the observed value
- Solved – Why does one report statistical power only when results are non significant
- Solved – What does Posterior tail-area probability mean in Causal Impact