I have binomial outcome data for three groups that I am analyzing with Cochran's Q test. As discussed at this site, Cochran.
However, I have to repeat this seven different times because I am analyzing performance of three diagnostic tests to see if they can find resistance to seven different drugs.
When there are significant differences between the three groups, I perform a series of three paired post-hoc McNemar tests to further analyze the data.
However, my question is when there are not significant differences, do I still perform the three paired post-hoc tests ?
Part of me says why not, these McNemar's are valid tests. However, another part of me figures that if this is the case, why do the Cochran's Q test at all ?
Could some one point me in the correct direction ?
Thanks !
Best Answer
Cochran's Q test is an analog to the repeated measures ANOVA for binary outcomes. As such it is an omnibus test with a null hypothesis that the population proportions behind each of the $k$ samples (three in your case) are all equal—H$_{0}text{: }p_{1} = p_{2} = dots = p_{k}$—with the alternative hypothesis that at least one population proportion is different than at least one other population proportion. Those who use such one-way omnibus tests typically proceed thus:
- Conduct the omnibus test;
- If you did not reject the omnibus test stop, you are done; otherwise:
- If you did reject the omnibus test conduct multiple post hoc pairwise tests to determine which pairs have different population proportions (McNemar's test is the appropriate test to use, although it is exactly equivalent to Cochran's Q for two groups, so you can you either for the post hoc tests); and
- If you are feeling extra competent, perform either family-wise error rate or false discovery rate multiple comparisons adjustments.
So how would that work out in your case?
- Conduct Cochran's Q test for your three groups.
- If you do not reject (1) stop you conclude you have no evidence that none of the population proportions behind your three samples are different from one another; otherwise:
- Conduct three pairwise tests (if you have groups A, B and C, these would be tests of A vs. B, A vs. C, and B vs. C). You can conduct these three tests using either McNemar's test or Cochran's Q for two groups;
- If you are feeling extra competent (and I just bet you are), you can perform a Bonferroni, Holm-Sidak, Benjamini-Hochberg or some other adjustment for multiple comparisons. At this point you may or may not determine that any numbers of these pairs are significantly different.
- Profit!
Why do people bother with the omnibus tests in the first place? Because laziness is a good thing, and the total number of possible tests, $m$, grows quite big as $k$ increases ($m$ can be as large as $frac{k(k-1)}{2}$). Thus if we do not reject the omnibus test's null hypothesis we do not need to carry on with the arduous, achingly banal tedium of conducting more tests.