I'm trying to carry out a hypothesis test for data that has the following structure.
Total number of correct answers out of 6 questions for each subject that only has a HS diploma:
3, 2, 3, 5, … (there are 16 total subjects)
Total number of correct answers out of 6 questions for each subject that has a College degree:
5, 5, 3, 6, … (there are 12 total subjects)
I'm not sure if this is relevant for my question but the data was obtained by asking 28 random subjects their education level and then recording how many out of 6 questions they answered correctly.
I used a 2 sample t-test to see if the population means differ for both groups. Is there a better method for this type of data? I'm asking because the data is discrete and I'm not sure if the t-test is best suited for this kind of data.
Also the t-test would tell me if the population means differ but lets say the question of interest is whether or not education (HS vs College) is associated with literacy (total number of correct questions), what kind of test would best answer that question?
Thank you.
Best Answer
Right, considering the modest sample sizes and discreteness of the data, there could be some question about applying the two-sample t test here. The t test formally assumes that the data is normally distributed, which of course is not true here since the data is discrete. For large sample sizes, the t test is robust against violations of the normality assumption, but for smaller sample sizes it is less so. It may still be all right here, since the sample size is not excessively small, and especially because the data is bounded (between 0 and 6) so there can be no concerns about heavy tails or outliers (which otherwise would pose the biggest threats to the accuracy of the t test). The Wilcoxon rank sum test is an alternative which could certainly be applied, and which would avoid any concerns about distributional assumptions, although it may provide slightly less power than the t test. If you use the Wilcoxon rank sum test here, be sure to use an implementation that properly handles ties, as there are many ties here.
On the other hand, summarizing the data as counts of correct answers already discards some potentially valuable information. Particularly if the questions have varying degrees of difficulty and quality, you may be able to get more out of the data and construct a more powerful hypothesis test by applying a method based on item response theory, although this would require a more complex analysis.
Edit: As another answer mentions, the chi-square test is another option here. A drawback of the chi-square test is that it ignores the ordering of the 7 possible responses (0-6), which could lead to a less powerful test. The results of the chi-square test are also less straightforward to interpret, since if the test rejects the null hypothesis, it might be because the two populations have different means, or it could be because they have different variances, or because there is some other kind of difference between the two distributions. Another way of looking at it is that the chi-square test has a more specific null hypothesis, namely that the two distributions are identical, compared to the null hypothesis of the t test that the two distributions have the same mean. The consequence of this is that the chi-square test also has a broader alternative hypothesis, one which is less focused on detecting differences in the means, and this explains why it would be expected to be less powerful if differences in the means is what you are looking for.