I have two processes, lets call them A and B. I want to infer whether their population means are equal for their experimental outcomes. Both of these processes generate 1s and 0s only. That is, the outcome is binary.
My situation is similar to using two different processes to flip a coin a few times. The idea is to compare the experimental outcomes of these two processes.
I will show two simple cases to illustrate the problem. In the first case, function t.test() of the R language's stats package runs just fine when t-testing these two outcomes:
> t.test(rep(1,5), c(rep(1,4),0)) Welch Two Sample t-test data: rep(1, 5) and c(rep(1, 4), 0) t = 1, df = 4, p-value = 0.3739 ... IN the second case, for some reason the t.test will not run, and an error is returned, when t-testing these two outcomes:
> t.test(rep(1,5), rep(1,5)) Error in t.test.default(rep(1, 5), rep(1, 5)) : data are essentially constant I will need a workaround to effectively run a t-test in case 2. I do not have a human available to always look at the data before deciding whether to run the t.test.
I have done some research and saw that some people write exception handling code in these situations. Do I really need to write my own t.test implementation, such as by wrapping exception handling code around the standard stats t.test and then doing some things depending on various things in the data (I don't know what exactly)? If there is a different implementation available somewhere then that might be the best solution. I am not confident in my ability to write new t.test implementations accurately.
These two cases represent entirely possible and realistic experimental data in my situation.
Edit: Additional background seems to be interesting to the community. Here you go.
Thanks to all for so many questions! Let me see if I can provide some of the context the community seems to be wanting. In the experiments in question, we might have a scenario where my users wish to infer whether technique A for studying for SAT exams is better than technique B. IN this case there will be (effectively) continuous outcomes in the form of the mean SAT score by groups A and B. In a different scenario my users want to analyze, there may be an AB test of a web site wherein the user wants to try to get more people to open the email and to do so the firm sends out an A subject on some emails, and a B subject on emails to different group of customers. Whether the customer opened the email is a success, encoded as 1, failure being the recipient did not open the email and this would be encoded most naturally as 0. A comparison of the group means would be interesting, as a significant difference in mean success rate of opening emails could cause the user's firm to change they way they write email subject lines. Hope this helps.
Best Answer
Based on your comments, I think I have an answer for you. It seems like for the processes you currently have, you have binary outcomes, but that there will be other processes which will not. You would like to use a t.test to analyze the results.
The issue that you're having is that the t.test is not designed for use on binary outcomes. It's designed for use on continuous outcomes. The formula for a t-test is:
t = (x1-x2)/(s*sqrt(2/n))
s represents the pooled standard deviation. You're encountering the error you have because the two samples in part 2 have no standard deviation – they're constant. This means that to generate the test statistics you're dividing by 0. Which, you know, makes the universe explode. If s is too small then the t value will be huge no matter what the difference between the two sample means (unless the sample means are identical, as in your example). The t-test becomes a moot point because either the sample means are identical, in which case the p-value would be 1, or the sample means are different by any amount, in which case the p-value would be so small as to be functionally 0.
In other words, you cannot effectively run a t-test in case 2. I'd be interested in seeing the sources you have found where people do workarounds for this.
Your examples at the end do help a lot. In the success/failure example (which is analogous to the examples you provided) you should use a proportion test instead of a t-test. In the SAT test, with continuous scores, you should use the t-test. It is highly unlikely that you would encounter a situation where there was so little variability in the SAT scores that you would see this error happen again. It's possible, but highly, highly, highly unlikely except with a very, very small sample size. In which case my advice is to increase the sample size 🙂
Similar Posts:
- Solved – How to test if two sets of vectors are statistically different from each other
- Solved – How to test if two sets of vectors are statistically different from each other
- Solved – What analysis to use with two groups, a pre- and post-test, and three outcomes
- Solved – Non-parametric equivalent of the 1-sample t-test (not paired)
- Solved – How to model a binary outcomes in time series using logistic regression