I'm working at an online agency, where we run a lot of AB testing in order to test differences in proportion between two groups (test vs. control). Standard practice in the industry for testing difference of proportions is either based on the normal distribution or chi-squared.
Chi base $lambda$ tests tend to use a lot of data, which you don't always have, while normal distribution tests are problematic, since proportions are bound by $0$ and $1$, unlike the normal approximation. I claimed to my colleagues that a test that uses the beta distribution should always over perform both normal / chi options, since its built for proportions.
Here is my R code to perform the test, this test shows that test over performed the control group (95%):
library(ggplot2) number_of_success_test <- 46 number_of_success_controll <- 33 number_of_failures_test <- 2643 number_of_failures_controll <- 2579 test1 <- rbeta(100000, number_of_success_test, number_of_failures_test, ncp=0) test2 <- rbeta(100000, number_of_success_controll, number_of_failures_controll, ncp=0) test <- data.frame(test1, test2) quantile(test2, 0.95) g <- ggplot(data=test, aes(x=test1)) + geom_density(color="red", bindwidth=0.0000001) + geom_density(aes(x=test2), bindwidth=0.0000001) + geom_vline(xintercept=quantile(test2, 0.95)) + geom_vline(xintercept=quantile(test1, 0.5), color="red") g + xlab("CR") + geom_text(label="95 pecentile - control group", x=quantile(test2, 0.95), y=15000) + geom_text(label="50 pecentile - test group", x=quantile(test1, 0.5), y=12000, color="red") Am I right? Is it really always better to use beta distribution over chi / normal distribution, when dealing with difference in proportions? (Also, is my approach in the R code right?)
Best Answer
From your code (and my knowledge of AB testing), I gather your proportions come in discrete increments. That is, for every person who visits a site, they end up categorized as a "success" or a "failure". In other words, your proportions come from a finite number of Bernoulli trials; they are not continuous proportions. As a result, the beta distribution (which is for continuous proportions) is not really appropriate here. Instead, you should use the binomial distribution. Provided your $n$'s are large enough relative to the proportion of successes, the normal approximation is quite acceptable (the standard rule of thumb is that the lesser of $np$ and $n(1-p)$ should be $>5$, in your case those values are $46$ and $33$). I would go with the chi-squared test in your situation, and not use the beta distribution.
If you didn't have enough successes to trust the normal approximation, you could use a permutation test, as @jbowman discusses here: The $z$-test vs. the $chi^2$-test for comparing the odds of catching a cold in two groups.
On the other hand, if your proportions were continuous (e.g., the mass of a tumor as a proportion of the mass of an organ), the beta distribution would be preferable. You could use beta regression in an ANOVA-ish way (i.e., only having categorical predictor variables). I have a simple example of beta regression in R that could be adapted to such a situation here: Remove effect of a factor on continuous proportion data using regression in R.
Similar Posts:
- Solved – Formula for Bayesian A/B Testing doesn’t make any sense
- Solved – Expected number of failures preceding the first success
- Solved – Test whether difference in proportions differs from a non-zero constant
- Solved – GLM or alternatve model for proportion data in R
- Solved – the difference between “count proportions” and “continuous proportions”