I am running an A/B test, and the following are samples of how many questions a user has answered. I am trying to figure out which test is better (A or B), and how confident we are that it is better. For example, in Sample A, first user answered 4 questions, next two users answered 5 questions.
I know how to calculate statistical significance for A/B tests, but am not sure how to do it for a range of numbers. I am trying to get users to answers the maximum questions possible.
Sorry for my incorrect lingo but i'm not a stat guy, so i'm not sure what you refer to these terms. Essentially, I would like to figure out it if my A/B test, A sample or B sample yields a higher number.
Sample A : 4, 5, 5, 9, 11, 14, 15, 15, 16, 19, 27, 30, 31, 32, 58, 65, 67, 79, 98, 99, 100, 103, 106, 204, 232, 341, 354, 359, 360 Med: 58 Avg: 98 Sample B: 1, 3, 4, 4, 4, 5, 9, 11, 12, 15, 19, 28, 37, 48, 50, 54, 59, 72, 74, 78, 80, 81, 89, 91, 99, 101, 103, 103, 104, 120, 121, 174, 203 Med: 59 Avg: 62 Best Answer
I would like to figure out it if my A/B test, A sample or B sample yields a higher number.
This is the sort of question I can answer.
but I see two issues:
1) your data appear ordered from small to large. Is this a feature of the data itself (and then why is it ordered? – is it cumulated??), or did you sort your data for some reason?
2) your responses are effectively counts, so equal-variance assumptions probably wouldn't hold. Nonparametric approaches might have issues with the heavy level of ties. A GLM might be okay.
Edit: Looking at the data on the square root scale:
The data are far too variable to be from a Poisson with constant mean. A negative binomial may be reasonable.
However, I am still quite concerned that there's some facts about this data we should know about but don't, such as something that indicates that the mean is in fact not constant, or that the data are not independent. Here's an index plot for the two samples:
There's so much here that's left unsaid, still.
Similar Posts:
- Solved – Cumulative predicted probabilities in logistic regression
- Solved – How to calculate average EER and ROC
- Solved – Testing significance of cross-correlated series
- Solved – Testing significance of cross-correlated series
- Solved – Is measuring click-through probability required when the experiment’s unit of diversion is a user