In the statistical practice of experimental design, you separate your tests by 'blocks' if you can control the factors (say routing to one page or another, or categorizing by browser) or through randomization of trials to counteract variables you don't control.
When doing A/B testing, you obviously can't control who comes to your site when. How can good statistical practice be done?
Best Answer
In high volume sites just randomly sending the user to one of your test pages or control (let's call them factor levels) usually works fine. There may be some variability in the mix of browsers between the factor levels but you'll have a sample size that's large enough for it not to matter so much. This is also very easy to implement, even if you randomly load-balance or shard across multiple servers.
For low-volume sites where you have $n$ factor levels you could generate random permutations and force the next $n$ users to the factor level according to the permutation. You could do this within each group you define (e.g. browser type). This does require keeping track of a global state, and assumes you can do so fast enough that you can observe an order in which the users arrive. I don't know of anyone that does this.
Similar Posts:
- Solved – Split-split-plot design and lme
- Solved – Split-split-plot design and lme
- Solved – Advantages/disadvantages of fractional factorial design vs completely randomized design
- Solved – Advantages/disadvantages of fractional factorial design vs completely randomized design
- Solved – Split plot with factorial design vs three way ANOVA