Solved – How to get proper randomization in website A/B testing

In the statistical practice of experimental design, you separate your tests by 'blocks' if you can control the factors (say routing to one page or another, or categorizing by browser) or through randomization of trials to counteract variables you don't control.

When doing A/B testing, you obviously can't control who comes to your site when. How can good statistical practice be done?

In high volume sites just randomly sending the user to one of your test pages or control (let's call them factor levels) usually works fine. There may be some variability in the mix of browsers between the factor levels but you'll have a sample size that's large enough for it not to matter so much. This is also very easy to implement, even if you randomly load-balance or shard across multiple servers.

For low-volume sites where you have $n$ factor levels you could generate random permutations and force the next $n$ users to the factor level according to the permutation. You could do this within each group you define (e.g. browser type). This does require keeping track of a global state, and assumes you can do so fast enough that you can observe an order in which the users arrive. I don't know of anyone that does this.

Similar Posts:

Rate this post

Leave a Comment