Solved – Pre-Post net of control vs ANCOVA

Pre-Post net of control is a nice method because it handles seasonality in sales and as long as you pick a representative group and a comparable control group, you should get unbiased estimates. However, one challenge with Pre-Post net of control is you are assuming your data are independent. Sales data usually has auto-correlation, which means you need to account for that or handle it appropriately if you want unbiased standard errors or p-values.

One way to handle this would be to use a mixed model, as you suggested. This would allow you to include random effects such as day, week, and/or month to handle the auto-correlated nature of your data. You could also include other covariates (such as a store ID, demographics, etc), as random effects.

Another way to analyze this (that gets you out of all data assumptions) would be to use a non-parametric method such as simulation (provided you have historical data). For example, you could pick a random time for the test, randomly pick a control and treatment group, and then run 1000 tests using historical data to get a distribution of possible effect sizes due to chance. Then, you could compare the effect size seen in your experiment to the distribution of random effects to see whether your test falls in the 97.5% or 2.5% tails, indicating significance at the 95% confidence level.