Analysis
Control group
Also called baseline variant.
Control group is the portion of an experiment that gets the unchanged experience. Every variant is measured against it.
How it is measured
Define the control before launch: the current page, price, or email, with nothing else changed. Keep it running at the same time as the variants, in the same traffic mix, for the whole test.
Verify that the control matches what was live before the test. If the control changed after assignment began, the comparison is broken. Log which version each person saw.
Worked example
A tea retailer tests a free-shipping banner on a store with a $45 shipping threshold. Control sees the store as it is. 2,480 control visitors place 96 orders at an average of $52.
Some weeks earlier, the retailer had compared the banner week against the quiet week before and 'saw' a 40 percent gain. A concurrent control showed the real lift was 9 percent, because that week also had a newsletter send.
How it differs
A control group is the baseline arm. An A/B test is the whole design that includes a control and a variant. A holdout that never sees any change serves as a control but for a longer time.
Common errors
Using last month's data as the control. Letting the control drift. Giving the control less traffic than variants. Running the control for a shorter window. Throwing away the control when results look clear.
In practice
For every test, write down what the control is in one line. Run it at the same time as the variant. Resist the urge to compare against history.