Analysis
A/B test
Also called split test.
A/B test is an experiment that splits traffic at random between a control and one or more variants, then compares one outcome. The split has to be random and the outcome has to be chosen before the test starts.
How it is measured
Assign each visitor to an arm once, by a hash or a server-side flag, and log the assignment next to the outcome event. The unit is usually the visitor or the session, the window is a fixed run measured in whole weeks, and the outcome is a rate such as signups per assigned visitor.
Check that arms received roughly the split you asked for. A 50/50 test that lands at 54/46 points to a flag bug or a redirect that drops people, and the result should not be trusted until that is fixed. A cookieless setup assigns by request-time rules rather than by a stored id, so decide up front how a returning person keeps the same arm.
Worked example
A bike-parts shop tests a shorter checkout form. Control gets 6,210 visitors and 187 orders. The variant gets 6,184 visitors and 221 orders. That is 3.0 percent against 3.6 percent, a lift that looks real, but the shop planned for two full weeks and only ten days have passed.
They wait out the remaining four days. The gap narrows to 3.0 against 3.3 percent, and the interval around the difference now includes zero. The early read came from a weekend promotion that hit the variant arm harder by chance.
How it differs
An A/B test has one change and one control. A multivariate test changes several elements at once and needs far more traffic to separate their effects. An A/B test does not tell you which part of a bundled change did the work.
Common errors
Stopping the moment the dashboard turns green. Changing the variant mid-run. Testing five things in one 'variant' and calling it one idea. Running two overlapping tests on the same page. Judging on clicks when the goal was paid orders.
In practice
Write the hypothesis and the stop date in the ticket before launch. Size the run from your baseline rate and the smallest lift you would act on, then leave it alone until the date. If traffic is too thin to detect that lift, ship on judgment and say so.