Analysis
Multivariate test
Also called MVT.
Multivariate test is an experiment that changes several page elements at once and measures every combination. It reveals which elements matter and how they interact.
How it is measured
Each element has a few options and the test serves every combination. Three headlines and two button colors make six cells. Traffic is split across all of them and each cell needs its own sample.
Count the cells before you start and divide your expected traffic by that number. If each cell would get fewer than a few hundred visitors, the test will not settle.
Worked example
A real-estate listing site wants to test two hero photos, three headline lines, and two search-box layouts. That is 12 cells. At 900 visitors a day, each cell sees 75 daily and needs about six weeks.
The team cuts the plan to a headline test and a hero test run one after the other, each finishing in nine days. They give up learning about interactions in exchange for an answer before the spring season.
How it differs
A multivariate test varies several elements in one run. An A/B test changes one thing at a time. The multivariate design finds interactions and costs far more traffic.
Common errors
Starting with too many cells. Calling a pair of A/B tests multivariate. Ignoring interactions in the results. Stopping at the first winning cell. Running it on a low-traffic page.
In practice
Use it only if the page has several thousand daily visitors. Otherwise run A/B tests in sequence. Write the cell count and run length down before launch.