Analysis
Sample size
Also called n.
Sample size is the number of independent observations behind a rate or a mean. It controls how much the number can wobble.
How it is measured
Count independent units, such as visitors or orders, not page loads. The uncertainty of a rate shrinks with the square root of n, so quadrupling traffic only halves the margin.
Before a test, compute the n you need from the baseline rate and the smallest difference you care about. Afterward, report n next to each rate.
Worked example
A winery's email test sends subject line A to 300 people and B to 300. A gets 21 clicks and B gets 33, 7.0 against 11.0 percent. The gap looks large.
With n of 300, the margin on each rate is about 3 points, so the ranges overlap. The winery resends the winning line to 4,000 people and sees 8.6 percent, between the two earlier results.
How it differs
Sample size is how many observations you have. Statistical significance is the verdict on whether a difference is beyond chance given that size. Small n alone can make a real effect fail the test.
Common errors
Counting pageviews as observations. Using the sample of one weekday. Stopping when the difference looks big. Treating the same visitor twice as two. Ignoring segment size after slicing.
In practice
Compute needed n before launch and write it in the plan. After slicing by segment, check n again. Report it beside each rate.