Experimentation and A/B Testing

A randomized controlled experiment that splits users between a control (A) and a treatment (B) to measure the causal effect of a change on metrics.

It balances known and unknown factors across groups, so metric differences can be attributed to the change rather than confounders.

It states the change, the expected effect on a specific metric, the affected users, and the reasoning, e.g., "If we shorten signup to one step, new-user activation will rise because friction drops."

The entity assigned to a variant: typically user, but sometimes session, page view, device, or cluster (e.g., city).

Users see a consistent experience, and user-level metrics like retention remain valid; page-level randomization can cause inconsistent experiences and correlated observations.

The pre-registered metric that determines whether the treatment ships, aligned with the product goal.