Calculate how many samples you need per variant for a statistically valid A/B test before you start.
Running an A/B test with too few samples is one of the most common and costly mistakes in experimentation. With insufficient sample size, random noise can easily produce a result that looks like a real effect but isn't, or genuinely mask a real effect that exists. Calculating required sample size before launching a test, rather than just "running it until it feels done," is what separates statistically valid experimentation from essentially reading tea leaves in your analytics dashboard.
Baseline conversion rate is your current performance, lower baseline rates require larger samples to detect the same relative effect, since rare events need more observations to measure reliably. Minimum detectable effect (MDE) is the smallest improvement you actually care about detecting, chasing tiny effects requires dramatically larger samples than chasing large ones, there's a real trade-off between sensitivity and practical sample size. Significance level (α) controls your tolerance for false positives, 95% confidence is the conventional default. Statistical power controls your tolerance for false negatives, missing a real effect that's actually there, 80% power is standard, meaning you accept a 20% chance of missing a genuine effect of the specified size.
A common but statistically invalid practice is checking test results daily and stopping as soon as you see significance. This dramatically inflates your actual false positive rate beyond the stated significance level, because you're effectively running many small tests (one per day of peeking) and the chance that at least one shows a spurious "significant" result by pure luck grows with every peek. Calculate your required sample size upfront, and either wait until you hit it before analyzing, or use a sequential testing method specifically designed to allow valid early stopping.
This calculator's day estimate assumes constant daily traffic, in reality, always run tests for at least one full week (ideally two), even if you hit your sample size target sooner, to account for day-of-week effects, weekday versus weekend behavior differs meaningfully for most products, and a test that only ran Monday through Wednesday may not represent your real user base.