Common A/B testing mistakes and how to avoid them

1. Peeking at results and stopping early

This is the single most common mistake. You check your dashboard daily, see p < 0.05 on day 3, and call the test. The problem: at a nominal 5% significance level, peeking daily for two weeks gives you roughly a 25–30% false positive rate.

Why it happens — P-values fluctuate naturally during data collection. With enough checks, random noise will eventually cross the 0.05 threshold.

The fix — Either commit to a fixed sample size and do not look until done, or use sequential testing which is specifically designed for continuous monitoring.

2. Running underpowered tests

An underpowered test is one that does not have enough visitors to detect the effect you care about. If your test has 40% power, it will miss a real effect 60% of the time — you will call winners losers and vice versa.

The fix — Always calculate sample size before launching. Know what MDE your traffic can support. If you need 50,000 visitors per variant and only get 1,000 per week, either accept a larger MDE or test on a higher-traffic page.

3. Testing too many variants without correction

Every additional variant increases the chance of a false positive. Testing 4 variants against a control at α = 0.05 gives you roughly a 19% chance of at least one false positive, not 5%.

The fix — Use the Multi-variant Calculator which applies Bonferroni or Holm corrections automatically. Or limit yourself to one variant at a time.

4. Optimizing the wrong metric

Optimizing click-through rate when your business cares about revenue can lead to changes that increase clicks but decrease purchases. A button that says "Free iPhone" will get more clicks but fewer real conversions.

The fix — Choose a primary metric that directly ties to business value. Track secondary metrics for guardrails (e.g., make sure revenue per visitor doesn't drop while optimizing conversion rate).

5. Testing without a hypothesis

Random changes ("let's try a blue button") occasionally produce wins, but you learn nothing from them. Without a hypothesis, you cannot build on successes or understand failures.

The fix — Before each test, write: "We believe [change] will [effect] because [reason]." This makes results interpretable regardless of outcome.

6. Ignoring losers

Teams celebrate wins and ignore losses. But failed experiments contain valuable information about user behavior. A variant that decreased conversions by 15% tells you something important about what users care about.

The fix — Document every test result — wins, losses, and inconclusive. Build a knowledge base. Review past failures before designing new experiments.

7. Flawed implementation

Even perfect statistical methodology cannot save a broken test setup. Common implementation issues:

• Sample ratio mismatch — unequal split indicating randomization bugs
• Caching showing control to returning variant users (or vice versa)
• The change not actually loading for some users due to JS errors
• Bot traffic inflating one variant

The fix — Always check for sample ratio mismatch before analyzing results. Run an A/A test (identical variants) first to validate your setup.

The checklist

1. Calculate sample size before launching
2. Do not peek — or use sequential testing
3. Correct for multiple comparisons
4. Pre-register your primary metric
5. Write a hypothesis
6. Check for SRM after the test
7. Document everything