A/B testing for small traffic sites

The reality of low-traffic testing

The math is unforgiving: detecting a 5% relative lift on a 3% conversion rate requires roughly 85,000 visitors per variant. At 250 visitors per day per variant, that is 340 days — nearly a year.

This does not mean A/B testing is impossible with low traffic. It means you need to be strategic about what you test and how.

Strategies that work

• Test bigger changes — Subtle tweaks (button color, copy changes) produce small effects that need massive samples to detect. Instead, test fundamentally different approaches: entirely new page layouts, different value propositions, different pricing structures. Larger effects need fewer visitors to detect.
• Focus on high-traffic pages — Concentrate testing on your highest-traffic pages. A test on the homepage with 80% of traffic will complete much faster than one on a settings page with 2%.
• Use fewer variants — Every variant divides your traffic. With 500 visitors/day, an A/B test (2 variants) gives you 250 per variant. An A/B/C/D test gives you only 125 per variant — making detection even harder.
• Accept a larger MDE — Instead of trying to detect a 5% relative lift, set your MDE to 15–20%. You will miss small improvements, but you will catch the big wins — which matter most anyway. Use the Sample Size Calculator to see how this affects required traffic.
• Run longer tests — With 250 daily visitors per variant, a test for a 20% relative MDE on a 5% baseline needs about 2,500 visitors — 10 days. That is perfectly feasible. Plan for 2–4 week tests rather than 3-day sprints.

Statistical methods for small samples

• Bayesian analysis — The Bayesian Calculator works well with smaller samples because it gives you a probability rather than a binary significant/not-significant answer. "72% chance B is better" is still useful information, even if a frequentist test would say "not significant."
• Sequential testing — The Sequential Calculator lets you monitor results continuously and stop early if there is a clear winner. This can save weeks of testing time when the effect is large.
• Fisher's exact test — For conversion rate tests with very small samples (under 100 per group), Fisher's exact test is more reliable than the standard z-test. The Conversions Calculator offers this option.

What not to do

• Do not lower your significance threshold — Using α = 0.20 instead of 0.05 to get more significant results just means you will ship more false positives. You will make changes that have no real effect — or make things worse.
• Do not ignore sample size requirements — Running a test for 3 days with 200 visitors and declaring a winner is worse than not testing at all — it gives false confidence in a random result.
• Do not combine non-comparable traffic — Pooling traffic across very different pages or user segments to reach sample size faster introduces confounds that invalidate the test.

A practical framework for small sites

1. Pick your highest-traffic page
2. Test a meaningful change (not a tweak)
3. Set MDE to 15–20% relative
4. Use the Duration Calculator to estimate test length
5. Run for at least 2 full weeks
6. Analyze with the Bayesian Calculator for the most useful interpretation

Small traffic does not mean you cannot experiment. It means each experiment should count — test bold ideas, accept larger MDE thresholds, and focus on learning.