Bayesian vs frequentist A/B testing
These two statistical frameworks answer different questions about your A/B test. Neither is universally better β the right choice depends on your goals and constraints.
The frequentist approach
Frequentist testing is the traditional approach. You set up a null hypothesis ("no difference"), collect data, and compute a p-value.
It asks:
"If there were no real difference, how likely would I be to see data this extreme?"
Strengths
- Well-established theory with decades of research
- Fixed false positive rate (Ξ±) is guaranteed if you follow the protocol
- Easy to pre-register: commit to sample size, run test, analyze once
Limitations
- Cannot say "there is an X% chance B is better" β only "we reject/fail to reject the null"
- Peeking at results invalidates guarantees without correction
- Requires fixed sample size decided upfront
The Bayesian approach
Bayesian testing starts with a prior belief and updates it with observed data to produce a posterior distribution.
It asks:
"Given the data I have observed, what is the probability that B is better than A?"
Strengths
- Gives direct probability statements ("92% chance B is better")
- Naturally handles peeking β you can check results anytime
- Intuitive interpretation that matches how people think
- Can incorporate prior knowledge from previous experiments
Limitations
- Results depend on the prior β different priors give different answers
- No fixed false positive rate guarantee
- Can be overconfident with small samples if the prior is too strong
Side-by-side comparison
| Aspect | Frequentist | Bayesian |
|---|---|---|
| Core question | Is the difference real or random noise? | What is the probability B beats A? |
| Primary output | P-value and confidence interval | Posterior probability and credible interval |
| Peeking at results | Inflates error rates without correction | Safe β probability updates continuously |
| Sample size | Must be fixed before the test | Flexible β can stop when probability is high enough |
| Interpretation | "We reject the null hypothesis at Ξ± = 0.05" | "There is a 96% chance B is better than A" |
When to use which
Use frequentist when:
- You need guaranteed false positive control (e.g. regulatory contexts)
- You can commit to a fixed sample size and run the full test
- You want a simple yes/no decision framework
Use Bayesian when:
- You want to know the probability that a variant wins
- You need to monitor results continuously and stop early
- You run many tests and want intuitive reporting for stakeholders
Use sequential testing when:
- You want frequentist guarantees but need to peek at results
- You want early stopping with controlled error rates
Try both approaches
Run your data through the Conversions Calculator for a frequentist result and the Bayesian Calculator for a Bayesian result. Comparing both can give you a fuller picture of what your data says.