What is the difference between Bayesian and frequentist A/B testing?

Frequentist tests produce p-values. Bayesian tests give direct probability that one variant is better.

What is expected loss in Bayesian A/B testing?

Expected loss is the average conversion rate you would lose by choosing the wrong variant.

What are credible intervals?

A 95% credible interval means there is a 95% probability the true value falls within that range.

Bayesian A/B test calculator

Question: What is the probability that variant B is better than A?

Analyze your A/B test results using Bayesian statistics. Get the probability that each variant is the best, the expected loss from choosing the wrong variant, and credible intervals.

How to use this calculator

Enter the number of visitors and conversions for your control and variant. The calculator uses a Beta-Binomial model with uninformative priors (Beta(1,1)) and runs 50,000 Monte Carlo simulations to estimate the probability that each variant is superior. Results update instantly as you change inputs.

Bayesian vs frequentist A/B testing

Unlike frequentist methods that produce p-values, Bayesian analysis gives you a direct probability that one variant is better. The posterior distribution for each variant's conversion rate is modeled as a Beta distribution. The probability to be best, expected loss, and credible intervals are computed from Monte Carlo samples drawn from these posteriors. This approach is more intuitive for business decisions because you get statements like "there is a 95% probability that Variant B is better" instead of "the result is significant at the 5% level."

When to use Bayesian analysis

Bayesian A/B testing is especially useful when you need intuitive probability statements for stakeholders, when you want to make decisions based on expected loss rather than statistical significance, or when your sample sizes are small. It also allows you to incorporate prior knowledge about conversion rates if you have it. Many modern experimentation platforms use Bayesian methods as their default.

Common mistakes in Bayesian A/B testing

Using overly informative priors that bias results toward your expectations is a common mistake. Uninformative priors (like Beta(1,1)) are safest unless you have strong prior data. Another mistake is confusing the probability to be best with the magnitude of the improvement — a variant can have 99% probability of being better but only improve conversion by 0.01%. Always check the expected loss and credible interval width alongside the probability.