Learn
Both — the two schools answer different questions, and a dashboard worth reading shows both answers. This page defines each school in plain English, works one test through both lenses, and explains why ProveBetter renders a frequentist and a Bayesian engine on every result. Both dashboards become readable in plain language by the end.
Every A/B test asks one question: is B better than A? The two statistical schools that dominate testing software answer it with different machinery, and the numbers they produce look similar enough that people misread them.
Frequentist testing starts from a skeptical baseline — assume there is no real difference between the variants — then measures how surprising the observed data is under that assumption. Its headline number is the p-value:
A p-value is the probability of seeing data at least this extreme if there were truly no difference between the variants. It is not the probability that the variants are equal, and it is not the probability that B is better.
Bayesian testing starts from a belief (the prior), updates it with the data, and outputs a full picture of where each variant's true rate plausibly sits. Its headline numbers are the credible interval ("given the data, B's true rate lies in this range with 95% probability"), the chance to beat control ("given the data, B's true rate is higher than A's with X% probability"), and expected loss ("what am I risking if I decide right now?").
The platforms line up cleanly on the two schools: Optimizely runs a sequential frequentist engine (p-values, its Stats Engine — the comparison page goes deeper), while VWO leads with a Bayesian engine that reports "chance to beat" instead of p-values and — in VWO's own framing — "dispenses with the need of a p-value altogether." Two honest answers to two different questions look like one number, and that's where the confusion starts.
A low p-value means the data is unlikely under "no difference" — evidence against A and B performing the same. The common misreadings: "1 − p" is not B's chance of winning, and the p-value never weighs A against B directly.
The significance threshold. Before the test, you pick the false-positive rate you tolerate — commonly 5% — and plan to reject "no difference" only if the p-value drops below it. That number is only valid for one look at the planned sample size — which is the problem every live dashboard has.
Sequential testing and the peeking problem. A fixed-sample p-value is valid for one look at one planned sample size. Dashboards update continuously, so humans peek — and peeking rewrites the math. Under continuous monitoring of a true null, a fixed-horizon p-value is guaranteed to cross any threshold eventually. Johansson, Shalit & Sontag measured the false-positive rate inflated 5–10× at just 10,000 observations (Peeking at A/B Tests, KDD 2017 — the paper behind Optimizely's Stats Engine). Adobe corroborated in production: 15% of 2,089 real tests that looked significant under peeking with fixed-horizon analysis were not significant under an anytime-valid analysis (arXiv 2302.10108). The industry answer is sequential machinery — Optimizely's Stats Engine reports always-valid p-values that stay valid no matter when you look.
SRM is the validity check that has nothing to do with p-values. Sample Ratio Mismatch means the observed visitor split drifted from the configured split — a 50/50 test that actually delivered 4,500 vs 5,500 visitors. Assignment broke (redirect bug, snippet drop), so every comparative number on the dashboard is invalid: p-value, credible interval, chance-to-beat, all of it. The check — observed vs configured split (the practitioner-standard framing is Lukas Vermeer's, KDD 2019) — applies to both schools equally.
The prior is what you believed before the test. The neutral default is a uniform prior — "no opinion." It matters at small sample sizes and fades as data accumulates; for conversion rates the update is friendly arithmetic in which each variant's belief is fully described by its own visitor and conversion counts.
The posterior is the prior updated by the data: the full probability picture of where the variant's true rate could be.
The credible interval is a range that contains the variant's true rate with a stated probability, given the data. "Given these 1,000 visitors and 58 conversions, B's true rate is between 4.5% and 7.4% with 95% probability." That is the sentence people intuitively want from a test — and it's one a frequentist confidence interval cannot say, because a frequentist interval is a property of the procedure, not a probability statement about the parameter. The p-value vs credible interval distinction is the fastest way to tell which school you're looking at.
Chance to beat control is the probability, given the data and the prior, that B's true rate is higher than A's. It is the direct Bayesian answer to "is B better than A?" — the headline readout VWO ships.
Expected loss is what you stand to lose, in conversion-rate points, if you decide now and you're wrong — averaged over everything the data says is still possible. It answers "what am I risking if I decide right now?" and gives Bayesian engines a natural stopping rule: keep running until the expected loss of your chosen variant falls below a tolerance you set. VWO's SmartStats is exactly this design: no p-value, chance-to-beat plus expected loss, stop when the risk is below tolerance; VWO claims this reaches decisions up to 50% faster than p-value engines — a vendor claim, quoted as such.
Same experiment, two dashboards. The test: 1,000 visitors per variant; A converts 52 (5.2%), B converts 58 (5.8%) — illustrative numbers chosen for clean arithmetic.
| What you see | Frequentist engine (Optimizely-style) | Bayesian engine (VWO-style) |
|---|---|---|
| Headline number | p-value 0.56 | chance to beat control ≈ 72% |
| Range shown | 95% confidence interval for the difference: −1.4 to +2.6 percentage points | 95% credible intervals: A 4.0–6.8%, B 4.5–7.4% |
| Decision rule | reject when p < pre-set threshold (commonly 0.05), at the planned sample size | stop when the expected loss of your chosen variant drops below your tolerance (say 0.2 percentage points) |
| What the headline means | "if A and B were truly equal, a gap this large would appear 56% of the time by chance" | "given the data, B's true rate beats A's with 72% probability" |
| Verdict at this sample size | not significant — keep collecting | leans B; expected loss of calling B now ≈ 0.18 percentage points — a decision only if that's under your tolerance |
Both engines read the same data and both honestly report that the evidence is weak — but the numbers mean different things, and "should I ship B?" gets different-shaped answers.
The same test through each lens, so you can see where the table's numbers come from. All figures are illustrative.
The gap is 0.6 percentage points (5.8 minus 5.2). At 1,000 visitors per variant, conversion estimates wobble by about 1 percentage point either way, so the engine divides the gap by that noise and derives a probability from the result: a two-sided p-value of 0.56. Reading it: "if A and B truly performed the same, a gap of 0.6 points or more in either direction would appear by chance 56% of the time." That isn't small, so there is no evidence to reject "no difference." The interval form tells the same story: the 95% confidence interval for the difference runs from −1.4 to +2.6 percentage points — it straddles zero. Verdict: not significant, keep collecting.
The trap: 1 − p = 0.44 is often read as "44% chance B is better." Wrong — the p-value never compares A and B; it only rates the data against the assumption that they're equal.
Same counts, same neutral prior, two posteriors: A's true rate averages ≈ 5.3% (95% credible interval 4.0–6.8%); B's ≈ 5.9% (4.5–7.4%). The intervals overlap — the Bayesian way of saying the data can't yet separate the variants. Chance to beat control: ≈ 72% — B is favored, but 72/28 is "leaning," not "proven." Expected loss prices the decision: declare B now and the expected regret is ≈ 0.18 percentage points of conversion rate; declare A now and you're risking ≈ 0.78 points. At a 0.2-point tolerance a Bayesian decision rule ships B; at 0.1 it doesn't — the engine prices the decision, you pick the tolerance.
Neither read says "proven." The frequentist says not significant; the Bayesian says the data leans B at 72% with bounded risk either way. When dashboards "disagree," this is most often what's happening: not a contradiction in the data, but two decision policies asking two different questions — is the effect real? versus how likely is B to win, and what does being wrong cost? The numbers aren't comparable either — 0.56 and 72% are different scales, a confusion VWO's own help docs record.
Different teams ask different questions. A data team wants to know whether an effect is real before signing off — the frequentist discipline, the p-value and confidence interval. A growth owner wants to know how likely B is to actually win and what deciding now costs — the Bayesian chance-to-beat and expected loss. Both are legitimate; a platform that hides one forces every team into a single school's answer.
ProveBetter's design decision is to print both, on every result, with the method labeled:
Sticky assignment keeps each visitor in their variant when traffic percentages change live; the honest-results banner keeps the history visible. Client-side and server-side tests resolve through the same two engines (guide, API reference). The design bet: neither school owns the truth — one dataset, two questions, both answers on the page. Run your own test on the free tier and watch both engines converge on the same data.
Frequentist and Bayesian testing are two schools answering two questions: is this effect real? versus how likely is B to win, and what does being wrong cost? Same data, different numbers: p-values and sequential corrections versus credible intervals, chance-to-beat and expected loss. Neither school owns the truth; pick the question you're asking and watch the validity checks on both. ProveBetter prints both engines on every result with a plain-language verdict. See how A/B testing is priced or start a test on the free tier.