Learn
A test with three, four, or five variants is a different tool from multivariate testing, and the difference decides whether your experiment produces decisions or noise. This page separates the three — A/B, multi-variant A/B/C/D/E, and true MVT — shows what each costs in traffic, and walks through the failure modes of multi-arm tests. The honest version up front: ProveBetter supports 2–5 variants per test but is not a multivariate testing platform.
A/B testing runs one challenger against the control — one comparison, one decision: keep the control or ship B (definition).
Multi-variant testing (A/B/C/D/E) runs two to five whole versions of the same page — headline, body, and layout travel together as a unit — and answers "which version wins." Every challenger is compared against the control, so a five-arm test means four comparisons (definition).
Multivariate testing (MVT) serves combinations of independent changes — headlines × images × button labels — so you can separate each element's effect from the interactions between elements. A 2×2×2 design is eight combinations, and sample requirements grow multiplicatively, which is why MVT needs serious traffic (definition).
| A/B | A/B/C/D/E (multi-variant) | MVT | |
|---|---|---|---|
| What changes | One whole version | 2–5 whole versions | Combinations of elements |
| Comparisons vs control | 1 | k − 1 (up to 4) | One per element, plus interactions |
| Question it answers | Is B better than A? | Which version wins? | Which element matters, and do they interact? |
| Traffic cost | Baseline | Scales with the arm count | Multiplicative |
| Who sells it | Everyone | Everyone | Optimizely (enterprise-gated); VWO keeps its multivariate engine frequentist-only |
What extra arms actually cost. Every arm-vs-control comparison needs the same per-arm sample as a plain A/B test — roughly 31,200 visitors per arm detect a lift from 5% to 5.5% at 80% power — and a five-arm test needs that for every arm. With an even split, a five-arm test takes about 2.5× the total traffic of an A/B test to reach the same power on every comparison. MVT's cost is different and worse: eight combinations don't just mean eight versions of the same sample, they mean enough data to estimate every element and every interaction, which is why MVT is a high-traffic tool in practice.
Correction raises the bar too. When you correct four comparisons down to Bonferroni's 0.0125, detecting the same 5% → 5.5% lift needs about 44,000 visitors per arm instead of 31,200 — roughly 40% more — because the threshold you're trying to cross got stricter. The uncorrected fifth arm is the cheapest thing to add to a test and the most expensive at the end: it multiplies comparisons, dilutes traffic, and tightens every threshold at once.
They pay off when the question is bounded and you can name the credible answers up front:
They don't pay off when:
The fifth variant exists to be used, not to be filled. Two challengers is often the right answer; the arm count should match the question, not the idea count.
ProveBetter's multi-arm support is deliberately flat: 2–5 variants, labeled A/B/C/D/E, on every tier — no enterprise gate on the number of arms.
Deployment doesn't change either: client-side tests are one snippet (see the install page), and server-side tests run over plain HTTP with no SDK (API reference).
The honest part. ProveBetter is not an MVT platform: variants are whole versions, not element combinations, and there is no factorial design. If your question is "which element matters," MVT lives in platforms built for combinatorics — Optimizely sells it (behind enterprise plans) and VWO runs its multivariate engine frequentist-only while its Bayesian SmartStats covers A/B and split tests (ProveBetter vs Optimizely shows the capability difference in full).
Multiple comparisons. A five-arm test makes four treatment-vs-control comparisons. Treat each at the usual 5% line and the family-wide chance of at least one false positive is roughly 20% — 4 × 5%. The standard correction divides the threshold: at four comparisons, Bonferroni sets it at 0.05 ÷ 4 = 0.0125 per comparison. ProveBetter applies multiple-comparison correction across variants on its frequentist engine, so the results panel doesn't pretend each arm is an independent test (multiple comparisons).
Dilution. Every added arm spreads the same conversions thinner and shrinks control's share — 20% of an even five-way split versus 50% of an A/B test — so per-arm conversion estimates get noisier at the same total sample. The guardrails: don't change the variant set mid-test (adding or removing arms is different from rebalancing percentages), and if you must rebalance, sticky assignment plus the recorded-change flag keeps the history readable.
Premature stopping. Five lines on a dashboard multiply the temptation to peek, and one of them will cross a threshold first. Peeking under continuous monitoring inflates false-positive rates 5–10× by 10,000 observations (Johansson, Shalit & Sontag, KDD 2017 — the figure printed on the homepage FAQ), so an early blink is a reason to keep running, not to ship (peeking). ProveBetter's trust layer is built for exactly this: an SRM watch against the configured split (sample ratio mismatch), a small-sample validity banner, the honest-results banner when traffic changed mid-test, and both engines on the verdict. When the two engines agree and the checks are green, you have a decision; before that, you have a trend.
Multi-variant testing answers "which version wins" with 2–5 whole versions; multivariate testing answers "which elements matter and how they interact" — a different tool with multiplicative traffic needs, and not one ProveBetter claims. On ProveBetter, variants are 2–5 whole versions per test on every tier, with per-variant traffic control, safe live rebalancing, one declared metric, and both engines on the verdict. See how the pricing compares (A/B testing cost) or start a test on the free tier.