Learn
A/B testing comes with its own vocabulary, and most of it gets thrown around loosely — p-values, credible intervals, SRM, uplift, CUPED. This glossary defines the terms product teams actually meet when they set up, run, and read an experiment: neutral definitions, a concrete example for each, and pointers to related terms. Everything is grouped into statistics (how the numbers work), methodology (how experiments are run), and platform concepts (how testing tools behave). Where a term names a ProveBetter feature, the entry says exactly that — nothing more.
A school of statistics that treats the conversion rate as a random variable with a distribution of plausible values rather than a single unknown number. You start from a prior — a belief before the data — observe the experiment, and update to a posterior that combines both. Decisions come from that posterior: the chance a variant beats control, the expected loss of picking wrong, and a credible interval for the true rate. The math stays tractable for A/B tests because a beta prior combined with binomial data produces another beta posterior. Example: with 600 conversions in 10,000 visitors, the posterior centers near 6% and tightens as more visitors accumulate. Related: prior, posterior, credible interval, chance to beat, expected loss. ProveBetter runs a Bayesian engine on every test beside its frequentist one; the plain‑language explainer walks through both.
The Bayesian probability that one variant's true conversion rate exceeds another's, computed from their posterior distributions. A chance to beat of 97% means: given the model, the prior, and the data so far, the variant wins 97 times out of 100. It is not the same quantity as a p-value — the two are computed differently and should not be compared digit-for-digit. Example: a dashboard reading "B beats A with 72% probability" is reporting chance to beat, not statistical significance. Related: Bayesian testing, expected loss, p-value, credible interval. ProveBetter prints each variant's chance to beat control on every result, labeled with the method — see the explainer.
A range, computed from the data, estimating where a quantity plausibly lies — a conversion rate, or the difference between two variants. Under the frequentist definition, if you repeated the experiment many times, 95% of intervals computed this way would contain the true value; it is not the probability that this particular interval contains the truth. A 95% confidence interval for the difference that excludes zero is the same statement as significance at the 5% level. Example: variant B is +0.6 percentage points versus control with a 95% interval of −1.4 to +2.6 — it crosses zero, so the difference is not significant. Related: confidence level, credible interval, p-value, significance level. ProveBetter's frequentist read prints a 95% confidence interval for the difference against control — explainer.
The long-run coverage chosen for confidence intervals and significance tests, almost always 95% in A/B testing. It trades against both error types: a higher level (say 99%) makes false positives rarer but demands more data to detect the same effect, while a lower level finds effects faster and cries wolf more often. The complement of the confidence level is the significance level α. Example: at the standard 95% level, a test aimed at 10% relative lift from a 5% baseline at 80% power needs roughly 31,000 visitors per variant. Related: significance level, sample size, statistical power, confidence interval. ProveBetter prints the confidence level on the frequentist read — explainer.
A sequence of confidence intervals that stays valid no matter how often, or when, you look at the data. Where a fixed-horizon interval is only correct for one pre-planned look, a confidence sequence allows continuous monitoring and stopping at any time without inflating the false-positive rate — the price is wider intervals early on. Optimizely's Stats Engine is built on this idea (a sequential average-likelihood-ratio test), and GrowthBook and Convert ship confidence sequences as their engine. Example: with a 95% confidence sequence, every time you check, the current interval has 95% coverage — stopping early does not invalidate the math. Related: sequential testing, peeking, confidence interval, false positive.
The Bayesian counterpart of a confidence interval — the range containing a chosen share of the posterior distribution for a rate. Where the frequentist interval is a statement about a procedure repeated many times, a 95% credible interval carries the natural reading: given the model and the data, there is a 95% chance the true rate lies in this range. Example: a variant's 95% credible interval of 4.5% to 7.4% means the posterior puts 95% of its probability mass between those two rates. Related: Bayesian testing, posterior, chance to beat, confidence interval. ProveBetter's Bayesian engine reports 95% credible intervals for every variant — explainer.
Controlled experiments Using Pre-Experiment Data — a variance-reduction technique that adjusts each visitor's outcome with a covariate measured before the experiment, typically the same metric's pre-test value. Visitors with high pre-test values carry that signal into the adjusted outcome, so individual noise shrinks: confidence intervals narrow and tests reach significance faster at the same sample size. Statsig, GrowthBook, and Eppo market CUPED in their engines; the technique is named for the 2013 paper by Deng, Xu, Kohavi, and Walker at Microsoft. Example: adjusting post-test spending by each user's pre-test spending strips out most individual difference, so the experiment's effect shows up in fewer observations. Related: confidence interval, sample size, statistical power.
A Bayesian decision quantity: what the data says it would cost — in conversion-rate points — to pick a variant that turns out not to be the best. Formally, it is the expected value of (true best rate − chosen rate) over the posterior; in practice it powers a stopping rule — keep collecting data until the expected loss of declaring the current leader drops below a tolerance you set in advance. Example: if declaring control now carries an expected loss of 0.18 percentage points and a 0.2-point loss is acceptable to you, the cost of deciding now is already negligible. Related: chance to beat, Bayesian testing, credible interval. ProveBetter reports expected loss alongside chance to beat so the Bayesian read ends in an explicit decision rule — explainer.
Failing to detect an effect that is really there — concluding "no significant difference" when the variant does actually differ. False negatives are governed by statistical power: with a test built for 80% power, the designed-for effect is missed 20% of the time by construction. They are the mirror image of false positives, and the two can only be balanced by sample size, not by wishes. Example: a test designed to catch a one-point lift keeps missing it because the sample is a third of what the power calculation required. Related: statistical power, sample size, false positive.
Declaring a difference significant when no real difference exists. The significance level you choose (α = 0.05) is the per-test false-positive budget: even a true-null experiment looks significant 5% of the time. The rate compounds — peeking at results repeatedly and running several comparisons both inflate the effective false-positive rate far beyond the nominal 5%. Example: monitoring a fixed-horizon p-value continuously under a true null is almost guaranteed to cross the threshold eventually, which is why "wait until p < 0.05" is not a strategy. Related: significance level, peeking, multiple comparisons, p-value.
The classical school most A/B testing starts from: assume there is no real difference, then measure how surprising the observed data is under that assumption. The headline output is the p-value, and the standard workhorse for conversion counts is a two-proportion z-test. The method's validity conditions matter — randomized assignment, reasonably large per-cell counts, and a single pre-planned analysis — because peeking breaks the math. Example: 10,000 visitors per variant with control at 5.0% and variant at 6.0% gives z ≈ 3.10 and p ≈ 0.0019 — significant at the 5% level. Related: p-value, null hypothesis, significance level, confidence interval. ProveBetter's frequentist engine is this two-proportion z-test, printed on every result with the method labeled — explainer.
The smallest true difference in the metric that a test is designed to reliably find, chosen before the test runs. Sample size, power, baseline rate, and the minimum detectable effect are locked together: detecting a smaller effect at the same power needs a larger sample, roughly by the inverse square of the effect size. Teams that skip this choice end up with tests that are either underpowered (can only find huge effects) or overpriced (built to find effects they do not care about). Example: detecting a 10% relative lift from a 5% baseline at 80% power needs about 31,000 visitors per variant — halving the target lift roughly quadruples the sample. Related: sample size, statistical power, significance level.
The inflation of false positives when several comparisons are tested at once — in A/B testing, several variants against one control, or several metrics. Each comparison carries its own false-positive chance, so the family-wide rate climbs past the nominal 5%. Corrections exist for this: Bonferroni divides α by the number of comparisons, Holm–Bonferroni is less conservative, and Benjamini–Hochberg controls the false-discovery rate instead of family-wide error. Example: four challengers against a control means four comparisons — at a naive 5% threshold each, the chance at least one looks significant by luck is roughly 20%. Related: false positive, significance level, multi-variant testing, p-value. ProveBetter applies a multiple-comparison correction across variants on the frequentist read — explainer; the multi-variant testing guide walks through what four comparisons actually cost.
The statistical claim a test tries to disprove — in A/B testing, that the variant and control have the same true conversion rate. The p-value is computed under this assumption: among all the data the null could produce, how extreme is what you actually saw? Rejecting the null is the formal meaning of "statistically significant"; failing to reject it means the data did not make the case, not that the variants are equal. Example: for a test of challenger B against control A, the null is "no difference" and a p-value of 0.0019 says data this lopsided is very unlikely under that claim. Related: p-value, significance level, false positive, frequentist testing.
The probability of seeing data at least as extreme as the observed data if there were truly no difference between the variants. It is not the probability that the variants are equal, and it is not the probability that B is better — the two most common misreadings. A p-value below the significance level (usually 0.05) means the data is surprising under the null, which is evidence against it but not proof of a real effect. Example: p ≈ 0.0019 on a 5.0% versus 6.0% comparison means data this lopsided would arise under the null less than twice in a thousand experiments. Related: significance level, null hypothesis, false positive, frequentist testing. On ProveBetter the p-value is printed on every result, side by side with the Bayesian read — explainer.
Checking experiment results before the planned sample size is reached. A fixed-horizon p-value is valid for exactly one look; under continuous monitoring of a true null, the false-positive rate climbs — the research behind Optimizely's Stats Engine measured inflation of 5–10× at just 10,000 observations. The remedy is sequential machinery that stays valid at any stopping time, or discipline plus honest labels on early reads. Example: refreshing the dashboard every day and declaring victory the first time p dips below 0.05 is the classic false-positive factory. Related: sequential testing, confidence sequence, false positive, p-value. ProveBetter prints validity banners — small-sample and honest-results notices — so early reads are labeled as early reads — explainer.
The Bayesian probability distribution over a rate after combining the prior with the observed data. For A/B tests it is a beta distribution whose parameters are the prior parameters plus the observed successes and failures — which is why each variant needs only two counters: conversions and visitors. The posterior is the entire answer in Bayesian testing: all intervals and probabilities are read off it. Example: with a uniform prior, 60 conversions in 1,000 visitors produce a Beta(61, 941) posterior whose mean sits just above 6%. Related: prior, Bayesian testing, credible interval, chance to beat.
The Bayesian starting belief about a rate before any experiment data arrives, expressed as a probability distribution. The neutral default in A/B testing is the uniform prior — a beta distribution with parameters 1 and 1, under which every rate from 0% to 100% is equally plausible — so the posterior is essentially data-driven. Jeffreys' prior is a common less-informative alternative. Example: choosing the uniform prior means the posterior mean is nearly the plain observed conversion rate, so the choice rarely flips a verdict. Related: posterior, Bayesian testing, credible interval.
A mismatch between the traffic split you configured and the visitors actually observed per variant — for example, a 50/50 test where 54% of traffic landed in one arm. It is detected with a chi-square goodness-of-fit test against the configured split, and practitioners flag real problems around p < 0.001 rather than 0.05 because the check runs continuously. A skewed split means assignment is broken — and the comparative statistics are invalid, no matter how significant the conversion numbers look. Example: an assignment bug that drops visitors on slow connections quietly starves one arm and produces a mismatch. Related: randomization, traffic split, assignment, false positive. ProveBetter runs an SRM watch against each test's configured split and flags mismatches on the dashboard — guide.
The number of visitors per variant an experiment needs to reliably detect its target effect. It is a function of the baseline conversion rate, the minimum detectable effect, the significance level, and the desired power — small effects, low baselines, and high confidence all push it up. Running fewer visitors than the calculation demands is the most common reason tests linger with wide, unhelpful confidence intervals. Example: at a 5% baseline, detecting a 10% relative lift at 80% power and 95% confidence needs roughly 31,000 visitors per variant. Related: statistical power, minimum detectable effect, confidence level, significance level. ProveBetter shows a small-sample validity banner when per-variant counts are too thin for the frequentist read to be trusted — guide. Per-arm sample is the same in a two-arm or five-arm test — see the multi-variant testing guide.
A family of designs that allow checking results at planned points without inflating the false-positive rate. Group sequential designs pre-register a small number of looks and spend the error budget across them via an alpha-spending function; fully sequential methods like confidence sequences allow stopping at any time. The trade-offs: sequential designs pay a modest sample-size premium or produce wider early intervals, and unscheduled looks still break them — the looks must be pre-registered. Example: a five-look O'Brien–Fleming design keeps the final threshold near z = 2.04 instead of 1.96 and costs only about 3% extra sample for the same power. Related: peeking, confidence sequence, false positive, p-value.
The false-positive threshold chosen for a test — the probability of declaring a significant difference when none exists. A/B testing almost always uses α = 0.05, meaning one in twenty true-null tests will look significant by chance. It is a dial, not a law: 0.01 is stricter (fewer false positives, larger samples needed), and the level interacts with peeking and multiple comparisons, both of which inflate the effective rate. Example: with α = 0.05 and four variants against a control, the naive family-wide false-positive rate is about 20%, which is why multiple-comparison corrections exist. Related: false positive, p-value, multiple comparisons, confidence level.
The probability that a test detects an effect of a given size when that effect is real — one minus the false-negative rate. Power depends on sample size, the effect size, and the significance level: at 80% power, the designed-for effect is missed one time in five, and raising power to 90% or shrinking the target effect both require more traffic. Sample-size calculations exist to hit a chosen power; without one, power is just whatever the traffic happened to allow. Example: a sample-size calculation at 80% power means the test is engineered to find its target effect four times out of five. Related: sample size, minimum detectable effect, false negative. Power per arm is the same whether a test has two arms or five — the multi-variant testing guide covers the per-arm cost of extra variants.
The improvement a variant delivers over control, expressed absolutely or relatively. Absolute uplift is the difference in conversion rates — B at 6.0% versus A at 5.0% is a one-point absolute lift. Relative uplift divides the difference by the control rate — the same gap is a 20% relative improvement, and it is what people usually mean by "a 15% lift." Because relative numbers depend on the baseline, always check what the percentage is computed against before celebrating. Example: moving from 5.0% to 5.5% conversion is a 0.5-point absolute lift and a 10% relative lift. Related: conversion rate, control group, confidence interval.
A randomized controlled experiment that splits traffic between a control experience and one or more challengers, then compares an outcome — conversion, signup, revenue — across the groups. Randomization is the point: with enough visitors, the only systematic difference between the groups is the experience, so a measured difference in the outcome can be attributed to it. Tests with more arms follow the same logic: A/B/C, or A/B/C/D/E. Example: sending half of checkout traffic to a redesigned page and comparing completion rates is an A/B test — the redesign wins only if the difference beats the noise. Related: control group, variant, randomization, conversion event. ProveBetter runs 2–5 variants per test (A–E), client-side and server-side alike — homepage.
The mechanism that decides which variant each visitor gets — typically a deterministic hash of a stable visitor ID across the configured traffic percentages. Assignment shapes the denominators of every metric, so it must be stable (same visitor, same variant), random-looking (no self-selection), and complete (every eligible visitor assigned exactly once). A broken assignment is the root cause of sample ratio mismatches and quietly biased results. Example: hashing visitor ID 42 against a 50/50 split lands that visitor in control on every visit, not bouncing between arms. Related: sticky assignment, visitor ID, traffic split, randomization. Server-side ProveBetter tests assign through POST /api/assign — API reference.
The fraction of visitors who reach one funnel step who also reach the next — per variant, per step. It isolates where in a funnel variants separate: a variant that improves signup but hurts activation shows a strong first continuation and a worse second. Continuation rates must be paired with the counts behind them — 100% of three users is noise — and because steps share visitors, only the declared primary goal should drive the headline verdict. Example: of 1,000 visitors who reached the pricing page, 120 in variant B reached checkout versus 90 in control — B progresses better at that step. Related: conversion event, guardrail metric, exposure.
The baseline experience a test compares against — variant A in most tools. The control should be the current, unchanged experience, drawn from the same traffic at the same time, because only comparison against a simultaneous baseline neutralizes day-of-week and campaign effects. Everything else — uplift, p-value, chance to beat — is measured relative to it. Example: in a headline test, the existing headline is the control and the two candidates are challengers B and C. Related: variant, A/B test, randomization, uplift. Every ProveBetter test pins variant A as the control with up to four challengers (B–E) — homepage.
The countable outcome a test is designed to move: a purchase, signup, click, form fill, or custom action. Conversions are counted against the exposed visitors of each variant to build conversion rates, and the event definition must be identical across variants or the comparison is invalid. Choosing the conversion event — and declaring it before the test — is what turns an experiment from an exploration into a decision. Example: a pricing-page test may define the event as "clicked Start trial," a step ahead of payment that still tracks the page's job. Related: conversion rate, metric targeting, exposure. ProveBetter's goal picker covers call, click, form fill, sale, signup, retention, and custom events — API reference.
The proportion of exposed visitors who complete the conversion event — conversions divided by visitors, usually shown as a percentage. It is the primary number in most A/B tests, and small differences between rates are exactly what inferential statistics are for: a 5.2% versus 5.8% gap in 1,000-visitor arms is well within noise (p ≈ 0.56), while 5.0% versus 6.0% at 10,000 visitors per arm is highly significant (p ≈ 0.002). The same percentage gap means different things at different sample sizes. Example: 58 conversions among 1,000 exposed visitors is a 5.8% conversion rate. Related: conversion event, uplift, exposure, sample size.
The moment a visitor is assigned a variant and enters the experiment's denominator — the population against which conversions are counted. Exposure must be the earliest and cheapest event in the funnel, because selection happens at the door: if exposure fires late or only on fast pages, the visitor populations skew between arms and the test can show a sample ratio mismatch without anyone touching the assignment code. Example: a visitor lands, the snippet assigns variant B and records the exposure — that visitor is now part of B's denominator whether or not they convert. Related: conversion event, visitor ID, assignment, sample ratio mismatch. ProveBetter counts an exposure only when the snippet actually serves a variant — on slow connections visitors see control and are not counted — install.
A metric monitored not to win a test but to catch harm — the safety checks that keep a conversion win from hiding a support-ticket flood or a revenue collapse. Guardrails are declared alongside the primary metric and read with a different bar: any unambiguous movement is a reason to stop and investigate, not a reason to ship. Example: a checkout test that lifts completion but tanks average order value fails its guardrail and does not ship. Related: metric targeting, conversion event, novelty effect.
A portion of traffic deliberately kept out of an experiment — either assigned to a permanent global control group or reserved for future tests. Holdouts give a baseline that never experiences any variant, which is how long-run and network effects get measured, not just immediate conversion. Many large experimentation programs keep a small permanent holdout for exactly this reason. Example: a 5% holdout never sees an experiment variant, so after a year the team can compare ever-experimented users against never-experimented users on retention. Related: control group, guardrail metric, partition testing.
Choosing, at test creation, the single metric the test will be judged on, rather than treating every dashboard number as the result. Pre-declaring the primary metric prevents the habit of hunting through all the metrics for the one that happens to look significant — itself a multiple-comparisons trap — and it is what makes the eventual verdict legible. Example: a signup-flow test targets "signup completion"; the time-on-page bump in the data is context, not the verdict. Related: conversion event, multiple comparisons, guardrail metric. ProveBetter's builder asks for a goal metric at creation and the dashboard reports in that metric's language — guide.
Running more than one challenger against a control — A/B/C or A/B/C/D/E — to answer "which of these versions wins" in one experiment. The cost is statistical: every variant adds a comparison (see multiple comparisons) and dilutes the traffic per arm. Practical guardrails: cap the number of variants at what your traffic can support, and do not change the variant set mid-test. Example: three headline candidates against the current headline is a four-arm test answering one question — which headline converts best. Related: multivariate testing, multiple comparisons, traffic split, control group. ProveBetter supports 2–5 variants labeled A–E with per-variant traffic percentages — homepage; a full walkthrough lives in the multi-variant testing guide.
Testing combinations of independent changes at once — headlines × images × button labels — to learn not just what wins but what interacts. A 2×2×2 design is eight combinations, and sample requirements grow multiplicatively, which is why multivariate testing needs serious traffic. It is a different tool from multi-variant testing: MVT tests combinations of elements, while A/B/C tests whole page versions. Some platforms sell MVT (Optimizely is the notable one), and VWO keeps its multivariate engine frequentist-only. Example: if headline, image, and button each have two candidates, MVT serves all eight combinations to separate each element's effect from the interactions. Related: multi-variant testing, sample size, multiple comparisons. ProveBetter is not an MVT platform — its 2–5 variant tests are whole-version A/B/C/D/E tests, a distinction the Optimizely comparison spells out, as does the multi-variant testing guide.
The short-lived change in behavior that comes purely from a feature being new — visitors click the shiny button — and fades as the novelty wears off. Early data overstates the effect of visible changes, so the standard guard is to let novelty decay before judging and to treat early lifts skeptically when the change is prominent or announced. Example: a redesigned checkout shows a 12% lift in week one that settles to 2% by week four — week one was mostly novelty. Related: peeking, guardrail metric, holdout.
The practice of dividing eligible traffic into disjoint partitions so concurrent experiments never share visitors — each partition hosts its own experiment, and a visitor in partition one cannot be assigned to partition two's test. It prevents interference, where one experiment's treatment contaminates another's control and both results distort. Partitioning is how organizations with many concurrent tests keep their experiments independent. Example: team A's pricing test runs on partition one (10% of users) while team B's onboarding test runs on a different 10% — no user is in both. Related: holdout, traffic split, randomization.
Assigning visitors to variants by chance rather than by choice, login state, or referral path — the step that makes groups statistically comparable. Proper randomization means the only systematic difference between arms is the variant itself; anything self-selected (users who opt into the "new" experience) breaks comparison. In practice it is a deterministic hash of a stable visitor ID, which looks random but is reproducible per visitor. Example: hashing the visitor ID into a 70/30 split is randomization; letting users choose their experience is not. Related: assignment, control group, visitor ID.
The statistical effect where an aggregate result reverses direction when the data is split into groups — and a classic failure mode for mid-test traffic changes. If the split changes mid-test, early and late visitors form different cohorts with different time-on-test; each cohort can prefer variant B while the pooled result favors A. The practical rule: changing traffic allocation mid-test mixes cohorts and biases conversion and continuation rates, which is why mid-test changes need to be explicit and recorded. Example: control loses every daily segment yet wins the pooled total — the classic signature of a mid-test split change. Related: sample ratio mismatch, traffic split, peeking. ProveBetter allows live traffic adjustment but flags it — the dashboard prints "traffic changed mid-test — results may be biased" — guide.
The property that a visitor keeps the same variant across sessions, page loads, and API requests — the assignment sticks to their ID. Stickiness matters twice: for experience (a user should not see variant A on one page and variant B on the next) and for statistics (per-visitor conversion, where one user can convert many times, requires a stable bucket). Without it, visitors relabeled mid-test quietly corrupt denominators. Example: a visitor bucketed into variant B on Monday is still variant B on Wednesday, even though the traffic percentages were rebalanced on Tuesday. Related: assignment, visitor ID, traffic split, exposure. ProveBetter assignment is sticky — server-side tests pass a stable visitor ID to /api/assign and get the same variant back — API reference.
The configured percentages deciding how much eligible traffic goes to each variant — 50/50 for two arms, or uneven splits when control needs a bigger baseline. The split sets the denominators of per-variant rates, so changing it mid-test must be done deliberately — with sticky assignment and cohort awareness — rather than casually. Example: a test set to A 50% / B 30% / C 20% serves the control to half of all eligible visitors. Related: assignment, randomization, sample ratio mismatch, sticky assignment. ProveBetter sets per-variant traffic percentages at creation and lets you adjust them live with sticky assignment — guide.
One of the experiences in a test: the control (usually A) plus each challenger (B, C, D, E). Each variant is served to its share of traffic and measured on the same metric. The word is shorthand for "variation," and it is interchangeable with treatment arm or bucket depending on the tool. Example: in a three-arm test, variant B might be the headline rewrite while variant C changes the button color. Related: control group, A/B test, traffic split.
A stable identifier for a visitor — cookie, login ID, or device ID — used to assign variants deterministically and to deduplicate events. Stability is the whole game: if the ID changes between requests, the visitor looks like a new person, stickiness breaks, and one real user inflates the sample. Privacy-conscious teams trade some stickiness for anonymity, but the trade should be explicit. Example: server-side testing sends the signed-in user's ID to the assignment API so their variant survives reloads and other devices. Related: sticky assignment, assignment, exposure, randomization. ProveBetter's server-side API takes a stable visitor ID that you supply — API reference.
Techniques that prevent the flash of original content — the visible blink of the old page before the variant applies. The standard approach is a boot snippet that hides the page while the variant decision is made and the change applied, with a timeout so the page always renders even if the snippet is slow — hiding forever is worse than flickering. Example: a boot snippet hides the headline for a few hundred milliseconds while variant B's copy loads, then reveals B — no blink for the visitor. Related: exposure, SDK and snippet. ProveBetter's boot snippet (~1.4 KB gzipped) hides up to 2 seconds; on slow connections visitors see control and are not counted — install.
An append-only record of who changed what in a test and when — traffic percentages, variant content, goal metrics, schedule changes. The audit trail is what lets a team reconstruct why a test's history looks the way it does, turning "this result looks odd" from a mystery into a lookup. Example: a results review finds traffic was rebalanced from 50/50 to 80/20 last Thursday — the audit trail shows who did it, and the results page now carries the honest-results banner. Related: scheduling, winner rollout, traffic split. ProveBetter records test changes — including goal-metric edits — in a full audit trail — guide.
The ability to pull raw experiment data — assignments, exposures, conversions — out of the testing tool into your own warehouse or files for independent analysis, custom modeling, or joins with internal data. Export is a trust feature: a platform that lets you take your data out is also one that lets you verify its numbers. Example: a nightly sync pushes yesterday's events to the warehouse, where the data team joins them with billing records. Related: event metering, audit trail, conversion event. Every ProveBetter tier includes CSV/JSON and API export; Enterprise adds scheduled warehouse sync — pricing.
A pre-built, read-only experiment shipped with a platform so new users can click through a real test — variants, results, verdicts — before creating their own. It is the fastest way to see what the product actually outputs, and it doubles as a testbed for reading statistics. Example: a demo headline test shows variant A versus B with conversion counts and a plain-language verdict, no account needed. Related: A/B test, variant, winner rollout. ProveBetter includes a demo experiment you can explore before creating your own — homepage.
Pricing or limiting a testing tool by the number of events it receives — each assignment (exposure) and each tracked goal or custom event counts. Event-based billing scales the invoice with actual experimentation volume rather than audience size, which suits small teams and traffic spikes alike; the mirror side is that event limits need honest overage signaling — warn first, then gate — so tests never silently lose data. Example: a 1M-event plan charges nothing extra until the month's exposures and conversions pass 1M, and warns at 85%. Related: rate limit, conversion event, data export. ProveBetter prices by event volume — Free 1M, Small 5M, Medium 25M, Enterprise 100M events per month, with a warning at 85% of plan usage — pricing; the cost explainer compares the billing models across tools.
A way to inspect what each variant looks like on the real page before or during a test, without polluting the data. Preview is the quality gate for variant content — typos, layout breaks, wrong copy — and it must be invisible to the statistics: previewed views must not count as exposures or conversions. Example: pressing the preview shortcut on the live site cycles through variant B's rendered page so you can verify the hero copy before real traffic sees it. Related: exposure, anti-flicker, variant. ProveBetter's preview (Ctrl+Shift+Alt+T) shows each variant on your page, and previewed views are suppressed from exposure and conversion counts — install.
A cap on how many requests a client can send to an API within a time window — a protective measure that keeps a misbehaving snippet, a bot, or a buggy integration from flooding the event pipeline. Rate limits are part of data integrity: garbage requests are not just load, they are potential fake exposures and conversions that can tilt a test. Example: an event endpoint accepting 600 requests per minute per test answers a runaway loop with 429 responses instead of recording phantom events. Related: event metering, conversion event, SDK and snippet. ProveBetter rate-limits the event and assignment endpoints per test (600 requests/minute) so experiment data stays clean — API reference.
Setting a test to start or stop automatically at future times instead of flipping it by hand. Scheduling matters for experiments tied to launches, seasons, or campaigns — and it interacts with the statistics, because a schedule is a commitment to a pre-planned observation window, which is exactly what fixed-horizon methods assume. Example: a Black Friday checkout test is scheduled to start November 1 and end November 30, with no one on call either night. Related: winner rollout, audit trail, sample size. ProveBetter supports future scheduling of test start and stop — API reference.
The code a platform hands you for client-side testing: usually a small boot snippet in the page head — to lock in the variant before render — plus a fuller SDK that assigns variants and reports events. Server-side testing flips this: no snippet at all, the platform exposes an HTTP API, and your backend decides the variant before the page renders. Example: a boot snippet briefly blocks rendering while it fetches the variant, then the SDK records the exposure and later the conversion. Related: anti-flicker, exposure, event metering. ProveBetter's client-side SDK is a ~1.4 KB boot snippet plus ~4.2 KB SDK (gzipped); server-side tests need no SDK, just HTTP — install.
The act of shipping the winning variant to everyone once a test concludes, instead of leaving traffic split. A clean rollout ends the experiment, serves the winner to 100% of the test's traffic, and records the decision. Some platforms add progressive rollout (10% → 50% → 100%) to catch fleet-wide surprises; the key is that the winner decision and the rollout are separate, recorded steps. Example: after six weeks, B beats control decisively — one click rolls B out to 100% and the test is archived as a decision. Related: holdout, scheduling, audit trail, multi-variant testing. ProveBetter's winner rollout is one click to 100% of in-test traffic — homepage FAQ; rolling out a five-arm winner is the same one click, covered in the multi-variant testing guide.