A/B Tests in Plain English

What you'll learn

Explain what an A/B test is, and why random assignment is what makes it trustworthy
Name the parts: control, treatment, outcome metric, conversion rate
Know the rules that keep a test honest — one change, similar groups, enough time, success defined up front

Two lessons ago you learned the hard truth that a pattern in your data — feature users churn less, app users spend more — can’t prove the feature caused anything. Here’s the good news that resolves it: there’s a clean way to actually find out, and companies run thousands of them a day. It’s called an A/B test, and once you understand the shape of it, “did that change actually work?” stops being an argument and becomes something you can settle. No math required — just the logic of a fair comparison.

Change one thing, split the room in two

An A/B test shows two versions to two randomly chosen groups at the same time, then compares what happened. Randomness is the whole trick.

Suppose a website wants a better sign-up button. Version A (the current one) says “Create Account.” Version B (the challenger) says “Start Free.” Instead of arguing in a meeting about which is better, the site simply shows A to a random half of visitors and B to the other random half, at the same time, and counts how many in each group actually sign up. Whichever wins, wins on evidence. The magic ingredient is that the two groups are chosen at random, so they’re alike in every other way — same mix of new and returning, mobile and desktop, morning and evening. Because the only systematic difference between them is the button, any difference in sign-ups can finally be blamed on the button. That’s the causation proof correlation could never give you.

The words that matter

A/B test: Show version A to a random half of users and version B to the other half, then compare an outcome. Also called a split test or experiment.
Control group: The half that sees the current version (A) — your baseline for comparison.
Treatment group: The half that sees the change (B). Also called the variant.
Random assignment: Deciding who sees which version by chance, so the two groups are alike except for the change. This is what makes the test trustworthy.
Outcome metric: The one thing you’re measuring to decide the winner — here, whether someone signed up. Chosen before the test starts.
Conversion rate: The share of people who took the action you wanted — sign-ups ÷ visitors. The most common outcome metric in A/B tests.

Random assignment makes the two groups alike, so any gap in sign-ups can be pinned on the one thing you changed.

Text description of this diagram

Visitors arrive on the left and hit a random split that sends 50% one way and 50% the other. The control group (A) sees the current “Create Account” button and signs up at 4.0%; the treatment group (B) sees the new “Start Free” button and signs up at 4.6%. Because assignment was random, the two groups are otherwise alike, so the two rates can be compared directly and the difference attributed to the button. The banner underlines the design: same visitors, one difference: the button. Change only one thing, split fairly, measure a pre-agreed outcome — that’s the entire logic of an A/B test.

The rules that keep a test honest

The shape is simple; the discipline is where tests go right or wrong. Four rules matter. Keep the groups similar — random assignment does this automatically, which is exactly why you don’t let people self-select into the new version. Change only one thing — if B has a new button and a new headline and a new price, a win tells you nothing about which change caused it. Run it long enough — a test that stops after a day is measuring a Tuesday, not your customers; behavior varies by day, week, and season. And decide what success means before you look — pick the outcome metric and the finish line up front, because a team that starts hunting through the data after the fact will always find something that looks like a win.

That last rule has a name-worthy failure mode: define “better” as “more sign-ups” before the test, or you’ll be tempted to declare victory on whatever number happens to be green when you peek. Discipline before data is what separates an experiment from a story.

Put the test in order

A clean A/B test runs in a fixed sequence. Tap each card to check the step.

Common misunderstanding

“We changed the whole page and conversions went up, so the new button works.” If you changed several things at once, you’ve learned that something helped — not what. Maybe the headline did all the work and the button actually hurt. This is why serious tests isolate one variable at a time; a bundle of changes tested together is a mystery with a happy ending, not evidence. One change per test is what makes the result mean something.

Try this at work

When someone proposes “let’s just try the new version,” ask the two questions that turn a guess into an experiment: “What’s the one outcome we’ll judge it on, and how will we split users fairly?” If the answer is “we’ll roll it out to everyone and see,” there’s no control group — and no way to know whether any change was the cause. Reflect: think of a recent change your team shipped. Could you actually prove it helped, or did you just ship it and hope?

The bottom line

An A/B test splits users randomly, shows each group one version, changes just one thing, and compares a pre-agreed outcome. Random assignment makes the groups alike, so the difference can be blamed on the change — the clean causation proof a correlation can never provide.

Why it matters

Experiments are how modern companies decide what to ship — which button, which price, which flow — without betting the business on an opinion.

Understanding the shape of an A/B test lets you ask whether a “proven” result was actually tested fairly: one change, random split, enough time, success defined up front. But there’s one more question hiding in that 4.0% vs 4.6% result — could the gap just be luck? A different set of visitors on a different week might have flipped it. Deciding when a difference is real and when it’s noise is the job of the phrase everyone drops in meetings and few can define: statistically significant. That’s the next lesson.

Quick check

1. What makes an A/B test able to show causation when a plain correlation can't?

2. Version B changed the button, the headline, and the price all at once, and won. What did you learn?

3. Why decide the outcome metric before the test starts?

Answers explained

B is correct — randomization makes the control and treatment groups alike in every way except the change, so a difference in outcome can be attributed to that change. (If you picked A: size helps confidence, not causation. If you picked C: who runs it doesn’t create the logic — the random split does.)
C is correct — bundling changes means a win can’t be pinned to any one of them; you’d need to isolate variables to know. (If you picked A or B: either could be the cause — or could even have hurt while another change carried the win.)
A is correct — pre-committing to the outcome and finish line stops after-the-fact cherry-picking of a favorable number. (If you picked B: metrics can be measured later — the issue is honesty, not capability. If you picked C: it doesn’t change the test’s speed.)