Significant Isn't the Same as Important

What you'll learn

Separate statistical significance (is it real?) from business significance (is it worth doing?)
See why a tiny percentage at huge scale can beat a huge percentage at tiny scale
Weigh evidence, impact, and cost together before acting on a result

Last lesson ended on a cliffhanger: a result can be statistically significant — luck-proof, real — and still not worth a single hour of anyone’s time. “Significant” answers is it real? It says nothing about the question a budget owner actually cares about: is it worth doing? Those are two separate dials, and a data-literate person reads both. This short lesson is about the second dial — and it’s where a surprising number of expensive mistakes (and missed opportunities) hide.

Two dials, not one

Statistical significance tells you a result is real. Business significance tells you it’s worth acting on. High on one dial says nothing about the other.

The words that matter

Statistical significance: Whether a result is bigger than random luck — the “is it real?” dial from last lesson.
Practical significance: Whether the effect is large enough to make a real-world difference, regardless of what the statistics say.
Business significance: Whether acting on the result creates enough value — after cost, effort, and risk — to be worth doing.
Effect size: How big the change actually is (0.1 points? 40 points?), separate from whether it’s statistically real.
Scale: How many people or transactions the effect applies to. A tiny effect times a huge number can be enormous.
Return on investment (ROI): The value a change delivers weighed against what it costs to build and run. The final arbiter of “worth it.”

Two real examples make the gap obvious. Example 1: an online retailer runs a test on millions of checkouts and finds a 0.1% conversion improvement — a rounding error to the eye, but rock-solid statistically. Across millions of orders, that “trivial” 0.1% might be worth millions of dollars a year. Tiny effect, gigantic scale, huge business value. Example 2: a pilot shows a 50% improvement — but it went from 2 customers to 3. Enormous percentage, thrilling slide, and almost meaningless: one more customer isn’t dependable evidence of anything, won’t reliably repeat, and doesn’t move the business. Big effect, tiny scale, near-zero value. The percentages point in exactly the wrong directions from the value.

Business value comes from evidence × effect × scale, minus cost — not from the size of the percentage.

Text description of this diagram

The visual is a two-row scorecard. Each result is rated across four columns: Evidence (is it real?), Effect size (how big is the change?), Scale (how many does it apply to?), and Business value (the verdict). Row one — +0.1% conversion across millions of orders — scores strong evidence, a tiny effect, but huge scale, landing on HIGH — ship. Row two — +50% adoption, from 2 to 3 customers — scores weak evidence, a big-looking effect, but tiny scale, landing on LOW — wait. The amber banner names the twist: the headline percentage points the opposite way to the value. The 0.1% is the valuable one; the 50% is the empty one. Business value is evidence times effect times scale, minus what it costs to act — and the percentage on the slide tells you almost none of that.

Real, but is it worth it?

Even a perfectly real, well-powered result has to clear one more bar before it deserves action: is the value worth the cost? A significant 0.3% lift that would take six months of engineering, add operational risk, and confuse customers may simply not be worth building — the ROI isn’t there. Meanwhile a small, cheap, low-risk change with a modest effect might be an easy yes. Strength of evidence, size of impact, scale, cost, effort, risk, and customer impact all go into the decision. Statistics settle exactly one of those inputs; the rest are judgment. That’s why “it’s significant” is the start of a business conversation, never the end of one.

Common misunderstanding

“It’s statistically significant, so we should do it.” Significance clears only the “is it real?” bar. Whether to act depends on the size of the effect, how many it touches, and what it costs in money, effort, and risk to implement. A real result with terrible ROI is a pass; a small real result that’s cheap and safe is often a yes. Never let “significant” auto-approve a decision — it’s one input, not the verdict.

Try this at work

When a significant result is used to justify action, ask the business-significance question: “Real — agreed. How big is the effect, how many does it touch, and what will it cost us to act?” That reframes the room from “is it true?” to “is it worth it?”, which is the decision that actually spends money. Reflect: think of a flashy percentage you’ve seen in a deck — was it big because the effect mattered, or big because the starting number was tiny?

The bottom line

Statistical significance says a result is real; business significance says it’s worth doing — and they’re independent. A tiny effect at huge scale can be hugely valuable; a huge percentage at tiny scale can be worthless. Value is evidence × effect × scale, minus cost — and the headline percentage tells you almost none of it.

Why it matters

The gap between “real” and “worth it” is where budgets are won and wasted.

Hold both dials and you stop being swayed by a big percentage or a small p-value on its own, and start asking the fuller question that leads to good calls. You now have every individual skill in the course — sources, metrics, averages, percentages, charts, causation, surveys, experiments, significance, and importance. The final lesson ties them into a single habit: the short list of questions that turns any number thrown at you into something you can actually judge.

Quick check

1. A 0.1% conversion lift across millions of orders is statistically significant. Is it worth acting on?

2. A pilot shows "+50% adoption" — from 2 customers to 3. The right reaction is…

3. A result is real and significant, but acting on it needs six months of work and adds risk. What decides whether to do it?

Answers explained

B is correct — effect size times scale is what creates value, so a tiny per-order lift across millions can be worth a fortune. (If you picked A: judging by the percentage alone is the exact mistake. If you picked C: a bigger percentage on a smaller base could be worth far less.)
C is correct — a jump of one customer is neither dependable evidence nor meaningful scale, so the eye-catching 50% is nearly empty. (If you picked A or B: the percentage is inflated by a tiny baseline and proves little.)
A is correct — once a result is real, the decision turns on business significance: value versus cost, effort, and risk. (If you picked B or C: significance settles “is it real?”, not “is it worth it?”.)