← Data Literacy for Everyone
Module 12 Free 7 min

Before You Trust the Number: Asking Better Questions

The whole course, distilled into the questions a data-literate person asks — worked through one real claim ("satisfaction up 30%!") that quietly breaks every rule, plus a checklist to keep.

What you'll learn

  • Run the core questions that pressure-test any number before you act on it
  • Take apart a real-sounding claim that hides a baseline, a sample, a chart, and a causation problem at once
  • Keep a reusable "Before You Trust the Number" checklist for your own meetings

You’ve now met every skill in the course one at a time. This final lesson wires them together into a single habit — because in real meetings the problems don’t arrive labeled and separated. They arrive bundled inside one confident sentence, like this one, which we’ll take apart piece by piece:

“Our new customer-service process increased satisfaction by 30%.”

It sounds like great news. By the end of this lesson you’ll see that it quietly trips almost every wire you’ve learned to watch — and, more importantly, you’ll have the questions that expose it without needing to be the smartest person in the room. Just the most curious one.

The questions that do the work

You don’t need to memorize these. They fall into a few natural groups, and asking even one or two out loud usually changes the conversation.

Faced with any number, the data-literate reflex is to ask, gently: Where did this come from and how recent is it? (source, lesson 2) How was it defined? (lesson 3) What’s the baseline, and is that a percentage or a percentage-point change? (lesson 5) Average or median? (lesson 4) Is the chart’s scale fair? (lesson 6) How big was the sample, and who’s missing? (lesson 8) Is this correlation or causation — was there a control group? (lessons 7 & 9) Could it be luck, and is the difference big enough to matter? (lessons 10 & 11) None of these is aggressive. Each is just “help me understand the number” — and each one, applied to our satisfaction claim, finds something.

The moves, one line each

Check the source
Which system, defined how, as of when? Numbers drift between systems.
Check the baseline
“Up 30%” from what to what — and is that points or percent?
Check the summary
Average or median, and what does the distribution hide?
Check the chart
Where does the axis start, and what time window was chosen?
Check the sample
How many answered, who was invited, who’s missing?
Check the cause
Correlation or a real controlled comparison? What else changed?
Check the worth
Could it be luck, and is it big enough to bother acting on?

Take the claim apart

Now watch “satisfaction increased by 30%” surrender its secrets as we ask. Each new fact makes it look weaker — not because anyone lied, but because the headline hid the context.

An annotated dashboard card showing the satisfaction claim surrounded by red-flag questionsA dashboard card headlines "Customer satisfaction plus 30 percent" with a small two-bar chart whose axis starts at 48 percent, exaggerating a rise from 50 to 65 percent. Six red flags point at it: the real change is 15 points not 30 percent; only 40 people answered; only customers whose issue was resolved were surveyed; the axis starts at 48; the question wording changed; and there was no control group during the company's quietest month.Customer satisfaction+30% ▲48%5065axis starts at 48% ✎⚑ "30%" = 50→65 = 15 points⚑ only 40 people answered⚑ only "resolved" customers⚑ question wording changed⚑ no control group⚑ quietest month of the yearone confident sentence, six unanswered questionsReal read: a 15-point rise, from 40 hand-picked replies, on a zoomed-in chart, with nothing to compare against.

The same "+30%" claim, annotated with the questions this course taught you to ask. Every flag is a rule from an earlier lesson.

Text description of this diagram

At the center is a dashboard card headlining “Customer satisfaction +30% ▲” above a small two-bar chart. The chart’s axis starts at 48%, so a rise from 50 to 65 looks dramatic. Around the card sit six red flags, each a rule from an earlier lesson: “30%” is really 50→65, a 15-point rise (baseline & points vs percent, lesson 5); only 40 people answered (sample size, lesson 8); only customers whose issue was resolved were surveyed (selection bias, lesson 8); the axis starts at 48% (truncated axis, lesson 6); the question wording changed (survey design, lesson 8) — which also means the new survey isn’t comparable to the old one that surveyed all customers; and no control group, during the company’s quietest month (causation & confounding, lessons 7 and 9). The honest reading, printed at the bottom: a 15-point rise, from 40 hand-picked replies, on a zoomed-in chart, with nothing to compare against. The number wasn’t a lie — it was a headline with all its context removed.

Sort the red flags

Every problem in that claim maps to a skill you learned. Drag each flag to the kind of problem it is — or tap a flag, then a category.

Baseline & percentagesfrom what, points or percent?
Sample & surveywho was asked, how many?
Chartis the scale fair?
Causationwhat else could explain it?

Tip: drag with a mouse, or tap an item then tap a category on touch screens. Get one wrong and the answer key appears.

So what would a data-literate employee actually say? Not “that’s wrong” — but: “Encouraging! Before we roll it out — what was the baseline, and is that 30% or 15 points? How many customers answered, and were they the ones whose issues we’d already fixed? Is the chart zoomed in? And since it was our quiet month, is there a comparison group that rules out seasonality?” Every question is friendly, and together they turn a headline back into evidence.

Before You Trust the Number

Keep this. It’s the whole course as a checklist you can run in any meeting — screenshot it, print it, or paste it into your notes.

A checklist card titled Before You Trust the NumberA checklist card grouping the key questions into four areas: the number itself (source, definition, baseline, average or median), the picture (fair chart scale, time window, counts shown with percentages), the evidence (sample size, who's missing, bias, control group), and the meaning (could it be luck, is it big enough to matter, what decision it supports).✓ Before You Trust the NumberThe number☐ Where's it from & how recent?☐ How was it defined?☐ Baseline — points or percent?☐ Average or median?The picture☐ Does the axis start at zero?☐ What time window?☐ Are counts shown, not just %?☐ Any outliers hidden?The evidence☐ How big is the sample?☐ Who was included / missing?☐ Could it be biased?☐ Was there a control group?The meaning☐ Could it be luck?☐ Correlation or cause?☐ Big enough to matter?☐ What decision does it drive?The one-liner if you remember nothing else:"From what, to what, out of how many — and compared with what?"Ask it kindly. It turns any headline number back into evidence.

The whole course on one card — four groups of questions to run before any number drives a decision.

Text description of this diagram — the full checklist

A card titled “Before You Trust the Number” groups the questions into four areas. The number: Where’s it from and how recent? How was it defined? What’s the baseline — points or percent? Average or median? The picture: Does the axis start at zero? What time window? Are counts shown, not just percentages? Any outliers hidden? The evidence: How big is the sample? Who was included or missing? Could it be biased? Was there a control group? The meaning: Could it be luck? Correlation or cause? Big enough to matter? What decision does it drive? The green footer holds the one line to remember: “From what, to what, out of how many — and compared with what?” — asked kindly, it turns any headline number back into evidence. The complete 20-question version is written out in full just below, so you can copy or print it.

Here’s the complete checklist in plain text, ready to copy or print:

  1. Where did the data come from?
  2. How recent is it?
  3. How was the metric defined?
  4. What is the baseline?
  5. What time period is being shown?
  6. Are counts shown along with percentages?
  7. Is this an average or a median?
  8. Are there unusual values or outliers?
  9. Is the sample large enough?
  10. Who was included?
  11. Who may be missing?
  12. Could the survey or measurement be biased?
  13. Does the chart use a fair scale?
  14. Is this correlation or evidence of causation?
  15. Was there a comparison or control group?
  16. Could the result be random variation?
  17. Is the difference statistically reliable?
  18. Is the difference large enough to matter?
  19. What other explanation could there be?
  20. What decision will this data support?

Common misunderstanding

“Asking all these questions means I distrust data or I’m being difficult.” The opposite is true: people who ask these questions use data better, because they act on the numbers that survive scrutiny and quietly set aside the ones that don’t. Being data-literate isn’t rejecting numbers or blindly accepting them — it’s the calm middle. The questions are curiosity, not cynicism, and framed kindly they make you the most useful person in the meeting, not the most annoying.

Try this at work

Pick just two questions from the checklist and use them in your very next data meeting — “what’s the baseline?” and “how many people is that?” are the highest-value pair to start with. You’ll be surprised how often the honest answer is “let me check.” Reflect: which single question on this list would most change the decisions your team makes if everyone asked it every time?

The bottom line

Real claims arrive as one confident sentence with the context stripped out. The data-literate move is to put the context back by asking a few friendly questions — source, baseline, sample, chart, cause, and worth. You don’t need to be a statistician; you need to be curious out loud.

Why it matters

This is the habit the whole course was building toward — and it’s the one that will still be paying off years from now.

You won’t remember every term. You will remember to ask “from what, to what, out of how many?” — and that single reflex will keep you from acting on dressed-up numbers, and help you spot the good ones faster than people who just nod. One lesson remains: a glossary to look up anything you meet again, a capstone practice you can run yourself, and the final check that earns your certificate.

Quick check

1. "Satisfaction rose 30%" turns out to be a move from 50% to 65%. That's…

2. Only customers whose issues were already resolved were surveyed. This is mainly a problem of…

3. The improvement happened during the company's quietest month, with no control group. That threatens which claim?

Answers explained
  1. B is correct — 50 to 65 is a 15-point absolute rise, and 15 on a base of 50 is a 30% relative increase; both describe it, and the bigger-sounding one got the headline. (If you picked A: that would be 50 to 80. If you picked C: it’s very interpretable once you separate points from percent.)
  2. C is correct — surveying only customers whose problems were fixed hand-picks the satisfied ones, the textbook selection bias. (If you picked A: the axis is a separate flaw. If you picked B: this is about who was chosen, not luck.)
  3. A is correct — with no control group and a naturally calmer month, seasonality is an unruled-out alternative cause; you can’t credit the new process. (If you picked B or C: sample size and baseline are other issues, but the quiet month attacks causation specifically.)