The Data Scientist: Give the Agent Access to the Market

On Tuesday morning Anika Raghunathan has Wes Okafor’s product truth document open on one screen and an empty configuration file on the other. The document tells her exactly what Supply Signal does: monitors 1.4m suppliers, alerts on nine risk signals, does not predict shortages, has no native SAP connector at launch. Her job is to find out what the market around that product looks like — who else sells into it, what buyers complain about, what words they use, what changed last quarter — in three days rather than the three weeks an analyst would need.

She could have an answer in four minutes: paste the nine research topics into a chat window and receive fifteen pages of confident, well-organised prose about the supply chain risk market, a substantial fraction of it wrong in ways nobody downstream could detect. Almost everything Anika does this week is designed to make that outcome impossible.

What lands on Anika’s desk

Two things arrive. The first is Wes’s document. Without it the agent would research “supply chain software” — a category containing freight forwarding, warehouse robotics and customs paperwork — and return a thousand accurate facts about the wrong market.

The second is Delia’s rule from module 1: an agent may produce anything, and a named human owns every claim that reaches a customer. Anika is that human here, and a system whose output you must personally defend is a different system from one whose output you merely pass along.

The team has told her what it needs to know. Nine things: competitors, industry trends, customer problems, product announcements, pricing changes, market terminology, recent news, relevant statistics, and what customers say to each other in reviews and forums.

What a data scientist does on a marketing campaign

Most people expect a data scientist to build a predictive model. Here Anika builds something less glamorous and more useful: a research pipeline. She writes the instructions telling an AI agent what to look for, connects it to tools that let it read the live internet, and — the part that takes most of the three days — designs the checks that decide which of its output survives.

Everything turns on one distinction: a model answering from memory versus an agent that goes and looks. A large language model has compressed an enormous amount of text into statistical patterns. It does not store facts the way a database does; it stores the shape of language about facts. Ask it who competes with Supply Signal and it produces a fluent paragraph naming plausible companies with plausible positioning — but it cannot tell you that its training data ended months ago, that Nodefield changed its pricing in April, or that one of the companies it just named does not exist.

The software on Anika’s desk

The search API is the piece that makes any of this honest: it gives the model access to what exists now, rather than to what it happened to read during training. Around it, Python implements a loop that is unglamorous by design — issue the query, fetch the page, extract the specific claim, attach the URL and the date, store it. The loop is where the discipline lives, not the model.

Source scoring is the step that distinguishes research from confident noise. A regulator’s filing and an anonymous forum post are both text; treating them as equal evidence is how a market report ends up asserting a competitor’s marketing copy as fact. And the warehouse holds the result where Joel and Lena can query it later — 340 rows that survived, every one traceable. The 224 statements dropped for having no traceable source are the module’s real product: those are the sentences that would otherwise have been quietly believed.

The four decisions that make the agent trustworthy

Retrieval, not recall

The fix for a model that invents things is not a better model — better models invent less often and more convincingly, which is arguably worse. The fix is to change the job: stop asking the agent what it knows, and ask it to find, read and quote.

So Anika connects it to a web search API and rewrites its instructions around a different task. Not “tell me about the supply chain risk market”, but “search for this, open the results, extract what these documents actually say, and attribute every statement to the document it came from”. The agent’s value is not knowledge but breadth of reading and discipline of attribution — a very fast, very literal research assistant who is never tempted to fill a gap from memory.

The rule she writes into the pipeline is why the whole thing works: no claim leaves the agent without a URL and a date attached. Not “industry reports suggest” — a link that resolves and a publication date. The second half of the rule is the half most teams skip: a claim that cannot be sourced is deleted, not softened. Softening is how an unsourced assertion survives. “Some analysts believe” is not a weaker claim than “analysts say”; it is the same claim with the evidence removed and the confidence hidden.

What to ask it — turning nine topics into searches

“Research the market” is a useless instruction to an agent for precisely the reason it is useless to a graduate in their first week: it does not say what would count as an answer. The person who gave it will know an unsatisfactory result when they see one and cannot describe a satisfactory one in advance. An agent, unlike a graduate, will not come back and ask.

So Anika decomposes the nine topics into structured, repeatable queries. Competitors splits into named searches — Arbor Risk, Nodefield, each by product name, pricing page, announcements, job postings, reviews — and into category searches for the competitors nobody at Cadence has thought of yet. Both are necessary and they fail differently: named searches miss entrants, category searches drown in irrelevance.

She also separates questions that have answers from questions that have opinions. “What does Arbor Risk charge for its mid-market tier?” has an answer, and the agent’s job is to find the published price or report its absence. “Is Arbor Risk too expensive?” has only opinions, and there the agent’s job is to collect who said it, where and when — never to adjudicate. Blurring the two is how a research deck ends up asserting a competitor is overpriced on the evidence of four annoyed people on a forum.

And she saves the query set as a versioned artefact rather than a one-off prompt, so it can be re-run on a schedule. Fourteen weeks is long enough for the market to move underneath the campaign; when it does, the team refreshes the picture in an afternoon instead of rebuilding it, and can see exactly what changed between runs.

Source quality, and the tiers

This is where the real judgement lives. The agent will happily treat a regulator’s filing and an anonymous forum post as the same kind of thing, because to a language model both are text. The difference is not credibility on a single scale; they are different kinds of evidence, good for different purposes.

Anika stores the tier alongside every claim, so anyone downstream can see not just what was said but what sort of thing said it. She is strictest about the vendor rule, and applies it symmetrically. “Arbor Risk deploys in four weeks” is never recorded as a fact; it is recorded as Arbor Risk states, on its pricing page, on 14 May, that it deploys in four weeks. That reformulation is the whole difference between competitive intelligence and repeating a competitor’s marketing to your own board. Cadence gets the same treatment — its own site is vendor material about its own product, which is why Wes’s document exists.

Reviews and forums get a particular instruction: harvest them for language and complaints, never for statistics. If eleven of forty reviewers say alerts arrive too late, that is not eleven-fortieths of the market and must never become a percentage. It is a theme worth investigating, and the exact words those eleven people used are worth more to Naomi in module 7 than any number in the dataset. “Supplier risk” is what vendors say; “getting blindsided” is what buyers say.

What the agent must never do

The last decision is a set of prohibitions, written into the agent’s instructions and enforced by checks over its output rather than trusted to good behaviour. Each exists because of something specific that breaks without it.

Never assert without a source, because three weeks later an unsourced sentence is indistinguishable from a sourced one — by which time it is on a landing page and nobody remembers where it came from. Never fuse two sources into one sentence that neither supports, the most dangerous failure of all, because the result looks better-researched than either original and is supported by nothing. Never repeat a competitor’s claim as fact, because Cadence would then be publishing Arbor Risk’s marketing under its own name, with Cadence’s legal exposure attached. Never scrape what a site’s terms of service forbid, because cheap automated collection is not thereby lawful and Miriam will ask in module 9. And never put customer data into a public API — no account names, no contract values, no ticket text — because the moment confidential information leaves the building through a research tool, it stops being a marketing problem.

What the agent is genuinely good at, and what it cannot do

The agent runs 2,400 searches in three days, and the breadth is real value: it reads the ninth page of results and the trade publication nobody at Cadence subscribes to, applies the same extraction format to the four-hundredth source as to the first, and does not get bored at four in the afternoon. It is unusually good at surfacing the terminology a market actually uses, having no professional pride invested in the terminology Cadence prefers.

What it cannot do is decide which finding matters. It returns that Arbor Risk is enterprise-grade and slow, and that Nodefield is cheap with shallow coverage — both sourced, both weighted identically — and has no view on the fact that the most important competitor here is nobody doing anything at all, most buyers still running the problem on a spreadsheet. It cannot notice that a whole segment is missing, because absence leaves no trace in a search result. And it cannot feel that a review is angrier than it reads. That is Joel’s job in module 4.

Where this goes wrong

The common failure in real companies is a research agent deployed without the verification step, because verification is boring and the output looks finished without it. Somebody runs the nine topics through a model and circulates forty pages of fluent market analysis that reads better than anything the analyst team has produced in years. Three weeks later a statistic from page nineteen is on a landing page, a prospect asks where it came from, and nobody knows — it was in the document when the document arrived.

Anika’s pipeline produces 1,180 candidate claims; 340 survive with a source and a date attached. Most of the loss is duplication and irrelevance, the ordinary waste of broad searching. But 19% of everything the agent produced — some 224 statements — was dropped for one specific reason: it could not be traced to any source that actually said it. That is the most important number in this module. Those 224 sentences were fluent, specific and plausible, and every one would have been quietly believed downstream if nobody had built the check.

What Anika hands on

Joel Brennan receives 340 sourced claims, each carrying its URL, its publication date and its source tier, organised against the nine original topics. He also receives the query set, so the picture can be refreshed rather than rebuilt when the market moves in week nine, and a written list of the gaps — the questions the agent searched for and could not answer, which is information rather than failure.

The constraint he inherits is a standing warning Anika writes at the top of the file, and it shapes his entire module: this dataset describes what the market says, not what customers feel. Every claim in it was published by somebody with a reason to publish. The people who never wrote a review, never posted in a forum and never spoke to a journalist are not in here at all — and they are most of the market.

Designing this desk’s agent: the market research agent

Everything Anika built comes down to one refusal: a sentence with no source is deleted rather than softened. The rest of the design exists to make that refusal enforceable at the scale of 2,400 searches.

One concrete design choice. Make provenance a column rather than a convention. The claim table has no nullable url and no nullable retrieved_at, so an unsourced claim cannot physically be written — which is a stronger guarantee than any instruction in a prompt.

{
  "claim_id": "c-01847",
  "text": "Arbor Risk lists a 12-week implementation on its pricing page",
  "url": "https://...",
  "published_at": "2026-05-02",
  "retrieved_at": "2026-06-14T09:12:00Z",
  "source_tier": "vendor_self_reported",
  "is_fact": false,
  "superseded_by": null
}

The metric to track. Two. Survival rate — 1,180 candidates down to 340 kept — which tells you how much of what an unconstrained agent produces would not have withstood checking. And spot-check precision: pull thirty surviving claims at random each cycle and verify them by hand. If two of thirty are wrong, roughly twenty-three of the 340 are, and you now know that rather than hoping.

Sourced, or just fluent?

Read each one and decide whether it survives Anika’s check, then tap a card.