The ML Engineer: From Notebook to Monday Morning

Kofi’s desk on a Monday morning holds three things that do not yet fit together. On one screen is Priya’s notebook, which produces an excellent ranked list of at-risk accounts in about ninety seconds, provided you run the cells in the right order on Priya’s laptop with Priya’s credentials. Taped to the side of the monitor is a printed page of Elena’s conditions from module 9 — human review with the reason shown, role-based access, monthly monitoring with a named owner. And at the top of his inbox is a cheerful message from Sofia with the subject line “three small additions”.

Everything in this project so far has been analysis, judgement and permission. None of it has yet done anything for a customer. Kofi’s job is the least visible translation in the chain and probably the most decisive: taking a thing that works when a clever person is watching, and turning it into a thing that works when nobody is. A notebook is judged on its best day. A production system is judged on its worst.

What lands on Kofi’s desk

Four inheritances, none of them his. From Priya in module 8: the model and its scoring code, versioned properly, with a threshold that produces roughly sixteen alerts a week for Maya’s team of six. From Elena in module 9: approval to deploy, conditional on three controls being built rather than promised. From Aisha in module 7: a pipeline that refreshes the feature table at 2am every night, which is now the heartbeat Kofi’s job has to beat in time with. And from Omar in module 3: an envelope of $250k that is nearly spent, inside a 26-week timeline with no slack left in it either.

Omar set that envelope in week two partly so this exact choice would already be constrained by the time anyone reached it, writing then that Kofi would face a decision between a simple nightly job and an elegant real-time system costing three times as much. This is what a constraint travelling downstream feels like from the receiving end: not an argument, just an option that quietly is not available.

Sofia’s PRD from module 4 supplies the target: scores inside the customer success platform the team already works in, refreshed within 24 hours of new usage data arriving, a reason beside every account, one-click logging of what the rep did, and a weekly email digest.

What an ML engineer actually does

Productionising a model means five unglamorous things: packaging it so it runs identically on a server as on a laptop, scheduling it so it runs unattended, integrating it so answers land where people already are, monitoring it so a human learns it broke before a customer does, and making it reversible so a bad release can be undone by someone who is not the author, at an hour when the author is asleep.

The craft is almost entirely restraint. Every extra capability is more surface to secure, monitor, debug and pay for — not once, but every month for as long as the thing exists. Each clever piece is another thing capable of paging someone at 3am. In production, boring is a compliment.

The software on Kofi’s desk

Git and GitHub provide the two things production work cannot do without: every change is reviewed by another person before it ships, and every change is recorded and reversible. When something breaks at an awkward hour, the recovery is usually “return to the last known good version”, and that option only exists because of this tool.

GitHub Actions is the automated road from a change to production — build, run the tests, deploy — with the property that a failing test simply stops the journey. This is what “dev, test, production” looks like as software rather than as a diagram. Databricks Jobs (or Airflow) runs the nightly scoring after Aisha’s pipeline has landed and writes the results into the customer success platform. Grafana watches all of it and pages a human when the job fails or the model’s behaviour drifts from what it was trained on. None of these four tools makes the model more accurate. All four decide whether anyone can rely on it on a Tuesday.

The four decisions Kofi makes

How the scores reach people

The first decision is physical: by what mechanism does a number calculated on a server appear in front of a human who might act on it. The candidates are a real-time scoring API — a doorway letting one system ask another a question and get an immediate answer — and a nightly scored file delivered into the customer success platform. The API is the impressive option, the one leadership half-expects because a competitor demonstrated it, and it would answer at any moment: what is this account’s score right this second?

That question is the whole argument, because nobody in this workflow asks it. Churn moves over weeks — Ben established in module 6 that leavers’ logins start sliding about ten weeks before they cancel. Aisha’s data refreshes once a night. Jade’s team make calls in office hours from a list they read with their coffee. A score updating every second would feed fresh answers to a process that consumes them daily, at roughly $30k and two weeks against an envelope nearly gone and a clock nearly out.

So Kofi chooses the nightly scored file, written immediately after Aisha’s 2am pipeline finishes and imported before anyone is awake. It matches the rhythm of the data, honours the PRD’s 24-hour promise with hours to spare, costs a fraction of the alternative, and — the part that matters most in month eighteen — can be diagnosed by reading a file. When Jade’s list looks wrong on a Thursday, someone can open last night’s output and see exactly what the system believed.

Where mistakes are allowed to happen

The second decision is about geography — not where the system runs, but where its errors are permitted to land. Software normally lives in three places. Development is Kofi’s own copy, where things are half-built and breaking them costs nothing. Test is a faithful replica with realistic data, where a change is proved before anyone depends on it. Production is the real one, where a mistake reaches Maya’s team and, indirectly, Bramford Logistics. Under a deadline, test is always the first thing someone proposes to skip, on the reasoning that it is only a pilot and time is short.

Kofi’s answer is the most useful sentence in this module. Skipping a test environment does not remove testing from the project. It relocates it — to production, in front of the users whose trust the product depends on, at the exact moment they are forming their first opinion of it. Every system gets tested. The only choice is whether it happens before or after the users arrive.

He builds all three, and adds a pilot flag so the first release is switched on for one willing team and nobody else. It costs very little and buys two things: the blast radius of a bad week becomes six people who agreed to be early rather than the whole organisation, and the pilot can be switched off in seconds. A pilot you cannot switch off is not a pilot; it is a rollout with optimistic vocabulary.

What happens at 2am when it fails

The third decision separates engineers from enthusiasts, because it starts from the assumption that the system will fail. Not might. Will — a source will go down, a schema will change, a credential will expire. Aisha lived this in module 7, at 6:12am, when a vendor renamed a column overnight.

The framing that matters is this: the choice is not between working and broken, but between three kinds of broken. Silence is the job failing and nothing happening — no alert, yesterday’s list still sitting there looking as authoritative as it did when it was fresh. Fiction is worse: the job half-runs, produces scores from partial data, and presents them as current. The third is an honest reduced service: the system tells a human immediately, and shows users yesterday’s scores clearly labelled as stale.

Kofi builds the third. An alert fires to a named on-call person the moment scoring fails, and the platform falls back automatically to the previous day’s scores with a visible marker saying so. This is graceful degradation — deciding what the system does when it cannot do everything, so that reduced service is a designed state rather than an accident. He rejects the tempting middle option of retrying forever, because a retry loop hides the failure while hammering a broken source: the job technically “ran” for six hours and nobody learned anything.

The reason this matters is not technical. The argument against alerting is always “someone will notice”, and someone always does — here it means Maya noticing three days later, having worked a list she now knows was wrong. What she loses is trust in the list, and trust in the list is the product’s only currency. The model does not save customers; a rep acting on a list they believe saves customers.

What actually ships

This is where the project is really decided, and it takes about forty minutes at a whiteboard. Sprint one holds twenty story points — a story point being a rough unit of effort used to size work against other work, so a plan can be argued about honestly rather than in optimistic hours.

The arithmetic is brutal. The nightly scoring job and its delivery cost six. Elena’s three conditions — the review flow showing score and reason, role-based access, and monitoring with stale-score alerting — cost four, three and three. That is sixteen of the twenty gone before a single feature is considered. Governance is not free; it is prepaid, and this is the invoice.

Four points remain, and Sofia has arrived with three requests: trend arrows on each account (three points), Slack notifications when a score jumps (four), and configurable dashboard widgets (six). Thirteen points of good ideas competing for four. Every one would improve the product, which is exactly what makes this hard; nobody wrecks a release with obviously bad suggestions.

Kofi spends three of the four on one-click intervention logging, which was in the PRD and which Sofia herself refused to trade away in module 4. It records what the rep actually did about each flagged account: the least exciting item on the board and the only one the project cannot survive without, because without it Nadia has nothing to measure in module 12. Retention numbers with no record of interventions are a story, not evidence. He then defers all three of Sofia’s requests to the next release, in writing, with the reasoning attached.

That written note is the decision, not the private judgement behind it. Saying yes to everything is not generosity. It is a decision to be late, taken on behalf of people who have not been told. What you cut is a plan; what you quietly fail to finish is a surprise.

The same honesty runs the other way. Reading the PRD line by line, Kofi finds the weekly email digest still listed as release-one scope and concludes it no longer makes sense: the pilot is one team reading a fresh ranked list in their own platform each morning, so a weekly summary duplicates what they already have, for two points he has not got. He says so and the PRD is amended. Had it still carried the real-time scoring line leadership pushed for — Sofia refused that in writing in module 4, which is why it is not a problem now — this is the moment it would have had to be raised aloud rather than quietly dropped. Changing your mind in public is cheap; discovering in week twenty-four that a written promise was abandoned in week eighteen is not.

Where this goes wrong

It happens like this. The model impresses people in a meeting, somebody says it would be a shame to wait, and the notebook is pointed at live data while a couple of reps use the output. There is no test environment because there was never a decision to build one, no monitoring because nothing has broken yet, no rollback because there has never been anything to roll back to. For a few weeks it works beautifully.

Then a credential expires and the list quietly goes stale. Nobody is alerted, because alerting was going to be added later. A rep calls Hale & Porter LLP about a crisis that resolved itself a fortnight ago, and nobody calls Bramford Logistics, who cancel in week nine. The system is fixed within a day of someone noticing; the trust is not. Six months later the settled conclusion is not “we deployed carelessly” but “the AI didn’t work”, and that sentence outlives everyone who was in the room.

The quieter version is more common: scope holds until the final fortnight, three late additions are absorbed without a conversation, everything runs a week over, and the item dropped in the scramble is the boring one nobody was defending — the intervention logging. The pilot then runs perfectly and proves nothing at all.

What Kofi hands on

What leaves the desk is a deployed pilot and a release checklist: nightly scoring into the customer success platform on Aisha’s 2am rhythm, a review flow showing score and reason, role-based access, monitoring with stale-score alerting and a named owner, a practised rollback, one-click intervention logging, and a pilot flag switched on for exactly one team. Alongside it, in writing, the deferred list and the amended PRD.

It goes to Jade, CS Operations, in module 11: it is live for your pilot team; every morning there is a ranked list with a reason per account, and what you do with it is the part that actually saves customers. No customer will ever see Kofi’s work, and every customer outcome now depends on it.

Three constraints travel on. The nightly rhythm inherited from Aisha is what puts a fresh list on Jade’s screen at 8am on Monday, sixteen names long, ready before the working day rather than during it. The pilot flag is what makes module 11 a genuine trial rather than an irreversible rollout — a thing that can be stopped is a thing that can be honestly evaluated. And the intervention logging that survived the sprint by three points is the single reason Nadia can measure anything at all in module 12.

Spot the engineering call

Read each situation, decide what Kofi would do, then tap a card to check.