Latest / MBA Training Data / ML in production: why models die after the notebook
Transcript
- 0:00Host: Welcome to the Leaders Insights Podcast.
- 0:07Host: Today's episode, the ML to Production Field Guide, who shapes what breaks and why.
- 0:13Expert: Here's a stat that should terrify anyone funding an AI budget.
- 0:17Expert: Most machine learning projects never make it out of the lab.
- 0:20Expert: They die somewhere between a data scientist notebook and an actual working product.
- 0:24Expert: Why?
- 0:25Expert: Because a notebook is a sandbox and production is a war zone.
- 0:29Expert: In the notebook, that's the interactive coding environment where data scientists prototype.
- 0:35Expert: Everything is clean, static, and forgiving.
- 0:37Expert: In production, the data is late.
- 0:39Expert: The users are weird.
- 0:41Expert: And nobody warned you the upstream system changed a column name at 2 a.m.
- 0:45Expert: So, it's not that the models are bad, it's everything around them.
- 0:49Expert: The model is maybe 10% of the work.
- 0:51Expert: The other 90 is plumbing.
- 0:53Expert: Pipelines that move data, monitoring that tells you when it rots, and the boring governance
- 0:58Expert: of who owns what.
- 1:00Expert: O'Reilly's survey work has been saying this for years.
- 1:04Expert: The top barriers to ML adoption aren't algorithms, they're skills gaps and lack of data infrastructure.
- 1:10Host: Give me the concrete failure.
- 1:12Host: Where does the notebook to production handoff actually snap?
- 1:16Expert: Three places.
- 1:17Expert: One, the data drifts.
- 1:19Expert: The real world inputs slowly stop looking like the training data, so a fraud model
- 1:23Expert: trained on 20-24 spending patterns quietly goes senile.
- 1:27Expert: Two, nobody owns the pipeline after launch, so it breaks and sits broken.
- 1:32Expert: Three, the thing was never designed to be retrained, so fixing it means rebuilding it.
- 1:37Host: Go senile.
- 1:38Host: That's the drift problem.
- 1:40Host: How fast does that happen?
- 1:42Expert: Faster than people expect.
- 1:44Expert: A recommendation model can degrade meaningfully in weeks if behavior shifts.
- 1:48Expert: And here's the trap.
- 1:49Expert: It doesn't crash.
- 1:51Expert: It keeps returning confident answers that are increasingly wrong.
- 1:55Expert: A broken server pages you at midnight.
- 1:58Expert: A drifting model just costs you money silently for a quarter.
- 2:02Host: So who are the people who actually fix this?
- 2:04Expert: The field guide is about roles.
- 2:06Expert: The key hire nobody budgets for is the ML engineer.
- 2:10Expert: The person who lives between the data scientist and the software team, whose whole job is
- 2:14Expert: making a model survive contact with reality.
- 2:18Expert: Then data engineers who build the pipes.
- 2:21Expert: Use those two and you've hired a Formula One driver and forgotten to build a track.
- 2:25Host: Let's talk numbers.
- 2:26Host: I've seen vendors claim huge productivity gains from better tooling.
- 2:30Host: How much do you trust that?
- 2:32Expert: Carefully.
- 2:33Expert: DPD labs.
- 2:34Expert: And note.
- 2:35Expert: They sell data transformation software, so they've got skin in the game.
- 2:39Expert: Publishes analytics engineering surveys showing data teams spend enormous time just cleaning
- 2:44Expert: and reshaping data before any modeling happens.
- 2:48Expert: Directionally true, but cross check it against something independent.
- 2:52Expert: MIT Sloan Management Review has done more sober work here and their line is consistent.
- 2:59Expert: The winners treat data as a managed product, not a byproduct.
- 3:03Host: Data as a product.
- 3:05Host: Unpack that because it sounds like a slogan.
- 3:07Expert: It means someone owns a dataset the way a product manager owns an app.
- 3:11Expert: With a quality standard, documentation, and a promise it won't silently change.
- 3:17Expert: Most companies treat data like leftovers in the office fridge.
- 3:20Expert: Technically available, ownership unclear, increasingly dangerous.
- 3:25Host: So an exec hears all this and thinks, I need to buy a platform.
- 3:29Host: Are they right?
- 3:30Expert: That's the reflex, and it's usually wrong.
- 3:33Expert: Tools don't fix an org that hasn't decided who's accountable when the model breaks
- 3:36Expert: at 3 a.m.
- 3:38Expert: MIT Sloan's research keeps landing on culture and clear ownership over any specific
- 3:43Expert: tool.
- 3:44Expert: You can buy the best monitoring in the world and still ignore the alerts.
- 3:48Host: Give me the uncomfortable question then.
- 3:50Host: What's the one thing companies fake?
- 3:53Expert: They fake done.
- 3:54Expert: A demo works, everyone claps, the press release goes out.
- 3:58Expert: And then it never actually runs reliably for real users.
- 4:02Expert: Working in the demo and working in production are as different as a wedding and a marriage.
- 4:07Host: So for the exec listening who's about to greenlight an AI project, what do they
- 4:11Host: do Monday morning?
- 4:12Expert: Before you approve the model, ask one question.
- 4:15Expert: Who is on the hook to keep this running six months after launch?
- 4:19Expert: And what's their name?
- 4:21Expert: If there's no name, you're not funding a product.
- 4:23Expert: You're funding a very expensive demo.
- 4:25Host: A name, not a platform.
- 4:27Host: That's the takeaway.
- 4:28Host: Thanks for this.
- 4:29Expert: Anytime.
- 4:30Host: Sources for today's episode.
- 4:32Host: Towards Data Science, KD Nuggets, The New Stack, DBT Labs, Vendor, Data Tooling,
- 4:38Host: O'Reilly Radar, MIT Sloan Management Review.
- 4:41Host: That's a wrap.
- 4:42Host: Watch CDO briefings drop daily at MBA-training.com.