Latest / MBA Training Data / Feature stores and the ML data supply chain: what CDOs actually need to understand
Transcript
- 0:00Welcome to the Leaders Insights Podcast.
- 0:08Today's episode features stores the missing infrastructure layer in your ML data supply
- 0:12chain.
- 0:13Everyone blames the model when their AI project dies.
- 0:16You're telling me that's usually wrong.
- 0:18Almost always wrong.
- 0:20When I audit failed machine learning projects, systems that predict something from data,
- 0:25maybe one in 10 died because the model was bad.
- 0:28The rest choked on the data supply chain, the plumbing that gets numbers to the model.
- 0:33That sounds like a convenient excuse for data scientists who can't build good models.
- 0:37It would be, except the failure mode is depressingly specific.
- 0:41It's called training serving skew.
- 0:44In plain terms, the data you trained your model on doesn't match the data it sees
- 0:48in production.
- 0:49So the thing that looked brilliant in the lab quietly rots in the real world.
- 0:53Give me a concrete version of that.
- 0:55Fraud detection.
- 0:57You build a feature, a computed signal, called average transaction size over the last 30
- 1:02days.
- 1:03In training, your data scientist calculates it one way in Python.
- 1:08In production, an engineer re-implements it in Java under time pressure and rounds differently
- 1:14or uses a slightly different 30-day window.
- 1:17Now your model is being fed a number that means something subtly different from what it
- 1:21learned on.
- 1:22And nobody notices because the system still returns an answer.
- 1:26It's not really.
- 1:27It doesn't crash.
- 1:28It just gets quietly worse.
- 1:30Your fraud catch rate drops three points, you blame the model, you retrain, and you
- 1:35fix nothing because the pipe is still leaking.
- 1:38So the feature store is supposed to be the fix.
- 1:41Sell me on why it's not just another dashboard nobody opens.
- 1:44A feature store is one place where you define a feature once, the actual computation,
- 1:50and both training and production pull from that same definition.
- 1:54One source of truth for average transaction size.
- 1:57The whole class of skew bugs I just described disappears because there's no second implementation
- 2:03to disagree with the first.
- 2:05One place, one definition.
- 2:07That's it?
- 2:08That's the missing infrastructure layer everyone's excited about?
- 2:10The idea is boring.
- 2:12The discipline is rare.
- 2:14Uber built the first serious one, Michelangelo, back around 2017 because they had hundreds
- 2:20of models and every team was reinventing the same features.
- 2:24They reported cutting the time to get a new model into production from months to days.
- 2:29That's their number.
- 2:30So read it as a vendor would sell it.
- 2:32But the direction is real.
- 2:34Months to days sounds like the kind of figure a vendor prints on a billboard.
- 2:37It does.
- 2:38And I discount the headline.
- 2:40But the durable lesson underneath is what matters.
- 2:43Features are reusable assets, not one-off code you throw away after each project.
- 2:49Once customer lifetime value is defined properly once, the next 12 models use it for free.
- 2:55The value compounds.
- 2:57That's the part serious operators internalized years ago.
- 3:00So why doesn't everyone have one?
- 3:02Because it's infrastructure.
- 3:04And infrastructure has no demo.
- 3:06You can't show a feature store to the board and get applause.
- 3:09It's a seat belt.
- 3:10Nobody thanks you for it until the crash you prevented never happens.
- 3:14So it loses budget fights to the shiny model that gets a press release.
- 3:17Let me push harder.
- 3:18If I'm a midsize company with five models, not 500, am I building a cathedral for a village?
- 3:25Fair.
- 3:26At five models, a managed one, Databricks and Tekton both sell hosted versions.
- 3:31Makes more sense than building your own.
- 3:34And honestly, at two models, a well-documented shared library might be enough.
- 3:39The mistake isn't skipping the store.
- 3:41It's letting every team recompute the same feature in isolation and calling that a strategy.
- 3:47So the real product is in the software.
- 3:49It's the rule.
- 3:50The rule is a feature gets defined once and consumed everywhere.
- 3:55The store is just how you enforce it at scale.
- 3:58Companies fixate on buying the tool and skip the governance.
- 4:02Then wonder why they have three conflicting definitions of active user living inside
- 4:06their expensive new store.
- 4:08Give me the one thing a listener does Monday morning.
- 4:11Pick your most important production model.
- 4:14Make its top five features and ask one question.
- 4:17Is each computed by the exact same code in training and in serving?
- 4:21If you can't answer that in 10 minutes, you've found your next outage.
- 4:25Fix that before you buy anything.
- 4:27That's your five.
- 4:28The full CDO track, structured, no fluff, is at MBA-training.com.