Latest / MBA Training Data / Model monitoring, drift detection, and retraining triggers: a CDO playbook
Transcript
- 0:00Welcome to the Leaders Insights Podcast.
- 0:08Today's episode, Model Monitoring, Drift Detection, and Retraining Triggers.
- 0:12A CDO playbook.
- 0:14Your models are rotting right now and you don't know it.
- 0:17That's your claim.
- 0:18Defend it.
- 0:19It's not a claim.
- 0:20It's arithmetic.
- 0:21A fraud model trained on 2024 spending patterns is scoring 2026 transactions, and people spend
- 0:28differently now.
- 0:29The model didn't break.
- 0:31The world moved and the model stood still.
- 0:33But if it's scoring fine and nobody's complaining, where's the harm?
- 0:37That's the trap.
- 0:38It scores fine right up until the quarter closes badly.
- 0:42Zillow's home buying arm learned this the hard way.
- 0:45Their pricing model drifted against a shifting market and they wrote down roughly $300 million
- 0:50in 2021.
- 0:52Nobody got an alert saying your model is now wrong.
- 0:56The losses were the alert.
- 0:57So define drift, plainly, what is actually moving.
- 1:01Two things and keep them separate.
- 1:03Data drift, weeee.
- 1:05The inputs change.
- 1:07Your customers are younger or a new product launched so the numbers going in don't look
- 1:11like the training data anymore.
- 1:14Then concept drift.
- 1:18The relationship changes.
- 1:20The same inputs now mean something different.
- 1:23During 2020 lockdowns, customers suddenly stopped spending, flipped from a churn signal
- 1:28to totally normal overnight.
- 1:30Fine.
- 1:31How do I catch either one before it cost me $300 million?
- 1:34You monitor three layers, cheapest to most expensive.
- 1:38Layer one, the inputs.
- 1:41Are the features drifting from what the model trained on?
- 1:44You measure that with something like a population stability index, a single number for how
- 1:48far today's data has slid from the baseline.
- 1:52Of 0.2, you investigate.
- 1:55That sounds like a lot of dashboards nobody reads.
- 1:57Which is why layer one alone is useless.
- 2:00Layer two is the outputs.
- 2:02Is the model's score distribution shifting?
- 2:05If a loan model that used to approve 60% suddenly approves 40%, something moved even
- 2:10if every input looks normal.
- 2:13Layer three, the one that actually matters.
- 2:17Outputs outcomes.
- 2:18Actual defaults.
- 2:19Actual conversions.
- 2:21Actual fraud you missed.
- 2:22Why not just watch layer three and skip the rest?
- 2:25Because layer three arrives late.
- 2:27You don't know a loan defaulted for months.
- 2:30That's the whole point.
- 2:32Inputs move first, outputs second, money last.
- 2:35Watching only the money is like checking your smoke detector by waiting for the
- 2:38fire.
- 2:39There's your joke.
- 2:40Okay, I've got signals.
- 2:42When do I actually retrain?
- 2:44Retriggers and you decide the thresholds in advance.
- 2:47In daylight.
- 2:48Before anyone's panicking.
- 2:50One.
- 2:51Scheduled.
- 2:52Retrain every 30 days regardless.
- 2:54A floor, not a strategy.
- 2:56Two.
- 2:58Performance-based.
- 2:59Accuracy drops below a line you set.
- 3:01Say five points.
- 3:02You retrain.
- 3:03Three.
- 3:04Drift-based.
- 3:06That stability index crosses your threshold.
- 3:09You retrain even if performance looks okay.
- 3:12Because you can see the wave coming.
- 3:14Retraining constantly sounds expensive and slightly insane.
- 3:17It is.
- 3:18Which is why retrain always is as lazy as retrain never.
- 3:23Retraining on bad or poison data just bakes in a fresh mistake faster.
- 3:28The trigger isn't drift detected go.
- 3:30It's drift detected.
- 3:32A human looks then decides.
- 3:35Automation flags.
- 3:36People pull the lever.
- 3:38That's governance.
- 3:39Which usually means a committee that meets monthly and decides nothing.
- 3:43Only if you build it that way.
- 3:45The governance that works is narrow.
- 3:47Every production model has a named owner.
- 3:50One human, not a department.
- 3:52That model has a documented baseline, defined thresholds, and a runbook.
- 3:57When this alert fires, this person does this thing within this many hours.
- 4:02Amazon scrapped a hiring model in 2018 because it drifted against women.
- 4:07The failure wasn't the drift.
- 4:09It was that nobody owned the decision to kill it fast.
- 4:12So the CDO's job here is in the math.
- 4:14It's the org chart.
- 4:16The CDO's job is making sure that when a model quietly goes wrong at 2 a.m., there's
- 4:21a named person, a threshold, and a runbook, instead of a surprise on a slide three months
- 4:27later.
- 4:28Vendors like Fiddler and Arise will sell you the monitoring layer, and it's genuinely
- 4:33useful.
- 4:34But a dashboard with no owner is just an expensive way to be surprised more precisely.
- 4:39Give me the one thing to do Monday.
- 4:41Pick your highest stakes production model and ask one question.
- 4:44If it's silently degraded today, who gets paged and what number tells them?
- 4:49If you can't name both in 10 seconds, that's your project.
- 4:53Start there.
- 4:54That's a wrap.
- 4:55Fresh CDO briefings drop daily at MBA-training.com.