Latest / MBA Training Data / Data observability: catching bad data before it reaches decisions
Transcript
- 0:00Welcome to the Leaders Insights Podcast.
- 0:07Today's episode, Data Observability, catching bad data before it reaches decisions.
- 0:13Every CDO I talk to swears their dashboards are clean.
- 0:16Then a number blows up in a board meeting.
- 0:19So which is it?
- 0:20Are they lying or do they just not know?
- 0:22They don't know.
- 0:24And that's the whole point.
- 0:25Bad data is polite.
- 0:27It doesn't crash anything.
- 0:28Your pipeline runs green, the report loads, the chart renders.
- 0:32It's just quietly wrong and nobody notices until finance asks why revenue jumped 40% overnight.
- 0:39So the durable lesson here isn't buy a data observability tool.
- 0:42What is it?
- 0:43The lesson is, assume your data is broken until something proves it isn't.
- 0:49Observability, instrumenting your pipelines so they tell you when they misbehave.
- 0:53The way engineers monitor servers is just the mechanism.
- 0:57The mindset is treating data quality as a runtime problem, not a cleanup project you
- 1:02do once a quarter.
- 1:03That sounds like a rebrand of the data quality tools people already bought and ignored.
- 1:08Fair hit.
- 1:09The old tools check data at rest, côse gant.
- 1:13Someone wrote rules, ran a batch job, filed a report nobody read.
- 1:17The shift is checking data in motion continuously and alerting a human before the bad number
- 1:22reaches a decision.
- 1:24Monte Carlo, the vendor that basically coined data observability, puts the average time to
- 1:30detect a data incident at four days without it.
- 1:33Four days.
- 1:35Your quarterly forecast is already out the door.
- 1:37Give me a real one.
- 1:39Where did silent bad data actually cost someone?
- 1:41Unity, the game engine company.
- 1:44Back in 2022, they told investors a bad data ingestion problem in their ad targeting model
- 1:50cost them roughly $110 million in a single year and knocked a chunk off the stock.
- 1:56The model wasn't broken.
- 1:58It was fed garbage from a large customer and kept optimizing confidently against it.
- 2:04Nobody was watching the input.
- 2:05So the model did exactly what it was told, with poison.
- 2:09Precisely.
- 2:10And this is why I'm allergic to teams that obsess over model accuracy and ignore data
- 2:14freshness, whether the data actually arrived when it should.
- 2:19A 99% accurate model on stale data is a very precise liar.
- 2:23Okay, I'm a CDO with a modest budget and 40 pipelines.
- 2:27Where do I actually start?
- 2:29Not with all 40.
- 2:31Start with the five that feed decisions someone would get fired over.
- 2:35Board reporting, regulatory filing, the churn model, the share of customers who walk, anything
- 2:41where a wrong number has a name attached to the consequence.
- 2:45Instrument those first.
- 2:46And instrument how?
- 2:47Be concrete.
- 2:48Board checks boring on purpose.
- 2:51Freshness.
- 2:52Did the data land on time?
- 2:54Volume.
- 2:55Did roughly the expected number of rows show up?
- 2:57Not 10 million yesterday and 8,000 today.
- 3:01Schema.
- 3:02Did someone rename a column and silently break everything downstream?
- 3:06And distribution?
- 3:07Does the shape of the values look like it did last week?
- 3:10That's 80% of real incidents right there.
- 3:13Distribution checks sound like where you drown in false alarms.
- 3:17Three marketers urge an alert nobody reads.
- 3:19You're right, and that's how observability dies.
- 3:23Alert fatigue kills more of these programs than bad tech.
- 3:26The fix is ownership.
- 3:28Every alert routes to one named person who can actually fix it.
- 3:32Not a channel with 300 people muting it.
- 3:35If an alert has no owner, delete the check.
- 3:38A test nobody acts on is just anxiety with a dashboard.
- 3:42What's the number that tells a CDO this is working versus theater?
- 3:46Time to detection and time to resolution.
- 3:49How fast you catch it.
- 3:50How fast you fix it.
- 3:52If you can't say both in minutes and hours, you don't have observability.
- 3:56You have hope.
- 3:57Good teams catch it before the business does.
- 4:00That's the entire game.
- 4:01You find out before your CEO does.
- 4:04And if they can't afford the fancy platforms?
- 4:06Then write the freshness and volume checks yourself in whatever you already run.
- 4:11DPT, the transformation tool half the industry uses.
- 4:15Ships basic tests free.
- 4:17This is not a licensing problem.
- 4:19It's a nobody decided to own it problem.
- 4:21One thing they can do Monday morning?
- 4:23Pick your single most important report.
- 4:26Trace it back to the source table.
- 4:28Add one freshness check and one volume check on that table.
- 4:31Routed to one person.
- 4:33If it breaks this week, you just caught something that would have reached the board.
- 4:36Start there.
- 4:37That's your five.
- 4:38The full CDO track, structured no fluff, is at MBA-training.com.