Latest / MBA Training Data / Natural-language BI and the analyst's new role: why the "democratisation" story is only half true
Transcript
- 0:00Welcome to the Leaders Insights podcast.
- 0:07Today's episode, Natural Language BI and the Analyst New Role, why the democratization
- 0:12story is only half true.
- 0:14Every vendor pitch I've heard this year says the same thing.
- 0:18Type a question in plain English, get a chart back, and you can finally fire your analytics
- 0:22team.
- 0:23You're telling me that's wrong.
- 0:25I'm telling you it's the most expensive half-truth in the industry right now.
- 0:29The typing a question part works.
- 0:31The firing the team part is where CDOs, E's, chief data officers, the people who own a company's
- 0:37data strategy, light money on fire.
- 0:40Strong start, but be concrete.
- 0:42What actually breaks?
- 0:43The question isn't the hard part.
- 0:45The definition behind it is, ask what was revenue last quarter?
- 0:50And the tool has to know, recognized revenue or booked?
- 0:53Does it include refunds?
- 0:55Which quarter?
- 0:56Calendar or fiscal?
- 0:58But one retailer I worked with, the same question gave finance and marketing numbers
- 1:02that differed by 11 percent.
- 1:04Because the Natural Language tool guessed the join between two tables and guessed wrong.
- 1:09But the vendors say the semantic layer fixes exactly that.
- 1:13The dictionary that tells the tool what revenue means.
- 1:16Right, and I love the semantic layer.
- 1:18The shared dictionary of business definitions the machine reads before answering.
- 1:23Here's the catch, nobody prints on the box.
- 1:26Someone has to build and maintain that dictionary.
- 1:28That someone is an analyst.
- 1:30So the tool that supposedly kills the analyst runs entirely on the analyst's work.
- 1:34So you've just moved the bottleneck, not removed it?
- 1:38Exactly.
- 1:39You've moved it upstream, where it's less visible and more dangerous.
- 1:42When a human analyst gave you a wrong number, they hedged.
- 1:45This looks off, let me check.
- 1:47The tool answers in confident prose with a nice bar chart.
- 1:51It has the tone of certainty and none of the doubt.
- 1:54That's how bad decisions get made fast.
- 1:56Give me a number.
- 1:57How often are these things actually right?
- 1:59The honest benchmarks are sobering.
- 2:02There's public research.
- 2:03The spider benchmark, an academic test of turning English into database queries, where
- 2:08even the best 2025 models landed around 80 to 90 percent accuracy on clean, well-labeled
- 2:15data.
- 2:16Sounds great until you realize real company data is filthy and a 15 percent error rate
- 2:22on financial reporting is not a rounding issue.
- 2:25It's a resignation letter.
- 2:27Vendors would say those numbers are old and their tool is better.
- 2:30Of course they would.
- 2:31And their internal benchmarks, which they ran themselves so great accordingly, always show
- 2:3695 plus.
- 2:38I've watched those same tools confidently sum a column of customer IDs and report
- 2:42it as a metric.
- 2:44It's like hiring an intern who never says, I don't know and never sleeps.
- 2:48So who's actually winning with these tools, if anyone?
- 2:51The companies that treat it as an amplifier, not a replacement.
- 2:55Spotify and a few others have talked openly about internal query assistance.
- 3:00But the point is they sit on top of a governed data model built by a real team.
- 3:05The tool handles the boring 70 percent.
- 3:08How many active users in Germany last week?
- 3:11The analyst is freed for the 30 percent that pays their salary.
- 3:15The why did they drop question no natural language box can answer.
- 3:19Because that requires knowing the marketing calendar and the app outage on Tuesday.
- 3:24So the analyst's job changes.
- 3:26Into what exactly?
- 3:27Two things.
- 3:28First, a definer.
- 3:29The person who decides and documents what every metric means so the tool can't improvise.
- 3:35Second, an investigator.
- 3:37The one who takes the tool's answer and asks whether it's actually true.
- 3:42The pure report runner, the person whose whole day was building the same dashboard
- 3:46for the fourth time.
- 3:47Yes, that role shrinks.
- 3:49But that was never analysis.
- 3:51That was clerical work wearing a data badge.
- 3:54So what's the structural mistake a CDO makes if they believe the simple story?
- 3:58They cut headcount first and build the semantic layer never.
- 4:02Then the tool spreads confident wrong answers across the whole company.
- 4:06And now everyone's a self-served analyst producing garbage at scale.
- 4:10You didn't democratize insight.
- 4:12You democratized error.
- 4:14Give me the one thing to do, Monday.
- 4:16Before you buy any natural language tool, ask your team to write down the definition
- 4:20of your top five metrics.
- 4:23If they can't agree in a room, the tool will just automate the disagreement, faster,
- 4:28and with prettier charts.
- 4:29Done.
- 4:30Want to know where you actually stand?
- 4:32Take the CDO self-assessment at MBA-training.com.