Latest / MBA Training Data / How Airbnb rebuilt metric consistency with a semantic layer
Transcript
- 0:00Welcome to the Leaders Insights Podcast.
- 0:07Today's episode, how Airbnb rebuilt metric consistency with a semantic layer.
- 0:13Airbnb just published a teardown of a system they call Minerva, and the headline confession
- 0:17is brutal.
- 0:18Their own analysts were producing conflicting numbers for the same business question.
- 0:23Two people ask how many bookings last quarter and get two different answers.
- 0:27How does a data first company end up there?
- 0:29The same way everyone does.
- 0:31Every team writes its own skill query.
- 0:34That's just the code that pulls data out of a database, we'll see.
- 0:37And every query bakes in a slightly different definition.
- 0:41One analyst counts a booking when the guest clicks reserve.
- 0:45Another counts it when payment clears.
- 0:47Another excludes cancellations.
- 0:49Another doesn't.
- 0:50Nobody's wrong.
- 0:51They're just all using a private dictionary.
- 0:54So it's not bad data, it's a definitions problem.
- 0:57Exactly, the pipes were fine.
- 0:59The vocabulary was chaos, and it compounds.
- 1:03Airbnb had thousands of these queries floating around, each one a fork in the road.
- 1:08By the time it reaches a board deck, you've got numbers that are off by percentage points,
- 1:13and nobody can reconstruct why.
- 1:15And the fix is this thing called a semantic layer.
- 1:18Translate that before I let you keep going.
- 1:20A semantic layer is a single, central place where you define what a metric actually means.
- 1:25One canonical definition of booking, active user, revenue.
- 1:30And then every dashboard, every report, every query pulls from that one definition, instead
- 1:35of reinventing it.
- 1:37Think of it as the company dictionary that everyone is forced to use.
- 1:41Minerva is Airbnb's version.
- 1:44Forced is the interesting word.
- 1:46Because the reason those private queries exist is analysts didn't want to wait for
- 1:50central approval.
- 1:52Isn't a semantic layer just re-centralizing a bottleneck people already escaped once?
- 1:56That's the trap, and it's why most of these projects die.
- 2:00If your semantic layer means every new metric needs a ticket and a two-week wait, analysts
- 2:04route around it inside a month.
- 2:07What made Minerva work is they let teams define metrics in a config file.
- 2:11A simple structured text file that gets reviewed, versioned, and then computed centrally.
- 2:17You own your definition, but once it's in, it's the definition.
- 2:21There's no shadow copy.
- 2:23Version meaning what, in plain terms?
- 2:25Meaning, if you change how booking is calculated in March, there's a record of it, and every
- 2:30downstream number moves together.
- 2:33The alternative, what most companies live with, is someone silently edits a query.
- 2:38Q1 no longer matches Q2.
- 2:41And you spend a board meeting arguing about whether growth is real or a definition change.
- 2:46Airbnb killed that entire category of argument.
- 2:49Let me push.
- 2:51Airbnb has enormous engineering resources.
- 2:53A CDO, a chief data officer, at a mid-size company hears we built Minerva and reasonably
- 2:59thinks that's not my life.
- 3:01Fair.
- 3:02And you shouldn't build Minerva.
- 3:03You don't need to.
- 3:05The blueprint is transferable.
- 3:06The code isn't.
- 3:08There are off-the-shelf semantic layers now, on our DBT has one, cube.
- 3:13Lookers had Lookmill for years, which is just their language for defining metrics
- 3:17once.
- 3:18The expensive part was never the software.
- 3:21It's the governance decision, agreeing that there is one definition of a metric, and someone
- 3:26owns it.
- 3:27So where does a CDO actually start Monday morning?
- 3:30Because define all your metrics is a five-year project.
- 3:33Don't do all of them.
- 3:35Find the 20 metrics that show up in your executive reporting and your board deck.
- 3:39That's your real exposure.
- 3:40Lock those first.
- 3:42In practice, when a company runs this exercise, they discover they have four definitions of
- 3:47revenue and nobody knew.
- 3:50Just surfacing that is worth the quarter.
- 3:52And the trust problem underneath all this, the reason Airbnb bothered?
- 3:57That's the whole game.
- 3:58Once executives catch data being wrong twice, they stop using it and start going with their
- 4:03gut.
- 4:04A semantic layer is really a trust recovery project wearing an engineering costume.
- 4:09The metrics are the excuse.
- 4:11The outcome is that people believe the dashboard again.
- 4:14Give me the one thing to do this week.
- 4:16Make your single most quoted number, the one in every all hands, and ask three teams
- 4:21to pull it independently.
- 4:23If they disagree, you've found your Minerva project and you've found it before your board
- 4:27does.
- 4:28That's it.
- 4:29The CDO Decision Tools are live at MBA-training.com.