Latest / MBA Training Data / Open data portals: publish fewer datasets as products
Transcript
- 0:00Host: Welcome to the Leaders Insights Podcast.
- 0:08Host: Today's episode, publishing open data that citizens, journalists, and oversight bodies
- 0:12Host: actually use.
- 0:14Host: There are government open data portals with thousands of datasets, and most of them get
- 0:18Host: downloaded once, probably by the person who uploaded them.
- 0:23Host: Why do we keep building these graveyards?
- 0:25Expert: Because publishing looks like progress, a politician cuts a ribbon on a portal with
- 0:30Expert: 4,000 datasets and calls it transparency.
- 0:34Expert: Nobody measures whether a single journalist ever opened file number 3,000.
- 0:38Expert: It's theater with a CSV.
- 0:40Expert: That's a comma-separated file, plain text spreadsheet, attached.
- 0:44Host: So the number of datasets is a vanity metric?
- 0:47Host: What should they be counting instead?
- 0:49Expert: Repeat use.
- 0:50Expert: Does anyone come back to a dataset a second time?
- 0:54Expert: Data.gov, comma, the US federal portal, has hosted well over 300,000 datasets, and internal
- 1:00Expert: reviews have shown the long tail is basically dead weight, thousands with essentially zero
- 1:06Expert: downloads.
- 1:07Expert: The health of a portal isn't how much you dumped in, it's how many datasets have a
- 1:11Expert: living audience.
- 1:12Host: Fine, but living audience sounds like a slogan.
- 1:16Host: Who actually uses this stuff, and what do they need?
- 1:18Expert: Three groups, and they want opposite things.
- 1:22Expert: Advocates want a story-shaped file today, clean enough to open and excel without crying.
- 1:27Expert: Advocates want a stable series they can track over years.
- 1:30Expert: Oversight bodies, auditors, inspectors general, want completeness and provenance, meaning
- 1:36Expert: where every number came from.
- 1:38Expert: If you design for all three at once, you serve none.
- 1:42Expert: You pick.
- 1:43Host: That's uncomfortable.
- 1:44Host: You're telling public agencies to prioritize instead of publishing everything?
- 1:48Expert: Yes, because publish everything is how you get the graveyard.
- 1:52Expert: The durable lesson the good operators internalized years ago, treat a dataset like a product,
- 1:58Expert: not a file.
- 1:59Expert: A product has an owner, a release schedule, documentation, and someone whose job is ruined
- 2:04Expert: if it breaks.
- 2:05Expert: A file just sits there rotting.
- 2:08Host: Give me a concrete case where someone did this right.
- 2:11Expert: Transport for London, the transit agency.
- 2:14Expert: They didn't just dump timetables.
- 2:16Expert: They published a live feed with a real contract behind it, stable formats, and they let developers
- 2:21Expert: build on it.
- 2:23Expert: CityMapper, the journey planning app, was built on that data.
- 2:26Expert: TFL estimated the economic value to travelers and the city in the tens of millions of
- 2:32Expert: pounds a year.
- 2:34Expert: That's what happens when you maintain data like it matters instead of archiving it.
- 2:38Host: The counter argument, agencies are broke and understaffed.
- 2:42Host: Maintaining 100 datasets as products is a fantasy.
- 2:45Expert: Which is exactly why you kill 90 of them.
- 2:48Expert: Take your budget and pour it into the 10 datasets people actually fight over.
- 2:52Expert: Spending, permits, inspections, police stops.
- 2:55Expert: DBBT Labs, the data tooling vendor, argues that maintenance eats something like half
- 3:01Expert: of a data team's time, though they sell the software that promises to cut that, so
- 3:05Expert: cross-check it against independent work.
- 3:08Expert: MIT Sloan Management Review has been blunt for years that data quality neglect is where
- 3:14Expert: value quietly dies.
- 3:16Expert: The point stands either way.
- 3:18Expert: Fewer things kept alive.
- 3:20Host: How do you know which 10?
- 3:22Host: Every department will claim theirs is essential.
- 3:25Expert: Don't ask departments.
- 3:26Expert: Ask the users.
- 3:27Expert: Look at what journalists file public records requests for over and over.
- 3:32Expert: That's your backlog, handed to you free.
- 3:34Expert: If reporters keep suing you for the same inspection data, publish it well, and you've
- 3:39Expert: turned a legal headache into a product.
- 3:41Expert: The demand signal is already screaming at you.
- 3:44Expert: Agencies just refuse to listen.
- 3:46Host: And documentation.
- 3:48Host: Everyone says it matters.
- 3:49Host: Nobody does it.
- 3:51Host: Why does it actually change usage?
- 3:53Expert: Because an undocumented column is a landmine.
- 3:56Expert: If a field is called status underscore two and nobody knows what value four means, a
- 4:01Expert: journalist either guesses wrong and prints something false or walks away.
- 4:06Expert: A data dictionary, a plain language description of every field, is the difference between
- 4:11Expert: a data set that gets cited in an investigation and one that gets abandoned in an afternoon.
- 4:17Expert: It's boring.
- 4:18Expert: It's also the whole game.
- 4:20Host: So if a CDO listening today does one thing this week, what is it?
- 4:24Expert: Open your portal's download logs, find the five data sets people actually return to,
- 4:30Expert: and assign each one a named human owner with a publishing schedule.
- 4:34Expert: Then let the dead one stay dead.
- 4:36Expert: Serve the five you have brilliantly before you publish a sixth nobody wanted.
- 4:41Host: Sources for today's episode.
- 4:43Host: The new stack towards data science, MIT technology review, MIT Sloan Management Review,
- 4:49Host: DBT Labs, vendor data tooling.
- 4:53Host: That's your five.
- 4:54Host: The full CDO track, structured no fluff, is at MBA-training.com.