Latest / MBA Training Data / How Cloudflare rebuilt its data stack around dbt, Fivetran, and Airflow
Transcript
- 0:00Welcome to the Leaders Insights Podcast.
- 0:07Today's episode how Cloudflare rebuilt its data stack around DBT 5-Tran and Airflow.
- 0:13Cloudflare just published a case study on how they tore out their old data pipelines
- 0:17and rebuilt everything around DBT 5-Tran and Airflow.
- 0:21For a company routing something like 20% of web traffic, that's not a small remodel.
- 0:26Why should anyone outside their data team care?
- 0:28Because Cloudflare hit the exact wall every scaling company hits, just louder.
- 0:33They had hand-coded SQL pipelines and a mess of ingestion scripts.
- 0:38All qi xin.
- 0:40The little programs that pull data from one system into another, and nobody wanted to
- 0:44touch them.
- 0:45When your data plumbing becomes haunted, analysis stops.
- 0:49That's the part that matters.
- 0:51Broken pipelines don't just annoy engineers, they starve the business of answers.
- 0:56Cloudflare is doing a lot of work there.
- 0:58What actually breaks when SQL is hand-coded?
- 1:01Fragility and fear.
- 1:03Someone writes a 400-line query in 2021, they leave, and now it's load-bearing and undocumented.
- 1:09Change one column upstream and three dashboards silently lie to you.
- 1:14Cloudflare's own write-up describes scripts no engineer wanted to touch.
- 1:19That's the tell.
- 1:20When the fix is scarier than the bug, you've got a graveyard, not a data stack.
- 1:25So they move to ELT, spell that out before I fall asleep.
- 1:29ELT, Extract, Load, Transform.
- 1:33The old way was ETL.
- 1:35You transform the data before it lands in the warehouse, meaning you shape it while
- 1:39it's in flight, which is brittle.
- 1:41ELT flips the last two steps.
- 1:43You dump the raw data into the warehouse first, then transform it there with SQL.
- 1:49You can actually see and test.
- 1:51The warehouse is cheap and fast enough now that hoarding raw data isn't the sin it was
- 1:55ten years ago.
- 1:56Fine, but that's three vendors stitched together.
- 1:595tran, dbt, airflow?
- 2:02Isn't that just trading haunted scripts for a subscription pile?
- 2:05It can be, if you're lazy about it.
- 2:07But each tool kills a specific pain.
- 2:105tran does the extract and load, the boring breakable connector work between, say,
- 2:15Salesforce and your warehouse.
- 2:18That's the stuff engineers hate maintaining, so you rent it.
- 2:22dbt, data build tool, handles the transform.
- 2:26It lets analysts write modular SQL with version control and tests so a change gets caught before
- 2:31it poisons a dashboard.
- 2:33An airflow?
- 2:34Airflow is the orchestrator, the scheduler that decides what runs, in what order, and
- 2:40screams when something fails.
- 2:42Think of it as air traffic control for your data jobs.
- 2:46Without it, you've got a hundred cron jobs firing blind and no idea which one broke the
- 2:50morning report.
- 2:51Here's the uncomfortable one.
- 2:535tran charges by data volume by monthly active rows.
- 2:57For a company at CloudFlare's scale, doesn't that bill get insane?
- 3:00It absolutely can.
- 3:02And that's the honest catch nobody in the vendor blog post wants to dwell on.
- 3:065tran's pricing is consumption-based, so a chatty source table can quietly cost you
- 3:11five figures a month.
- 3:13The trade you're making is engineering time for money.
- 3:16If your engineers are worth more building your product than babysitting a sales force connector,
- 3:21you pay 5tran.
- 3:23If you're a startup with cheap time and expensive cash, maybe you don't.
- 3:28CloudFlare made the first call.
- 3:30Not everyone should.
- 3:31So, what did this shift actually require in practice?
- 3:35Beyond swiping the corporate card?
- 3:37Discipline mostly.
- 3:38dbt only works if people write tests and document their models.
- 3:43Otherwise you've rebuilt the graveyard with nicer tools.
- 3:46The real work at CloudFlare wasn't installing software.
- 3:49It was defining ownership.
- 3:51Who owns which data model?
- 3:53Who gets paged when it breaks?
- 3:55Tooling doesn't fix culture.
- 3:57It just exposes whether you have any.
- 3:59That's the part everyone skips.
- 4:01Every time.
- 4:02People think buying dbt makes them modern.
- 4:05Buying dbt makes you a company with dbt.
- 4:08The governance, knowing what's trustworthy and who's accountable, is the actual product.
- 4:13Give me the one thing a listener does Monday morning.
- 4:16Find your scariest query.
- 4:18The one nobody will touch.
- 4:20Open it and try to answer three questions.
- 4:23What feeds it, what depends on it, and who owns it.
- 4:26If you can't answer all three in 10 minutes, you've found your haunted house.
- 4:30And that's where your rebuild starts.
- 4:32Not with a vendor demo.
- 4:34Start with the thing you're afraid of.
- 4:36I'll take it.
- 4:37That's it.
- 4:38The CDO Decision Tools are live at MBA-training.com.