Latest / MBA Training Data / ELT stack: where dbt, ingestion and orchestration break
Transcript
- 0:00Welcome to the Leaders Insights podcast.
- 0:07Today's episode, the modern ELT stack, explain DBT, ingestion, and orchestration working together.
- 0:14Everyone talks about ELT like it's obvious, but half the executives I know can't tell
- 0:19me what the letters even mean.
- 0:20So start there.
- 0:21ELT is Extract, Load, Transform.
- 0:25You pull data from your sources, dump it into the warehouse first, and clean it up
- 0:29afterwards.
- 0:30The old way, ETL, transformed the data before loading it.
- 0:34You scrubbed it in transit.
- 0:36Why does the order of two letters matter enough to build an industry around it?
- 0:40Because storage got absurdly cheap, and warehouses got absurdly fast.
- 0:45Snowflake, BigQuery, Databricks.
- 0:48These engines can crunch transformations at a scale that made the old approach look like
- 0:52doing your taxes with an abacus.
- 0:54So you load everything raw, mess and all, and sort it out later.
- 0:58Basically, you keep the raw data forever.
- 1:01If your business logic changes, say marketing redefines what counts as an active user, you
- 1:07just rerun the transformation.
- 1:09You don't have to re-extract anything, that's the whole trick.
- 1:13Fine, you mentioned three layers earlier.
- 1:15Walk me through them one at a time.
- 1:17Layer one is ingestion, getting the data in.
- 1:20Tools like 5tran or Airbyte connect to your Salesforce, your Stripe, your Postgres
- 1:26database, and copy that data into the warehouse on a schedule.
- 1:30And these just work?
- 1:31You point them at a source and walk away?
- 1:34Mostly, until a source changes its structure and the connector quietly breaks.
- 1:395tran's own materials say they handle schema changes automatically, schema being the shape
- 1:44of the data, the columns and types.
- 1:47In practice, you still get surprises.
- 1:49But it's a solved-ish problem.
- 1:51Layer two.
- 1:52Transformation.
- 1:54This is DBT.
- 1:56Data build tool.
- 1:58And it's the star of the modern stack.
- 2:01DBT lets analysts write transformations in SQL.
- 2:04The standard query language, but treats that SQL like real software.
- 2:09Meaning what?
- 2:10Concretely?
- 2:11Meaning version control, testing, documentation.
- 2:14Before DBT, transformation logic lived in some analyst's head, or a spreadsheet named
- 2:19final underscore v3 underscore real dot xlsx.
- 2:24DBT puts it in one place where everyone can see it, test it, and trust it.
- 2:29That final underscore v3 file exists in every company on earth.
- 2:33And it's usually the one running payroll.
- 2:36DBT kills that pattern.
- 2:37You define a model once, wa, say, revenue by region, and it becomes the single definition
- 2:43everyone inherits.
- 2:44Third layer.
- 2:45Orchestration, the conductor.
- 2:48Something has to decide the order.
- 2:50Ingest first, then transform, then refresh the dashboards.
- 2:54Tools like Airflow or DAGSTER do that.
- 2:56They run the steps, in sequence, and scream when one fails.
- 2:59Why can't the other tools just do this themselves?
- 3:02Because they don't know about each other.
- 3:055-tran doesn't know DBT exists.
- 3:07DBT doesn't know your dashboard needs refreshing.
- 3:10The orchestrator is the only thing holding the whole chain of dependencies in its head.
- 3:15So where do leaders actually get burned?
- 3:17Give me the real tradeoff, not the brochure.
- 3:20The tradeoff is cost versus control.
- 3:225-tran charges by rows of data change to receive 2 or more monthly active rows.
- 3:27Load a high traffic source and the bill balloons.
- 3:30I've seen teams go from 4,000 a month to 40,000 because one table updated constantly.
- 3:35So the cheap storage promise has a catch.
- 3:38Storage is cheap.
- 3:39Movement and compute aren't.
- 3:41And because ELT loads everything, you're now paying to transform data nobody asked
- 3:45for.
- 3:46The discipline you skipped at ingestion.
- 3:49You pay for it at the warehouse.
- 3:51That sounds like the whole model quietly undermines itself.
- 3:54It rewards laziness and then invoices you for it.
- 3:57The fix is boring.
- 3:59Only load what you'll use and set spending alerts.
- 4:02Most teams discover their bill the way you discover a leak.
- 4:06When the ceiling caves in.
- 4:07If an exec remembers one thing from this, what is it?
- 4:11Ask your data team a single question.
- 4:13When our definition of a key metric changes, how long until every dashboard reflects it?
- 4:19If the answer is days, your stack is broken.
- 4:21If it's minutes, you built the three layers right.
- 4:24And you can prove your money went somewhere real.
- 4:26Minutes, not days.
- 4:28That's the test.
- 4:29That's the test.
- 4:30That's a wrap.
- 4:31Fresh CDO briefings drop daily at MBA-training.com.