Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How to Test Data Pipelines Without Trusting the Output
Transcript
- Lucas: So there's this story from a financial services firm I heard about a few months back. They had a reporting pipeline that pulled exchange rates from an external API, and one day that API changed its response format slightly — the field for the rate went from a string to a float. The pipeline didn't crash, it just silently converted the float to a string in the output. For six months, every client report that used that rate was off by a factor of ten on certain currency pairs. Nobody noticed because the numbers looked plausible. Luna: Six months? That's terrifying. And I bet the tests they had were all green the whole time. Lucas: Exactly. The unit tests passed because they mocked the API response with the old format. Integration tests passed because they used sample data that also used the old format. The pipeline's output looked fine. It's the classic problem with data pipelines — the code works, but the data doesn't. And that's what I want to dig into today: how do you test the data itself, not just the code that moves it? Luna: So essentially we're talking about verifying the quality of the output at every stage, right? Lucas: Right. And there's a framework that's been gaining traction — it's called data contracts, schema validation, and data profiling. The idea is you define explicit expectations about what the data should look like, and you check those expectations on every batch. It's like a safety net that catches the silent changes before they propagate. Luna: Can you walk us through a concrete example? I think that helps people understand how to actually implement this. Lucas: Sure. Let's take a simple e-commerce pipeline. You have an order table with columns like order_id, customer_id, total_amount, and currency. The pipeline ingests raw events, enriches them with customer data, and outputs a daily summary. A naive approach would be to just check that the row count matches expectations. But that doesn't catch a currency field that's suddenly a string instead of a float. Luna: So what do you do instead? Lucas: You define a schema contract. You specify that total_amount must be a float, currency must be a three-letter ISO code, order_id must be unique and non-null. Then you run a schema validation step at the beginning of the pipeline. If the input doesn't match the contract, you fail fast. That alone would have caught the exchange rate bug immediately. Luna: So schema validation checks the types and the structure. But what about the values themselves? Like, what if the currency code is 'ZZZ'? Lucas: That's where data profiling comes in. You write expectations like: the set of currency codes must be within a known list, the average order value should be within a certain range, the number of null customer_ids should be zero. You run these checks on the output of each stage. If the profile deviates from what you expect, you know something changed. Luna: And you run these checks on every batch? That could be a lot of overhead. Lucas: It can be, but you can be smart about it. You don't need to profile every row. You can sample, or you can use a tool that does it incrementally. The point is to catch the big issues early. For a daily batch, a few seconds of validation is worth it if it prevents a six-month silent bug. Luna: That's a good trade. But what about pipelines that run in real time? Is this approach still applicable? Lucas: It's trickier, but you can adapt it. For streaming, you can validate windows of data, or you can do a quick schema check on each event. Some frameworks let you attach expectations to the stream itself. But the core principle holds: you're defining what 'good' looks like and you're checking against it continuously. Luna: And what about the people dimension? I imagine getting the business to agree on what 'good' looks like is half the battle. Lucas: It's true. You need to sit down with the data consumers — the analysts, the finance team — and ask them, what would make you trust this data? What checks would you run before you use it? Then you codify those checks into the pipeline. It's a collaboration, not a solo engineering task. Luna: So it's almost like you're building tests with the users, not just for them. Lucas: Exactly. And that's why this approach works better than just writing more unit tests. It forces you to think about the data's meaning and its business context. Like, the exchange rate bug — the unit test would never have caught it because the test was written to match the existing format. But a data contract that says 'this field must be a float' would have caught it the moment the API changed. Luna: It's a fundamentally different mindset, isn't it? You're not testing code in isolation, you're testing the system's behavior with real data. Lucas: Right. And that mindset shift is what separates teams that have these silent data bugs from teams that catch them in minutes. It's about being suspicious of your data, not just your code. Luna: I think that's a great note to end on for our main discussion. But before we wrap up, I know that this kind of practical conversation is exactly why many of you listen to this show. And if you've gotten something useful from episodes like this, and you want to keep them coming without ads, there's a way to support us directly. Lucas: Yeah, it's a small thing, but it makes a difference. If you're in a position to, you can buy us a coffee at buy me a coffee dot com slash fexingo. Just a simple way to keep the show running and ad-free. No pressure, but we appreciate it. Luna: And we'll get back to the topic in a second, but that's really all it is. Lucas: So, to circle back — the key takeaway is that testing data pipelines is about defining and checking data expectations, not just testing code. Start with a schema contract, then add profiling checks on key fields, and involve your data consumers in the process. Luna: And if you want to see this in action, there are open-source tools like Great Expectations and dbt tests that can help you implement these checks without building everything from scratch. Lucas: Great point. Those tools let you write expectations in a declarative way, and they integrate into your CI/CD pipeline. So you can automate the whole thing. And that's the key — consistency. It's not a one-time effort, it's an ongoing discipline. Luna: So, next time you see a pipeline that's been running for months, ask yourself: would I trust the data it produces? If the answer is not a confident yes, maybe it's time to add some contracts. Lucas: And that's the question we should all be asking. Until next time.