Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How Testing Data Pipelines Prevents Costly Data Errors
Transcript
- Lucas: You know how everyone talks about testing the software that users interact with—the front end, the APIs, the mobile app? But there's this whole other layer that often gets way less attention, and that's the software that moves and transforms data behind the scenes. Luna: Data pipelines. The invisible circulatory system of every modern company, right? Lucas: Exactly. And when those pipelines break, the consequences can be just as bad as a crashed website—sometimes worse because you might not even notice for weeks. I'm thinking of a specific fintech example from a couple years ago. A mid-sized payments company had a pipeline that aggregated daily transaction summaries for their revenue reports. Luna: Sounds pretty standard. What went wrong? Lucas: A developer pushed a change to the refund-processing logic. The old code correctly subtracted refunds from gross revenue. The new code inadvertently added refunds back in—a plus sign where there should have been a minus. The pipeline ran fine, no errors, no alerts. But for an entire quarter, the company reported revenue that was about three percent too high. Luna: Ouch. So their books were wrong, and they had to restate earnings. That's the kind of thing that can tank investor confidence. Lucas: Right. They estimated the direct cost—restatement fees, regulatory fines, lost business—at around two million dollars. And it all traced back to a pipeline that had unit tests for the individual components but no end to end tests that validated the actual data outputs. Luna: So what should they have done differently? I mean, unit tests caught syntax errors, but they didn't catch business logic errors. Lucas: This is where data pipeline testing gets really interesting. The key insight is that you need to test the data itself, not just the code. That means writing assertions about the outputs: 'the total refund amount should never exceed the gross revenue for a given day,' 'the number of transactions should match the source count,' things like that. Luna: So data quality assertions, kind of like Great Expectations or dbt tests. Lucas: Exactly. Great Expectations is a great example. You define expectations for your datasets—like 'column X should have no nulls' or 'column Y should be between 0 and 100'—and the framework runs those checks every time the pipeline executes. If an expectation fails, you can alert or even halt the pipeline. Luna: That would have caught the refund bug immediately, because the refund amount would have been positive when it should have been negative or zero. Lucas: Right. But there's another layer: contract testing between data producers and consumers. In many companies, data pipelines are owned by separate teams. The payments team produces a transactions table, and the finance team consumes it. If the payments team changes the schema or the semantics without telling finance, things break silently. Luna: That's where something like a schema registry comes in, right? Like Confluent's Schema Registry for Kafka. Lucas: Precisely. A schema registry enforces compatibility rules. If the payments team tries to change the refund field from an integer to a float, the registry blocks the change unless it's backward compatible. That prevents the kind of silent data corruption where a consumer suddenly starts reading nulls because the type changed. Luna: But schema changes aren't the only risk. What about data that's correct at the source but gets corrupted during transformation? Lucas: That's the hardest part. I've seen teams adopt a technique called 'record and replay' for deterministic testing. You capture a small set of real input records—say, a thousand transactions from production—and store them along with the expected output. Every time you change the pipeline code, you replay those records and compare the output. Luna: So it's like snapshot testing, but for data. You're making sure that known inputs produce known outputs. Lucas: Yes. And it catches exactly the kind of regression that happened in that fintech case. The developer probably didn't consider the refund logic change because the unit tests passed. But if they had a replay test with a transaction that included a refund, the output would have diverged from the expected snapshot. Luna: That sounds powerful. But I imagine it's tricky to maintain those snapshots. Data evolves—new fields, new business rules. Lucas: Absolutely. That's the trade-off. You have to update the snapshots whenever the expected behavior changes intentionally. But teams that do this religiously—I've seen a fintech startup that reduced data incidents by 95 percent after implementing record and replay plus data quality assertions. Luna: Wow. So the combination—contract testing for schema evolution, deterministic replay for transformation logic, and data quality assertions for output correctness—covers almost all the bases. Lucas: That's the trifecta. And one more thing: don't forget about environment parity. The pipeline that ran in staging was using a tiny fraction of the data volume. The production bug in the refund logic only surfaced when processing refunds from high-volume days, which the staging test never exercised. Luna: So they needed to use representative data volumes in their test environment. Or use a technique like traffic mirroring to run production data through the new code in parallel. Lucas: Exactly. Shadow testing, some call it. You clone a portion of production traffic and feed it to a staging pipeline to see if the outputs match. It's a bit more infrastructure, but for critical financial pipelines, it's worth it. Luna: That's a solid approach. And it ties back to the core theme: data is the new gold, but only if it's accurate. Speaking of which, we've been digging into this topic for a while now, and I think it's a great example of why we keep this show ad-free. We want to deliver useful, deep content without interruptions. Lucas: Absolutely. And if that resonates with you, supporting the show is simple: you can go to buy me a coffee dot com slash fexingo. It helps us keep producing episodes like this one. Luna: Yeah, it really makes a difference. Now, back to pipelines—one last thought: what about testing for data privacy? Ensuring that personally identifiable information isn't leaking through transformations. Lucas: Great point. That's a whole other angle. But for today, the takeaway is: don't let your data pipelines be the untested black box. Invest in contract testing, replay testing, and quality assertions. The cost of getting it wrong, as we saw, can be millions. Luna: Thanks, everyone. Next time we'll talk about how to test serverless applications effectively. That's a tricky one. Lucas: See you then.