Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / Why Your Test Suite Needs Data Contracts for Production Data
Transcript
- Lucas: You know that sinking feeling when your test suite passes locally, passes in CI, and then fails in staging because a column that used to be non-null suddenly has nulls? Luna: Oh, I've lived that. It's usually a data producer upstream changed something, and your tests had no way of knowing. Lucas: Right. And the typical response is to add more defensive assertions—check for nulls, check types, check ranges. But that's reactive. What you really want is a contract between the data producer and consumer that says, 'I will always give you data that looks like this.' That's the idea behind data contracts. Luna: So it's like an API contract, but for data pipelines instead of microservices? Lucas: Exactly. A data contract formally specifies schema, freshness, maybe row count ranges, and even semantic rules like 'this column must be unique' or 'this field must be a valid email.' The producer publishes that contract, and the consumer—your test suite—validates against it before even running a single assertion. Luna: I can see how that prevents the null-column disaster. But who writes the contract? The producer or the consumer? Lucas: Ideally, both. The producer defines what they can guarantee, the consumer defines what they need, and they negotiate the middle. Uber's Data Quality Platform, or DQP, did something like this at scale. They had over ten thousand data pipelines producing tens of petabytes daily, and they needed a way to enforce quality without manual reviews. They introduced contracts that included schema, freshness, and even data provenance checks. Luna: Provenance—like, which system generated this data and through which transformations? Lucas: Yep. So if a pipeline suddenly sources from a new table, the contract flags it. Uber reported that within six months of rolling out their contract-based validation, data incidents dropped by about 70 percent. And that's not just production incidents—that includes test suite failures caused by unexpected data changes. Luna: That's huge. But I'm wondering about the overhead. If every team has to negotiate contracts, doesn't that slow things down? Lucas: It's a valid concern. The key is to start small and automate contract enforcement. You don't need a formal committee. You can define contracts in something as simple as a YAML file in the producer's repo, and the consumer's test pipeline picks it up. The contract itself can evolve—versioned like any other artifact. Luna: So the producer changes the contract, the consumer's test suite breaks, and that's the signal to update the contract or fix the data. That's actually faster than discovering a mismatch at two in the morning during a regression run. Lucas: Exactly. Let me give you a concrete example. A major online retailer had a data pipeline that fed a recommendation model's test suite. The pipeline team switched from one internal data source to another that had slightly different null-handling logic. The test suite started failing intermittently—took them three weeks to trace it back. Luna: Because the test suite had no contract to check that the 'product_rating' column should never be null. Lucas: Right. After they introduced a data contract, that same change would have been caught at contract validation time, before the test suite even ran. The fix was a simple contract update—the new source actually had valid data, just different null conventions. But without the contract, it was a latent bug. Luna: So the contract acts as a trust boundary. The test suite trusts that the data conforms, and if it doesn't, the contract fails first. Lucas: That's exactly the framing I use. It's the same principle as contract testing for microservices: you define the interface, and each side can change independently as long as the contract holds. Data contracts bring that same decoupling to data pipelines. Luna: What about freshness? I've seen test suites that fail because a daily batch job ran late, not because the data was wrong. Lucas: Freshness is a classic contract clause. You can specify that a table must be updated within the last 24 hours, or that a stream must have records within the last five minutes. If the data is stale, the contract fails, and the test suite doesn't even try to run. That saves time and prevents flaky failures. Luna: So the test suite only runs when it knows the data is valid and fresh. That's a big shift from 'run tests and hope the data is okay.' Lucas: It is. And it changes the culture around data ownership. Producers become accountable for the quality of the data they publish, not just for making it available. Consumers have a clear path to raise issues without blaming each other. Luna: That sounds like a mature data organization. But for teams just starting out, what's the simplest first step? A single contract for one critical table? Lucas: Absolutely. Pick your most-used data source—the one that feeds the most test suites—and write a contract that checks schema and maybe two or three critical columns. Then automate the validation in your CI pipeline. Once you see the value, you'll expand naturally. Luna: And what about tools? Are there off-the-shelf solutions or do most teams build their own? Lucas: There are a few emerging open-source libraries, like Great Expectations and Soda Core, that let you define expectations in Python or YAML. They integrate with CI and can send notifications when a contract fails. Uber built DQP internally, but the concept is now accessible to smaller teams. Luna: Good. Because I think the pattern is solid, but if it requires a dedicated platform team, it's not going to take off in most companies. Lucas: That's the right filter. And I'd argue the biggest return on investment comes not from the tooling but from the discipline of defining what you expect from your data. That discipline pays off even if you implement it with a simple script that checks a few SQL queries. Luna: So the contract is more about communication than technology. Lucas: Exactly. And that ties back to something we've touched on before—test suite reliability is often a people problem disguised as a code problem. Data contracts are a way to make the implicit explicit. Luna: Speaking of making things explicit... Lucas, we've been doing these episodes for a while now, and I think it's worth mentioning that the show stays ad-free because a small group of listeners chips in. If you've found today's conversation useful, you can support the show at buy me a coffee dot com slash fexingo. It genuinely helps us keep doing deep dives like this one. Lucas: Yeah, and we don't have a big marketing machine behind us—it's really just listener support that makes this sustainable. So if data contracts or any of our past episodes have saved you a late-night debugging session, that's the kind of thing that keeps us going. Luna: Exactly. And now back to the topic—because I want to talk about how data contracts handle schema evolution. What happens when a producer needs to add a column? Lucas: That's where contract versioning comes in. You can specify that the contract is backward-compatible—new columns are optional, and existing columns can't change type or nullability unless the contract version bumps. Consumers can choose to adopt the new version when they're ready. Luna: So it's like a semver for data. That makes evolution manageable. Lucas: Exactly. And if you automate contract validation in CI, then a producer that tries to push a breaking change gets immediate feedback. They can't accidentally break downstream test suites. Luna: I think the key takeaway for me is that data contracts shift the burden from 'fix the test suite after the data breaks' to 'ensure the data meets expectations before the test suite runs.' Lucas: That's a great summary. And it applies whether you're testing a machine learning model, a reporting dashboard, or a simple data pipeline. If you rely on data, you need a contract. Luna: Alright, Lucas. Next time we should look at how data contracts fit into a broader data mesh architecture. I think there's a lot of synergy there. Lucas: Absolutely. That's a perfect follow-up. For now, I think we've given listeners a concrete pattern they can try on Monday. Find your most critical data source, write a simple contract, and see how it changes your test suite reliability. Luna: And if you do, let us know how it goes. We'd love to hear your stories.