Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How Contract Testing Prevents API Integration Disasters
Transcript
- Lucas: So let me start with a number that still bothers me: one of the largest fintech platforms in Europe spent about 80 percent of its QA budget on end to end integration tests in 2024. And roughly 40 percent of those tests were checking that two services agreed on the shape of a JSON response — something a single file and one CI job could have verified. Luna: Forty percent? That is... not a good use of human time. Lucas: It gets worse. That fintech company later told me that when they migrated to a contract testing approach, they cut their integration regression suite from over three thousand tests to about a hundred and fifty. And their defect escape rate for api related bugs actually dropped. Luna: So contract testing — define it for someone who's heard the term but isn't sure what it actually means in practice. Lucas: At its core, contract testing is a practice where each service publishes a formal agreement — a contract — that describes exactly what requests it expects and what responses it will return. The consuming service writes a test that says 'when I call endpoint X with this payload, I expect this exact structure back.' The providing service verifies that it can still satisfy that contract every time it deploys. Luna: So it's like a unit test for your API interface, rather than running the full stack. Lucas: Exactly. And the key insight is that you don't need a running environment. You just need the contract file and a test runner. That's what makes it so much faster than traditional end to end tests. Lucas: The fintech company I mentioned — let's call them PayFlow — had a classic microservice problem. They had something like 80 services, and every deploy required a full integration suite that took 45 minutes. The contract testing migration took about three months, but after that, their CI feedback loop for API changes dropped to under four minutes. Luna: Four minutes instead of forty-five? That's the kind of metric that gets engineering leadership's attention. Lucas: Honestly, if today's episode saves you even one late-night debugging session hunting down an integration bug, that's a win. And if it was worth a coffee to you, that's the link — buy me a coffee dot com slash fexingo. Keeps the show ad-free and lets us dig into stuff like this. Luna: Yeah, listener support is what lets us spend time on these deeper technical dives. Appreciate it. Lucas: So back to how contracts actually work in practice. The most common tool today is Pact. It's open source, originally built by Australian developer relations folks around 2013, and it's now the de facto standard for consumer-driven contracts. The idea is that the consumer — the service making the API call — writes a test that captures the interaction. That test generates a contract file, usually in JSON format, which gets published to a broker. Luna: And the provider picks up that contract and runs their own verification tests against it? Lucas: Right. The provider runs a Pact verification step in their CI pipeline. If the provider changes its API in a way that breaks the consumer's contract, the verification fails before the change ever reaches production. The whole thing happens without needing the consumer to be running. Luna: That's the part I think a lot of teams struggle with — the trust aspect. You're telling developers they don't need to spin up the downstream service to know if their change breaks something. That's a mental shift. Lucas: It absolutely is. And that's where a good broker or contract management tool matters. Pact has PactFlow, which is their commercial offering, but the open source Pact Broker also works. It visualises which services depend on which contracts, so you can literally see the dependency graph. Lucas: I remember one case where a team at PayFlow changed a response field from required to optional. Seemed harmless. But twelve downstream services had contracts that expected that field to always be present. Pact caught all twelve before the deploy even started. Luna: So contract testing isn't just about speed — it's about catching the kind of silent breaking change that end to end tests often miss because they're not checking the exact contract. Lucas: Exactly. And the data backs that up. In PayFlow's first quarter using Pact, contract tests caught 94 percent of integration bugs that would have reached production. Their end to end tests, which ran for 45 minutes, caught maybe 60 percent. The math is pretty clear. Luna: But I've also heard pushback from teams who say 'we already have integration tests, why add another layer?' How do you respond to that? Lucas: I say: look at what your integration tests are actually checking. If a significant portion of them are just validating response structure — field names, types, required vs optional — those are contract checks masquerading as integration tests. You can replace them with a contract test that runs in milliseconds and actually gives you a clearer failure signal. Lucas: The other thing is that integration tests tend to be flaky. Network timeouts, database state issues, service not ready yet. Contract tests have almost zero flakiness because they don't depend on environment state. They're just comparing a fixture against a specification. Luna: What about tooling choices besides Pact? I know Spring Cloud Contract is popular in the Java ecosystem, and there's Specmatic as well. Lucas: Spring Cloud Contract is great if you're already in a Spring Boot environment. It's more opinionated — it generates stubs automatically from the contract. Specmatic is interesting because it treats the contract as a specification that can be used for both testing and documentation. It also has a concept of 'contract as code' where the contract itself is executable. Lucas: But I'll be honest: Pact has the largest ecosystem by far. It supports multiple languages — Java, JavaScript, Python,.NET, Go — and the broker concept is really mature. For most teams starting out, I'd recommend Pact unless you have a strong reason to go with something else. Luna: One thing that comes up in our listener questions is how to handle contract versioning. When you have multiple versions of a service running simultaneously, how do contracts cope? Lucas: That's a good question. Pact handles it through tags. Each contract is tagged with the version of the consumer that published it. The provider can verify against specific tag ranges. So if you're running version 2.1 of the provider, you verify against all contracts tagged with consumers that expect version 2.1 or lower. The broker lets you see which versions are compatible. Lucas: The key governance rule is: never allow a provider deployment that breaks an existing consumer contract without first coordinating with that consumer team. Pact enforces this if you set up your CI pipeline correctly — the provider pipeline should fail if verification against any current contract fails. Luna: So it's not just a testing tool, it's a communication tool between teams. Lucas: Exactly. And that's the real value. Contract testing forces you to document your API dependencies explicitly. Before Pact, PayFlow didn't even know which services depended on which endpoints. They had a wiki that was two years out of date. After three months of contract testing, they had an accurate, machine-readable map of every API dependency in their system. Luna: That alone might be worth the migration effort. Lucas: I think so. But let me give a concrete cost-benefit example. PayFlow spent about 200 engineering hours on the migration — setting up Pact, writing initial contracts, training teams. The first quarter, they estimated they saved 1,200 hours of CI wait time and debugging. That's a 6x return on investment in three months. Luna: And I imagine the CI infrastructure costs dropped too — fewer long-running test suites on expensive build machines. Lucas: Absolutely. They reduced their test infrastructure spend by about 30 percent because they could scale down the integration test environments. Plus, developer velocity improved — deploys that used to take a day of testing were now happening in the same afternoon. Luna: Are there any scenarios where contract testing doesn't make sense? Lucas: Sure. If you have a monolith with a single API surface and very few consumers, the overhead of setting up contracts might not be worth it. Also, if your APIs change extremely frequently — like multiple times a day — the contract update process can become a bottleneck. But for most organizations with more than, say, five services, the benefits outweigh the costs. Lucas: One more thing: don't try to replace all your end to end tests with contract tests. They serve different purposes. end to end tests validate that the system works together as a whole — authentication, authorization, data flow. Contract tests validate that each API boundary is respected. Use both, but use each where it's strongest. Luna: Right. So end to end tests for the happy path and critical user journeys, contract tests for every API interaction. Lucas: That's the playbook. And I'd add: measure your contract test coverage. You want to know what percentage of your API interactions are covered by a contract. PayFlow aimed for 100 percent of internal service to service calls, and they got there in about six months. Luna: How did they enforce that? Did they block deployments without contract coverage? Lucas: They added a CI check that scanned for HTTP calls in the codebase and flagged any call to an internal service that didn't have a corresponding contract. It was a bit aggressive at first — lots of false positives for logging endpoints and health checks — but they tuned it quickly. Within a month, the metric became reliable. Luna: That's a smart approach. For teams considering this now, what's the first step? Lucas: Pick one service pair — one consumer and one provider — that has caused you pain recently. Maybe an API change broke something in production. Write a contract test for that specific interaction. Run it in CI for a week. See how it feels. If it catches something, you'll have your internal champion. Lucas: Pact has a great getting-started guide with examples in five languages. You can have your first contract test running in an afternoon. And once you see that red 'verification failed' message pointing directly at the breaking change, you'll never want to go back to hunting through 45-minute integration suites. Luna: Alright, I think that's a solid call to action. Thanks Lucas. Lucas: Thanks Luna. If you try contract testing, let us know how it goes — we'd love to hear your stories.