Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How Semantic Versioning Breaks Your Automated Tests
Transcript
- Lucas: So here’s a scenario that might sound familiar: your CI pipeline has been green for weeks. You push a branch that only changes a README, but suddenly half the test suite fails. No code changed. What gives? Luna: Is the answer 'it’s always the dependencies'? Because that’s my batting average. Lucas: You’re not far off. In this case, the culprit was semantic versioning. A library you depend on bumped from 2.1.4 to 2.2.0 — a 'minor' version according to semver, so supposedly backward compatible. But your tests caught something the version number didn’t promise. Luna: So the semver contract broke, and the tests took the blame. Lucas: Exactly. And this isn’t a one-off. A major e-commerce platform — I’m not naming names, but think big — had a production outage traced back to this exact pattern. They updated a logging library from 3.1.0 to 3.2.0. The new version changed the internal format of a log context object. Their monitoring system parsed those logs, and suddenly dashboards went dark. Luna: Wait — a minor version changed internal behavior? That’s a bug in the semver promise, not in the tests. Lucas: Right. The library author didn’t consider that log format was part of the contract. But here’s the hard truth: in practice, the semver spec is aspirational. The real contract is what your tests actually validate. And when a dependency bumps, your tests become the canary. Luna: So what’s the fix? Never trust semver, always pin everything? That’s its own mess. Lucas: You hit it. Pinning to exact versions — like '3.1.0' in your package.json or requirements.txt — buys you reproducibility but at the cost of security patches and bug fixes. You freeze the world. The middle ground is lockfiles. Tools like npm’s package-lock.json or Bundler’s Gemfile.lock record the exact resolved tree. When you run 'npm install', you get the same versions every time. Luna: But that doesn’t solve the problem that semver can be wrong. You still get surprised when you actually update. Lucas: True. So you need a second layer: contract testing. For every dependency your code interacts with — especially internal microservices — you define a formal contract: expected inputs, outputs, side effects. Then your test suite includes a set of 'canary' tests that directly validate those contracts against the actual library version. Luna: So instead of hundreds of integration tests failing cryptically, you get one clear failure: 'Contract violation: logging library version 3.2.0 changed log format.' Lucas: Exactly. And that failure tells you immediately: this isn’t a bug in your code. It’s a dependency change that needs review. You can then decide: adapt your code, pin to the old version, or escalate to the library maintainer. Luna: Let’s talk about microservices, because that’s where this really spirals. When you have 50 services, each publishing their own API with semver, the matrix gets insane. Lucas: Right. Every team publishes a version. Consumer teams pin to ranges like '^2.0.0' meaning 'any 2.x version'. And then a 'minor' bump in one service can break ten downstream services. I’ve seen a case where a team changed a field name from 'customerId' to 'customer_id' in a minor version — they considered it internal. But three services relied on that field directly. Luna: That’s not a semver problem, that’s a governance problem. Lucas: It’s both. The semver spec says nothing about what constitutes the 'public API' in a microservice context. Is the HTTP endpoint the only contract? Or the database schema? Or the async event payload? The lines blur. Luna: So what do you recommend teams actually do? I know some organizations mandate that every dependency version change triggers a full integration test suite. Lucas: That’s a start, but it’s expensive. A smarter approach is to use 'dependency diffs' — tools that compare the resolved dependency tree across environments. When you promote a build from staging to production, the diff tells you exactly which libraries changed. Then you can run a targeted set of contract tests for those specific libraries. Luna: And if those tests pass, you have high confidence the bump is safe? Lucas: Higher than just trusting semver, anyway. But I’ll be honest — no system is perfect. The fundamental tension is between reproducibility and freshness. You can pin everything and never update, but then you miss security patches. Or you can auto-update and accept occasional breakage. Luna: Where do you land personally? Lucas: I lean toward pinning with lockfiles, plus a scheduled periodic update — say every two weeks — where you deliberately bump dependencies and run a full CI suite. That way, breakage happens on your terms, not when you’re trying to ship a feature. Luna: That’s sensible. And it gives you time to write those contract tests for the critical libraries. Lucas: Yeah. And one more thought: if you’re the library maintainer, treat every change as potentially breaking unless you can prove otherwise. Write migration guides. Use feature flags. And for the love of everything, don’t change internal formats without a major version bump. Luna: Alright, I want to circle back to something you said earlier about the e-commerce platform. They had a monitoring system that parsed logs — that’s an internal contract, not just an API. Lucas: Right. And that’s actually the most dangerous class of dependencies: the ones you don’t explicitly declare. You might not list 'log format' in your package.json, but your entire observability pipeline depends on it. So when a library changes its log structure, your alerts go silent. Luna: That’s terrifying. How do you test for implicit contracts like that? Lucas: You have to make them explicit. Write a contract test that sends a log line and validates that your log aggregator parses it correctly. Same for any serialization format, any metrics endpoint, any health check response. If it’s consumed, it’s a contract. Luna: So in effect, you’re expanding the scope of what your tests cover beyond just functional correctness. Lucas: Exactly. And once you start thinking that way, you realize that semver is just a naming convention. The real source of truth is your test suite. It’s the only thing that tells you whether a version bump actually works in your specific context. Luna: Okay, I want to shift slightly. We’ve been talking about dependencies you pull in from outside. But what about internal libraries? Your own team’s modules? Lucas: Great point. Internal packages are even more prone to semver abuse because there’s no public accountability. A team might bump 'minor' on a library used by 20 other teams, thinking they’re being safe, but they changed a function signature. Luna: I’ve seen that. The fix is to enforce the same contract testing for internal packages as external ones. No exceptions. Lucas: And that’s where a tool like a 'version manifest' can help. Instead of each consumer specifying their own version range, you centralize the dependency map. When a library updates, you run a pre-merge CI job that checks every consumer’s contract tests against the new version. Luna: That’s a lot of infrastructure. But for a platform team, probably worth it. Lucas: Yeah, it’s an investment. But the ROI comes from avoiding the kind of silent failures we talked about. One production outage from a bad semver bump can cost way more than setting up the tooling. Luna: By the way, speaking of keeping things running smoothly — you know what makes all these deep dives possible? The listeners who chip in a few bucks a month. We keep the show ad-free, and that’s because of support from the audience. Lucas: Yeah, it really makes a difference. If you’ve gotten value from episodes like this one, you can find us at buy me a coffee dot com slash fexingo. That literally helps fund the research and production. No pressure, just if it feels right. Luna: Exactly. And it lets us keep diving into topics like versioning hell without any ad breaks. Lucas: Alright, back to the topic. One last thing I want to mention: the concept of 'semantic versioning for test suites'. Some teams version their test suites independently from the application code. That way, a test can say 'I am compatible with app version 2.3.x'. If the app bumps to 2.4.0 and the test suite hasn’t been updated, you get a clear mismatch. Luna: That’s clever. It’s like applying the same principle to the tests themselves. Lucas: Right. Because tests are also code, and they also have dependencies — on the app’s internal APIs. So versioning them explicitly can prevent false negatives. Luna: I think the takeaway here is: don’t trust the version number. Trust your tests, and make your contracts explicit. Lucas: Exactly. Semantic versioning is a useful shorthand, but it’s no substitute for a well-designed test suite that validates the actual behavior you depend on.