Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How Testing Your Test Infrastructure Prevents Silent Failures
Transcript
- Lucas: Luna, let me start with a question. When was the last time you actually tested your test infrastructure itself — not the application code, but the machinery that runs your tests? Luna: Honestly? Never. I've tested the tests themselves, like mutation testing, but I don't think I've ever validated the CI runner or the Docker image that executes them. Lucas: That's exactly the blind spot. A friend of mine at a mid-size fintech company — they process payments for about two million users — had a nightmare scenario last year. Their regression suite had been passing with 99.8 percent success for six straight weeks. Everyone was feeling great. Luna: I can already tell this is going to end badly. Lucas: It does. One day, a junior engineer accidentally introduces a bug that breaks credit card tokenization. The test suite runs, all green, ships to production. Within an hour, three thousand payment failures. They roll back. Then they dig into why the tests didn't catch it. Luna: Let me guess — the tests never actually ran? Lucas: Close. They ran, but a misconfigured Docker image — a base image had been auto-updated and the new version had a broken SQLite driver — caused the test database connection to silently fail. The test framework, pytest in this case, treated a failed connection setup as a skip rather than a failure. So every integration test just skipped. The unit tests passed, but the integration layer was essentially untested for weeks. Luna: That's terrifying. And it's not even a flaky test problem — it's an infrastructure invisibility problem. Nobody noticed because the pipeline reported 'all tests passed' with a green check. Lucas: Exactly. That's what I want to drill into today: testing your test infrastructure. Most teams have robust application test coverage, but they never verify that the CI pipeline, the test runner, the database fixtures, the network mocks — all of that — are actually working as intended. Luna: So what's the first thing teams should do? I mean, where do you even start? Lucas: Start with the simplest check: assert test execution count. In every CI run, compare the number of tests actually executed against a known baseline. If your regression suite has 4,200 tests and today it reports 4,180, that should be a yellow flag. Maybe a test was skipped, maybe a module wasn't discovered. That mismatch is the earliest indicator of infrastructure drift. Luna: That's a great lightweight check. But you also have to trust that the baseline itself is correct. If someone updated the test suite and legitimately added or removed tests, the baseline shifts. Lucas: Right, so you'd want a versioned baseline — tied to a tag or a known good commit. The comparison should be against the previous passing run on the same branch, not some static number. But the principle holds: if the count changes by more than one percent, alert someone. Luna: Okay, what's step two? Lucas: Step two: validate baseline data integrity. Many integration tests depend on seed data in a test database. If that seed data gets corrupted — maybe a migration script ran against it, maybe a cleanup job deleted a table — the tests might pass because they're testing against empty tables. The fintech case I mentioned: their seed data was intact, but the connection was broken. So you need to add a 'health check' test that verifies the test environment is in a known state. Luna: Something like — assert that the test database contains exactly the expected number of rows in core tables? Lucas: Exactly. Or assert that a known query returns a specific value. It's a smoke test for the infrastructure itself. Run it first, before any application tests. If it fails, stop the pipeline and send an alert to the DevOps team. In the fintech's case, that health check would have caught the dead database connection immediately. Luna: Step three? Lucas: Step three is what I call a 'test of the tests.' Periodically inject a known-breaking change into a feature branch and verify that your test suite actually fails. It sounds obvious, but most teams never do this. You can automate it: have a CI job that takes the current branch, applies a simple mutation — like flipping a boolean in a critical function — runs the test suite, and asserts that at least one test fails. Luna: That's basically mutation testing, but applied to the infrastructure layer rather than the code. Lucas: Right. Mutation testing checks if your tests catch code mutations. This checks if the pipeline catches a known failure. If the pipeline stays green despite a known-breaking change, something is wrong with your test infrastructure. Maybe the test runner is skipping files, maybe a mock isn't wired correctly, maybe the build artifact is stale. Luna: I can see how that would build confidence. But it also adds complexity to the CI pipeline. Teams with limited DevOps bandwidth might push back. Lucas: They might, but the alternative is shipping a broken build with a green checkmark. And that's far more expensive. The fintech company's incident cost them roughly three hundred thousand dollars in chargebacks and lost transaction fees. Plus the reputational damage. Luna: So what did they do after the incident? Lucas: They implemented exactly these three checks. Took about two sprints. They added a health-check test that asserts database connectivity and schema integrity before any other tests. They added an execution count assertion that alerts if the count deviates by more than one percent from the previous build on the same branch. And they set up a weekly cron job that creates a branch with a deliberate bug, runs the full pipeline, and emails the team if the pipeline passes. Luna: Has it caught anything since? Lucas: Yes. About three months later, the health check caught a missing environment variable after a Kubernetes pod update. The test suite would have run fine with a default value, but the production behavior would have been wrong. They fixed it before it reached any customer-facing tests. Luna: That's a great example. And it's not just about databases and Docker. You could apply this to any part of the infrastructure: API mock servers, test message queues, even the CI runner itself. Lucas: Absolutely. One team I know tests their CI runner by periodically running a pipeline on a known bad commit and verifying that it fails. If it passes, they know the runner is misconfigured. It's the same principle. Luna: My only hesitation is that these checks themselves could become flaky. If the health check ping to the test database times out once every hundred runs, you'll get alert fatigue. Lucas: That's a valid concern. The key is to make the health checks deterministic. Instead of pinging a live database that might be transiently unavailable, validate against a local fixture or a snapshot. And the test of the tests — the known-breaking commit — should be completely deterministic. No randomness. You're testing the machinery, not the application behavior. Luna: So you'd want the health check to run against a dedicated test database that's only used for that purpose, not shared with other CI jobs. Lucas: Exactly. And you'd want the known-breaking commit to be a simple, well-understood change that you know will cause a specific test to fail. Keep it simple. Luna: I think a lot of listeners might be in the same position as that fintech — trusting a green pipeline without questioning the infrastructure underneath. It feels like one of those things you only think about after an incident. Lucas: That's often how it goes. And that's why we talk about it. Because if this episode makes even one team add an execution count assertion or a health check, we've probably saved them a production outage. Luna: You know, speaking of saving time and avoiding issues — we deliberately keep this show ad-free. No sponsors, no mid-roll interruptions. It's a choice we make because we want the conversation to be exactly as long as it needs to be, without any breaks for things we don't believe in. Lucas: Exactly. If today's tech conversation gave you something usable — maybe you're already thinking about adding that test infrastructure health check — and you want to support that choice, the link is buy me a coffee dot com slash fexingo. No pressure, just an option for those who find value here. Luna: It's a small way to keep this independent and focused on the content, not on ad revenue. So thank you to anyone who considers it. Lucas: Alright, back to the nuts and bolts. One more thing I want to touch on: how to design these checks so they don't become a maintenance burden themselves. Luna: Yes, because if they're brittle, they'll be disabled within a month. Lucas: Right. The number one rule: keep the health check code separate from application test code. Put it in a dedicated directory, like 'ci_health' or 'infra_smoke'. That way it's clear this isn't testing the product, it's testing the pipeline. And use a separate test runner configuration — maybe a small pytest.ini that only runs those tests — so they don't get mixed with flaky application tests. Luna: And how do you handle failures from those health checks? Do you block the pipeline? Lucas: For the health check and the execution count assertion, yes — block the pipeline. A failing health check means the test environment is corrupt. If you let the tests run anyway, you'll get false positives or false negatives. For the test of the tests — the known-breaking commit — that should not block the pipeline. It's an out-of-band verification. If it fails, send a notification to the QA and DevOps channels. Luna: That makes sense. You don't want a false alarm on the weekly cron to block every developer's PR. Lucas: Exactly. And finally, one more concrete recommendation: log the exact command and environment variables for every test run. The fintech team eventually realized that their CI runner was using a cached Docker image from six months ago because the pull policy was set to 'if not present'. Tracking the image digest would have caught that. So include image digests, dependency versions, and commit hashes in your test report. Luna: That's a great habit. It turns an opaque pipeline into a transparent one. You can always go back and say 'what was the exact environment when this test passed?' Lucas: Precisely. And with that level of detail, you can start building dashboards that monitor test infrastructure health over time. Look for trends: are we seeing more skips? Are certain modules consistently dropping tests? That's proactive, not reactive. Luna: I think the takeaway for me is that trust in automation is earned, not given. A green pipeline is not a guarantee — it's a hypothesis that you need to validate. Lucas: That's a perfect way to put it. So this week, maybe take a look at your pipeline. See how many tests you expect to run, and verify that number. Check that your test database is wired correctly. And maybe schedule a weekly known-breaking commit test. It's a small investment that pays back the first time it catches a silent failure. Luna: I know I'll be adding that execution count assertion to our main repo this afternoon. Lucas: Great. And if you do, let us know how it goes. We'd love to hear from listeners who've caught something with these techniques.