Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / Why Your Tests Fail Due to Flaky Infrastructure
Transcript
- Lucas: You've been in the CI queue all morning, your pull request is finally green, and then — rerun the same suite and it's red. No code change. Just a test that decides to fail for no reason. Luna: Classic flaky test. And the first instinct is always to blame the test itself. But that's not always fair. Lucas: Right. So today I want to talk about one of the most underrated sources of flakiness: your infrastructure. Not your assertions, not your mocks — the actual machine or container where your tests run. Luna: Okay, I'm listening. Because I've definitely blamed a test for weeks only to find out the CI runner was running out of memory at peak hours. Lucas: Exactly that. Let's use a concrete case. A mid-size fintech company I consulted for — they processed payments, so testing was critical. They had about 3,000 end to end tests running on a shared CI cluster. Every afternoon, around 2 PM, a handful of tests would fail. Always different tests. Always between 1 and 3 PM. Luna: That timing is suspicious. Were they running other jobs at the same time? Lucas: Yes. The CI cluster was shared across all teams. At 2 PM, the data engineering team kicked off their nightly batch jobs — massive memory hogs. They weren't even tests, just data pipelines. But they'd consume enough RAM that the test runners got squeezed. Luna: So the tests weren't flaky — the environment was. Lucas: Exactly. When we dug into the failing tests, they all had one thing in common: they loaded a moderate amount of test data, maybe a few megabytes, and then performed some operation that required additional memory briefly. Under normal conditions, fine. Under memory pressure, the OS would start swapping, and a timing-sensitive assertion would fail. Luna: How did you find that? Just looking at the test output? Lucas: Not initially. The test logs showed generic timeout errors. But we added a small metadata capture at the start of every test run: CPU load, available memory, number of concurrent CI jobs. Once we correlated that with failures, the pattern was obvious. Tests failed when available memory dropped below 512 megabytes. Luna: So the fix was to isolate the test runners? Lucas: Partly. They moved to containerized runners with resource guarantees. Each test pod got a dedicated memory limit — 2 gigabytes — and the CI scheduler was configured to never overcommit. That alone cut flaky test failures by 80 percent. Luna: Eighty percent. That's huge. And it wasn't a code change at all. Lucas: Not a single line of test code changed. The lesson is: before you mark a test as flaky and add a retry annotation, check if the environment is the variable. Because retries mask the problem — they don't fix it. Luna: That's a good point. Retries just make tests slower and give false confidence. If a test fails one in ten times, and you retry three times, you'll almost never see the failure in CI. But it's still there. Lucas: Right. And you're shipping code that might break in production under similar conditions. So let's talk about how to systematically diagnose infrastructure-induced flakiness. Luna: I'm ready. What's the first step? Lucas: Instrument your test runners. At the very least, log CPU, memory, disk I/O, and network latency at the start and end of each test. Store it alongside the test result. Then you can query: 'show me all failed tests where memory was below X at start time.' You'll find patterns fast. Luna: And you need a baseline. What's 'normal' for your environment? Lucas: Exactly. If you don't know your baseline, you can't detect anomalies. Most CI platforms give you some of this data out of the box, but rarely at the test level. You have to capture it yourself. Luna: What about shared vs. dedicated runners? A lot of teams use shared runners to save money. Lucas: Shared runners are fine as long as you have resource limits and you monitor them. The problem is when the scheduler treats all jobs equally, even though some are memory-intensive and some are cpu bound. You need to tag your jobs with their resource profile and let the scheduler bin-pack accordingly. Luna: Donation segment? Lucas: Yeah, good call. If today's episode was useful to you and you want to keep it ad-free, buy me a coffee dot com slash fexingo helps. Luna: Even a couple of dollars a month means we can keep producing these. Just saying. Lucas: Alright. So let's go back to diagnosing flaky infrastructure. Another big culprit is network latency. If your tests hit external APIs or databases, a slow network can cause timeouts that look like test bugs. Luna: We had that at my last job. Our CI ran in us-east-1, but our staging database was in us-west-2. Cross-region latency was wildly variable. Some tests would pass at 3 AM but fail at 3 PM. Lucas: Classic. The fix there is either co-locate your test infrastructure with the services you're testing, or use deterministic network simulation. Some teams use tools like Toxiproxy to inject controlled latency and ensure their tests handle it gracefully. Luna: But that adds complexity. Is it always worth it? Lucas: Depends on your risk tolerance. If you're a fintech company processing millions of dollars, yes. If you're a small SaaS tool, maybe not. But you should at least measure the latency and set appropriate timeouts, not the default 30-second timeout that came with your HTTP client. Luna: Right. Default timeouts are never right. So what's the one change a team can make this week to reduce infrastructure flakiness? Lucas: Add resource quotas to your CI jobs. Give each test job a guaranteed minimum memory and CPU. Most CI providers support this. If your job exceeds the quota, it gets killed cleanly instead of silently degrading. Then you know the job needed more resources, not that the test is flaky. Luna: And that forces you to either optimize the test or increase the quota. Either way, you're fixing the real issue. Lucas: Exactly. And for teams using containers, you can also use cgroups to enforce limits. That's what the fintech company did. They set memory.max to 2GB per test container. Suddenly, the intermittent failures stopped. Luna: It's almost like treating your test infrastructure as a first-class system, not just a utility. Lucas: That's the takeaway. Tests are code, and the environment they run in is infrastructure. If you wouldn't run production without monitoring and resource guarantees, why run tests that way? Luna: Good question. So next time a test fails randomly, check the infrastructure first. Lucas: And if you can't find the issue, add instrumentation. The data will tell you the story. Luna: Thanks, Lucas. That was a great angle on flaky tests. Lucas: Always a pleasure. Until next time.