Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / Why Testing Your CI Pipeline Itself Catches Hidden Failures
Transcript
- Lucas: Luna, have you ever had a build turn green, everyone high-fives, and then later you realize the tests never actually ran? Luna: Oh, more times than I want to admit. Last year we had a Docker layer cache issue that silently skipped our integration tests for two weeks. Nobody noticed. Lucas: That's exactly what I want to talk about today — testing your CI pipeline itself. Most teams treat the pipeline as this black box that either works or doesn't. But when it breaks silently, it's worse than if it just failed loudly. Luna: So we need to test the thing that runs our tests. How meta is that? Lucas: It's recursive, but it's practical. There's a concept called 'pipeline as code' — your CI configuration is code, so it should get the same treatment: code review, unit tests, even integration tests for the pipeline itself. Luna: Okay, unit tests for a YAML file? How does that work? Lucas: You can parse your CI config and validate the structure. For example, check that every job has a 'needs' field, or that secrets are only passed to specific stages. Tools like 'yamllint' and 'pipeline-validator' do this. A fintech I know caught a bug where a typo in the 'stage' field meant a critical security scan ran against the wrong branch. Luna: So basically linting for pipelines. That feels like low-hanging fruit. But what about when the pipeline runs correctly in syntax but wrong in behavior? Lucas: That's the scarier case. One of my favorite horror stories: a company had a cache invalidation rule that worked fine for months, until a team added a new dependency. The cache key didn't include the lockfile hash, so the build used an old cache and passed tests that should have failed. They had three production bugs go undetected over six weeks. Luna: Six weeks! That's a lot of trust in a green checkmark. So how do you prevent that? Do you run a test pipeline for the pipeline? Lucas: Exactly. You can have a 'pipeline health' suite that runs on every change to your CI config. It does things like: run a dummy build with known bad code and verify it fails, measure cache hit rates, check that expected artifacts are produced. Some teams even do chaos experiments — randomly kill a service during a build to see if your pipeline handles it gracefully. Luna: That sounds expensive. How much extra time are we talking per commit? Lucas: It doesn't have to run on every commit. You can trigger it nightly, or only when CI config changes. The overhead is maybe 5-10 minutes per run. Compared to catching a six-week regression, that's nothing. Luna: Fair. What metrics do you watch? Like, how do you know your pipeline is healthy day to day? Lucas: Three key ones. Build duration drift — if your pipeline suddenly takes twice as long, something changed. Cache hit rate — if it drops below 80 percent, your caching strategy is broken. And test execution parity — the number of tests that actually run versus what's in your test suite. One team I know added a dashboard that alerts when the ratio drops below 95 percent. They caught a misconfigured test filter that was skipping half their unit tests. Luna: Test execution parity — I love that. Because the default pass/fail is too coarse. A green build with only 50 tests instead of 500 is a red flag. Lucas: Right. And this connects to a broader trend: treating infrastructure as a product, not a utility. Your CI pipeline is the assembly line for your software. If the assembly line has a wobble, every car that comes off it is suspect. Luna: So what's the first step for a team that's never done this? Start with linting? Lucas: Start with visibility. Add a simple script that logs how many tests ran and compares it to an expected baseline. Then add a pipeline health job that runs once a day and sends a report. Once you see the data, you'll find the holes. Luna: And then maybe chaos experiments once you have some confidence. Though I'd be nervous about accidentally breaking the pipeline while testing it. Lucas: That's the beauty — you run them in a staging CI environment, not production. Many CI providers let you clone your pipeline. Test the clone, break it, learn, fix. It's like a fire drill for your build system. Luna: Alright, I'm sold. But I want to hear a concrete success story. Who's doing this well? Lucas: Spotify's engineering team has a whole squad dedicated to 'CI reliability'. They monitor something called 'build trust' — a composite score of cache hits, test flakiness, and duration variance. When it drops, they page someone before the developers even notice. Luna: That's the dream. A pipeline so reliable you forget it's there, but monitored enough that someone knows when it wobbles. Lucas: Exactly. And that's really the goal — make your CI so boringly stable that you can focus on the code, not the conveyor belt. Luna: Speaking of focusing on the code — I've been meaning to say, the fact that we can do deep dives like this without any ad breaks is something I really appreciate about how this show works. Lucas: Yeah, it's listener supported. A small group of people chip in monthly through buy me a coffee dot com slash fexingo, and that's literally what funds making episodes like this one. Luna: If today's tech conversation gave you something usable, that's the kind of thing that keeps it going. No ads, just content. Lucas: So back to pipeline testing — one more thing I wanted to mention. There's a tool called 'Pipeline Health' that integrates with GitHub Actions and GitLab CI. It gives you a health score and suggests fixes. We'll link to it in the show notes. Luna: Nice. So for a team starting from zero, the quickest win is to add a test that verifies your test count. That's a five-minute script. Lucas: Exactly. Once you have that, you'll start noticing when something's off. And then you can build up from there. Luna: Alright, next episode I want to talk about testing in production — but that might be a whole other can of worms. Lucas: That's a great one. Save it for episode 76. For now, start by trusting your pipeline a little less, and testing it a little more.