Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How Test Environment as Code Eliminates Configuration Drift
Transcript
- Lucas: There's a sentence that has probably cost the software industry more money than any single bug, and it's five words long: 'It worked on my machine.' Luna: I feel like I've said that myself maybe forty times this year alone. Lucas: Right. And the thing is, it's almost never a lie. It did work on your machine. The problem is that your machine — your local environment, your Docker compose file, your manually tweaked configuration — is not the same as the CI runner, which is not the same as staging, which is not the same as production. That gap is called configuration drift, and it's the silent killer of test reliability. Luna: So how do you actually stop it? I mean, we've talked about test environment drift before in episode 92, but that was more about diagnosing the symptoms. I want to know how you prevent it from happening in the first place. Lucas: That's exactly where I want to go today. The emerging answer is something called test environment as code. The idea is straightforward: you define every aspect of your test environment — operating system version, installed packages, environment variables, network rules, database schemas, the whole stack — in version-controlled configuration files. Then you deploy those environments from scratch, every time, using automation. Luna: So instead of ssh ing into a server and tweaking nginx.conf by hand, you write a Terraform file or an Ansible playbook, check it into Git, and let the tooling build the environment for you. Lucas: Exactly. And the key word there is 'from scratch.' A lot of teams think they're doing infrastructure as code because they have an Ansible script that updates a few config files on an existing server. But if that server has been running for six months with manual hotfixes, the script is applying changes on top of drift. Real environment as code means you destroy and rebuild the entire environment on every run. Immutable infrastructure. Luna: That sounds expensive. If you're rebuilding a full integration test environment every time you run a pipeline, aren't you burning a lot of cloud compute and time? Lucas: It can be, if you do it naively. But the cost of false failures is often much higher. Let me give you a concrete example. I talked to a team at a mid-sized e-commerce company — about forty engineers, maybe a dozen microservices. They were seeing about twenty to thirty percent of their CI test runs fail for environment-related reasons. Not code bugs. Things like a missing Redis config, a different Node version, a stale database migration. Their developers were spending, on average, forty-five minutes per failed run just debugging environment issues. Luna: That's brutal. Forty-five minutes of context switching per failure, and you multiply that by the whole team. Lucas: Right. They estimated it cost them about two hundred and fifty engineering hours a month. So they invested in environment as code. They defined their staging environment in Terraform, their integration test environment in a separate Terraform workspace, and used Ansible for the software provisioning layer. They made it so that every CI run spun up a fresh environment from the Terraform definitions, ran the tests, and then tore it down. Within six months, their environment-related false positives dropped by seventy-three percent. Luna: Seventy-three percent is huge. But I'm guessing that initial setup wasn't trivial. How long did it take them to get that working? Lucas: About three months to get the first environment fully defined and automated. But they started small — just the staging environment for one service. They proved it worked, then expanded to all services and added the integration test environment. The key is to start with a single, well-understood environment and version-control its definition. Once you have that, you can replicate it. Luna: So what's the actual tooling here? You mentioned Terraform and Ansible. Are there other options? Lucas: Terraform is great for the infrastructure layer — VPCs, subnets, EC2 instances, RDS databases. Ansible or Chef or Puppet handle the software configuration on top. But there's also Pulumi, which lets you write infrastructure definitions in general-purpose languages like Python or TypeScript. And for containerized environments, Docker Compose files and Kubernetes manifests are essentially environment as code for the container orchestration layer. Luna: I've seen teams use Docker Compose for local development environments, but then the CI environment is a completely different setup. That itself becomes a drift source. Lucas: That's a classic problem. The solution is to use the same Docker Compose file — or the same Kubernetes YAML — for local, CI, and staging. Maybe with slight variations for resource limits, but the core definition is identical. That way, if it works locally in your container, it'll work in CI, because the environment is byte for byte the same. Luna: What about detecting drift when it does happen? Because even with environment as code, you can still get drift if someone manually changes a resource, or if a cloud provider changes something on their end. Lucas: Great point. That's where drift detection tools come in. AWS Config, for example, can continuously monitor your resources against your desired configuration and alert you when something changes. There's also open-source tools like Driftctl — now part of the Snyk ecosystem — that compare your Terraform state to the actual cloud resources and report discrepancies. And you can run those checks as part of your CI pipeline, so any drift is caught before it causes test failures. Luna: So you're saying the CI pipeline itself should check that the environment hasn't drifted before it runs any tests. Lucas: Exactly. You add a drift scan step at the beginning of your pipeline. If drift is detected, the pipeline can either alert the team or automatically rebuild the environment from the defined configuration. It's a shift from reactive debugging to proactive prevention. Luna: Let me push back a little. For a small team — say, three developers working on a single service — does this level of automation really pay off? Isn't it overkill to have Terraform and drift detection for a simple Node.js app? Lucas: It depends on how often you deploy. If you're deploying once a month, manual environment setup might be fine. But if you're deploying daily or multiple times a day, the cost of manual configuration scales linearly with your deploy frequency. I'd say the tipping point is when you have more than five services or more than three environments — dev, test, staging. At that point, the cognitive load of keeping everything consistent by hand exceeds the cost of automation. Luna: So start with a single environment definition file. What would that look like for a simple web app? Lucas: A single Docker Compose file with your app container, a database container, and maybe a Redis container. That file is version-controlled. Your CI pipeline runs `docker-compose up` to spin up the environment, runs the tests, then `docker-compose down` to tear it down. That's the simplest form of environment as code. From there, you can graduate to Terraform for cloud infrastructure, or Kubernetes for orchestration at scale. Luna: I like that — start with Docker Compose, prove the concept, then expand. It's not an all-or-nothing investment. Lucas: Exactly. And the mindset shift is just as important as the tooling. The old way was: 'I have a server, I SSH into it, I tweak things until the tests pass.' The new way is: 'I have a definition file. If the tests fail, I fix the definition file, commit it, and rebuild.' The environment becomes a versioned artifact, not a snowflake. Luna: That's the 'treat your infrastructure like code' mantra. But I've seen teams struggle with that cultural change. Developers who are used to having root access on a box can feel constrained by immutable infrastructure. Lucas: It's a real challenge. The key is to give them a fast feedback loop. If you make rebuilding the environment take five minutes instead of five hours, they'll adopt it. Also, provide a local dev environment that's identical to CI — using the same Docker Compose file — so they can iterate locally and trust that it'll work in CI. Once they experience the reliability improvement, most developers don't want to go back. Luna: I think that's a good moment to mention something that's been on my mind. You know, conversations like this — where we dig into practical, real-world testing practices — are exactly why listeners tell us they appreciate the show. And we keep the podcast ad-free because of that listener support. Lucas: Yeah, and we want to keep it that way. So if you find these episodes useful — and we hope you do — one of the easiest ways to help is by supporting us at Buy Me a Coffee. That's buy me a coffee dot com slash fexingo. Luna: And it genuinely makes a difference. It helps us cover the hosting costs and free up time to research topics like environment as code. No pressure, just if you feel like it. Lucas: Alright, back to the main thread. So let's talk about a concrete way to measure whether your environment as code investment is paying off. One metric I like is the 'time to first green build' for a new service. Before environment as code, that might take days — setting up databases, configuring load balancers, debugging mismatches. After, it should be minutes. Luna: That's a great leading indicator. If a new developer on the team can spin up a full test environment and get a passing build in under an hour, you're in good shape. Lucas: And a lagging indicator is the percentage of test failures that are traced back to environment issues. If that number is above five percent, you've got a drift problem. The team I mentioned earlier got theirs down to under two percent after adopting environment as code. Luna: So the playbook is: pick your most painful environment, define it in code, automate the build and teardown, add drift detection, and then expand. And start with Docker Compose if you're small. Lucas: That's the whole strategy. The alternative is living with 'it worked on my machine' forever. And honestly, that sentence should be the industry's warning label, not its punchline. Luna: At this point, I wonder if we'll ever truly eliminate it, or just reduce its frequency. Lucas: Probably reduce, not eliminate. There's always some edge case — a network latency issue, a race condition that only happens in production. But if we can eliminate the configuration drift piece, we remove the majority of false failures. And that alone is worth the investment.