Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / Why Your Tests Fail in Staging but Pass in Production
Transcript
- Lucas: So we've all been there — tests pass beautifully in staging, you deploy to production, and suddenly customers are staring at error 500s. Or the reverse: a test fails in staging for what looks like a real bug, you block the release, spend hours investigating, and it turns out the staging environment was slightly different from the moment you started the test run. Luna: Right, the classic 'works on my machine' problem, but at scale. I've definitely wasted afternoons on false positives from staging that turned out to be environment drift. Lucas: Exactly. And it's not just a nuisance — it costs real money. Let me give you a concrete case. Back in November 2022, a mid-sized e-commerce platform — I won't name them since they're private — had a Black Friday sale that generated about $4 million in revenue. Their QA team ran a full suite of performance and regression tests on staging, all green. They deployed to production at midnight, and within two hours, the site was timing out on checkout. Luna: Oof. What went wrong? Lucas: The staging database had a different indexing scheme. In staging, they had a composite index on that made the checkout queries fast. In production, that index was missing — someone had manually dropped it during a schema migration a week earlier and forgot to update the staging copy. The production database spent minutes scanning rows instead of milliseconds. The site was effectively down for 90 minutes on Black Friday. They estimated lost revenue at $340,000. Luna: And the tests passed because they ran against a database that didn't match production. That's the textbook definition of environment drift. Lucas: Right. Environment drift is any difference between your staging, QA, or development environments and your production environment that can affect test outcomes. It could be database schemas, OS patches, middleware versions, network latency, DNS configurations, even the amount of disk space. The list is terrifyingly long. Luna: So what's the solution? Do you just try to keep everything identical? That seems like a losing battle if you have even a handful of services. Lucas: Well, the ideal is a concept called 'golden environment baseline.' You define every configuration parameter — every package version, every kernel setting, every database index — in a version-controlled file. Then you use infrastructure as code tools to spin up environments from that baseline. Terraform, Pulumi, AWS CDK — pick your poison. The key is that staging and production are created by the same script, from the same source of truth. Luna: But even then, drift can happen. Someone applies a hotfix directly to production, or a configuration management agent runs a different playbook. I've seen cases where the Terraform state file itself gets out of sync. Lucas: Absolutely. So you need a second layer: automated drift detection. Run a periodic comparison between the actual state of each environment and the golden baseline. There are tools for this, like CloudHealth or even custom scripts using the cloud provider's API. The point is to catch drift before a deployment. Luna: And then what? You block the deployment until the drift is fixed? Lucas: Exactly. Make environment parity a pre-deployment gate. Before any code goes to production, your CI/CD pipeline should check that staging is in sync with the baseline. If not, the pipeline fails immediately, and the developer gets a clear message: 'Staging database index mismatch. Run terraform apply on staging first.' It's not hard to implement if you're already using infrastructure as code. Luna: But that assumes you have the bandwidth to maintain that kind of infrastructure. For smaller teams — like a startup with three engineers — setting up Terraform and drift detection might feel like overkill. What would you recommend for them? Lucas: I'd say start small. Use Docker containers for local development, and use the same Docker Compose file for staging. That way, at least the application runtime environment is identical. Then gradually migrate to a more robust solution as you grow. The cost of one production incident — like that $340,000 Black Friday — quickly justifies the investment. Luna: That's a good point. And speaking of justifying investments, if you find this kind of practical, no-fluff content useful, consider supporting the show. It keeps us ad-free and independent. You can do that at buy me a coffee dot com slash fexingo. Just a simple way to say thanks and help us keep digging into topics like this. Lucas: Yeah, we really appreciate that. And it helps us spend more time on research and less time worrying about the bills. So thank you if you do. Luna: Absolutely. Alright, back to environment drift. Lucas, you mentioned that even with infrastructure as code, drift can still happen. What about the human factor? How do you prevent someone from manually ssh ing into a production server and changing something? Lucas: That's where immutable infrastructure comes in. The idea is that you never modify a running server. Instead, you bake a new machine image — an AMI on AWS, a container image — that includes all the configurations, and you deploy that image to replace the old one. If someone tries to SSH into a production instance, the instance should be designed so that any manual change is ephemeral: it gets wiped on the next deploy. Tools like Packer help create those images, and orchestration tools like Kubernetes or Nomad handle the rollout. Luna: So no more 'just this one tiny hotfix' on production? That sounds like a cultural shift as much as a technical one. Lucas: Absolutely. And it's one of the hardest parts. I've seen teams that have perfect infrastructure as code, but then a senior engineer has root access and makes a change 'just to get through the night.' The next morning, the automated tests pass because the test environments were also drifted manually, and suddenly you're back to square one. So you need change management processes — and that's more of a people problem than a tech problem. Luna: Speaking of processes, what about the opposite scenario: tests fail in staging but would pass in production? That's also environment drift, right? Lucas: Right. And that can be even more insidious because it erodes trust in the test suite. If developers see tests failing due to environment issues, they start ignoring failures, and eventually real bugs slip through. I've seen teams where the test suite has a 20% flake rate just from environment inconsistencies. Luna: So how do you catch staging-specific drift? Like, maybe staging has less memory, or a slower disk I/O. Lucas: Good question. One approach is to run environment-specific smoke tests that verify the infrastructure itself. For example, run a simple script that checks database connection strings, timeout settings, and resource limits. If staging has a 5-second timeout but production has 30 seconds, that's drift worth knowing about. Luna: And you'd run those as part of the CI pipeline, before the actual functional tests? Lucas: Exactly. A pre-flight check. It takes maybe 30 seconds to run and can save hours of debugging. I've seen teams implement this with a simple Python script that validates environment variables, database indexes, and even file permissions. It's low-tech but highly effective. Luna: Alright, let's talk about the elephant in the room: container orchestration. With Kubernetes, you can have a staging cluster and a production cluster, and if they're not running the same version of Kubernetes, or the same set of admission controllers, that's drift. How do you handle that? Lucas: That's a great point. Kubernetes is a powerful tool, but its complexity makes drift more likely. I recommend using the same Kubernetes version across environments, and using GitOps tools like ArgoCD or Flux to ensure that the desired state in Git matches the actual state in the cluster. If someone manually edits a ConfigMap in production, ArgoCD will revert it automatically. Luna: So the golden baseline extends to the cluster level. What about network-level drift? Like load balancer rules, or firewall settings? Lucas: Yes, network configuration is often overlooked. I've seen a case where staging had a load balancer that allowed HTTP/2, but production's load balancer was configured for HTTP/1.1 only. A test that relied on HTTP/2 multiplexing would pass in staging but fail in production. The fix is to manage load balancer configs as code — using Terraform for ALBs or Nginx configs, and including them in the golden baseline. Luna: So essentially, the golden baseline should cover everything from the kernel up to the application layer. That's a lot. Lucas: It is. But you don't need to do it all at once. Start with the most critical components: database schemas and application configurations. Then gradually add more layers. Each time you fix a drift-related incident, add that check to your baseline. Over time, you build a comprehensive system. Luna: And what about the human side? How do you get buy-in from developers who might see this as bureaucracy? Lucas: Show them the data. When a developer spends three hours debugging a test failure only to find it was a staging DB index issue, they'll quickly see the value. Also, make the process easy — automate the checks so they don't have to think about it. The golden rule: make the right thing the easy thing. Luna: That makes sense. Let's wrap up with a takeaway. For a team that's just starting to tackle environment drift, what's the one thing they should do this week? Lucas: This week, audit your staging and production databases for schema differences. Run a 'mysqldiff' or use whatever your database's comparison tool is. If you find any differences, that's your first drift incident. Fix it, and then write a script that checks for that difference every time you deploy. That's a concrete, immediate step that will save you headaches. Luna: And if they find no differences, they're probably in better shape than they think. Or they just haven't looked hard enough. Lucas: Exactly. Thanks for listening, everyone. We'll be back next week with another deep dive into the messy reality of software testing.