Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / Why Test Environments Drift and How to Stop It
Transcript
- Lucas: So there's this moment every QA engineer knows — you push a build to staging, you run the smoke tests, and something fails that passed locally. You dig in and it's not the code. It's the environment. The staging database is running a schema that's two migrations behind production. Luna: Oh, that's the classic. We've all been there. But it's one thing to shrug it off as a fluke and another thing to realize it's costing you actual release days. Lucas: Exactly. And that's the angle I want to dig into today. We're going to talk about test environment drift — what it is, why it keeps happening, and what a team can actually do about it without building some massive platform. I've got a concrete story from a mid-sized FinTech that lost an entire release day to this. Luna: Let's hear it. What happened? Lucas: So this company — let's call them FinFlow, a payments startup with maybe forty engineers — they were getting ready to ship a feature that touched their core ledger system. The QA team ran the regression suite on staging, and about fifteen percent of the tests just started failing. Not the same tests every time, but a consistent pattern of failures around anything that wrote to the transactions table. Luna: That smells like a schema mismatch right there. Lucas: It was. After a couple of hours of digging, they found that their staging database had been provisioned from a backup that was from the previous quarter. So the schema was missing a column that production had had for months — a column their new code was reading and writing. Every write to that table was throwing an error. Luna: And they only caught it in staging? That's terrifying because if they'd shipped, production would have been a mess. Lucas: Right. The scary part is they nearly did ship. They were on a Friday release cadence. They caught it on Thursday morning and spent the whole day fixing the environment, re-running the tests, and they still had to push the release to Monday. That's a full release day lost, plus the cost of the team context-switching to debug what was essentially an environment issue, not a code issue. Luna: So what was the root cause? Was it just a backup from who knows when? Lucas: It was a combination. The staging environment wasn't managed by anyone in particular. It was a shared resource that people spun up when they needed it, and there was no process for keeping it in sync with production. The database was restored from a backup that someone had taken months ago, and nobody had thought to apply the migration scripts since. Luna: So it's not just a one-off mistake. It's a systemic gap. There's no ownership, no automation, no checks. Lucas: Exactly. And that's why drift is so insidious. It doesn't announce itself. It just slowly accumulates until it bites you. The fix wasn't just to restore the database correctly — it was to build a habit of detecting drift before it caused a failure. Luna: So what did they actually do? I'm guessing they didn't go out and buy some expensive environment-management platform. Lucas: They didn't. They started small. First thing they did was add a drift check to their CI pipeline. After every deployment to staging, they ran a script that compared the schema of the staging database to the schema of the production database. If there were differences, the pipeline would fail and alert the team. Luna: That's a smart, low-tech solution. Just a schema diff. Lucas: Yeah, they used a simple open-source tool that dumps both schemas and does a diff. It's not perfect — it doesn't catch data drift or configuration drift — but it catches the most common and most dangerous kind. And it's something any team can set up in an afternoon. Luna: But schema drift is only one kind. What about configuration drift? Like, an environment variable that's different, or a feature flag that's set differently? Lucas: That's a whole other layer. And it's often even more subtle because it doesn't show up in a schema dump. A common approach is to have a config file checked into the repo, and then deploy that file to every environment. But you also need to verify that the running config actually matches what's in the repo. You can write a script that periodically fetches the runtime config from staging and production and compares them. Luna: So you're checking both the source of truth and the running state. That's a good practice. Lucas: And it doesn't have to be complex. The FinTech team started with a simple cron job that ran every night and posted a report to their Slack channel. If the schema was in sync, it posted a green checkmark. If not, it posted a red X with the specific differences. That visibility alone changed the culture. Suddenly, people cared about keeping staging clean. Luna: Visibility is huge. If no one notices the drift, it's like it doesn't exist. Lucas: Right. And once they had that baseline, they could start tackling the next layer. They made environment parity part of the definition of done for any feature that touched data. Before you could mark a ticket as done, you had to verify that the staging environment was in sync with production. Luna: That's a nice lightweight process. But I have to ask — what about the actual provisioning? If the environment is still being restored from old backups, you're just catching the problem later. Don't you want to prevent it in the first place? Lucas: That's the right question. And honestly, the ideal is to make environments ephemeral and fully automated. You define your database schema as code, you run migrations in CI, and you spin up a fresh staging environment from the latest code and the latest migrations every time you need one. Tools like Terraform and Docker make this possible. Luna: But that's a bigger investment. Is it worth it for a small team? Lucas: It depends. For a team that's shipping weekly and has a complex database, yeah, it's worth it. But you can get a lot of mileage out of the detection-first approach. The FinTech team didn't go all-in on ephemeral environments. They kept their shared staging, but they added the drift checks and they made the provisioning script idempotent — so re-running it always brought the environment to the correct state. Luna: So the detection caught the problem, and the idempotent script made it easy to fix. That's a pragmatic path. Lucas: Exactly. And I think that's the takeaway for most teams. Don't wait for the perfect platform. Start by measuring the delta between your environments. You can't fix what you can't see. Luna: And that delta can be more than just schema. What about the data itself? If staging has a tiny dataset, you might miss performance issues or edge cases that only show up with production-scale data. Lucas: That's a great point. Data drift is a whole other beast. The FinTech team dealt with that by having a separate performance test environment that they refreshed with anonymized production data quarterly. That way, they had a representative dataset without risking sensitive information. Luna: So they have three environments now: a clean staging for functional tests, a performance environment with realistic data, and production. That sounds like a good setup. Lucas: It is. And the key is that each environment has a clear purpose and a clear owner. The staging environment's owner is the QA lead. The performance environment's owner is the backend team. And the drift checks are the connective tissue that keeps them honest. Luna: I love the idea of ownership. But what about the human side? How do you get developers to actually care about environment parity? Lucas: That's often the hardest part. The FinTech team found that the best way was to make it visible and personal. When the drift check failed, they didn't just send a generic alert. They posted the names of the migrations that were missing and who wrote them. That created a bit of healthy pressure to keep things clean. Luna: So you're naming and shaming, in a friendly way. Lucas: A little bit. But it works. Because nobody wants to be the reason the pipeline is red. And once they started treating environment drift as a bug, not a nuisance, the culture shifted. Luna: And that's a good segue — thinking about it as a bug. Because if you treat it like a bug, you'll write tests for it. You'll automate the detection. You'll make it part of CI. Lucas: Right. And that's the mindset shift that turns environment management from a chore into a engineering discipline. And it's a discipline that can save you entire release days. Luna: It's funny, we talk a lot about testing the code, but we rarely talk about testing the environment itself. Lucas: And that's exactly what we're doing here. And you know, it's a bit like keeping your kitchen clean before you cook. If the counters are dirty, the food might still turn out fine, but you're taking a big risk. And we've seen teams take that risk for years without realizing it. Luna: Speaking of taking risks, we should mention — this show exists because a handful of listeners chip in monthly through buy me a coffee dot com slash fexingo. That's literally what funds making this many episodes. No ads, no corporate sponsors, just listener support. Lucas: Yeah, and we're really glad it works that way. It keeps us independent, and it lets us talk about whatever we find useful. If you've gotten something out of today's episode, and you want to keep this going, that's the way to do it. Luna: And we're back to the topic. So, to wrap up, what's the one thing you want listeners to take away from this? Lucas: I'd say: don't assume your staging environment matches production. Verify it. Set up a simple drift check in CI, make it visible, and make it someone's job to keep it clean. It's a small investment that can save you from a Friday afternoon of chaos. Luna: And if you're already doing that, maybe look at your data and your config, too. The same principle applies. Lucas: Absolutely. Because at the end of the day, testing is only as good as the environment you run it in. If the environment lies to you, your tests lie to you. And that's a risk no team should take. Luna: Well said, Lucas. And that's a wrap for today. Thanks for listening, everyone.