Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / Why Test Environments Need Ephemeral Infrastructure
Transcript
- Lucas: So there's this SaaS company — mid-size, maybe 300 engineers — and they were running a shared staging environment. One team deploys their branch, accidentally breaks a dependency, and suddenly every other team's integration tests fail. Classic Monday morning chaos. Luna: Right, the 'works on my machine' problem gets amplified when it's 'works in my test environment' and everyone shares the same one. Lucas: Exactly. And they decided to go all-in on ephemeral environments — spinning up a full, isolated stack per pull request. Not just a container for the microservice, but the databases, the message queues, the whole thing. And they told me it cut their test environment costs by 60 percent. Luna: Wait — costs went down? I would have guessed spinning up entire stacks on demand would be more expensive. Lucas: That's what I thought too. But here's the thing: shared staging environments are always running, always consuming resources, even at 2 a.m. when nobody's testing. Ephemeral environments only exist for the lifecycle of a PR — usually a few hours. So you're not paying for idle compute. Luna: Makes sense. And I imagine the isolation also eliminates that whole class of flaky tests caused by environment pollution. Lucas: Yeah, if today's tech conversation gave you something usable — a new angle on environments or even just a reminder to look at your CI costs — a handful of listeners chip in monthly through buy me a coffee dot com slash fexingo. And that's literally what funds making this many of these. It keeps the whole thing ad-free and independent. Luna: It's true. And we don't run sponsorships or tracking, so the listener community is what keeps the lights on. Lucas: Alright, back to ephemeral environments. The company I mentioned — they were using Kubernetes with Helm to orchestrate the stack per branch. Each PR triggered a pipeline that built the images, applied Terraform to provision a temporary namespace, seeded the databases with a snapshot of anonymized production data, and ran the full regression suite. Luna: That sounds like a lot of moving parts. How long did it take to spin up? Lucas: They got it down to about four minutes for a typical microservice. But the big win was developer velocity — they found bugs 40 percent faster because the environment was consistent. No more 'but it passed on staging' debates. Luna: Forty percent is massive. But I'm wondering about stateful services — databases, caches. If you're spinning up a fresh database each time, how do you keep the seed data manageable? Lucas: Great question. They used a combination of techniques. For the core reference data — like product catalog — they baked it into the container image. For transactional data, they took a daily snapshot of production, anonymized it, and stored it in object storage. The ephemeral environment would pull that snapshot on startup. Luna: So the data is a point-in-time copy. That means tests might fail because of data drift between the snapshot and the code changes. Lucas: Right, and that's a real trade-off. They accepted a six-hour lag on the snapshot. For most integration tests, that's fine. But for tests that need real-time data, they had a separate mechanism: they'd pipe a live stream of a subset of production events into a temporary Kafka topic. Luna: That's clever. So the ephemeral environment can either use a static snapshot or a live stream depending on the test scenario. Lucas: Exactly. And they automated that decision based on test tags. If a test is tagged 'recent-data', the pipeline provisions the streaming connection. Otherwise, it uses the snapshot. Luna: Alright, but what about third-party dependencies? If you're spinning up an ephemeral environment, you can't spin up a replica of Stripe or AWS S3. Lucas: They used a combination of sandbox accounts and service virtualization. For Stripe, they had a dedicated test account. For S3, they used a local MinIO instance. The key was that every ephemeral environment got its own isolated set of credentials and endpoints. Luna: So no sharing at all. That must have been a cultural shift for the developers — going from 'just deploy to staging' to 'I need to configure my own environment.' Lucas: It was. The first month, a lot of engineers complained about the four-minute wait. But then they realized they were spending way more time debugging environment-specific failures in the old model. Onboarding new hires also got faster — they could experiment without fear of breaking anything. Luna: I've also seen teams use ephemeral environments for manual QA — like a UAT sign-off before merging. That seems like a natural extension. Lucas: Absolutely. The same company did that. Product managers could click a link in the PR and see the feature running with realistic data. They'd catch UX issues before the code even hit main. Luna: So let's talk about the infrastructure cost more concretely. You said 60 percent savings — where does that come from? Lucas: Mainly from not running a full staging cluster 24/7. Their shared staging environment had four beefy nodes always on. With ephemeral environments, they scaled to zero when no PRs were active. They also used spot instances for the worker nodes, which cut compute costs by another 30 percent. Luna: But spot instances can be interrupted. If a test environment is yanked mid-run, doesn't that cause flakiness? Lucas: It can. Their solution was to mark those tests as 'non-deterministic' and rerun them on a small pool of on-demand nodes. But they measured that only about 2 percent of test runs were affected, so it was a worthwhile trade-off. Luna: Two percent is pretty low. Still, I can see some teams being nervous about that. What about teams that aren't on Kubernetes? Can you do ephemeral environments with serverless? Lucas: Yeah, there are serverless approaches. For example, you can use AWS Fargate or Google Cloud Run to spin up containers per request. But the challenge is stateful services — serverless databases like Aurora Serverless can scale to zero, but they have cold starts that can add seconds to the provisioning time. Luna: Right, so for teams that need sub-minute environment spin-up, Kubernetes with pre-warmed node pools might be the way to go. Lucas: Exactly. One pattern I've seen is having a pool of 'hot' namespaces that are pre-configured but idle. When a PR triggers a build, the pipeline just swaps in the code and seeds the data. That cuts spin-up to under a minute. Luna: That's a nice middle ground. Alright, let's talk about the elephant in the room: database schema migrations. If each ephemeral environment runs migrations on a fresh database, and you have dozens of PRs, you're running the same migrations over and over. Doesn't that slow things down? Lucas: It does, but only on the first run. They cached the database snapshot after running migrations on a base image. So new environments would start from a snapshot that already had the latest schema applied. Only the current PR's new migration would run. Luna: So the cache is invalidated when a migration is merged to main. That's efficient. But what about rollbacks? If a PR introduces a bad migration, the ephemeral environment just gets torn down — no harm done. Lucas: Exactly. That's one of the biggest safety benefits. You can test destructive schema changes without risk of corrupting a shared database. Luna: I want to circle back to the cost savings. You mentioned 60 percent — but that's not just compute, right? There's also the developer time saved from debugging environment issues. Lucas: Right, they estimated that the old shared staging environment caused about 15 percent of test failures to be false positives. That wasted maybe 20 developer hours per week. After ephemeral environments, that dropped to near zero. Luna: So the real ROI is the combination of infrastructure savings and developer productivity. That's a compelling case. Lucas: It is. But I should mention a pitfall: over-provisioning. Some teams spin up an entire replica of production — dozens of microservices — for every PR. That defeats the purpose. The key is to only include the services that are directly impacted by the change. Luna: So you need good dependency mapping. Maybe using a service mesh or tracing data to figure out what's actually needed. Lucas: Yeah. The company I mentioned used OpenTelemetry traces to build a dependency graph. Their pipeline would analyze the diff and automatically decide which services to include. If the diff only touched the billing service, it wouldn't spin up the recommendation engine. Luna: That's smart. So the environment is not just ephemeral but also minimal. I think that's the direction a lot of teams should be heading. Lucas: Agreed. The tooling is maturing fast — things like Telepresence, Tilt, and the new generation of CI/CD platforms are making this much more accessible. I wouldn't be surprised if, in a couple of years, shared staging environments start feeling like a relic. Luna: I think you're right. It's one of those shifts where the cost-benefit analysis is becoming undeniable, especially for teams already on Kubernetes. Lucas: And for teams that aren't, there are lighter-weight options. You can do it with Docker Compose and a CI runner that provisions a cloud VM. The principle is the same: isolate, automate, destroy. Luna: It really comes down to treating test environments as cattle, not pets. That old metaphor is actually perfect here. Lucas: Exactly. And once you internalize that, a lot of the classic testing pain points — flakiness, environment drift, contention — just evaporate. It's a shift that pays for itself.