Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How to Test Infrastructure as Code Before It Breaks Production
Transcript
- Lucas: So back in June, a major cloud provider went down for nearly four hours. The root cause? A misconfigured Terraform template. It passed every unit test they had, but when it hit production, it overwrote a critical load balancer configuration and took out half their European region. Luna: Wait — every unit test passed? How does a Terraform template pass unit tests and still break production that badly? Lucas: Because unit tests for infrastructure check the syntax and maybe a few resource attributes. They don't simulate real dependencies, they don't check for state corruption, and they almost never validate against the actual cloud provider's API quirks. That's the gap we need to talk about. Luna: So Infrastructure as Code — Terraform, CloudFormation, Pulumi — it's code, but we're not testing it like code. We're treating it like a script you run and hope for the best. Lucas: Exactly. And that's dangerous because infrastructure code has properties that application code doesn't. It's stateful — Terraform maintains a state file that can drift from reality. It's declarative — you say what you want, but the provider decides how to get there. And it's highly permission-sensitive. Luna: So where do we start? What's the first testing layer most teams are missing? Lucas: Static analysis. Tools like Checkov or tfsec scan your Terraform HCL for known misconfigurations — open security groups, unencrypted S3 buckets, weak IAM policies. That's the quick win, and it runs in seconds in CI. Luna: Right, but static analysis can't catch logic errors — like if your module creates a load balancer with the wrong health check path. That's where people get burned. Lucas: That's where policy as code comes in. With Open Policy Agent, you can write rules like 'every load balancer must have a health check path matching a regex' and enforce it before apply. It's a step beyond static scanning because it understands relationships between resources. Luna: So OPA is like a linter for architecture decisions. I've seen teams use that with Conftest to validate Terraform plans. Lucas: Exactly. But even that won't catch state corruption or dependency order issues. For that, you need integration tests that actually provision resources in a sandbox environment and verify they work together. Luna: And that's where Terratest comes in. You write Go tests that call Terraform init and plan, then apply, then assert on the real infrastructure. Lucas: Terratest is great. You can test that after provisioning, the load balancer returns a 200, that the database accepts connections, that the autoscaling group actually scales. It runs against a real cloud account — usually a separate testing AWS account or a local tool like LocalStack. Luna: But integration tests are slow and expensive. You can't run a full suite for every commit. Lucas: No, you tier them. Unit and static analysis on every commit. Policy checks on every pull request. Integration tests only on merged changes or scheduled nightly runs. And you make sure your integration tests are idempotent — they tear down everything after the test. Luna: That reminds me of a pattern I saw at a fintech company. They ran a 'plan-only' pipeline that compared the Terraform plan against a set of approved baselines. Any unexpected change blocked the pipeline. Lucas: That's a fantastic practice. It's essentially a diff review, but automated. The Terraform plan output is JSON, so you can parse it and flag things like 'resource X is being replaced when only a tag changed' — which is a common cause of downtime. Luna: And then there's the human review. Even with all these automated gates, someone should look at the plan. But the key is that the plan must be generated from the actual state, not mocked. Lucas: Right. One thing that often gets overlooked is testing state migration. Terraform state files can get corrupted or drift. There are tools like terraform state inspect, but teams rarely test the state handling itself. Luna: What about testing that infrastructure behaves correctly under failure? Like, if a load balancer fails, does the autoscaling group behave as expected? Lucas: That's moving into chaos engineering territory. But there's a simpler layer: test that your Terraform modules are idempotent. Apply twice — the second apply should produce no changes. That catches a surprising number of bugs where modules create resources with non-deterministic names. Luna: I've seen that. Tags that include timestamps, or random suffixes that change every apply. It's a nightmare for drift detection. Lucas: So to sum up: the stack for testing Infrastructure as Code includes static analysis, policy as code, plan diff automation, integration tests with real provisioning, and state consistency checks. And all of it should be automated in CI. Luna: That's a lot of layers, but it's not that different from testing application code. It's just that the tools and mindset are newer. Lucas: Exactly. And adopting even two of those layers — say, static analysis and plan-diff automation — would have prevented that June outage. The team had no static checks and no plan review. They just applied. Luna: And if today's tech conversation gave you something usable — a tool to try, a mindset shift, a way to prevent an outage — you should know that these episodes are ad-free and listener-supported. A couple of dollars a month is genuinely what keeps these going — buy me a coffee dot com slash fexingo, if you've gotten something out of them. Lucas: Yeah, it really makes a difference. And we don't run sponsors, so your support directly keeps the show independent. Luna: So going back to the testing stack — the next frontier I see is automated hypothesis testing for infrastructure. Tools that generate random AWS configurations and check if the Terraform code handles them gracefully. Lucas: That's essentially property-based testing for IaC. It's early, but I think in a couple of years, every CI pipeline will include a step that says 'run 100 random plan variations and report any errors.' We're not there yet, but the foundation is solid. Luna: And until then, static analysis, policy checks, and plan diffs are the low-hanging fruit. No excuse not to start today. Lucas: No excuse. Next episode, we're going to look at something completely different — testing for observability. How do you know your monitoring is actually catching the right signals? Luna: I'm already taking notes. See you next time.