Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How to Test Configuration Changes Without Breaking Production
Transcript
- Lucas: So Luna, you ever changed a feature flag in production and held your breath for a second? Luna: Every single time. It's like the system is one toggle away from chaos. Lucas: Exactly. And that's because configuration changes are some of the least-tested code paths in any system. We write unit tests for functions, integration tests for services, but a simple config change? Often it goes straight to prod with no safety net. Luna: And yet, some of the biggest outages in recent history have been caused by config changes, not code bugs. Lucas: Right. Take the 2023 AWS S3 outage. That was a configuration change that took down a huge chunk of the internet. And who can forget Knight Capital? A configuration error in their trading software led to a $440 million loss in 45 minutes. Luna: Ouch. So why is configuration so easy to get wrong? Lucas: Because it's invisible. You don't see it in code review the same way you see a logic error. It's often just a value in a YAML file or a database row. And it's easy to assume that if it was working before, it'll work now. Luna: So what's the fix? How do we test configuration before it bites us? Lucas: First, treat configuration as code. That means storing it in version control, subjecting it to code review, and applying the same testing principles we use for code. Luna: But config isn't code. It's data. How do you unit test data? Lucas: You test it the same way you test any data: by validating its structure and its values. Start with schema validation. If your config is JSON or YAML, you can define a schema and automatically validate that every change conforms to it. Luna: So you catch typos and missing required fields before they hit production. Lucas: Exactly. But you also need to validate the values themselves. For example, if you have a timeout setting, you might want to assert that it's within a reasonable range — not negative, not zero, not a string. Luna: And what about relationships between config values? Like, if you change a database connection string, you should also update the pool size. Lucas: That's a great point. That's where you can write what I call 'invariant checks.' These are tests that verify cross-cutting constraints across your entire configuration. For instance, every service endpoint that's referenced in your config should actually exist in your service registry. Luna: So you're building a safety net that's specific to your system's config. Lucas: Right. And once you have those checks, you run them in CI on every config change. That's the 'config as code' pipeline. But even that isn't enough, because some config only breaks in production. Luna: What do you mean? If your schema is good and your values are valid, what else can go wrong? Lucas: The most dangerous type of config bug is the one that's syntactically and semantically valid, but just wrong for the environment. For example, you might point a service to a staging database instead of production. No schema error, no value out of range, but your entire system is suddenly reading from the wrong data source. Luna: Oh, that's terrifying. How do you catch that? Lucas: That's where you need environment-specific testing. You need to test the config in a production-like environment, not just in dev. That's why staging environments exist — but they only work if they're actually representative of production. Luna: And we all know staging environments drift. Lucas: They do. So another key practice is to automatically compare your staging config to your production config. You can write a script that diffs the two and flags any unexpected differences. That's a form of drift detection. Luna: But even with all that, you can't test everything. How do you roll out a config change safely? Lucas: That's where canary releases come in. Instead of flipping a flag for 100% of users, you start with 1% or 5% of your traffic. Monitor the metrics — error rates, latency, business KPIs — and if nothing looks wrong, gradually increase the percentage. Luna: So you're treating config like a feature, with a rollout plan. Lucas: Exactly. And you should have an automated rollback plan. If a config change causes an error spike, you want to revert it in seconds, not minutes. That means storing the previous config and having a button or a command to flip back. Luna: Is that more common in the industry now? I feel like a lot of companies still just edit a file and hope for the best. Lucas: It's improving, but there's still a long way to go. I've seen teams that have rigorous testing for their code, but when it comes to config, they treat it like a sysadmin task from 1995. The gap is real. Luna: And that gap is exactly where incidents happen. Lucas: Yeah. And I think the root cause is a cultural mindset. We need to elevate configuration to be a first-class citizen in our testing strategy. It's not just an ops problem; it's an engineering problem. Luna: So what's the first step for a team that wants to get better at this? Lucas: Start by inventorying your config. Find every place where a change could cause an outage. Then prioritize based on impact. The most critical config is probably your feature flags and your service to service endpoints. Luna: And then build schema validation for those, and run it in CI. Lucas: Exactly. You don't have to solve the whole problem overnight. Just start with the most fragile parts. And once you have that safety net, you'll sleep better at night knowing your next config change isn't gambles. Luna: That's a good segue. We're in the business of helping engineers sleep better, right? And that's also why we keep this show ad-free. It's a deliberate choice, and we appreciate that you listen. If you want to support that choice, there's a link on our show page: buy me a coffee dot com slash fexingo. No pressure, just a way to keep this going. Lucas: Yeah, thanks for that, Luna. And it does tie in — because reliable software is about reducing stress, and we want to keep sharing these insights without any commercial breaks. So, back to config testing. One more thing I want to mention is the idea of 'test doubles' for configuration. You can inject fake config in your tests to simulate different scenarios, like a service that's down or a feature flag that's off. Luna: That's like mocking the config itself, so you can test how your code handles different config states. Lucas: Right. And that's actually a powerful technique because it lets you test the behavior of your system under config changes without actually changing the config in a real environment. You can write a test that says 'when this flag is false, the home page should not show the new banner' — and you test that in isolation. Luna: So you're testing the code that reads the config, not just the config itself. Lucas: Exactly. Because even if the config is perfect, your code might handle it incorrectly. So you need both sides: config validation and code-level tests that exercise different config states. Luna: That's a solid approach. So, to wrap up, what's the one takeaway for our listeners? Lucas: Treat configuration like the dangerous, invisible force it is. Give it the same respect as your code. Validate it, test it, roll it out carefully, and you'll avoid the next big outage. Luna: And if you do, you'll be a hero. Thanks for listening, everyone.