Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / Testing Shadow Mode Without Breaking Production
Transcript
- Lucas: The hardest part of shipping software isn't writing the code. It's knowing if that code will actually work when millions of people hit it at once. We talk a lot about canary releases and feature flags, but there is a third option that most teams ignore because it feels like cheating. It is called shadow testing. Luna: Right, where you run the new code in parallel with the old code but just throw away the results. It sounds simple enough on paper, but I always wonder about the cost of running two systems simultaneously. Lucas: That is the exact question. And the reason companies like Netflix do it is because they have billions of requests a day. Even if you are only shadowing one percent of traffic, that is still a massive amount of compute. But the insight you get from seeing how your new logic handles real user data is worth the price tag. Luna: So it is less about proving the code works and more about seeing if it breaks in weird ways that unit tests never catch. Like edge cases involving actual human behavior. Lucas: Exactly. Unit tests are great for logic. They tell you if a function returns the right number for a given input. But they do not tell you what happens when a user has a weirdly formatted address or a payment method that expires mid-transaction. Shadow testing exposes those messy, real-world friction points before you flip the switch. Luna: It reminds me of how airlines test new flight paths on cargo planes first. You want to see if the route is viable without risking passengers. Lucas: That is a perfect analogy. And unlike canary deployments, which slowly roll out the change to five percent of users, shadow testing gives you zero risk to the customer experience. If the shadowed request fails, the user never knows. They just see the result from the old, stable code path. It is the ultimate safety net. Luna: But doesn't that create a problem if the new code starts generating errors? You don't want those error logs flooding your monitoring system, do you? Lucas: You absolutely do not. That is the biggest operational headache with shadow testing. If you let every failed shadow request trigger an alert, your on-call engineer will be woken up by ghosts. You have to filter those logs aggressively so you can analyze them later, but not in real time. Luna: So you are essentially building a parallel universe inside your production environment. A sandbox that looks exactly like reality but leaves no footprints. Lucas: I love that description. A parallel universe. And the cool part is that you can use the data from that universe to tune your performance expectations. You might find that your new algorithm is correct but twenty percent slower. In a normal test, you might miss that latency spike until it hits production. Luna: Which brings us back to the cost. If you are paying for cloud resources based on usage, doubling your load for even a few hours can get expensive fast. Lucas: It can. But compare that to the cost of a production outage. One hour of downtime for a major e-commerce site can cost hundreds of thousands of dollars. The marginal cost of extra compute is usually a rounding error in comparison. The math favors caution here. Luna: True. Though I imagine smaller startups struggle with this. They do not have the infrastructure budget to spin up duplicate clusters just to test a checkout flow. Lucas: They do. And that is why many smaller teams skip straight to canary deployments. They accept a tiny bit of user risk because they cannot afford the infrastructure overhead. It is a trade-off between safety and speed. Larger enterprises prioritize safety because their scale makes the risk unacceptable. Luna: So it is really a function of company size and risk tolerance. Not necessarily technical superiority. Lucas: Precisely. Technical superiority is about having good tests. Operational maturity is about knowing which deployment strategy fits your current capacity. Shadow testing is a luxury of scale. You need enough traffic to make the sample size meaningful. Luna: What about data privacy? If you are mirroring live traffic, you are copying real user data into your shadow environment. Does that create compliance issues? Lucas: That is a critical point. You cannot just mirror everything. You have to scrub personally identifiable information before it hits the shadow service. If you are in Europe, GDPR makes this tricky. You need to ensure that the shadow environment does not become a data leak. Luna: So you need a proxy layer that strips out emails and names before the data even reaches the new code. It adds another moving part to your architecture. Lucas: It does. And that proxy layer becomes a single point of failure. If it slows down, you slow down the entire application. So you have to build it to be incredibly lightweight. It should not add more than a few milliseconds of latency. Luna: This feels like it requires a very specific type of engineering discipline. You cannot just wing it. You need strict contracts between the proxy and the services. Lucas: Absolutely. And you need automated checks to verify that the scrubbing process is working. If you miss one field, you could be exposing sensitive data. It is not just a performance test; it is a security audit disguised as a performance test. Luna: I guess that is why we see big tech companies investing so heavily in internal tooling. They build these frameworks once and reuse them across dozens of teams. Lucas: Exactly. Companies like Netflix and Uber have spent years refining their internal platforms to make shadow testing effortless. For a developer, it should be as easy as adding a header to their API call. They shouldn't have to think about the infrastructure behind it. Luna: So the barrier to entry isn't really the concept. It is the platform maturity required to support it safely. Lucas: Right. If you are a small team, you might be better off focusing on rigorous integration tests and maybe a limited canary release. Shadow testing is powerful, but it is heavy. It is a sledgehammer, not a scalpel. Luna: Speaking of tools and platforms, if today's deep dive into shadow testing gave you something usable for your next sprint... Lucas: and you found value in breaking down these complex engineering strategies, the way we keep these episodes ad-free and consistent is through listener support. It helps us cover the research and production costs. Luna: If you want to help keep the signal clear and the noise low, you can buy me a coffee dot com slash fexingo. It takes thirty seconds and keeps the show running. Lucas: Thanks for considering it. Now, back to the data. Let's talk about how you actually measure success in a shadow test. What does 'good' look like? Luna: It is not just about whether the code runs. It is about whether the output matches the old code. You need a diff engine that compares the results side by side. Lucas: Yes. And that diff engine needs to be smart. It cannot just do a byte for byte comparison. Sometimes the order of items in a list changes, or timestamps shift slightly due to network latency. You need semantic equality checks. Luna: So you are looking for logical equivalence, not literal identity. Two different JSON structures that represent the same financial transaction should be considered a match. Lucas: Exactly. And you have to define what constitutes a 'mismatch'. Is a difference in currency formatting a mismatch? Or is that acceptable? These rules need to be configurable per endpoint. Luna: This sounds like it requires a lot of upfront configuration. You have to teach the system what matters and what doesn't before you can trust the results. Lucas: It does. And that is the hidden cost. The setup time. If you spend three weeks configuring your shadow testing framework, you might not ship that feature for months. It slows down development velocity initially. Luna: But then you pay dividends later. Every subsequent release becomes safer and faster because the framework is already in place. Lucas: Precisely. It is an investment in future speed. Once the infrastructure is built, adding a new shadow test takes minutes instead of weeks. You are trading short-term effort for long-term stability. Luna: Do you see this becoming standard practice for all mid-sized companies, or will it remain a luxury for the giants? Lucas: I think it will become more common as cloud providers offer managed services for this. We might see AWS or Azure offering a 'shadow testing as a service' product soon. That would lower the barrier significantly. Luna: Managed infrastructure removes the ops burden. If I can just toggle a switch in my console to enable shadow testing for my API, I am much more likely to do it. Lucas: That is the trend. Making safety easier to adopt. Right now, it is still manual. But as the market matures, the tools will abstract away the complexity. You won't need to be a platform engineer to use it. Luna: It is interesting how the definition of 'testing' is evolving. It used to be about finding bugs. Now it is about validating behavior in a live environment. Lucas: That is a profound shift. We are moving from a mindset of prevention to a mindset of observation. We assume things will break, so we build systems to watch them break safely. It is a more humble, but ultimately more robust, approach to engineering. Luna: Humble engineering. I like that. It acknowledges that we cannot predict every outcome, so we build mirrors to see what happens anyway. Lucas: Exactly. And while we are watching those mirrors, we have to make sure we are not learning anything we shouldn't. Privacy remains paramount even in our safety nets. Luna: Always. Because the moment you compromise user trust for a smoother deploy, you have lost the war regardless of how many bugs you caught. Lucas: Well said. So the next time you are planning a major release, ask yourself if you are ready to build a mirror. Or if you are just hoping the glass holds.