Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How Figma Prevented a Catastrophic UI Collapse
Transcript
- Lucas: I want to start with a moment that every software engineer dreads, but few have lived through at scale: August twenty-first, two thousand twenty-one. It was a Tuesday afternoon, and Figma went dark for roughly three hours. No login, no editing, no collaboration. For a company whose entire value proposition rests on real-time co-creation, that wasn't just an inconvenience; it was an existential crisis. Luna: Right. And unlike a server crash that you can restart, their backend services were still running, but the frontend couldn't talk to them without causing a cascade failure. It was a logic trap, not a hardware problem. Lucas: Exactly. Most people blame the outage on a bad database migration or a rogue developer commit. But when we dig into the post-mortem, the root cause was actually architectural: a circular dependency in their internal package management system. A change in one core library triggered a rebuild loop that consumed all available CPU resources, effectively freezing the entire platform. It teaches us that the biggest risk in modern software isn't always the code itself, but how those pieces connect. Luna: So the bug wasn't in the feature they shipped, it was in the plumbing that held the features together. That feels like a much harder thing to spot during a standard code review. Lucas: It is. And this brings us to a concept that most tech leaders ignore until it's too late: dependency graph visibility. When you have a monorepo with thousands of microservices, knowing which service calls which becomes a game of telephone. If Service A depends on Service B, which depends on Service C, and Service C suddenly changes its API contract, Service A doesn't just fail gracefully; it might take down the user session layer. We see this happen constantly in enterprise environments where legacy systems are bolted onto new cloud infrastructure. Luna: It reminds me of the financial sector's reliance on SWIFT. If one node goes down, the whole settlement process stalls. But in software, the velocity is so much higher that these breaks happen faster. Lucas: Precisely. In finance, the speed limit is regulatory and operational. In tech, the speed limit is set by the next sprint cycle. Teams push updates daily, sometimes hourly. Without a map of those dependencies, you're flying blind. I've talked to several CTOs who admit they don't actually know their full dependency tree. They assume it's stable because nothing has broken yet. But stability is not the same as resilience. Luna: That assumption of stability is dangerous. It’s like assuming your car will drive forever because it started fine this morning, ignoring the wear on the transmission. Lucas: A perfect analogy. So, how do we fix this? How do you build a dependency map that actually reflects reality? The answer lies in static analysis combined with dynamic tracing. You need tools that scan your codebase to identify import relationships, but you also need runtime telemetry that shows you what's actually happening in production. Static analysis gives you the potential paths; dynamic tracing gives you the actual traffic patterns. When you combine them, you can simulate failures before they happen. Luna: So you're saying you shouldn't just test if the feature works, but test if the network of features holds up under stress? Lucas: Yes. This is where chaos engineering meets dependency management. Instead of randomly killing servers, you start by isolating a single dependency node and observing the ripple effect. Did the failure stay contained? Or did it propagate to the billing system? To the user dashboard? To the email notification service? By understanding the blast radius of each connection, you can design better boundaries around your services. Luna: It sounds like a lot of work to implement, especially for smaller teams who are just trying to ship products. Do they really have time for this level of introspection? Lucas: They do, because the cost of ignorance is exponentially higher than the cost of implementation. Let's look at the numbers. A minor outage in a mid-sized SaaS company can cost upwards of fifty thousand dollars per hour in lost productivity, support tickets, and reputational damage. Figma’s three-hour downtime likely cost them millions. Investing ten percent of your engineering capacity into mapping and testing dependencies pays for itself after one major incident avoided. Luna: And that’s before you factor in the trust erosion. Users leave platforms that feel unreliable, even if the downtime is brief. Lucas: Absolutely. Trust is the hardest asset to rebuild. But here’s the tricky part: many companies try to solve this with more manual checks. They add more QA engineers, more meetings, more sign-offs. But human beings are terrible at tracking complex networks. We get overwhelmed. Automation is the only way to handle the complexity of modern stacks. You need automated dependency scanners that run on every pull request. Luna: So the scanner blocks the merge if it detects a risky circular dependency? That seems like a hard stop. Lucas: Not necessarily a hard stop, but a strong warning. If you introduce a new dependency that creates a cycle, the tool should flag it immediately. It forces the developer to pause and think about the impact. Often, the solution is simple: refactor the module, move the logic, or create a shared interface. But you catch it in development, not in production. That shifts the cost of fixing the bug from hundreds of dollars to mere cents. Luna: I love that metric. Shifting the cost curve is exactly what good engineering management looks like. Lucas: It is. And this leads us to a broader point about organizational structure. Dependency issues are often symptoms of siloed teams. If Team Alpha owns the authentication service and Team Beta owns the payment gateway, and they don't communicate, their APIs will inevitably drift apart. You need cross-team coordination mechanisms, not just technical ones. Regular interface reviews, shared documentation, and clear ownership boundaries. Luna: So it’s as much a communication problem as it is a coding problem. You can’t code your way out of a cultural gap. Lucas: Correct. Technology amplifies whatever exists in the organization. If your teams are fragmented, your software will be fragile. If your teams are collaborative, your software will be resilient. This is why companies like Netflix and Amazon invest so heavily in their 'two-pizza teams' model. Small, autonomous groups that own their entire stack, including the dependencies. They control their own interfaces, so they can evolve faster without breaking others. Luna: But autonomy can lead to duplication if everyone builds their own version of the same wheel. How do you balance independence with consistency? Lucas: You balance it through platform engineering. You provide a standardized internal developer platform that handles the heavy lifting: logging, monitoring, security, and yes, dependency management. Teams can innovate on top of that platform without worrying about the underlying infrastructure. It’s like giving them a well-equipped workshop instead of making them build their own tools from scratch. Luna: That makes sense. Standardize the boring stuff, let them shine on the creative stuff. Lucas: Exactly. Now, before we dive deeper into the technical specifics of implementing these scanners, I want to take a quick moment. If today's conversation gave you something usable for your engineering strategy, the way these stays ad-free is listener support — buy me a coffee dot com slash fexingo. It helps keep the research deep and the insights sharp. Luna: Yeah, and if you find value in these detailed breakdowns, consider supporting the show. It directly funds the kind of rigorous analysis we bring to topics like this. Lucas: Thanks. Now, back to the mechanics. Let’s talk about how to actually visualize these dependencies. The best tool for this is a directed graph. Each node is a service or module, and each arrow represents a call. When you visualize this, you’ll often find 'star' patterns, where one central hub is connected to dozens of other nodes. That hub is your single point of failure. Luna: So you’re identifying the bottlenecks by looking at the density of connections? Lucas: Precisely. The hub with the highest degree centrality is your most critical asset. You need to protect it fiercely. That means redundancy, caching, and graceful degradation. If that hub goes down, the rest of the network should continue to function in a limited capacity, not collapse entirely. Think of it like a power grid. If one substation fails, the neighborhood loses light, but the city keeps running. Luna: Graceful degradation is key. Users can handle a slower experience, but they cannot handle a blank screen. Lucas: Right. And implementing graceful degradation requires foresight. You have to ask: what is the minimum viable experience if half our services are offline? Can we serve cached content? Can we disable non-essential features? Can we queue requests for later? These decisions must be made in peace, not in panic. That’s why dependency mapping is not just a technical exercise; it’s a product strategy exercise. Luna: It ties the code directly to the user experience. Which is where most tech teams lose sight of the bigger picture. Lucas: They do. They focus on the code, not the customer. But in the end, the customer doesn't care about your microservices architecture. They care if their work gets done. If Figma had implemented better dependency isolation, users might have seen a partial failure instead of a total blackout. They could have opened files in read-only mode, for example. That would have preserved trust, even during the outage. Luna: Read-only mode is a smart fallback. It turns a catastrophic failure into a manageable inconvenience. Lucas: It is. And that’s the goal of resilience engineering: to manage failure, not eliminate it. Failure is inevitable. Systems grow complex. Bugs happen. Hardware breaks. The question is not if you will fail, but how you will respond. By understanding your dependencies, you can design responses that are predictable, controlled, and user-friendly. Luna: So the lesson is: map your connections, protect your hubs, and plan for the worst-case scenario. Lucas: Exactly. Map, protect, plan. It’s a simple framework, but it requires discipline to implement. Most teams skip the mapping because it’s tedious. They skip the protection because it’s expensive. They skip the planning because it’s uncomfortable. But the cost of skipping is always higher than the cost of doing. Luna: Discomfort now versus catastrophe later. It’s a choice every engineering leader has to make. Lucas: It is. And as we look toward the future, with AI integration becoming more common, these dependency chains will only grow longer and more opaque. An AI model calling a data service calling a recommendation engine calling a payment gateway. The surface area for failure expands rapidly. We need these foundational practices more than ever. Luna: AI adds another layer of unpredictability. If the model behaves unexpectedly, can your dependency map help you contain that? Lucas: It can. If you’ve isolated the AI component behind a clear API boundary, you can throttle its requests, cache its outputs, or route it to a fallback model if it starts hallucinating. The isolation protects the rest of the system. Without it, a bad AI response could corrupt your database or trigger a false payment. The boundaries are your shield. Luna: Boundaries as shields. I like that framing. It turns abstract architecture into a defensive strategy. Lucas: It does. And finally, remember that this isn't a one-time project. Dependency maps decay over time. New services are added, old ones are deprecated, APIs change. You need a living document, continuously updated by your CI/CD pipeline. Make it part of your daily rhythm, not a quarterly audit. Luna: Continuous mapping. It’s like maintaining a garden rather than building a monument. It requires ongoing care. Lucas: Perfectly put. Care for your dependencies, and they will care for your users. What do you think is the first step a team should take if they’re starting from zero on this front? Luna: Start small. Pick one critical service and map its dependencies manually. Just see what’s there. Awareness comes before action. Lucas: Awareness is the hardest part. Once you see the web, you can’t unsee it. And then you can start to strengthen the knots. Thanks for listening.