Latest / The API Podcast with Fexingo: REST, GraphQL, and Modern Web APIs / How Async APIs Handle Failure Gracefully
Transcript
- Lucas: Most developers still think of an API as a phone call: you dial, they answer, and if they don't pick up, you assume the line is dead. But in modern distributed systems, that model collapses under scale. The real question isn't whether the service responds instantly, but how the system behaves when it doesn't. Luna: Right, because hanging on a timeout for thirty seconds is worse than just admitting you'll handle it later. It turns out waiting is expensive. Lucas: Exactly. And the cost isn't just latency; it's resource exhaustion. When a synchronous endpoint hangs, it ties up database connections, memory, and thread pools until the request finally times out. That cascade effect is what takes down entire platforms during peak traffic. So we need to talk about shifting from synchronous calls to asynchronous message passing, where the sender says 'I sent this' and moves on, rather than waiting for a receipt. Luna: But doesn't that create a trust problem? If I send a payment request and get back a simple 'received', how do I know it actually processed? Lucas: That is the core architectural shift: moving from immediate consistency to eventual consistency. You're not asking for a guarantee right now; you're asking for a mechanism to verify status later. Think about Shopify's order processing pipeline. When a customer checks out, the frontend gets an immediate confirmation that their order was accepted. Behind the scenes, that order event is pushed into a message queue. Inventory checks, fraud detection, and shipping label generation happen asynchronously. If fraud detection is slow, the user doesn't stare at a loading spinner; they get a notification when it's ready. This decoupling is why Shopify can handle Black Friday volumes without crashing. Luna: So the user experience actually improves because you remove the bottleneck of waiting for every single downstream step to finish before showing any result. Lucas: Precisely. And it saves money too. In a synchronous model, you have to provision enough servers to handle the worst-case scenario all at once. With async, you can buffer requests in a queue like Amazon SQS or Kafka. You process them at a steady rate your infrastructure can handle, regardless of how many hit the API at once. That smooths out the spikes and lowers your cloud bill significantly. Luna: It’s basically financial hedging for server load. But what happens when the downstream service actually crashes? Does the message just vanish into the void? Lucas: This is where most async implementations fail, and it’s the most critical part of the design. You never just send a message and forget it. You implement a retry policy with exponential backoff. If the inventory service is down, the message stays in the queue, and the worker tries again in ten seconds, then thirty, then five minutes. If it fails after a set number of attempts, the message moves to a Dead Letter Queue. This isn't a trash bin; it's a staging area for manual intervention. An engineer logs in, sees which order failed, fixes the data issue, and pushes it back into the pipeline. No data is lost; it’s just paused. Luna: That sounds like a massive operational overhead though. Do small teams really have the bandwidth to monitor dead letter queues daily? Lucas: They shouldn't have to. Modern platform engineering automates this. You set up alerts so that if messages accumulate in the Dead Letter Queue faster than they are cleared, you get an alert immediately. For most healthy systems, the DLQ is empty. It only becomes active when something genuinely breaks. The key is treating the DLQ as a feature, not a bug. It gives you visibility into systemic failures that synchronous timeouts hide because they just return a generic error code to the user. Luna: So instead of the user seeing '503 Service Unavailable', the backend quietly retries and eventually succeeds? That’s a much better brand impression. Lucas: Absolutely. It transforms a potential outage into a background maintenance task. But there’s a trade-off. Asynchronous systems are harder to debug. If a transaction is missing three days later, tracing the log requires following the message ID through multiple services and queues. It’s not as straightforward as checking a single server’s error log. You need distributed tracing tools like Jaeger or OpenTelemetry to stitch those events together. Luna: Which means you need more observability infrastructure upfront. Is that worth the complexity for a startup just starting out? Lucas: For a small team, maybe not initially. If you’re doing ten transactions an hour, synchronous is fine. But once you hit concurrency, the complexity pays for itself. The real value isn't just technical; it’s organizational. Async APIs force you to define clear boundaries between services. You stop building monolithic blocks of logic and start building discrete, testable units. That makes your codebase easier to maintain and your team easier to scale. Luna: It’s interesting that the technical decision drives the company structure. Decoupled systems lead to decoupled teams. Lucas: Right. Conway’s Law applies here. If your API is synchronous and tightly coupled, your teams have to coordinate constantly to avoid breaking each other. If your API is async and event-based, teams can deploy independently. The payment team can update their fraud logic without worrying about breaking the shipping team’s workflow, as long as the event schema remains compatible. Luna: That independence is huge for velocity. How do you handle schema changes in an async world? If the sender updates the event format, does the receiver break? Lucas: This brings us back to Ep 186, but with higher stakes. In sync APIs, you can often roll out changes quickly because the caller and callee are connected directly. In async, the producer and consumer are decoupled in time. They might not even be running the same version of the software. You need strict schema contracts, usually managed via a registry like Apicurio or Confluent Schema Registry. You enforce backward compatibility rules: you can add new fields, but you cannot delete or rename existing ones. This allows consumers to upgrade at their own pace while still accepting older events. Luna: So the API contract becomes a permanent record of truth, not just a temporary handshake. That requires serious discipline from the engineering side. Lucas: It does. And it shifts the culture. Engineers can’t just push code and hope it works. They have to document the event payload, test against the schema, and ensure idempotency. Because in an async system, a message might be delivered twice. Your consumer has to be able to process the same event twice without creating two orders. That’s why idempotency keys are non-negotiable in event-driven architectures. Luna: I remember discussing idempotency keys earlier, but in a sync context, it felt optional. In async, it feels like the only way to survive. Lucas: It is the only way to survive. Without it, you get duplicate charges, double shipments, and corrupted state. The network is unreliable. Queues are unreliable. Your own workers might crash mid-processing and restart. Idempotency is the insurance policy that ensures your business logic holds up under those conditions. Luna: So if I’m a CTO looking at my tech debt, should I be prioritizing async migration? Lucas: Look at your bottlenecks. If your users complain about slow load times during peak hours, or if your servers crash when a third-party service goes down, then yes. Start by identifying one critical path—like order creation—and refactor it to async. Introduce a message broker, decouple the validation steps, and measure the improvement in throughput and error rates. The ROI is usually visible within a few sprints. Luna: And the learning curve? Are there good entry points for teams used to REST? Lucas: The best entry point is using an existing managed service. Don’t build your own queue. Use AWS SNS, Google Pub/Sub, or Azure Service Bus. These platforms handle the reliability, the scaling, and the DLQ management for you. You just write the producer and consumer functions. It lets you learn the pattern without drowning in infrastructure plumbing. Once you’re comfortable with the flow, then you can consider self-hosted solutions like Kafka if you need more control. Luna: Managed services make it feel less like a revolution and more like a tool upgrade. That’s probably easier to sell to stakeholders. Lucas: Exactly. It’s not about replacing REST; it’s about complementing it. You keep synchronous endpoints for simple queries and real-time interactions, but you offload heavy, complex, or unreliable processes to async workflows. It’s a hybrid approach that balances speed with resilience. Luna: I think that’s the nuance people miss. They think it’s either sync or async, but the smart move is mixing them based on the task. Lucas: Right. A search query is sync. You want the answer now. But sending an email after a purchase is async. It doesn’t matter if it arrives in three seconds or three minutes. Matching the protocol to the requirement is the mark of mature engineering. Luna: So, before we dive deeper into the implementation details, I want to take a quick moment. If today's tech conversation gave you something usable, like a concrete way to think about system reliability, a couple of dollars a month is genuinely what keeps these going. Buy me a coffee dot com slash fexingo, if you've gotten something out of them. It helps us keep the episodes ad-free and focused on the details that matter. Lucas: We appreciate it. Every bit helps cover the hosting costs and the research time. Now, back to the queue. Luna: Let’s talk about the actual implementation. What’s the biggest mistake teams make when they first try to implement a retry loop? Lucas: They implement linear retries. They try once, wait a second, try again, wait another second. This creates thundering herd problems where everyone retries at the exact same moment, overwhelming the service further. Exponential backoff with jitter is essential. You randomize the wait time so the retries spread out over time. It’s a simple change that prevents secondary outages. Luna: Jitter seems like such a small thing, but it solves a huge coordination problem. It’s the difference between a controlled recovery and a cascading failure. Lucas: Exactly. And it’s free. It doesn’t require new hardware or new licenses. It just requires thinking about the timing of your operations. Most bugs aren’t code bugs; they’re timing bugs. Luna: So the lesson is: design for failure, assume duplication, and spread out your retries. Lucas: Design for failure, assume duplication, and spread out your retries. Those three principles will save you more nights than almost anything else. The goal isn't to prevent all errors; it's to ensure the system survives them gracefully. Luna: And that survival capability is what separates fragile apps from robust platforms. Lucas: Precisely. When you build for the worst case, you end up with a system that handles the best case efficiently too. It’s counterintuitive, but resilient systems are often simpler to operate because they don’t panic under pressure. Luna: Do you think we’ll see more tools emerge that abstract away the complexity of message queues entirely? Lucas: We already are. Serverless functions and event-driven frameworks are making it easier to trigger code based on events without managing the queue infrastructure. The abstraction layer is rising, but the underlying principles of decoupling and eventual consistency remain. You can hide the pipes, but you can’t hide the logic. Luna: So the fundamentals stay the same, even if the tools get fancier. That’s reassuring. Lucas: It is. Technology changes, but the physics of distributed systems don’t. Latency, concurrency, and fault tolerance are constant constraints. Mastering the async mindset is mastering those constraints. Luna: If you could go back and tell your younger self one thing about API design, what would it be? Lucas: I’d tell myself to stop optimizing for the happy path. Everyone designs for success. Design for the moment the database locks up. Design for the moment the network drops packets. That’s where the real engineering happens. Luna: Optimizing for failure is the ultimate form of optimism about the system’s ability to recover. Lucas: It is. And it’s what keeps the lights on when everything else goes dark.