Latest / The API Podcast with Fexingo: REST, GraphQL, and Modern Web APIs / API Webhooks And The Push Versus Pull Debate
Transcript
- Lucas: If you built your first API around twenty fifteen, you probably started with a simple REST endpoint that a client could poll until it got what it needed. It was safe, predictable, and entirely passive on the server side. Luna: Right, because the client controlled the rhythm. If the network dropped, the client just tried again. No one had to chase down a missing packet in transit. Lucas: Exactly. But then the market shifted toward real-time expectations, and everyone started adopting webhooks to push data instead of waiting for pulls. Now we have millions of endpoints trying to deliver events asynchronously, and half of them are failing silently. Luna: Silently failing is worse than failing loudly because nobody knows the data is broken until their dashboard shows stale numbers. Lucas: That is the core tension here. We are moving from a model where the consumer bears the cost of checking for updates to a model where the provider bears the cost of ensuring delivery. And when that delivery fails, the blame game starts immediately between the sender and the receiver. Luna: It feels like we traded latency for complexity. Polling is slow but dumbly reliable. Webhooks are fast but fragile. Lucas: Let us look at why fragility happens. Most webhook implementations treat delivery as a fire and forget operation after the initial handshake. They send the payload, get a two hundred okay, and assume success. But what happens if the recipient’s server goes down three seconds later? Luna: The event is gone. Unless there is some kind of retry mechanism or a log somewhere, that state change is lost forever. Lucas: Most teams do implement retries, but they often get the strategy wrong. They will retry with exponential backoff, which sounds smart, but they forget to make the retry idempotent. So you end up with a user getting charged twice because the webhook fired, failed, and then succeeded on the third attempt without checking if it had already processed. Luna: Double charging is the classic nightmare scenario. How do big platforms actually solve this without adding massive overhead to every single transaction? Lucas: They rely heavily on idempotency keys embedded in the event payload. Think of it like a unique fingerprint for each event. When your system receives the webhook, you check that key against your database. If it is already there, you return a two hundred okay immediately, effectively telling the sender stop trying. You skip the actual processing logic. Luna: So the receipt becomes the source of truth for duplication, not the processing pipeline itself. That shifts the burden to the database lookup rather than the application logic. Lucas: Precisely. And it forces you to think about storage costs early on. Do you keep those keys forever? Probably not. You might purge them after thirty days, accepting that an event older than that is too stale to matter anyway. Luna: But what about the security angle? If anyone can call your webhook endpoint with a fake idempotency key, they can trigger actions on behalf of users. Lucas: That is why signature verification is non-negotiable. Every major player now sends a hash signature in the header, usually calculated using a secret shared only between the sender and receiver. Your code must verify that signature before you even look at the idempotency key. Luna: Verify the identity first, then check for duplicates. That order matters because if the signature is bad, you should reject the request entirely rather than wasting cycles on processing. Lucas: Agreed. Rejecting invalid signatures protects you from spoofing attacks. But it also protects you from accidental misconfigurations where a different team’s testing environment accidentally points to your production URL. Luna: We have seen that happen. A staging webhook pointing to prod can flood your system with garbage data. Signature verification acts as a gatekeeper against chaos. Lucas: Now let us talk about visibility. Even with retries and signatures, you need to know when things go wrong. Most teams build custom dashboards to track delivery rates, but that is a lot of maintenance. Luna: Why build your own monitoring tool when the problem is universal? Third-party services like Sentry or specialized webhook monitors exist for this exact reason. Lucas: They do, but they add another layer of dependency. Sometimes the simplest solution is just good logging combined with a dead-letter queue. If a webhook fails five times in a row, move it to a separate table or bucket for manual review. Do not let it block the main flow. Luna: Isolating failures is better than letting them cascade. But manual review doesn’t scale. You still need automated alerts so someone notices before the business impact grows. Lucas: Absolutely. Set up an alert if more than one percent of your webhooks fail over a ten-minute window. That gives you enough data to spot a trend without being noisy during normal variance. Luna: One percent seems low, but in high-volume systems, that is thousands of events. Missing that many orders or inventory updates is significant revenue leakage. Lucas: Which brings us to the economic side of this decision. Why did we switch to webhooks in the first place? It was usually to reduce server load on the client side or to enable faster user experiences. Luna: Polling requires the client to constantly ask questions. Webhooks answer proactively. It saves bandwidth and compute, but it trades that efficiency for operational complexity on the backend. Lucas: And sometimes the trade-off is not worth it. If your data changes infrequently, maybe polling every hour is cheaper and safer than maintaining a complex webhook infrastructure. Luna: Context is everything. Real-time stock prices need webhooks. Daily sales reports do not. Choosing the wrong pattern based on habit rather than need is a common architectural debt. Lucas: That is a sharp point. We adopt technologies because they are trendy, not because they fit the use case. Webhooks are everywhere now, even where a simple cron job would suffice. Luna: And cron jobs are boring. Developers love webhooks because they feel more modern. But boring is profitable if it keeps the lights on. Lucas: Before we wrap this out, I want to pause for a moment. Honestly, if today was worth a coffee to you, that's the link — buy me a coffee dot com slash fexingo. Luna: Support keeps the signal clean and ad-free. Thanks for listening. Lucas: Back to the technical reality. Let us discuss the future of this space. We are seeing a move toward protocol-level improvements, like HTTP/3 and QUIC, which handle connection drops much better than older TCP connections. Luna: Faster connection setup means fewer timeouts. If the underlying transport is more robust, the application layer has less work to do managing retries. Lucas: True, but application-level resilience is still required. Network improvements help, but they do not fix bad error handling in your code. You still need those idempotency checks. Luna: Technology evolves, but the fundamentals of distributed systems remain the same. Trust nothing, verify everything, and assume failure is inevitable. Lucas: Assume failure is inevitable, and design for recovery. That mindset shift is what separates mature engineering teams from the rest. Luna: Do you think we will ever see a standard protocol that makes webhooks as easy to use as simple REST calls? Lucas: Maybe, but ease of use often comes at the cost of flexibility. Webhooks are powerful precisely because they are flexible. Standardizing them might kill the very adaptability that makes them useful. Luna: Or it might force us to define clear boundaries. Standards usually emerge after years of pain. Lucas: Pain drives standards. Until then, we have to write the code ourselves. What is one thing you will change in your webhook implementation tomorrow? Luna: I will audit my idempotency keys. I suspect half of mine are not actually preventing duplicates effectively. Lucas: A good place to start. Verify the keys, tighten the logs, and sleep easier knowing your events are accounted for. Luna: And if the logs don't match the payments, we finally know who to call. Lucas: Exactly. Clear signals, clear accountability. That is how you build trust in a distributed system.