Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How Square Rebuilt Its Payments Stack for Offline Reliability
Transcript
- Lucas: If these conversations are useful for what you're building or running, you're in the right place. Lucas: A few years back, Square published a technical post about how they rebuilt their payments stack for the real world. And the real world, as Square found out, has terrible internet. Luna: Especially at a farmers' market or a food truck. That's where Square's original use case lived. Lucas: Exactly. The original Square reader plugged into the headphone jack — you'd swipe a card, the app would send the data up to Square's servers, and you'd wait. If the connection dropped, the transaction failed. No receipt, no sale, angry customer. Luna: So they had to build an offline mode. But payments are not like syncing a Google Doc. You can't just merge conflicts later. Lucas: Right. The constraint is brutal: you have to decide, right there at the counter, whether to take the money or not. Square's solution became a case study in offline-first design. They literally re-architected their client-side stack to run a mini payments engine on the device itself. Luna: So the phone becomes a point of sale terminal, even offline. How does that work with the bank networks? Lucas: The key insight is that Square doesn't actually process the payment on the device. What it does is locally authorize it, encrypt the card data, store it in a local queue, and then, when connectivity returns, it forwards the batch to Square's servers for actual processing. The risk is that the merchant might hand over goods before the bank clears the transaction. Luna: Which means Square takes on the fraud risk. That's a big bet. Lucas: It is. Square decided that the cost of losing a sale due to connectivity was higher than the cost of occasional fraud losses. So they built a risk engine that runs on the device — a set of rules that scores each transaction locally. Is the card present? Is the amount below a threshold? Has this merchant been reliable? If the score passes, the app says 'approved' and stores the transaction locally. Luna: But storing locally means you have to handle conflicts. What happens if the same transaction gets submitted twice? Lucas: That's where the data structure matters. Square uses a form of conflict-free replicated data type — a CRDT — for the transaction queue. Each transaction gets a unique ID generated on the device, and the server deduplicates based on that ID. So even if the merchant hits 'charge' twice, only one goes through. Luna: So it's not just caching. It's a full offline-first data model. Lucas: Exactly. The local store is not a cache you flush when online. It's the authoritative source until the server confirms receipt. Square even handles the case where the device goes offline mid-transaction — the card data is partially sent, the app rolls back, and the customer can retry without re-entering everything. Luna: That's interesting because most offline-first apps just fail on partial writes. So Square must have built some kind of state machine. Lucas: They did. The transaction lifecycle is modeled as a finite state machine: idle, reading card, processing, awaiting confirmation, completed or failed. Each state transition is atomic on the device. If the network drops while in 'awaiting confirmation,' the local state machine keeps the transaction in 'pending' and retries on reconnection. The user sees a consistent status. Luna: And what about the backend? How does it handle a flood of delayed transactions from thousands of devices all coming back online at once? Lucas: That's the other half of the story. Square's backend is designed for eventual consistency with strict ordering guarantees per merchant. When a device reconnects, it opens a long-lived WebSocket and replays its local queue in order. The server processes each transaction idempotently — if it sees a duplicate ID, it skips it and returns the already-stored result. The merchant sees a reconciled batch at the end of the day. Luna: That sounds a lot like the architecture behind Google Docs' offline mode. But with money. Lucas: Similar principles. Operational transforms in Docs, CRDTs in Square. Both have to handle concurrent edits and resolve without conflict. The difference is that in Docs, a conflict means a weird sentence. In Square, it means double-charging someone's card. So Square's conflict resolution is more conservative — they always prefer the server's view if there's any ambiguity. Luna: One thing I wonder — how do they handle regulatory compliance offline? Like, you need to print a receipt with certain details, and the card data can't just sit on the device unencrypted. Lucas: Great question. Square encrypts the card data at rest on the device using a device-specific key. The merchant can't access the raw PAN. And the receipt is generated locally with a truncated card number, so it's compliant. The full encrypted data is only decrypted on Square's servers after it's received. So the device never holds decrypted card data. Luna: That's the kind of detail that makes this architecture interesting. It's not just a tech problem — it's a compliance problem with a tech solution. Lucas: Right. And they also had to think about storage limits. A busy food truck might process hundreds of transactions in a day. If they're offline for hours, the local queue could grow large. Square imposes a soft limit — if you exceed, say, 500 offline transactions, the app starts warning the merchant. And the queue is capped at a thousand. After that, new transactions are declined. Luna: So there's a hard ceiling. That makes sense. You don't want unbounded local storage. Lucas: Exactly. And the risk model adjusts dynamically. If a merchant suddenly goes offline for a long time, the per-transaction risk threshold drops. Smaller amounts only. It's all configurable on the server and pushed down to the device. Luna: Honestly, if today was worth a coffee to you, that's the link — buy me a coffee dot com slash fexingo. It's how we keep the show ad-free. Lucas: Yeah, and it's the smallest way to say this kind of deep dive matters. We appreciate every single one. Lucas: So back to the architecture — one of the cleverest parts is how Square handles the transition from offline to online. The device doesn't just dump everything at once. It uses a backoff strategy: send a few transactions, wait for acknowledgement, send more. If the server is overloaded, it sends a 'slow down' signal, and the device backs off. Luna: That's like TCP congestion control, but at the application layer. Lucas: Exactly. And it's necessary because when a major event ends — like a festival — thousands of Square devices all come online simultaneously. Without that backoff, their backend would get thundering-herded. Luna: Did they publish numbers on how much this improved reliability? Lucas: They did. In their 2019 blog post, they said that after rolling out the offline mode, transactions completed successfully even with intermittent connectivity increased by over 30 percent. And the fraud rate remained below their target. So the bet paid off. Luna: Thirty percent is huge for a payments company. That's directly more revenue for Square and happier merchants. Lucas: Right. And it changed how they think about the device. The Square reader is no longer just a card swiper — it's a distributed node in a payment network. Every device is a mini data center. Luna: One more thing — did they open-source any of this? Lucas: Not the payments-specific code, obviously. But they've contributed to the CRDT library they used, and they've published several talks and blog posts detailing the design. So the patterns are out there for any team building offline-first systems. Luna: That's the kind of sharing that helps the whole industry level up. Lucas: Definitely. And it's a reminder that sometimes the most impactful engineering isn't a new feature — it's making the existing feature work when everything else fails. Luna: Yeah. Connectivity is a feature. And Square proved that offline doesn't mean off limits. Lucas: For next time, I want to look at how another payments company — maybe Stripe — handles a different hard problem: multi-currency settlement in real time. Because that's a whole other can of worms. Luna: I'll bring the coffee.