Latest / The Edge Computing Podcast with Fexingo: Local Compute, CDNs, and Distributed Infrastructure / How Edge Computing Puts a Supercomputer in Your Pocket
Transcript
- Lucas: So, there's this number that's been rattling around in my head for a few weeks now: thirty-eight trillion operations per second. That's the peak performance of the neural engine in Apple's A18 chip. Luna: Thirty-eight trillion — that's a lot. What does that actually mean in practice? Lucas: It means your iPhone can run deep neural networks in real time without phoning home to a cloud server. And this isn't just Apple — Qualcomm's latest Snapdragon, Samsung's Exynos, Google's Tensor chip — they're all pushing similar numbers. The real story here isn't the hardware spec, though; it's what developers are now doing with it. Luna: So we're moving from cloud-dependent AI to on-device AI. What's driving that shift? Lucas: Three things, really. Latency — if you're doing real-time translation, you can't afford a round trip to a data center. Privacy — you don't want your face or your voice recordings leaving your phone. And cost — inference in the cloud costs the provider money. If you can run it on the device, that's free. Luna: Right, but there's still stuff that needs the cloud. You can't train those models on a phone. Lucas: Exactly — training stays in the cloud. But inference, the act of actually using the model, that's moving to the edge. And 'edge' in this context means your pocket. Let me give you a concrete example: real-time language translation. Google Translate used to send your voice to a server, get the translation back, and play it. That took maybe a second. With on-device models, it's under two hundred milliseconds. You barely notice the pause. Luna: I've seen that. It makes conversation feel more natural, less like you're waiting for a robot to think. Lucas: And it works offline. That's a huge deal for travelers or people in areas with poor connectivity. But translation is just the beginning. The bigger play is augmented reality — Apple's Vision Pro and Meta's Quest headsets are doing real-time environment mapping, hand tracking, object recognition, all on device. The A18 in the latest iPhone can run a semantic segmentation model — that's where the phone labels every pixel in a camera frame as 'sky', 'road', 'person', 'car' — at sixty frames per second. Luna: That's basically a real-time 3D understanding of the world. For AR, that's crucial. Lucas: Right. And you can't do that by streaming video to the cloud — the bandwidth would be insane, and the latency would make you sick. So the edge is the only viable path. But here's the thing that I find fascinating: developers are now having to decide how much to put on device and how much to leave in the cloud. It's a new design trade-off. Luna: How do they make that call? Is it just about speed versus accuracy? Lucas: It's a bunch of factors: model size, battery consumption, how often the model needs updating, how sensitive the data is. A good rule of thumb is: if the task needs sub-hundred-millisecond response and the model fits in, say, a hundred megabytes, do it on device. If you need the full power of a giant language model or access to a massive database, you go cloud. More and more apps are splitting the workload — simple inference on device, complex queries to the cloud. Luna: Like a virtual assistant — wake word detection on device, then the actual query goes to the cloud. Lucas: Exactly. Siri, Google Assistant, Alexa — they all do that now. The wake word model is tiny, runs continuously on a low-power core, consumes almost no battery. Only when you say 'Hey Siri' does it send the audio up. That's edge computing in action, and it's been running for years without us thinking about it. Luna: So what's new? What's changed recently that makes this a bigger deal? Lucas: Two things. First, the models themselves are getting more efficient. Techniques like quantization — reducing the precision of the numbers in the model — can shrink a model by four times with minimal accuracy loss. And neural architecture search, where AI designs AI, is creating models that are tailor-made for mobile chips. Second, the chips are getting dedicated hardware for specific tasks. The A18 has a sixteen-core neural engine, but it also has what Apple calls the 'media engine' that handles video encoding and decoding separately. So you can process video streams without tying up the main CPU. Luna: So the hardware is becoming specialized for this type of workload. Lucas: And that specialization is exactly what edge computing needs. In a data center, you have GPUs and TPUs that are power-hungry but incredibly fast. On a phone, you have a fraction of the power budget, but the neural engine is so efficient that it can do the same kind of work at a different scale. And this isn't just phones — it's earbuds, smart glasses, even smart home devices. The Apple AirPods Pro have a chip that runs noise cancellation and adaptive audio entirely on device. Luna: Right, but does that mean the cloud becomes less important? Or does it just shift the role? Lucas: It shifts the role. The cloud becomes the brain for training and for tasks that need huge models or collective intelligence. But the edge becomes the nervous system — fast, local, responsive. The real opportunity is in the interplay between them. For instance, federated learning: your phone trains a model locally on your data, then sends only the updated parameters — not your data — to the cloud. Google uses this for its keyboard, Gboard, to improve predictive text without ever seeing what you type. Luna: That's elegant. Privacy preserving, but still getting better over time. Lucas: And it's a perfect example of edge computing that doesn't feel like edge computing. It's just how your phone works. I think in the next two to three years, we'll see this split become the default architecture for most apps. Developers who ignore on-device inference will find their apps feel sluggish compared to ones that do it right. Luna: It's almost like the pendulum is swinging back from pure cloud to a hybrid model. We went to the cloud because we needed compute we didn't have locally, now local compute is catching up. Lucas: Exactly. And the companies that win will be the ones that design for that hybrid world. Apple is obviously all-in on device-side intelligence. Google is interesting because they have both the cloud and the device — Tensor chips in Pixels, TPUs in the cloud. They can optimize across the whole stack. Qualcomm is trying to be the platform that enables everyone else. Luna: This episode is actually a good example of something that makes me think about the value of independent podcasts. We get to go deep on a specific angle without any ads or sponsor constraints. Lucas: Yeah, and that's entirely thanks to listeners who chip in. If today's tech conversation gave you something usable, buying us a coffee is a great way to keep this ad-free. It's buy me a coffee dot com slash fexingo. Luna: Even a small contribution makes a big difference. And it keeps us independent. Lucas: So back to the hybrid architecture: one challenge that often gets overlooked is the update problem. On-device models aren't static — they need to be updated when new data comes in or when the cloud trains a better version. That's a distribution challenge. Luna: How do you update a model on millions of devices without causing issues? Lucas: Apple and Google have built systems for this — Apple's Core ML and Google's ML Kit allow developers to push model updates through the same channels as app updates. But the tricky part is when the model runs in a low-level component like the camera pipeline. You can't just swap it out without risking a crash. So there are versioning strategies, A/B testing, gradual rollouts. It's more complex than updating a cloud API. Luna: Because with a cloud API, you just swap the model on the server side and every client gets the new behavior instantly. Lucas: Right. With on-device, you have to wait for the user to update their app, or you use a silent background update. And if something breaks, you can't roll back as easily. So the operational complexity is higher. But the payoff in user experience and privacy is substantial. Luna: Are there any specific apps you've seen recently that do this well? Lucas: One that blew me away is the app 'Arc Search' — it uses on-device AI to summarize web pages and answer questions without sending your query to a server. Another is 'Photomath' for math problems: it recognizes handwritten equations on device, solves them, and shows steps. And of course, Apple's Live Text — that reads text from images in real time — that's all on device. Luna: I use Live Text all the time. I didn't realize it was running locally. Lucas: Most people don't. And that's the sign of good edge computing — it's invisible. You just point your camera at a sign, and it becomes selectable text. No loading spinner, no internet required. That's the promise of putting a supercomputer in your pocket. We're not quite there yet, but the trajectory is clear. Luna: What's the one thing you'd tell a developer who's thinking about moving some inference to the edge? Lucas: Start small. Pick a single model that's critical for latency or privacy, and deploy it on device. Use tools like Core ML or TensorFlow Lite. Measure the performance improvement. And then iterate. The biggest mistake is trying to do everything on device at once — you'll run into memory constraints or battery drain. Find the one use case where the edge gives you a clear win, and build from there. Luna: That's practical advice. I think we'll see a lot more of this in the next year. Lucas: Absolutely. And it's one of those rare areas where the hardware is actually ahead of the software. The chips are ready. Now it's up to developers to take advantage.