Latest / Hardware Startups with Fexingo: Devices, Robotics, and Manufacturing Tech Companies / How Hardware Startups Fix Firmware Updates
Transcript
- Lucas: So we’re looking at hardware startups today, but not the usual angles of supply chains or packaging. We need to talk about the thing that keeps founders up at night once the device leaves the factory. It’s firmware updates. Specifically, the architectural decision of how you deliver them, verify them, and roll them back when things go wrong. Luna: Right. Because most teams think of firmware as just code on a chip, not a living service. They build the hardware first and bolt the software on later. That’s where the brick rate starts. Lucas: Exactly. And by brick rate, I mean devices that stop functioning entirely after an update fails. You’ve got thousands of units out there, maybe in people’s homes or factories, and suddenly they’re paperweights. The cost isn’t just the hardware; it’s the trust. Once a customer thinks your device is unreliable, you lose the brand before you even scale. Luna: It’s funny how often we see this. A startup launches with a sleek app, but then a minor bug requires a full reset. The support tickets explode overnight. Lucas: Let’s look at a concrete example. Take a company like Sonos, early on. They didn’t just push a binary file to every speaker. They built a dual-bank flash system. So the new firmware downloads to Bank B while Bank A runs the current version. If Bank B fails verification, the device just boots Bank A again. No brick. Just a retry. Luna: That’s the kind of redundancy that costs a few extra dollars in bill of materials but saves millions in support calls. Why do so many startups skip that? Lucas: Cost and time. Early on, founders want to ship. Adding a second partition doubles the flash memory requirement, which might mean switching from a cheap microcontroller to a slightly more expensive one. Or worse, adding external SPI flash, which takes board space. But here’s the thing: if you don’t have that redundancy, you have no safety net. Luna: Yeah, and it’s not just about saving money on chips. It’s about the psychological comfort of knowing you can recover from a mistake. Lucas: Comfort? It’s survival. Let’s talk about the handshake. When a device receives an update, it doesn’t just install it. It needs to authenticate the payload. If you’re using a generic bootloader, anyone can push malicious code. That’s not just a bug; that’s a liability. Luna: So you’re saying the security layer is part of the update pipeline, not an add-on. How do you actually implement that without slowing down the process? Lucas: You use asymmetric cryptography. The device has a public key embedded in the bootloader. The server signs the firmware with a private key. The device verifies the signature before writing anything. If the signature doesn’t match, it rejects the update. Simple. But you need to manage those keys. What happens when you need to rotate them? Luna: Rotation. Right. Because if that private key gets leaked, the whole ecosystem is compromised. How do you handle key rotation without bricking the existing fleet? Lucas: You design for it from day one. You don’t hardcode a single key. You have a root of trust, and you allow for multiple valid signatures during a transition period. Maybe you accept both the old key and the new key for six months. That way, you can switch safely. But most startups don’t plan for six months of overlap. They plan for launch week. Luna: That’s a classic trap. Optimizing for speed over resilience. And what about the user experience? Does the device tell the user it’s updating, or does it just disappear? Lucas: This is where product design meets engineering. If the device goes dark for five minutes with no indication, users think it’s broken. They try to power cycle it, which can corrupt the update mid-stream. You need a clear state machine. Flashing amber light. A progress bar on the screen. Or even a voice prompt. Make it obvious that work is being done. Luna: So visibility is as important as the technical mechanism. Because if the user intervenes, you’ve lost control. Lucas: Precisely. User intervention is the enemy of a clean update. You want to guide them, not let them guess. Now, let’s talk about partial updates. Do you ship the whole image every time, or just the delta? Luna: Delta updates save bandwidth and battery. But calculating the delta adds complexity. Is it worth it for most startups? Lucas: For IoT devices on cellular or limited Wi-Fi, absolutely. A full firmware image might be ten megabytes. A delta could be two hundred kilobytes. That’s a huge difference in data costs and download time. But you need a robust diff tool. And you need to handle cases where the delta fails. That brings us back to the dual-bank system. Luna: Because if the delta doesn’t apply cleanly, you need a fallback. Which again, points to having that redundant storage. Lucas: Yes. And here’s a nuanced point: you should test the delta application on the device itself, not just simulate it. Sometimes, memory wear or specific hardware quirks cause the merge to fail. You need a pre-install validation step that checks the target block integrity. Luna: So you’re verifying the destination before you start writing. That’s smart. It prevents corruption from propagating. Lucas: It does. And let’s not forget the rollback trigger. If the new firmware crashes twice in a row, the device must automatically revert to the previous version. No human interaction. No app required. Just a hard rule in the bootloader. If it fails, go back. Luna: Automatic rollback is non-negotiable. Otherwise, you’re relying on customers to send their devices back to you. Which nobody wants to do. Lucas: Exactly. Returns destroy margins. And they destroy reputation. If I buy a $200 robot vacuum and it bricks after one update, I’m never buying from that brand again. I’m telling my friends. I’m leaving bad reviews. The cost of a return is ten times the value of the device. Luna: Because of shipping, handling, and the loss of future revenue. It’s a massive hit. So, how do you monitor these updates remotely? You can’t just push and pray. Lucas: You need telemetry. Every device that receives an update must report its status. Success, failure, error code, timestamp. You aggregate that data in real-time. If you see a spike in failures for a specific region or batch, you kill the update immediately. Roll it back globally. Then investigate. Luna: Kill switches. So you have the ability to halt a bad rollout instantly. That requires a very tight feedback loop between the cloud and the device. Lucas: And it requires discipline. Founders love to push updates. They see a bug fix and they want it out yesterday. But you have to resist the urge to rush. Staged rollouts are better. Push to five percent of users. Wait twenty-four hours. Check the metrics. If all is well, push to twenty percent. Then fifty. Then everyone. Luna: Staged rollouts limit the blast radius. Instead of bricking a million devices, you only brick fifty thousand. And you find the issue before it scales. Lucas: Right. And you document everything. Version numbers, checksums, release notes. Transparency builds trust. If users know exactly what changed, they’re more likely to accept the update. Silence breeds suspicion. Luna: I agree. Clear communication is half the battle. If the update notes are vague, people assume something is wrong. Lucas: Vague notes imply hidden bugs. Be specific. 'Fixed a connectivity issue in weak signal areas.' That tells the user it’s relevant to them. It increases adoption rates. Luna: So, it’s not just engineering. It’s psychology. You’re managing user expectations as much as you’re managing code. Lucas: Always. And finally, think about legacy support. What happens when you release a major version that drops support for older hardware? You need a sunset policy. Give users a year to upgrade. Don’t leave them hanging. Luna: That’s good practice. Abandoning users creates resentment. Even if the tech moves on, treating customers fairly matters. Lucas: It does. And it protects your brand. Okay, before we dive deeper into the technical specifics of OTA protocols, I want to mention something quickly. If these conversations are useful for what you're building or running, honestly, if today was worth a coffee to you, that's the link — buy me a coffee dot com slash fexingo. It helps keep this show ad-free and independent. Luna: Yeah, we appreciate the support. It allows us to dig into these details rather than skimming the surface. Lucas: Absolutely. So, let’s get back to the protocol layer. When you’re choosing between HTTP, MQTT, or CoAP for delivering these updates, what are you weighing? Luna: CoAP is great for constrained devices. Low overhead. But HTTP is easier to integrate with existing CDNs. Which do you prefer for most startups? Lucas: It depends on the scale. For small batches, HTTP is fine. Easy to debug. But for mass deployment, you want something that handles concurrency well. MQTT can be tricky with large payloads. CoAP is efficient but less common in general web stacks. I’d argue for a custom HTTP endpoint with chunked transfer encoding. It gives you flexibility and broad compatibility. Luna: Chunked transfer means the device can resume if the connection drops. That’s crucial for reliability. Lucas: Exactly. Resume capability is mandatory. Network instability is a fact of life. Your update client must be able to pick up where it left off. Otherwise, a brief WiFi drop causes a total failure. Luna: So you’re designing for the worst-case network scenario. That’s a mindset shift for many hardware teams who assume stable connections. Lucas: They always assume stable connections until they don’t. And then they pay the price. So, final thought on this: invest in the update pipeline early. Don’t treat it as an afterthought. Build the redundancy, the security, the telemetry, and the rollback mechanisms from prototype phase. It will save you from nightmares later. Luna: Agreed. Because fixing it after launch is infinitely harder than building it right from the start. Lucas: Right. And that’s the core lesson. Resilience isn’t a feature you add; it’s a foundation you lay. As we look toward the next quarter, with more devices becoming truly autonomous, how do you think the standards for firmware security will evolve?