Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How Vercel Rebuilt Its Edge Network for Sub-50ms Cold Starts
Transcript
- Lucas: If these conversations are useful for what you're building or running, you've probably noticed we don't run ads on any of them. Luna: Yeah, that's a deliberate choice — keeps the focus on the engineering decisions, not on trying to sell you something. Lucas: Exactly. And if you want to support that choice, the simplest way is buy me a coffee dot com slash fexingo. No perks, no tiers — just helps keep the show ad-free. Luna: Alright, back to today's topic — cold starts at the edge. Lucas: So, cold starts. Anyone who's deployed a serverless function knows the pain: your function sits idle, a request comes in, and the platform has to spin up a new runtime from scratch. That can take anywhere from a few hundred milliseconds to multiple seconds. Luna: And for edge compute, where the promise is single-digit millisecond responses, that's a deal-breaker. Lucas: Right. Vercel, the platform behind Next.js and a lot of modern front-end deployments, has been working on this problem for years. Their Edge Functions are designed to run at the network edge — close to the user — but they need to start up fast enough that the cold start is invisible. Luna: What's their target? I've heard sub-50 milliseconds. Lucas: That's the goal, and they've actually achieved it for most workloads. But the architecture behind it is pretty different from a typical AWS Lambda or Cloudflare Worker. Vercel uses a combination of isolate pooling, WebAssembly sandboxing, and a custom runtime called Edge Runtime. Luna: Let's unpack isolate pooling first. What does that mean in practice? Lucas: Normally, when a function is invoked, the platform creates a fresh V8 isolate — a separate instance of the JavaScript engine — loads your code, and executes it. Vercel pre-creates a pool of warm isolates in each edge location, so when a request arrives, it grabs an already-initialized isolate and loads only your specific function handler into it. Luna: So the V8 engine is already running, the garbage collector is ready — it's just the code that's cold. Lucas: Exactly. That brings the cold start down from hundreds of milliseconds to around 10 to 20 milliseconds for JavaScript handlers. But there's a catch: if your function uses native dependencies, like a compiled Rust binary, you can't just drop that into a V8 isolate. Luna: That's where WebAssembly comes in, right? Vercel supports Wasm for those cases. Lucas: Yes. WebAssembly modules are compiled ahead of time and can be loaded into an isolate much faster than spinning up a separate process. Vercel uses Wasm as a sandbox for running non-JavaScript code, and they've optimised the instantiation path so that loading a Wasm module adds only a few milliseconds. Luna: But Wasm has its own cold start cost — initialising the linear memory, setting up imports. How do they keep that under 50 milliseconds? Lucas: They do a few things. First, they compile the Wasm module to native code ahead of time using a streaming compiler, so by the time the request arrives, the module is already compiled. Second, they cache the compiled module in the isolate pool, so if the same function is called again from the same region, the Wasm is already warm. Luna: That sounds similar to Cloudflare's approach with Workers — they also use isolate pooling and pre-warming. Lucas: It is similar, but Vercel has invested heavily in regional pre-warming. They maintain a baseline traffic pattern across their edge nodes, and they proactively spin up isolates in regions that are expecting load. For example, if traffic from Southeast Asia spikes at certain hours, Vercel pre-pools isolates in Singapore, Tokyo, and Sydney before the spike hits. Luna: Interesting. So it's not just reactive — it's predictive. Lucas: Right. They use a machine learning model that predicts request volume per region based on historical patterns, and they scale the isolate pool accordingly. That means even a completely new user hitting a function for the first time might get a warm isolate because the platform anticipated demand in that region. Luna: How does that interact with their partnership with Cloudflare? I know they collaborate on WinterJS, a JavaScript runtime built in Rust. Lucas: WinterJS is actually a key piece of the puzzle. It's a runtime that implements the Service Worker API, but it's written in Rust and compiled to Wasm. Vercel uses WinterJS as an alternative to the standard V8 runtime for certain workloads, especially when they need tighter sandboxing or lower memory overhead. Luna: And because WinterJS is itself Wasm, it can be instantiated inside an isolate without needing a separate process — so cold starts are even faster. Lucas: Exactly. Early benchmarks show that WinterJS cold starts are around 5 to 10 milliseconds, compared to 15 to 20 for V8 isolates. That's a big deal for functions that are invoked infrequently but need to respond instantly — think authentication checks, A/B testing, or personalisation at the edge. Luna: Let's talk about the trade-offs. What do you lose by going with Wasm-based runtimes instead of native V8? Lucas: You lose access to the full Node.js API. Wasm runtimes typically only support a subset — like fetch, request, response — but not filesystem operations, child processes, or common crypto libraries. So if your function needs to read a file or run a database query, you're better off with a traditional serverless function running in a container. Luna: Right, edge functions are really for lightweight, stateless logic. That's a constraint, but it's also why they can be so fast. Lucas: Exactly. Vercel makes that trade-off explicit: if you need Node.js APIs, use their Serverless Functions; if you need low latency globally, use Edge Functions. And they've built a smooth migration path between the two — you can change the config in your next.config.js file and redeploy. Luna: I want to go back to the numbers. You mentioned sub-50 milliseconds for cold starts — what does that look like in a real-world benchmark? Lucas: Vercel published a benchmark last year comparing cold start latencies across platforms. For a simple JavaScript edge function that returns 'Hello World', Vercel measured a median cold start of 22 milliseconds. Cloudflare Workers were around 28 milliseconds. AWS Lambda at the edge was over 200 milliseconds. For a Wasm-based function, Vercel's median was 12 milliseconds. Luna: That's a huge gap. Why is Lambda so much slower? Lucas: Lambda uses Firecracker micro-VMs, which boot a full Linux kernel for each invocation. Even with Firecracker's fast boot — around 125 milliseconds — you still have to initialise the runtime, load dependencies, and execute the handler. And Lambda's cold start includes the time to download the function code from S3 if it's not cached. Luna: So the isolate model is fundamentally faster because it avoids the kernel boot entirely. Lucas: Right. V8 isolates share the same OS process, so the kernel is already running. The question becomes: how many isolates can you fit in a single machine? Vercel runs tens of thousands of isolates per server, each with its own memory space but sharing the OS and the V8 binary. Luna: That sounds like a security nightmare if one isolate can escape and read another's memory. Lucas: Yes, and Vercel takes that seriously. They run each customer's code in a separate Linux user namespace, and they use seccomp filters to restrict system calls. The Wasm sandbox adds another layer — Wasm modules can't access the host's memory directly. And they've invested in a team that does continuous penetration testing on the edge infrastructure. Luna: So the trade-off is complexity on the platform side for speed on the developer side. That's a fair bargain. Lucas: It is. And the result is that developers can build real-time personalised experiences — like dynamic pricing, user-specific content, or fraud detection — without worrying about a cold start degrading the user experience. Luna: What's next for Vercel's edge compute? I've heard whispers about supporting more languages — maybe Python or Go. Lucas: They've talked about adding Python support via Wasm, but the challenge is the standard library — Python's standard library is huge and relies on C extensions that don't compile to Wasm easily. Go is more promising because the Go compiler has a Wasm target, but the Go runtime is still relatively large. Luna: And then there's the question of database access. Edge functions are stateless by design — how do you persist data without killing the latency? Lucas: That's the next frontier. Vercel recently launched Edge Config, a key-value store that's replicated to every edge location. Reads take under 10 milliseconds. They also integrate with Neon for serverless Postgres, but that adds 50 to 100 milliseconds per query. So the trend is to push more state to the edge — think replicated SQLite databases or crdt based stores. Luna: It feels like we're moving toward a world where the edge is the primary compute layer, and the 'origin' is just a source of truth for writes. Lucas: Exactly. And Vercel's bet with sub-50 millisecond cold starts is that developers will adopt that model faster if the friction of starting up is gone. I think they're right.