Serving the Machine: Speculative Decoding, Paged Attention and Disaggregated Inference

Deep End Era. Engineering with Sankalp · Oct 28, 2026 · 53m

At 3 a.m. the assistant is fast. At 9 a.m. it stutters. Ines Halvorsen and Rafael Mbeki (both fictional) find out why, without buying a single new GPU. It covers why decode leaves the arithmetic units idle, how a draft model can guess ahead without ever changing the output, why speculation can hurt at peak load, how an operating-system idea rescued serving memory, what goodput measures, and why the newest systems run prefill and decode on different machines. Cross-references: ep27 (KV cache), ep34 (inference economics), ep114 (transformers).

Chapters

00:00:00 Cold open: the dashboard at nine…