
Speculative Decoding: Theory and Implementation
Expert guide to speculative decoding for accelerating LLM inference: draft model selection, acceptance rates, rejection sampling, tree-based speculation… From the article "Speculative Decoding: Theory and Implementation" by Synor, published on Misar.Blog.
In this episode:
0:00 Speculative Decoding Accelerates LLM Inference by Using a Draft Model
0:49 At Acceptance Rates of 70-90% Speculative Decoding Yields 1.5-3× End-To-End
1:24 The Speedup Formula Is (Γ × Acceptance_rate)/
1:59 Rejection Sampling Guarantees the Output Distribution Is Identical
2:32 Draft Model Family Matters
3:05 Optimal…
The skinny
The skinny isn't ready yet — notes appear once the transcript is processed.