The Conversation Between Chips: All-Reduce and the Networks That Train Giant Models

Deep End Era. Engineering with Sankalp · Oct 28, 2026 · 1h 2m

At two forty in the morning, cluster reliability engineer Tomas Reyes watches a training run's step time creep from four point one to four point six seconds across twelve thousand GPUs. In episode one hundred and two we follow the conversation that turns separate chips into one machine: the all-reduce. We start with a hundred accountants and their seven-billion-line ledgers, learn the vocabulary of collectives from MPI, and price every message with two numbers, latency and bandwidth. We meet the ring all-reduce and why it is provably bandwidth optimal, then why it breaks at scale and how…