Attention, a Conveyor Belt and a Lot of Practice: Transformers from First Principles

Deep End Era. Engineering with Sankalp · Oct 28, 2026 · 54m

Margit Solberg is training a small transformer to predict the next event in fifteen years of gas-plant alarm logs, and reliability engineer Oyelaran Pike wants to know what the thing actually does. In episode one hundred and fourteen we build a transformer from first principles. We start with recurrent networks and their two problems, meet attention in 2014 and its takeover in 2017, then turn alarms into tokens and tokens into points on a map. We run attention by hand with three events and two-dimensional vectors, see why the scores are scaled, why the model must not peek at the future, and…