The Road to the Warehouse: FlashAttention and GPU Kernels

Deep End Era. Engineering with Sankalp · Oct 28, 2026 · 1h 1m

Fictional performance engineer Henrik Solberg finds his seismic transformer using a quarter of a GPU that promises nearly a thousand trillion operations per second. His colleague Mirela, a former finite-element specialist, diagnoses it in thirty seconds: a world-class reactor with a single-lane road to the tank farm. In episode one hundred and seventeen we follow GPUs from graphics cards to matrix engines, tour the H100's registers, shared memory, L2 and HBM, and use the roofline model to see why a vector add runs at a fraction of a percent of peak while a big matrix multiply runs near it. We…