Thirty Winners and a Flat Line: Bandits and Online Experimentation

Deep End Era. Engineering with Sankalp · Oct 28, 2026 · 1h 1m

A fictional online grocer runs sixty A/B tests in a year, ships thirty one "winners" claiming 22 percent uplift, and sees revenue per visitor rise two percent. Product lead Ruth Okafor and statistician Marco Bianchi find out why. Part fifteen of The Deep End's Machine Learning series covers bandit algorithms from Thompson's 1933 paper to contextual bandits, and the experiments that decide what you see online, with every way they go wrong and the unglamorous rules that fix them. Cross-references: parts 7 (probability), 9 (recommenders), 10 (learning to rank), 14 (RL), 16. Larder, Ruth and…