Latest / The Experimentation Edge
Why Twilio ships on signals instead of significance
Summary
Wanli Lau spent eleven years at Expedia Group, most recently running consumer product analytics on a program that shipped more than 1,500 A/B tests a year. Five months ago she moved to Twilio to lead global R&D analytics. In this episode she walks Ashley Stirrup through what carried over and what did not.
The centerpiece is a test that looked like a win. At Expedia, an account sign-up takeover on the front page drove conversion sharply up, and quietly harmed return rate and engagement across every geography. That result is where Wanli's insistence on guardrail metrics and a trade-off…
The skinny
The skinny isn't ready yet — notes appear once the transcript is processed.