Latest / The Experimentation Edge / The four questions Early Warning asks before any A/B test
Transcript
- The Experimentation Edge - Priya Singhee === Priya Singhee: [00:00:00] Hello, and welcome to today's episode. I'm excited to welcome Priya Singhee, VP of analytics and data science at Early Warning. Welcome, Priya. Thank you for having me. Yeah, I'm excited to have you on the show. Early Warning isn't a household name, but they do some pretty important stuff. Maybe you could start by just telling us a little bit about Early Warning. Yeah. So Early Warning it's been in business for about 30 years. We are owned by the seven largest US banks to fight fraud. So it's really a consortium that collects data on financial information on the consumer accounts across US, and we build machine learning models on top of that to identify identity and payment risk fraud. We are also the operators of Zelle. Zelle did a trillion dollars in paid transaction processing volume last year. We also have a new early business called Paze that is very similar Ashley Stirrup: very similar Priya Singhee: to a digital wallet, if you will. So yeah excited to be part of Early Warning. Yeah. Sounds like they're doing some incredible [00:01:00] stuff. A trillion dollars is nothing to sniff at. And earlier in your career, did you start at Wayfair Ashley Stirrup: Did you start at Wayfair Priya Singhee: in AB testing, or when did you get your start in AB testing? No. I've been at a lot of consumer tech startups, and I want to say, if you've ever been part of a growth team at a consumer tech startup, you are by default doing a lot of Yes. Yeah, that is definitely a best practice. Maybe you could tell us a little bit more about your time at Wayfair in particular. Yeah. So at Wayfair, I was the global head of storefront product analytics which meant from the time you search for a couch on Google to the time you check out, we are constantly trying to optimize that customer experience. And as part of optimizing that customer experience, we are testing every bit of that funnel, as you can imagine. From the landing pages to how you search for things, to how we check out. All of that is all being optimized and that's where we ~got-- ~get our biggest learnings and improvements from. Yeah. Terrific. And today's episode will focus a lot [00:02:00] on your learnings from that time. Maybe you could start things off by talking a little bit about your decision framework on when to A/B Yeah. So today I'd love to chat more about, ~h- ~how should... Let's say you are trying to set up your experimentation team. ~How should you put-- ~how should you think about experimentation as a platform, experimentation as a culture? And my learnings over the past, I want to say, 10 years or so, having done this at different companies, So the first thing I think before you even start any A/B test is really have a decision framework. When do you A/B test? And you have to ask yourself a couple of questions before you can launch into one. The first thing you should ask yourself is: Can you actually truly randomize? Because it truly only works when the randomization is clean. Like who gets treatment versus who is going to succeed anyway. If you can't cleanly randomize you don't really have a case for a standard A/B test. The second is: Is your effect size even plausible given your traffic? And this is something I've seen time and again. Most people's [00:03:00] guesses... Ashley Stirrup: Like, Priya Singhee: if you only get 5,000 checkout sessions a month, you're hoping for a, I don't know, half percent lift in conversion. You'll need months of data, and this is so important to get it right at the beginning. The third thing is: Is your change reversible and cheap to test? So oftentimes when you're trying to launch a huge thing, it's again not a... should we change our return policy? ~Th- ~that's going to get contaminated so much, and it makes a clean split really hard, so I wouldn't want to A/B test that. Finally, and this is again something I really lay importance on, is do you really have a hypothesis or are you just poking around? We'll just test something and see. A very bad idea. That's a red flag. A test without a specific falsifiable hypothesis is just like a fishing expedition in my mind, and frankly, this is where I've seen most of the So to me, it's really not about should we test, but it's more about can you test this validly? Yeah, that's a great question. When have you [00:04:00] run into challenges around being able to randomize? A lot... the most common ones are if the seeds are not randomized, if there's going to be cross-pollination tests that you should run. And if... It's... Another thing is what randomization unit are you using? Is it session? Is it user? Is it device? And I've often seen teams, they will choose a typical unit for randomization, but do a experiment readout on another unit. ~So wrong. So- ~Yes ... you have to be careful about all those Yeah. That's a very hot topic that whole thing of picking your randomization unit. had Khan Academy on, and for their products that are more designed for classrooms, it was really important to randomize at the classroom level so you didn't have the teacher dealing with half the students seeing one experience and half the students Do you have any best practices on how to set up a test? Yes. Sure. My first one would be, like I said you have to write down the hypothesis. For example, if you are trying to reduce [00:05:00] checkout from four steps to two steps what is it going to do to your conversion rate? Maybe it improves it by one point five points. Basically, I'm trying to force you to write it down because not only are you thinking about the mechanism, you're also going to check yourself against this later. It has to be very specific. And frankly, this will avoid a lot of the p-hacking that I've seen if you're very specific. The second piece, we already talked about this, randomization unit. You have to be very clear on what you're ~randomiz- ~randomizing on. The third one, I would say people should be Most people are very confident about the primary metric that they're trying to test on, but have you set up your guardrail metrics? That could... For us at Wayfair, for example, I remember it was page load times or exit rates. You can't afford to hurt them or, return rates. And for other ~com- ~others where I was especially on the email stuff, unsubscribe rate. ~A-and ~again, you have to be very deliberate about observing this. Don't just observe it in that one-week window. Look at it over a period of time because it [00:06:00] generally shows up later. And the fourth, I can't stress on this enough, the power and duration. You have to calculate this in advance. You can't be doing this as a afterthought. ~A-and ~finally is the pre-registration of the analysis plan. This is something, again, I've seen. You will start with your power and duration on a particular sample and a detectable size, but nevertheless, you'll then start wondering, "Oh, how would it do on new users? How would it do on just the mobile traffic?" You never pre-registered for that, so you can't be reading out your experiment results on those. So to me, these are all best practices that I would want to see on a test setup. Yeah, that all makes a ton of sense. One of the things that~ a, ~a previous guest said that I thought was just brilliant assume your test Ashley Stirrup: is Priya Singhee: going to fail, and then ask- ... yourself what data will you wish you had. Yes. And so then you make sure you design it with that in mind, and sometimes it might change your sample size if you wanna slice on a certain dimension and you [00:07:00] realize you don't have enough power to actually have reliable data on that, yeah. I almost want to see... so by the way, since we brought that topic up, 85 to 90% of tests fail. This is a very well-known statistic, yet in most companies there's so much pressure to win and hunt for that win. You have to understand the AB test is a learning agenda. If you've learned something, it's good enough. You... If you're winning too many, I would be very skeptical, because really 85 to 90% of them are supposed to fail. And so in any pre-registration analysis, what I want to see is what would you do if it succeeded? What would you do if it failed? What would you do if the primary metric was good but this guardrail metrics are going down? Like, all of that you should be very clear and get maybe your VP level approval so you're not chasing a ~ba- ~a loss, saying, "Oh, I'm still so desperately trying to prove it to be a win." To me, those are all things that I've learned if you're trying to, set up a test in the right way. Yeah. And next week we're gonna have the Philadelphia [00:08:00] Inquirer on a webinar at GrowthBook, it's interesting 'cause they used experimentation to create kind of a common, framework for evaluating the success of experiments. 'Cause they had an advertising team that kind of cares about ads and revenue, and then they have the content team that cares about engagement. And sometimes, You put an ugly ad on that page. Exactly. Yeah. You've got conflicting goals there. And so if you can Ashley Stirrup: kinda Priya Singhee: design the experiment as a joint team, then it helps really cut through all that and it helps people look at it from the company level, not just the yeah. I remember was it a beta or random where, if you start putting in a lot of sponsored ad products Ashley Stirrup: products, Priya Singhee: it will hurt some other team. So even when you are trying to, let's say, it was the ads team Ashley Stirrup: Let's say Priya Singhee: that was trying to put more of these sponsored ones, but it would have hurted the landing page or the search team's metrics. So again, you have to be very careful, and as you're running thousands of concurrent tests, try to understand who's running what on your [00:09:00] page and what might get impacted. Yeah. So you've mentioned some of the pitfalls of testing. Do you have other examples you wanna share? Yeah. The common ones that I've seen it... So we talked about, the randomization unit. The other one that I've seen often is seasonality and novelty effects. Reading it so reading it too early or the nove- you read out a novelty effect and it frankly just dies down after some time. So always trying to understand that during a proportional spike, if it doesn't generalize once you roll it out, you don't have a win there. So th- that would be my second one. The third one is ~j- the h- ~this is very nuanced, but a hidden heterogeneity behind an average effect. This is something, again, I have seen so many times. A change can be flat or slightly negative or average, and you're just so tempted to call it a win, but still really it w- could probably just be for that particular segment or the average is hiding it. So I truly think that you have to go deeper into when [00:10:00] you're ~ana- ~doing your analysis, that if you're just seeing Ashley Stirrup: flat Priya Singhee: or slightly negative changes, be skeptical. Dig deeper. Yeah. And then the biggest one is the multiple comparisons problem, right? If you're running thousands of experiments and you're ~c- ~checking all the time you either go for a Ashley Stirrup: a always valid Priya Singhee: P Ashley Stirrup: values. Priya Singhee: I know some experimentation platforms do this, so they'll Ashley Stirrup: they'll show Priya Singhee: you a confidence. If you're checking today, tomorrow, how is that confidence level improving? However, if you are going to do the multiple comparisons test, you are ~al- ~you are always going to have very high false positive rates. So it... to me, this is not something we should be doing, and Ashley Stirrup: we Priya Singhee: should be Ashley Stirrup: cognizant of that. Priya Singhee: Yeah, those are the big ones that I've seen. There's another one, Simpson's paradox, but I've not seen it as much on the experimentation Got it. Do you have any examples of when you saw a novelty effect at one of the companies you worked at? I have seen that actually. And typically it is with new user experimentation where you're trying to improve either the new user onboarding [00:11:00] flow or getting a new user to sign you know, put their card on file. I have definitely seen that novelty effect ~i- ~initially and then dies down. Because either we were running it with a promo or something and then it dies down. Got it. Yeah, 'cause I think a common challenge Ashley Stirrup: that Priya Singhee: a lot of companies you'll stack up a bunch of wins and then you get to the end of the year and you don't see the overall growth you expected. And I think, if you listen to Ronny Kohavi or some of the other- Mm-hmm ... ... experts in the industry, a 20% haircut, is appropriate for a variety of But I think some organizations see much worse they don't see half the lift they expected, so then clearly there's something else going on. Do you have any thoughts on, why people are experiencing I do actually. My thought process is it really depends on where you are in the journey. If you're already very mature as a company, you are going to see that number really the experiment... it's like you're maximizing the local min- local points, right? You... And [00:12:00] only so much you can do. You have to move to another hill, but you're constantly trying to, Ashley Stirrup: k- Priya Singhee: Optimize on that hill. So that's one reason. Yeah. Yeah, that's a good And so do you have any can you talk to the learning agenda around how to avoid p-hacking when the test fails? Yeah. So to me the in the learning agenda, you have to fix it before any test launch happens, right? And that's why we keep coming back to pre-registration. It's really where you are documenting as a team what is an analyst checklist, right? Your hypothesis, your mechanism, your primary metric, your exact statistical test you will run on it, and for which segment. Any subgroup analysis you plan to run on, right? To reduce those false positive This is the biggest thing that I've seen. When tests are going n- sideways, people come, start coming to the analysts, "Hey, can you look at new user? Can you look at mobile segment? Can you look at just this segment?" And to me, ~that's, that, ~that's been a very common problem that I've seen, and you should [00:13:00] have a pre-registered kill criteria. What would make you say Ashley Stirrup: out loud Priya Singhee: that this idea was wrong, we're not shipping it, and we are now going to hunt for a different subgroup that it might work The biggest thing for me in a learning agenda is what did you learn from this test? I don't care if it won or loss, how would you describe what you learned from this Yeah. Yeah, I think that's, it's such a shame with cultures where there's a lot of pressure to ship winners, that, the whole point of A/B testing is to learn, and if you're putting all the emphasis on the winners, you're probably not putting enough emphasis Totally. ~A-a-and ~I think the companies where I've seen they do it well is they're actually logging all their tests and logging the learnings from each, so you continuously build on those learnings. Ashley Stirrup: A- and Priya Singhee: to me, that is a world-class organization where, there's this reinforcement loop going on, "Oh, I tried, tested this," and it's for everyone to see. And that's how you learn as an organization, Yeah. And what I've heard across a [00:14:00] variety of different guests that, you kinda wanna understand your buyer's journey understand where removing friction adds value and where adding some friction or adding some context- ... or education can add value. And so as you start to do those things, you learn, and then that... you can help use that context for figuring out the next piece of friction you might add or remove. So I think that can be really powerful. How about when it's time to roll out a winning test? Any, any thoughts there? There my biggest learning has been how do you ramp gradually? Because it is... I've seen where we've gone straight to 100% only to realize there are some ~prode-~ Ashley Stirrup: Production Priya Singhee: challenges. We've hit some snags and snafus. So to me, going on a slow ramp, 25, 50, 75, does basically two things. You get to see how the backend service is behaving as you're ramping up, and it helps you watch for your guardrail metrics as you keep ramping. Yeah ... so those are the two things I want to watch [00:15:00] out for in the rolling. And then this is something I wish every company could do have a discipline of holding a long-running holdout. Very few companies do it because there's all sorts of contamination going on. But especially I want to say for marketing, if you could have a long... if you could have this very tiny group not touched by anybody, because marketing problems are the hairiest ones, I think, the way they are trying to do the, these tests. So if you could have a long-running holdout, I'm all for it. And this I mentioned, the third one I mentioned earlier, where you're watching guardrails o-on a longitudinal basis, not just on launch. I've been hit by those as well, where we didn't do that and- We were hit by guardrails dipping. Yeah. Yeah. That's... Those are great call-outs. Yeah, ~one, ~one thing I think is really interesting we've been doing with our customers is just giving them a lot more control. So ~the- ~one of the nice things with feature flagging is- you can roll a feature out into production but only have 5% of users see [00:16:00] it. Ashley Stirrup: So you start Priya Singhee: To see the impact of that feature in a fully, full production environment, and then gradually roll it out, tied to guardrail metrics so you can auto roll it back. Yep. I think those things are really critical, especially in today's world of AI, where everybody can ship so much faster. You need to automate all those safety And so what tools would you recommend or how would you recommend people think about it in situations where maybe they don't have enough power or it's- just a difficult situation to do AB So there are lots of causal inference methods, but I want to say it's very tricky in how you set these up and how you do a readout. And frankly this is something I'm trying to learn personally as well and get better at. But what I've seen in the past is people trying to set up synthetic controls, but there's so many caveats and assumptions that go with you trying to be able to set up the right synthetic control. Because as much as we [00:17:00] want two units to be stable and independent, it's very hard to find those. So we've tried where, you couldn't AB test, you use synthetic control. The other one is differences in differences, where instead of just doing a pre/post on a test, let's say you have to launch to the whole population. But maybe, ~A- ~actually, I think we did this at Voodoo, where we launched it just on Roku and then compared Roku users to a very similar group in another living room because I couldn't compare a Roku user to a Samsung user, for example. They're very different lifetime value. How do you choose stable units and launch it differently across those two units? So yeah, those were the typical causal tools that are available. Then there is the the surrogate index method. This was typically used also where you're using basically Bayesian outcome. So what you're doing is you have a real randomized experiments that you ca-... but the outcome is very slow to materialize. So what you can do is you can train a model on a separate observational data set to match [00:18:00] short-term behavior to the long-term outcome, so that you can then apply that mapping to your experiment short-term behavior and forecast that long-term. So that- Yeah ... that's another common way of doing it. But to me, this is the most interesting part of our~ cau- ~field, which is causal inference, because not everything can be AB tested Yeah. Yeah I particularly think the whole idea of measuring some short-term metrics, especially often you say, "Okay, what's the thing closest to the feature I've launched?" Maybe long-term I wanted to increase conversion or revenue, but short-term I'm gonna measure engagement. And I think it's very powerful to be measuring engagement, but sometimes it's hard to know the true correlation between engagement and revenue, Oh, yeah. And then you move to a ~long, ~long-term retention, customer lifetime value, things like that, and it's even harder. So I think there's a lot of opportunity for us to continue to innovate in that area but I have a lot of respect for the, the teams that work to that, build the models- Yeah ... and measure that, that [00:19:00] type of thing. I want to say it's very hard. All this extrapolation of trying to measure short-term impacts and extrapolating it to long-term GMV, which is what every team is asked to do. Yes. I have a lot of respect for that too. It's a lot of work and sometimes we... it's, we get it wrong. Yeah. That's absolutely right. Did you have any other best practices that you wanted to share with the group? I wish everything could be A/B tested. I'm a huge A/B testing fan because what you don't realize... one last thing I'll leave people with, imagine if you didn't A/B test, and if I'm telling you 85 to 90% fail, you'd actually be losing revenue. Meaning, if you were just simply to launch everything without testing, imagine the losses you were... you missed. So- Yes ... my sincere request is people do more A/B testing, not less. Yes, I totally agree with that. If you think about a let's say you ship 10 features and maybe two are winners and six are flat and two are losers if you can just avoid those two losers- Yeah ... and then... [00:20:00] and turn maybe one of the winners... or sorry, one of the flat ones into a winner, suddenly you're at plus three versus flat. Exactly. And so- I wish people would take that mindset, then they wouldn't be chasing a win every time. That's right. Losses can be useful. Yes. Yes. I've had some guests talk about loss avoidance or savings basically- exactly ... from not shipping things, Priya, thank you so much for being on today's episode. You shared a lot of great best practices with the audience. I'm sure they got a lot out of it. Thank you so much, Ashley, for having me on the show. Really enjoyed it. Awesome.