Latest / The Experimentation Edge / Twitch on why false negatives kill product ideas

Transcript

Twitch on why false negatives kill product ideas · The Experimentation Edge

• ashley_stirrup-db5c79db-3dfe-4098-9a03-1-CFR === Arun GUEST: [00:00:00] false negatives are the killer. As an example, right? Actually, what will happen is a product manager is gonna propose a product idea. We're gonna run an experiment, and if the result is a false negative, it doesn't matter. The false negative or the true negative is something that only a few people in the team would have visibility into. The exec team's gonna look at it and gonna say, "Oh, you guys tried it. It didn't work. Let's move on." And the worst thing is it gets institutionalized that, "Oh, we did try that intervention and it did not work." Welcome to the Experimentation Edge, where product managers, data scientists, and engineers talk about how they make smarter decisions. I'm Ashley Stirrup, the chief marketing officer for GrowthBook, and in each episode, I'll sit down with an executive to unpack how they use experimentation and A/B testing to make better decisions. This show is sponsored by GrowthBook, the open source [00:01:00] experimentation platform leader. Now let's jump in and get started with our next guest. Ashley HOST: Hello, and welcome to today's show. I'm excited to have Arun Bodapati, director of data science from Twitch on the show. Welcome, Arun. Arun GUEST: Thank you, RC. Great to be here Ashley HOST: Arun has a deep background in data science with experience at Schwab, Uber, and Twitch. Arun, could you tell us a little bit about how you'd compare experimentation at Twitch to your time at Uber and Schwab? Arun GUEST: Yeah that's a great question. I think let me start off chronologically. So firstly, actually great to be here. Thank you for the opportunity. And I currently lead product data science at Twitch. We're a fairly large team of data scientists, machine learning engineers economists and computer scientists. And, our primary focus is to help make Twitch a better platform for our viewers and creators. That aside I'll start with Schwab. Schwab was many years ago. With regards to [00:02:00] experimentation, this was right around the time when Schwab was investing quite a bit of marketing dollars in acquiring new users. There was not a lot of experimentation in the traditional sense. They had, their websites, and the app was just getting built up.- So there was very little experimentation with the product itself. But there was a fairly I would say extensive experimentation with marketing. They had a fairly large marketing budget to spend, and it was fairly state-of-the-art as far as marketing employing experimentation to figure out where those marketing dollars go. Ashley HOST: I was just gonna ask how what was your role like at Schwab versus the other roles? Arun GUEST: Yeah. So I think Schwab was-- I was primarily involved with marketing. We had done a bunch of stuff with marketing data science. So I think the, there were two key problems at Schwab. One was figuring out almost developing the marketing mix models so that you figure out [00:03:00] they had almost $100 to $120 million kind of budget. You had to figure out where those dollars go broad strokes. And then there were a lot of what are called incremental spend questions, which is, once those dollars are allocated whether it's on a quarterly basis or a half-yearly basis, there would be target optimization where you take those incremental spend, if you will, from campaigns that probably did not run to full effect. You take those dollars and have to figure out on the fly new targets and so on and so forth. And then there was downstream attribution exercises as to, what is all this acquisition cost doing relative to LTV? And Schwab had, Schwab and banking in general their LTV is relatively easy to model 'cause, we all are very sticky with our financial services products, so makes the problem a little more tractable. So in that sense that industry is well built or, it's a good vertical for your [00:04:00] canonical LTV to CAC kind of models and calculations. And thereby upstream, you could do a bunch of experiments to say, "How do I optimize this CAC and, lead to better LTV?" Ashley HOST: It's interesting. As a marketer myself, I know just how hard it is to calculate attribution and, yeah, where do you put that incremental dollar can be really challenging. I'm sure that was extremely meaningful work you were doing there. Arun GUEST: Yeah. Yeah Ashley HOST: And I'll bet it was pretty different at Uber. Arun GUEST: Uber was very different. Uber was and still is a digital native company, right? It was-- It's a mobile native company. And was a very different environment and something that as a data scientist absolutely loved it. The experimentation stack over there was extremely mature even by the time I'd gotten there. They had back then what was called Morpheus. That was the experimentation stack. And recently they moved on to Citrus over the past three years. I... No, maybe more than that, maybe five years, they've moved on to Ci- I, I-- And [00:05:00] I'm not, I'm not, I don't work at Uber anymore, but in either case, they-- the business itself was fairly optimization heavy. It was real-time matching of riders and drivers. And so the marketplace itself demanded a fair degree of empiricism in approach to the business itself. And it was fantastic. So the tech stack was ready-built for data science. And culturally, Uber was very different in that execs down onwards downwards, you would have a very heavy reliance on experimentation to make product decisions, business strategy. A lot of it depended on experiments and experimental data. So was a great learning for me, and I'd led different teams over there. But largely this idea that experiments can drive product strategy and improve outcomes in a very measurable way was great Ashley HOST: Yeah. That's interesting. I never thought of [00:06:00] Uber like that, but I would imagine it's not a surprise that experimentation was so important 'cause it's such a dynamic marketplace of both the buyers and the sellers or the people providing the rides time of day, all these different factors. And no human can say, "Oh, we should raise prices 10%," right? You just can't do that, Arun GUEST: that's right. That's right. Yeah. A lot of it is algorithmic. Now there's some heuristics and both from a legal standpoint and so on and so forth, but largely it's algorithmic as to how the matching works and there's heavy reliance of experimentation, not just in the marketplace side of the house, but in every function. Like you go to customer support, you go to operations, wherever you go to there's experimentation is is very native in the culture over there Ashley HOST: Yeah. So I bet things were a little different when you came to Twitch Arun GUEST: Twitch is a different, in some degrees it's very similar in that, it's a two-sided platform. We connect viewers to creators. And, in that sense it's similar to [00:07:00] Uber. But there are... it's a very different business altogether. It's fairly unique in that, we are very focused on live streaming and making sure that viewers and creators form communities while there's a live stream going on. So fundamentally very different business. Experimentation at Twitch it always existed in bits and pieces. Engineering teams would use it. Some engineering teams would use it to drive feature adoption and so on and so forth. But it was never a consistent approach to experiments to drive product strategy and how to think about impact that's being landed and yeah. There was never a holistic approach to it. And largely as a result of that the stack was also not as mature as Uber had it. But slowly we, we brought in Eppo, which was it's primarily for the measurement side of the house, just as a dashboard. And that changed a fair deal of culture where earlier [00:08:00] we would, data scientists would run experimental analysis in Python notebooks. It was extremely hard to do any peer review. And There was a lot of inconsistencies once you started went digging in those notebooks. And so once we brought this in it's less about the specific platform and more about just culturally adopting a standard across the company and saying, "Hey, this is where we're gonna look at experimental results, and this is how we'll measure." Just homogenizing all of this brought a lot of hygiene around, around experimentation. And then that slowly had an impact on communicating with the executive team. And it's, there's a lot of earned trust in that process. 'Cause it's one thing to say we do experiments and then, all you need is one misconfigured experiment to, to say, "You guys don't know what you're talking about." Ashley HOST: Yeah. Yeah, trust is just so important and having a consistent way everybody can look at the metrics and then just shared learnings a-across the organization Arun GUEST: Exactly. Ashley HOST: One of the areas you said you [00:09:00] worked on was looking at pricing there at Twitch. That was a new Arun GUEST: Yes. Yes. So pricing was one, area, where we used, a couple of experiments. So the problem statement was, we had to change prices across our, subscription products. We hadn't done that for a very long time, and, we had been fairly resistant to increasing prices even as, the world went through a period of, fairly large degrees of inflation, right? post-COVID, we went through inflation, but we as a company were very resistant, primarily rooted in the sentiment that, "Hey, we don't want to hurt our viewers, we don't want to hurt our con- creators," and so on and so forth. But the underlying hesitation was around, oh, what would that do to revenue, not for Twitch in itself, but, for our creators. Would creators lose revenue if we raise prices, right? Ultimately, we are a platform where we only make money if our [00:10:00] creators make money. Otherwise, we, so that we had to and this is a good point to raise. Unlike Uber, which has real-time elasticities being computed, we don't have that, right? We, you could think of us as Netflix, where, Netflix is not going to raise your prices, on the fly. And so we had to figure out what the elasticity was. and because we had never done this before we really didn't know what an increase in price would do. so we began to and the exec team wanted that data, right? The initial models that we built suggested that, raising prices could be extremely harmful to creator revenue. And but then, that was done-- the data was based in a timeframe when there was not, we were not going through that inflation, right? So this was now and that was then, so how do you actually generate data for this? So we approached it through a geo-fenced experiment where we said, we have some geos which can be matched with other geos, and say [00:11:00] we raise the prices here and then in the back end use causal inference to figure out what is the true elasticity. Laws of economics, you raise prices, of course you're going to lose some units, but can you actually make it up with the price increase? And that process of just demonstrating through data and experiments what the elasticity was and then that actually translating into the balance sheets and, into our financial reports gave everyone a lot of confidence in what in how science and experiments could actually inform a very critical component of what Twitch does, which is which is provide these subscriber services, subscription services for our viewers and creators. Ashley HOST: Got it. Arun GUEST: I think, yeah Ashley HOST: and so one of the things you were doing is what? Picking different countries or states and then having one as a control and one as a variant? Arun GUEST: Yeah, different countries. A good example is we we started with Great Britain, and Great [00:12:00] Britain was the UK, actually UK and Ireland, and that was comparable to the US and Canada to a large degree in terms of viewer composition and and so we, we got a good sense of what the elasticities could be in English-speaking markets. We started doing that by various language/time zone. Because the best way to think about it is you'll get a good read for-- from UK and I to the East Coast of the United States, because they're all in a - similar time zone, especially for Twitch, there's a large population which views it, say, after 5 or 6 PM. And so it's late night in the UK and early evening in, in, say, New York City. So you get a good read for what the effects could be because those populations can be matched and, So we did all of that. And then slowly, we moved to, another good examples are the German, Austria, and Switzerland markets, where they're all fairly homogenous. They watch similar [00:13:00] creators. And not just in German. They do watch a fair bit of English content as well. But the viewership mix is similar, so you could actually geofence some of this and do this. And pricing, Ashley HOST: you had a time zone element to it as well, right? 'Cause you're-- it's live stream content and Arun GUEST: That's right. Ashley HOST: Yeah, making sure you're comparing similar types of users viewing similar content's very important. And what were some of the takeaways from the experiments you did? Arun GUEST: The experiments were no surprises. Just econ 101. We raised prices we have to figure out what is high enough that the increase in price is good and that that it was net-net accretive to the business. I think the one surprise learning was we also have gifted subs which you know you should think of gifting. It's a very unique Twitch behavior where-- And it truly speaks to the sense of community on Twitch where someone in the community will gift subs to the rest of the viewers so that they also have all the benefits of subscriptions, generally [00:14:00] ad-free, and they get emotes and badges and so on and so forth. And generally those are bought not just in ones but, some folks buy 15, 20 of them. And we began to play around with promotions because the elasticities were suggesting that, oh, you raise prices on basic subscriptions, you're gonna lose some of these big gifters if you will. And so we played around with promotions. There was a lot of hesitation around promotions in the beginning. Again, this is and so once we started to demonstrate through experiments that promotions can actually drive incremental revenue it made it made a lot of sense and these-- the thing that I would distinguish is culturally the shift that happened was firstly, experiments were thought of as something you do and you finish and you move on versus experiments are a state of continual improvement and learning and that was the big cultural shift that we went [00:15:00] through. That, these are not some features you build for an academic exercise but these are actual product instantiations. You're making changes to fundamental user experience because, as an example, I'll take promotions as an example in the past we might have done an experiment where we would just change it for once and then that's it. You-- We ran an experiment and then we would move on. And the systems were built as such, right? It was built for a single experiment. Whereas we changed the paradigm and we said: Hey, listen, this is a lever that actually helps us, trade off profitability a bit if we wanted to for for increased sales or vice versa. And so that lever has to be built and the, we can call it an experiment, but essentially it's a measurement technique to figure out the efficacy of that lever. Ashley HOST: Yeah. And I think about it from the business side, and you took a business that was probably a little bit religiously afraid to change [00:16:00] prices, and then gave them this new insight. so it must have had tremendous visibility across the organization and been a huge change in terms of the way experimentation was looked at Twitch. Arun GUEST: It had a huge effect. It had a very huge effect on the exec team, more importantly on the ICs, on engineers, product managers. And now that whole area is so mature that, it's it's in a very different place where, you know, that they're talking about bandits and so on and so forth. It's gone from, thou shall not touch any of that to a very mature state of using empirical experiments to drive this. So it's it's fantastic. Ashley HOST: Yeah, no, I c- I can imagine how special that kind of experience would've been. Arun GUEST: Yeah. Ashley HOST: You've had a lot of experience setting up experiments and understanding how to make experiments trustworthy. Are there any kind of best practice gotchas you'd recommend that people double-check as they're building and deploying experiments? Arun GUEST: Yeah, I think some of the key things [00:17:00] is I always nudge product managers, engineers, and data scientists to spend more time ahead of, pushing the play button, making sure that, make sure that you're clear on the enrollment logic, write in plain English what the hypothesis is, and, have a hypothesis and an intervention that that are very important. Do not try to optimize ahead of the experiment. You can always optimize later. Don't go for, if the intervention in plain English is very weak, just don't do the experiment. You're just wasting time. And then there's a lot of experiment hygiene, which it's extremely difficult for some reason to codify into systems, but one has to be, this is where we need much more hygiene and enrollment logic. What is the actual trigger? Are the events that underlie the trigger reliable? The client-side events, especially on mobile devices, are notoriously unreliable. So how do you [00:18:00] make sure that you have a very reliable trigger so that you do not end up with any of these, do not end up with the false negatives? 'Cause false negatives are the killer. 'Cause as, as an example, right? Actually, what will happen is a product manager is gonna propose a product idea. We're gonna run an experiment, and we get a result, and if the result is a false negative, it doesn't matter. The false negative or the true negative is something that only a few people in the team would have visibility into. The exec team's gonna look at it and gonna say, "Oh, you guys tried it. It didn't work. Let's move on." And the worst thing is it gets institutionalized that, "Oh, we did try that intervention and it did not work." And so yeah, one has to be very careful in setting up these experiments to avoid those false negatives. Ashley HOST: Yeah. Do you think false negatives are worse than false positives or there's more risk of false negatives? Arun GUEST: There's more risk [00:19:00] of false negatives. False positives I worry about them, especially when the effects are large. I make sure that we have a clear mechanistic understanding, right? Because if I do get a positive I want-- the numerical result is less interesting than actually describe in plain English what the user behavior was. 'Cause you're trying to ultimately engender some different behavior from users, and what is that's happening? If you just don't have a clear understanding of that, then there's some statistical error going on or and that we need to caution against Ashley HOST: Yeah. I think in in either case, doing the analysis, try to understand the user behavior and how the feature impacted the user behavior so that you've-- you basically can triangulate in on what was the real learnings from this. Arun GUEST: That's right. That's Ashley HOST: when you don't think about false positives and false negatives, maybe you don't think that kind of additional analysis is important, but it's really critical. Arun GUEST: It is yeah very critical. And yeah, generally with positives, I have two [00:20:00] quick follow-ups that I ask of the team. One is give me a mechanistic understanding of why you think what happened, and then tell me which segments are contributing most to this lift that we see. Because it's generally a very small population which which is actually contributing to this and then we can have some subsequent product iterations focused on that. Yeah. Ashley HOST: That's a great, that's a great takeaway. I'll make sure , we note that for the audience. The other thing you mentioned was like around enrollment triggers and Arun GUEST: Yeah Ashley HOST: that can be very tricky and that you think you've created the right enrollment trigger but it doesn't really line up with the hypothesis. Is that right? Arun GUEST: That's right. That's right. Enrollment triggers are, Yeah, people-- , there's a fine balance. Sometimes I find that we get very specific with who we enroll because we have a specific hypothesis in our head. And my general guidance is, hey, you can have very specific mental models of what the [00:21:00] intervention is and who this works for, but you don't have to run the experiment just for them. You can always do the analysis ex post to figure out if your hypothesis was correct. But have a, run a general explore experiment, especially if it's a if it's the first time you're running that kind of an intervention. Because the risk of running a small population is A, it's a small population, you don't have much power to begin with. And so yeah, odds of false negative go up, so Ashley HOST: But I imagine that's something you have to be careful of if you're introducing a new feature, let's say it's for people that have poor network connections, and you run it against a broader audience, it, probably likely not to come backs- back stat sig, but could be stat sig for that really small segment. So then how do you try to, account for that? Arun GUEST: Yeah. That's-- It's a great question. I think so I would say there's two things. One is if the intervention is in any way, If it has [00:22:00] unintended sort of side effects to the general population, we should think deep and hard about running that experiment. But if the, if the effects are neutral then there's not really a huge cost to running the experiment to a broad population, and we could always do some heterogeneous treatment effects models and use statistical inference on the back end to figure out what the effect on a specific sort of subpopulation is. So it is a tricky balance one has... And this is where judgment comes in and having that conversation is important a-ahead of time. Ashley HOST: Yeah. it's funny 'cause it's so easy to think of an AB test as, I show them the blue button or I show them the red button. How hard can this be? But with a lot of tests where you're really trying to learn things about user behavior the rabbit hole can get very deep. Yeah. Yeah. Arun GUEST: . Yeah, and especially with modern stacks, places like Twitch and Uber, even a small experiment, you're influencing experiences for millions of customers. And so that's, I take it with a [00:23:00] lot of care and I make sure that all of us sort of respect that as we think about these experiments. Ashley HOST: Yeah, definitely a measure twice, cut once type of Arun GUEST: Yes. Yes. Yes Ashley HOST: So how do you see experimentation evolving at Twitch over time? Arun GUEST: Good question. I think so one immediate effect that I see is what AI is having. It's very top of mind for us. So we have-- , we use something called Kiro internally. You could think of it as , it's an AI system wrapped around Visual Studio. It's a interface where there's a chat window and, underneath that you could pick one of the Claude models and try to interact with anything and everything at Twitch. Ashley HOST: Is it like a data analysis tool or Arun GUEST: could use it for anything. Yeah, you could use it for data analysis. We have bunch of MCPs. There's, we have-- You could query data, you could, and so we also build an experimentation MCP into it. So now all of the experiments that are running, all of [00:24:00] the experiments of the past are within that MCP. And so you could just go into that system and say, "Hey, give me a summary of all the experiments that are running on ads," and it'll give you a whole summary of everything that's going on and measurement and so on and so forth. Now, one has to be cautious with these systems and make sure that there's not much hallucination going on. But, we're building in some guardrails through steering files and all of that. But that aside, I think it's liberated a lot of product managers to go and use these tools and get a quick understanding of what's going on, where things are. I think that I definitely see not just us, but everyone else lean more into. Ashley HOST: AI for the analytics side and just trying to understand the business better. Arun GUEST: That's right. That's right. I think the second-- if I take that, I'll do the whole loop of it, but the next step of it is now I see folks consume those results and then interact with some of the [00:25:00] AI systems to hypothesize about next steps because they're bringing in, at least I do this, where I'm beginning to say, "Oh, I get these results about ad systems. Can you talk to me about what's happening at YouTube and Netflix, and how could we get inspired by ideas from the outside? What should next experiments look like?" And have a conversation. And so there's this whole idea generation, which, whose cost has just dropped to zero. And so that I see-- So that idea generation informed by experiments and some of the prior art existing outside of Twitch getting melded is gonna be a big multiplier. I think that'll be very powerful. And then lastly, I think at least for Twitch, I see that we have a fairly mature experimentation system. We are beginning to think about bandits and employing bandits in a variety of problem spaces. So excited about it. Ashley HOST: Yeah. Arun GUEST: Yeah. Ashley HOST: How do you think about the trade-offs around bandits? [00:26:00] Obviously, you give up some power and maybe some confidence that you've actually landed on the right answer, but to move faster, right? Arun GUEST: Yeah, I think... So this is where I'm-- Yeah. Yeah you're absolutely right., We have carved out a few areas where we are willing to play around with it. So as a good example is ads where we have some heuristics around how much, how do we personalize the ad load for a viewer is the problem that we're trying to answer. Now, we have some heuristics based on experiments that we've run, and those ad policies work well. They, and we are constantly running experiments to figure it out. And the trade-off we make is, short-term ad revenue to long-term engagement. That's the basic trade-off, if you will. It's a problem theoretically well-suited for bandits. And we should be able to do that. Now could we actually run this get these systems to, to solve this problem is a very different proposition. It's less about the systems itself. [00:27:00] Even if you had very simple plain vanilla Thompson sampling I'm still waiting for full in-depth evidence that this could actually work. So I'm still, It's still very nascent, I would say. But the problem... To answer your question, I think it's the problem space. To make sure that the problem is well-suited for bandits is the key thing because it's... Yeah. Theoretically, it'll sound good for any any iterative experimentation setup. But yeah there's a lot of power that you trade off and, Ashley HOST: yeah. It's one of those very tempting things to do 'cause you get to an answer faster, but is it the right answer? That's the long-term question. Maybe we'll have to have you come back on after Arun GUEST: Yeah. Yeah Ashley HOST: to experiment with this, all right. I look forward to that. Thank you so much, Arun. We learned a ton from you today. Really enjoyed the show and thank you so much for coming on. Arun GUEST: Likewise. Thank you so much for having me. This was great. [00:28:00]