Latest / The Experimentation Edge / Why you're already wrong 75% of the time

Transcript

Why you're already wrong 75% of the time · The Experimentation Edge

• Graham (00:00) at my previous company we had a redesign. In my experience, redesigns are almost always catastrophic, but we redesigned the whole product and I was like, Okay, great, we're gonna do an A/B test. And they were like, no, no, we have a marketing campaign coming up in two days. Like we need it at a hundred percent. And I was very much against this. you know, you can always move marketing campaigns, But we ended up not testing it. I was overruled. And then six months later we start noticing like, wait, why is that conversion rate consistently down? Because it's sometimes it can be hard to notice because metrics there's so much noise in them that it's not a clean, kind of piecewise jump of your metrics. It'll be buried in there. And when we finally unpacked it we realized that like we had tanked like our new user creation rate by something like twenty percent for the last six months. And it, you know, cost the company millions and millions of dollars. So we're like, whoops, yeah, I should have tested that. Ashley Stirrup (01:27) Hello and welcome to today's episode. This is a special episode we're gonna take a look at many of the past guests we've had on the experimentation edge. And we have Graham McNicoll and Cris Dizon as guests who will share some of their experience. Why don't we start off by having each of you introduce yourselves and a little bit of your background? Graham, why don't we start with you? Graham (01:53) Yeah, hi everybody. I'm Graham. I'm one of the co founders of Growthbook. Started back in 2020. I've been doing experimentation for a lot longer than that. We built our own experimentation platform at my previous company. ran a lot of experiments there, figured out how to make the company successful ultimately. Ashley Stirrup (02:10) Awesome. And Cris? Cris (02:12) Hi everyone, I'm Cris. I'm head of customer success and strategy here at GrowthBook. And before my role here at Growthbook, I was enterprise side running experimentation programs. So I have a lot of experience using different tools, different methodologies, and then now I'm helping customers kind of get the best use out of Growthbook. Ashley Stirrup (02:31) Yeah, and you've been at a lot of great companies. Why don't you tell people a little bit about that? Cris (02:36) Yeah, so I have a lot of experience in e commerce. So I've been at a lot of the big box retailers, Home Depot, Walmart and so forth. And then also have a lot of experience within the streaming industry. And you know, before my time here I was previously at Peacock. Ashley Stirrup (02:51) Awesome. Yeah. So we're going to take a look at, as I mentioned, the many of the past guests. We're well over 50 guests now on the show at 50 episodes. And one of the most important things that stands out is the percentage of winners. It's you know kind of in the 25% range. it really varies quite a bit from company to company. But I know I think a real unlock for a lot of companies is when they realize just how low their win rate actually is, and that how much value they can get from experimentation to test all their features to understand which ones are losers that they should pull back on right away, whether regardless of whether they're going to iterate on them or not and hopefully turn them into winners. Just by pulling them back alone, they're saving their company in many cases millions of dollars. Graham (03:38) it's fascinating that that's sort of best case scenario, right? the more you optimize the the lower your win rate gets and it's really hard, like it's kind of a kick to your ego when you realize that like your best efforts are only winning twenty five percent of the time, right? but basically, if you're not doing A/B testing. you're testing anywhere. You're just testing it a hundred percent without actually measuring any results. Right? Like you have absolutely no idea what the effect is, but you're gonna figure it out, And then no one builds a feature they think will not win. So this is our our best effort number. it's quite a shock to people. Ashley Stirrup (04:17) Yeah. And to me, it's just one of those super powerful moments. I had kind of a similar one at a company a long time ago, in the 90s, where I had built this product and I just thought it was so amazing. And then we did a usability test. We had the glass two-way mirrors, and I was watching people and every single user missed this really important button in my product where they couldn't get to the other half of the functionality and it was just so humbling. But it was also so empowering because then you realize how much better you could be doing. So that's one of the things that gets me so excited about it. Graham (04:53) it's an important like cultural leap to make to build that environment where it's okay to fail because like you you're gonna be failing a fair amount of the time. Like and that humility is really important. it's also makes it a much nicer place to work when people aren't, you know, telling you you're wrong all the time. They're like, Yeah, that might work. you're sort of much more open to exploring and trying new things There's a lot of cultural benefits from doing this. Cris (05:17) Right. as your experimentation program gets more mature and you're running more experiments, your win rate in theory will start to decrease over time as you've plucked the low-hanging fruit. And so now flipping these failures into learnings and kind of considering them as wins because you're preventing yourself from launching a feature that will cost you know you tech debt moving forward indefinitely decreased in your KPIs moving forward. So it's a it's a good thing to get into and and continue to push. So as you scale your program, you have to kind of to think about wins for both losses and actually increases in your KPIs. Ashley Stirrup (05:57) Yeah. And these podcast guests really range, you know, like Fixers at the time is a thirty five million dollar business and Amazon's obviously multi billion and they're all in these ranges, so it's pretty incredible. And Dave Massey, who heads experimentation at UPS, you know, he's delivered over five hundred million in gains for for UPS. And yet he'll freely admit that his gut is wrong more often than it's right. And so that whole thing of test everything is just so important. Graham, do you have any examples in that area? Graham (06:31) yeah, at my previous company we had a a redesign. In my experience, redesigns are almost always catastrophic, but um we we we redesigned the whole product and I was like, Okay, great, we're gonna do an A/B test. And they were like, no, no, we have a marketing campaign coming up in two days. Like we we need it at a hundred percent. And I was very much against this. you know, you can always move marketing campaigns, right? Like there's nothing crazy about that. But we ended up not testing it. I was overruled. And then six months later we start noticing like, wait, why is that conversion rate consistently down? Because it's sometimes it can be hard to notice because metrics are so there's so much noise in them that it's not like a clean, kind of piecewise jump of your metrics. It'll be buried in there. And when we finally unpacked it we realized that like we had tanked like our our new user creation rate by something like twenty percent for the last six months. And it, you know, cost the company millions and millions of dollars. So we're like, whoops, yeah, I should have tested that. Ashley Stirrup (07:33) Brutal. Yeah. Yeah. that's just such a great example of losing being quietly expensive. Cris, do you have any other examples like that? Cris (07:42) I can second Graham's sentiment about big big bang redesigns. You know, so I I was at a previous role and you know there was a large homepage redesign. and we were able to test it to a small percentage of of traffic, but even though it showed a decrease, they decided to roll it out anyway because of the marketing campaigns and all the other efforts that were tied into it. But then we spent the next few months kind of picking it apart. and like reverse engineering what aspects of it actually was causing the decrease. And so you know it took a lot longer and we kind of got back to parity with the previous one, but that was after a longer period of time that we forced that through. that was expensive because we pushed forward on a redesign that decreased our key metric. And then we had to run three or four more experiments to kind of help isolate and back test but what about that redesign actually decreased our our metric and then kind of get resolved it. So it was expensive, you know, probably total time for I had to think about how long it took to get that redesign up and running. It's like a full quarter, and then the experiments after the fact, maybe another month, month or two. so that's a lot of time and effort by everyone involved. Ashley Stirrup (08:54) Yeah. It's such a great example. I mean, I I hear it all the time on the show where it's like, Yeah, marketing has this big launch or something and so had to make this huge change, right? And then you don't know which parts of it were winners and which part were losers and yeah, so hard to figure out the whole thing. Graham, I was thinking about this slide as as you were talking about the that change, right? So this is an example from JP Morgan. Kevin shared this with us, and this is something he uses in his class on teaching people about experimentation. Somewhere in here is a million-dollar change that improved their funnel. And so they're looking at a 90-day funnel. And Graham's already seen me present this. So Cris, I'll ask you, can you tell where in here they launched the feature that added a million dollars to their business Cris (09:43) I know, there's so many it's it's so volatile, right? There's so many ups and downs, it's kinda hard to tell. Ashley Stirrup (09:47) Yeah. Yeah, I mean it's basically impossible. so here you can see where they actually made the change. So this line here is the the control and then you know the remaining the line that was originally there is the treatment. And yeah, you can see that there's no way unless you have an A/B test. You're you're just not gonna be able to eyeball this and say, there's there's the million dollar win. So great great way to bring to life the the need to do A/B testing. another great example from Twitch where they hadn't changed pricing in a long time. And when Arun got there, he said, Hey, we need to test this again. I think I think there's a real opportunity. It's been years. It's crazy we haven't changed pricing. And so they they went ahead and they they ran the tests and lo and behold there was a real opportunity there. sometimes, either the business changes or perhaps there is a poorly designed experiment, you can create anti-knowledge, right? Like misconceptions that become self-reinforcing inside the company. I don't know if either of you've seen that. Yeah. Graham (10:49) definitely. we had at my previous company we were like a a content site. We had a certain amount of content you would get for free, and then you had to pay to s get more. And that threshold, I think we set it at seven. Seven pieces of content. And it would reset every month. And it was like we had tested it in the past and so we got that seven and we thought that was the best number. and then went years as like accepted tribal knowledge that this is the right number. And we actually deployed a completely unrelated feature. I think it was like a a game or it was comp it was very weird. It was like completely unrelated feature, but as as we always do, we always A/B test it. And part of the guardrail metrics was like, you know, signup rate and revenue. and suddenly this experiment was had like increased revenue 30% as a guardrail metric. And we like, what? Like this should have no effect on revenue at all. but we started looking into it and we realized that there was a bug in the experiment and in the code and it actually set the paywall to zero accidentally. Um and we suddenly made like way more money. We're like, wait. And so it it caused us to kind of reevaluate what were those assumptions that we had been holding dear that may not be relevant anymore. And that was one of them. We actually, you know, found another a different no, I think we ended up at three. but it was yeah, it was a it was a great test for us. Ashley Stirrup (12:09) I think it's such a magical moment when you uncover that new vein, right? that that was like this great signal, like, hey, dig here. And then after testing you figured out what the right number was. So but that's that that's pretty powerful. Cris, any similar experiences? Cris (12:23) Yeah, we were launching an onboarding experience. and so this is for a streaming app I was I was working on. And so we wanted to, as customers signed up, instead of just like dropping them into the experience, there was some onboarding. So what are your interests? here's some content, what would you select, genres that you would like. And so it was asking the customer to give us information so that in theory we would improve their actual experience once they start using the app. it was it was a big effort. we launched the experiment, there was zero impact. so it didn't work. and ultimately customers were selecting the type of content they were interested in or specific content that was popular at the time and some of those content would go into their watch list to save for later but didn't impact any type of engagement because you know within within streaming that first like month's engagement is key to to keep them around and so it was flat so it didn't roll out we came back to the feature later on when the personalizations team's capabilities increased. And so now that information that the customer was giving to us was then feeding into the cold start models. And so instead of just adding content into their watch later or saves list, now it was actually feeding their homepage personalization. And so that actually helped and increased some of our metrics for these newer customers to get them engaged early on in their experience. So that was an example of it didn't work because the experience wasn't quite ready to utilize all the information and then we came back to it and then it actually helped and and created like a measurable impact. Ashley Stirrup (14:03) Yeah, it can be so easy as a marketer to kinda like, we tried this, we tried this, you keep moving on and then everyone's a way you have to stop and like, wait, the product I'm marketing has changed. So maybe I should go back and re-look at some of the things I did in the past. So Graham (14:17) Yeah, it's one of the reasons we named Growthbook Growthbook was to help capture that knowledge that you have. what we would find would happen was like we would run a test. You would maybe modify, do some manual look at some of the data, then you would email it out, hey, like this is what happened. And then six months later, like someone's like, Wait, what did we decide with that? Like, why do we call that test? And then you're sorting through Jira tickets and emails and your experimentation platform and All of it was all in different systems and it was impossible to get like that clear picture. Growthbook we try to keep all that in-house there, but we also just added the the learning library to help capture some of that knowledge and understanding that that knowledge will change, right? So that you can change the status, you can update it, you can have it refresh to make sure those learnings still apply to your new users. Ashley Stirrup (15:06) Yeah. That's kind of a fun future feature is the, our product now does this. Which experiments should I go back and look at lots of great examples from the show in terms of companies that have had huge wins. I'd already mentioned UPS with the five hundred million, but JP Morgan over a billion, and that's on the commercial side of their business. And then Fanatics is a three billion dollar business and they're getting eight percent growth from experimentation every year. So it's like in the range of two hundred and fifty million a year, so pretty incredible. one great thing that I heard from Crystal at Disney was reframing things from wins and losses to gains and savings because A lot of companies have this mindset of we've we've only got to do winners and losers are bad. And that your experimentation program A helps you avoid shipping those losers like Graham was mentioning that are costing you a lot of money, but also just you can get a lot of learnings and that you want people swinging for the fences, because that's where you uncover the next hill to climb. any thoughts on that, guys? Cris (16:08) when people track their their gains or their incremental revenue from their their wins, it's also important to to track your savings. I never added those numbers together, but you know, always referred to them so you know we prevented ourselves from a point five percent loss by rolling out these features. but Ashley Stirrup (16:27) Yeah. Cris (16:27) you know, and when you start to think about it that way, you can then take kind of a deeper lens into that feature and start to understand well why and then can we iterate off of that? I think reframing it and kind of creating that like culture where okay this may have been a loss but let's take a look and understand why and then see if we want to iterate or if we want to move on to the next thing. Ashley Stirrup (16:49) Yeah, and have have either of you seen examples of companies that they ended up being more so risk averse that they were afraid to swing for the fences 'cause there was so much pressure to just do winners. Graham (17:01) Can't think of any examples offhand, but it's it's certainly we avoided putting win rate on the product for a long time because we didn't want folks focusing on that as like the success criteria for your experimentation program. Because it's ultimately what what you really want to capture is like your learning rate, right? a bit of a nebulous term, very hard to quantify. because the the ultimate goal is like if you run enough experiments, you start learning what your users do and and do not like. And then you can apply those best practices to all future projects. So it's not necessarily in the experiment itself where the value is delivered, but like all the way down the life cycle of your product, you're learning and you're applying the learnings. So like the baseline of that product is higher and you're gonna be more successful over time. Ashley Stirrup (17:44) That's such a great way to look at it. Yeah, so these are just some of the other examples of the types of companies that have been on the show. A really huge range. just to call out a few we haven't talked about. So chess.com is a fun one we'll talk about here in a minute. Signet Jewelers Box Stitch Fix. they got me an outfit for the episode. So I I particularly enjoyed that episode. LinkedIn, lots of great examples. Yeah, and one of the most consistent things about this is that all these teams really went into it with the mindset is we know we're going to be wrong more often than we're going to be right. And that experimentation is a way to kind of extract the learnings out of every single feature we ship. a great example of that was chess.com, where they have a feature where you can review your game and it'll help you identify, things that you did well or things where you went wrong. And Nafis was on the show and he talked about how, you know, initially, it was Positioned as a feature to help you review your losses. And it's a premium feature, you get access to three reviews. And then if you want to keep doing it, you need to pay for a subscription. chess.com is full of avid chess players, and you know, their whole mindset was, well, you want to learn from your losses. that's where all your focus should be. And so they didn't even want to test the let's let's position it as a review your wins feature. But they went ahead and tested it. And what they found was that they saw 25% more reviews. and that led to more subscriptions. So it was a huge winner for them. And they found that yeah, people much prefer to celebrate their their wins than they do analyzing their losses. So a little a little counterintuitive, but super important for their business. another great example was Groupon, where they were doing flash sales, trying to drive conversion. And what they found is that initially they they got a little bit of a win, and then over time they found that customers were just kept waiting for the next flash sale and the next flash sale. So it's actually slowing down conversions rather than increasing urgency. And they so they switched it to showing that there, hey, there's only three spots left. and that creating that scarcity, that was what triggered people to actually go ahead and buy versus just the the pricing difference. So those are a couple of great examples. Cris, do you have any examples of where you had like a a surprising win? Cris (20:17) Yeah, I mean this one is interesting. I've spent a lot of time in e commerce, you try to optimize for the the customer behavior that that you want, the ideal behavior. if you depend too much on sales or coupons or promo codes, you train your customers. And you know, that was kind of a big thing about you know Bed Bath and Beyond, right? where they ended up and then eventually came back to life eventually. So you know I was at a e commerce marketplace startup and one of the big promos they would regularly launch Was 20% off your three orders, and you could use it three times. And so ultimately, what we saw is that after they used their three promo codes, it would tail off and conversion would just kind of stop until they would get their next promo. So it was the same thing. You were training these customers to wait until they got a promo code this is a great example by Groupon where you're looking at other aspects that are important to the customers to help them kind of make their buying decision, one of those being scarcity. So yeah, Ashley Stirrup (21:20) Yeah. Cris (21:21) this is a is a great example. Ashley Stirrup (21:23) Yeah, it's it's really interesting to hear you talk about it that way 'cause I talk a lot on the show about trying to learn the the buyer's journey and then how do you optimize the journey. But in in your example, you're actually changing the buyer's journey by what you're doing with your marketing. So that's another example of how you have to think about things so carefully. another surprising one that comes up a lot is added friction. So in the case of Signet Jewelers, when you go to the all rings page, they require you to pick a ring type that you're you're interested in. Cause before what they they had was just all rings and be really overwhelming and 90% of the rings on any given page would be rings you don't care about. And so people would kind of click through three times and then give up and leave. And so by forcing them to tell you what type of ring you you're interested in, then they only show you the type of ring you're you're actually looking for, which is exactly like what you do in a store, right? You walk in the store, the sales would say, what type of ring are you looking for? They bring you a tray of those types of rings to go look at. but a great example and there's lots on different episodes where adding friction really helps with the buyer's journey. So even though we're all kind of constantly hearing examples of how you have to remove friction, there are times when friction can be really important. another fun example is from Danielle at Box. Danielle came in and did a lot of work on their pricing pages, started off by simplifying the different the way they were describing their different subscription offers and got some great wins there, and then reached a point where she realized maybe she'd gone too far because You know, at first you want to kind of guide people in the decision, but at some point they need to know exactly what's in each different package so that they can pick the right one. So then she added some more information on on the different subscriptions. And then the next test she ran was introducing a even kind of lower end edition so that when people that they didn't show on the pricing page, but then when people were unsubscribing, offering them this kind of you know, reduced version. to see if they could retain them on that. And that edition actually failed. but what it did do is actually reduce the amount of churn from the other versions. So it was kind of teaching people all the great things that they had with the edition they already had. and it actually led to increased retention. So she called this the wine effect where you put two wines on the menu, people pick the cheaper one. You put three wines on the menu, they pick the medium priced one. and so that was a great example of of an experiment where they learned something important about their users. Graham (23:57) We had an interesting similar case I can't remember who which department wanted some new field on our registration form. maybe it was legal. They wanted like some checkbox and we we were like really concerned that we would add friction and reduce our conversion rate. And then when we so we tested a few different variants of an opt-in checkbox when you actually click it and one with like way more text and some things we thought absolutely would lose. And the winner really surprised us. I think it was the one with the most text on it. And it improved conversion rates. Like didn't just not affect it. It increased conversion rates over just the Cris (24:30) Yeah. Graham (24:30) standard form. And we were like really surprised until we kind of dug into it a bit more. And what happened was that we were actually building trust with our users because we were putting context around like what we would do with their data and you would opting into some s privacy policy and that people apparently really liked it and so increased conversion rates. It was something very unexpected. Ashley Stirrup (24:51) Yeah, I had a guest, might have been Course Hero, where they talked about when they ran a test, they would try to do the opposite feature too. So if it's remove text, add text as well. And a surprising amount of times the opposite with the thing that won versus that. And other times they'd find that, well, the winner didn't actually move the needle. But at least the opposite moved the needle the wrong way. And so it showed them, okay, there is it is possible to move this metric. 'Cause they found with a lot of metrics it's just super hard to move the metric no matter what. they uncovered information even from a losing metric like that. Cris (25:28) Just kinda motivate them to keep trying. Ashley Stirrup (25:31) That's right. And give a sense of okay, we can dig over here and maybe we can figure out a variant that actually moves moves the metric in the right way. another fun one from Crystal at Disney. This is actually earlier in her career. and this is when the Kindle came out they were advertising the ability to do video calls on the Kindle. And this inspired an executive at a financial services company she was at to say, Hey, we should like start doing video customer service. You know, we're a premium brand in the industry and we should go do this and It's gonna be super expensive for them. They were gonna have to hire a team, they were gonna need to have all this equipment, they were gonna need to have office space. So it was gonna be a massive investment for them. And there were a number of people that had concerns, and so they ran a test where they added the button and they showed it to four million people, and 106 people clicked on it. So it's basically like people don't want video chat when it comes to like support with their credit card and so that was a huge way to save the company money by putting a button in place versus spending millions of dollars on a new call center. So Cris (26:34) Yeah, I'm I'm all for, these painted door type experiments. I've been at organizations where, you know, they'll get their user research, they'll talk to customers, but then they'll do something small like a painted door just to actually now gauge and see if what customers are saying that we've talked to does it match with what customers are actually doing kind of live in the product. but you gotta be careful about what happens after you click click here for video chat. Is it just kind of a dead link or do you say, well, we're we're investigating if this is a feature. Thanks for you know helping us and show interest or something like that. Ashley Stirrup (27:08) Yeah, yeah, or at least have a good off ramp. Like sorry, video chat's not available now, but here's a number you can call or click here to email. you can really tell that some of these guests we've had are just in a class of their own, just doing incredibly well. and a lot of it comes down to the cultural side of it. So, you know, they they do look at losses as a way to get, you know, information that they're encouraging people to to think big and try things and it's okay to fail. And the CEO or some executive is in the room, you know, leading by example showing how important this is. And it's really special when you hear stories like that. And so here's a few examples for an executive who wants to help build that culture of experimentation. The first is just reinforce how it's like everything you ship, put a test on it, measure the impact, and you know, try to get those learnings. second is like reward the savings. So celebrate the losses hey, we just saved ourselves a million dollars by pulling that feature back. next is plan for failure. So as you're building experiments, ask yourself, well, if this fails, what do I want to learn? So that it will tell me like, should I be iterating on this or not? And if I want to iterate, where should I iterate next? and then the culture of sharing is just so important. like sharing your wins, your losses, having group discussions over experiment design. So everybody's learning how to design experiments with rigor. just so important for creating that culture and that mindset of, hey, let's test everything and here's the right way to test. and then engaging personally. To me, I think it's like if an executive is demonstrating that, hey, I don't know and I want to run the experiment so I can learn. they're showing everybody that it's okay and that, you know, the results of these experiments can be super valuable to the company. So it's a great way to kind of lead by example. And then of course, protect the people that are taking the risks and swinging big because they're the ones that are gonna un uncover your next home run. Graham, any any comments on this? Graham (29:08) just one point on on your second one there. you know, rewarding the save there. I think the the right question to ask if you are in leadership is could we have learned this faster? Like what would be the minimum thing we could have built to get to this data more quickly? Because you may not have had to actually build the whole thing. You could have just put the button or, you know, do a painted door test or something simpler. but asking that question to make sure they're considering the right scope of of like where to optimize. Ashley Stirrup (29:37) Love that. Cris (29:38) Yeah, I think share the readout is is really important, especially if you're trying to build that culture and get all teams wanting to run experiments. one thing that I find really helpful is not just the readout, share the experiment design prior to launching. and you can have kind of a poll in your Slack channels or wherever your your teams kind of work together and say, well, pick the winner, right? People get invested and then they'll come. of look back on the results once the test is over and and see if they won. So like that kind of gets teams involved and also kind of has a little bit of meat in the game. it helps to your earlier point, Ashley, about well half the time or more than half the time we're wrong. there's a very lower percentage of picking winners. And so yeah, having the ability to share the readout, make it accessible. can have access to that data is is important and helps kind of generally get people on board and and wanting to run more experiments. Ashley Stirrup (30:31) Yeah. any way in which you're kind of helping expose more people on the best ways to experiment. Like some some guests have talked about having a buddy program where you kind of identify your best experimenters across the company and make sure they're tagged they're teamed up with you know the people that are newer to it. another another guest from Charlie Health talked about having a a a pre-mortem rather than a postmortem, a pre-mortem where you sit down, especially for the bigger things and say like what could go wrong here and let's you know kind of pretend it went wrong and then try to understand why it might have and then re-look at things and to me I think this combines really well with your question Graham of how much of this do we really need to build in order to to truly test this out so thanks Cris and and Graham for joining. This was a great discussion. just to leave everybody with this idea that you're already wrong 75% of the time. The only question is whether you're actually gonna measure and learn from that or not. great final point to end on. Thank you guys. Graham (31:31) Thank you. Cris (31:31) Thank you.