Latest / The Experimentation Edge / Why GoPro stopped judging A/B tests by win rate
Transcript
- The Experimentation Edge - Will Guyeskey Will Guyeskey (00:00) You should not be worried about a loser. Like, you should not be worried about a losing test. You should not be worried about a flat test. You should absolutely be worried about... Ashley Stirrup (00:11) Welcome to The Experimentation Edge, where product managers, data scientists, and engineers talk about how they make smarter decisions. I'm Ashley Stirrup, the chief marketing officer for GrowthBook, and in each episode, I'll sit down with an executive to unpack how they use experimentation and A/B testing to make better decisions. This show is sponsored by GrowthBook, the open source experimentation platform leader. Now let's jump in and get started with our next guest. Hello, and welcome to today's episode. I'm excited to have Will Guyeskey, director of digital product at GoPro on the show today. Welcome, Will. Will Guyeskey (00:51) Hey. Thanks so much. Great to be here. Ashley Stirrup (00:53) Yeah. And, it's fun to be talking about such a cool product, like the ones GoPro makes. Will Guyeskey (00:59) Yeah. We think so. Ashley Stirrup (01:01) Yeah. I think so too. I'm a big fan. Maybe we could kick things off by having you tell us a little bit about your role at GoPro. Will Guyeskey (01:09) Absolutely. So I'm director of digital product at GoPro, and feel like that means something different everywhere. But at GoPro, that just means we are in charge of the e-commerce portion of the website. So we have other areas of the site, you know, post login and the subscription, our app, but my team is responsible for the e-commerce portion. Ashley Stirrup (01:33) Got it. And you've been doing A/B testing for a long time now. Right? Will Guyeskey (01:38) I have. Yes. I started several years ago now at a little consultancy at the time called Brooks Bell, and that is where I really cut my teeth, so to speak, and just learned a lot very quickly. The agency was this boutique kind of agency focused exclusively on A/B testing and experimentation and personalization. And so you got to work with a lot of different clients all at once and across industries. So I worked with, like, you know, Under Armour and Barnes & Noble and Ralph Lauren in retail and then had some FSI clients as well, Fidelity and Oppenheimer, and even Touch Hospitality. IHG was one of my clients too. Ashley Stirrup (02:30) Yeah. I bet that must have been an amazing experience. You know, it's one of the things I love about this show is where everybody guess who comes on, they're dealing with a different customer and a different buyer's journey, and you got to do that at a much, you know, more detailed level than we do on the show. Will Guyeskey (02:46) Yeah. Yeah. I mean, consulting is a grind, I think, for you know, depending, I think, on where you are and what it's like. But I I think there's no better way to learn as quickly as you do because you get to see what works and what doesn't across all these other clients. And that's the fun part. You get to kind of unpack all of that in real time with your clients and then internally with your coworkers. So yeah. Yeah. I learned a lot. I'm really grateful for the experience. Ashley Stirrup (03:19) Yeah. Has that shaped at all your approach to, your role at GoPro? Will Guyeskey (03:23) Yeah. It's very humbling. It's a very humbling experience to be on the consulting side and to be the expert and try to kind of intuit what will work for each of these, clients and then thinking that a big win on Under Armour might be translatable to Ralph Lauren and finding out that's not the case. That's where I really learned it's so much about the audience and the uniqueness of your customer. And so that's what I brought. After that consulting experience, I went to Gap and led their personalization program there across Gap and Banana Republic, Old Navy, and Athleta within the digital transformation office, and then now at GoPro. But the through line for me is you've gotta know your customer, and and that's so much of what the experimentation is about. It's about experimenting all these hypotheses that might unlock for you a deeper understanding of who your customer is. Ashley Stirrup (04:29) Yeah. I love that. That's such a great point. And I'm guessing you have some pretty unique and different, segments of customers, that buy GoPro products. Will Guyeskey (04:40) We do. We do. I mean, the the really fantastic thing about GoPro is about our cameras and everything we make really is it's so durable and it's so rugged. And but that puts us in kind of a unique position because we certainly have, you know, many customers who will come back and just get the newest version of every camera, but they don't have to. They they last for a long, long time, and they don't break easily. And if anything does happen, our customer support is fantastic. So it's different than, you know, buying a book or buying an article of clothing. There's a longevity there that I think is unique. Ashley Stirrup (05:22) That's interesting. I hadn't thought about that. So how many experiments are you running there a quarter? Will Guyeskey (05:29) Kind a nimble team. Like, we're a small cohort here, and so we try to be really strategic about the tests that we do run. But generally, we're doing, you know, anywhere between one and three a month. It it just varies. And, you know, lots of other ways that we kind of glean insights through some more qualitative data and and, of course, looking at our quantitative. But in terms of testing, that's kind of our sweet spot. Ashley Stirrup (05:56) Got it. And are there a variety of teams that you're working with there? Will Guyeskey (06:01) Absolutely. Yeah. So we I mean, I had mentioned earlier the different kind of areas of the site post login subscription. And so, yes, we work closely with all of them both, you know, kind of within A/B testing and outside of A/B testing more broadly. And then, of course, our our engineering teams, our IT teams, depending on what we're trying to roll out on the site will be involved as well. Ashley Stirrup (06:29) Yeah. And, just thinking back to that kinda know your customer. Ideally, you want all those people to know the customer and know the customer journey. So how do you, as you're learning things, how do you try to share those and make sure everybody knows them? Will Guyeskey (06:43) Yeah. I I love that question because I think it's so important to disseminate, you know, all of the insights and learnings that you glean. So, you know, I don't really think there's a wrong way to do it. The way that we do it is typically we have a biweekly call that is really kind of a cross section of the organization. And so we have engineering and product and program operations, merchandising, IT, marketing. They all come to this biweekly call where we share out kind of updates about digital product and what we're focused on and what our priorities are. And that's where we bring in A/B test learnings and and share those out. So I think it's been, you know, really insightful to your point for everybody across the organization to learn about what's working and what's not, what we thought would win and didn't, or we thought it would be flat, and it's a big winner. And one thing we recently started doing with that is sharing the the incremental kind of along the way readouts. That was something that for a long time we we withheld doing, not because we didn't want people to see those results along the way, but because we were concerned about misinterpretation of, okay. Well, it's it's up today, and the expectation is it's gonna be up when it ends. And as we all know, everybody who does this on a regular basis, that is not the case. We frequently will share out results that are way down one week, and they'll be way up, you know, the next week. So it's I think ultimately where we landed with that is that's a helpful thing for them to understand too, that there is volatility along the way, that that's normal, And that's precisely why we believe in rigorous experimentation design and and not calling something until it reaches statistical significance. Ashley Stirrup (08:44) Yeah. It's such a great example of, you know, when you first hear about A/B testing, oh, how hard could it be? I have an A and a B. I slap it up there and Once you do it. Will Guyeskey (08:54) Yeah. Right? Ashley Stirrup (08:54) Yeah. And showing people that while the numbers could look one way and then actually be another, you know. Yeah. It, definitely opens some eyes and makes people realize, okay. There's there's a deep end to this pool. Will Guyeskey (09:12) Totally. Even for us. I mean, you know, no no matter how long you do it, it's so hard not if you see it way up one week, it's so hard to not jump to the conclusion, you know, this is gonna be a winner. But all the time we are reminded, we can't do that. Ashley Stirrup (09:32) Yeah. Ashley Stirrup (09:33) Hi everyone, just a quick interruption to today's episode to ask you all to like and subscribe. It would really help the show. So if you're enjoying this as much as I am, please do so. Thanks so much. Ashley Stirrup (09:45) So could you tell us about an experiment where you had a lot of learnings? Will Guyeskey (09:49) Yes. There've been a lot, but one that stands out to me is actually from the consulting days at Barnes & Noble. It was a lot of fun to work with Barnes & Noble. They had a really fantastic fantastic A/B testing program, and we worked with them over years. You know? So we we had the good fortune of being able to test every single page type multiple times, you know, dozens of times, like the home page, the PLP, the PDP, landing pages, cart checkout, search results, all of it. And and so at one point, we took a step back and we just said, okay. You know, what makes sense to test now? Plenty of full site A/B tests. But one place that we had never tested was the order confirmation page. And so that's what we started to look at. And whenever we would strategize for a test, we'd bring all the data in. So we looked at visitors to the page. We looked at where they were clicking and how long they were staying and whether they were scrolling or not. So we brought that all together. We had our little war room strategize. And what came out of that was the idea to test a recommendation module on the order confirmation page. And I was certain that it would be at best dead flat, probably a loser. And so we ran the test, and, of course, it was a big winner. I mean, a really significant winner. It killed it. And that was a very I mean, again, it's it's like I have been humbled so many times throughout all these experiments that we've run. So that was a really fun one. Ashley Stirrup (11:41) Yeah. I think that's actually one of the most powerful things is to be humbled because then you realize what the true opportunity is, not what you think the opportunity is. So Will Guyeskey (11:52) Yeah. Yeah. And good insights from that one too. Ashley Stirrup (11:56) And so, like, I I think one of the big things you realized from all that is that the Barnes & Noble buyer is a different kind of buyer, and they're more likely to be a repeat purchaser. Is that right? Will Guyeskey (12:08) No doubt. Yeah. No doubt. And it connects, you know, I I think there were a few takeaways for for our team from that. One was exactly what you said. It's that this audience, this visitor on their site is just unique in in a whole bunch of ways. And one is just the average order value. Right? I mean, they're buying books. These are not these are not $600 GoPro cameras. Right? We're talking about a a 15 or $20 book, and that makes it unique in some ways. But, also the audience of readers, like, the the very idea that these people are the core customer is a voracious reader really made a difference too. And so on the order confirmation page, we had learned so much up to that point about that customer, that particular person. We knew where they visited on the site. We knew what books they browsed. We knew where they lingered. And so by the time they got to the order confirmation page, we could put really fantastic recommendations right in front of them. And so because they're a reader, they didn't wanna lose that recommendation. They wanted to purchase right then. And because the AOV wasn't super high, you know, they felt like they could pull the trigger without too much hesitation. Ashley Stirrup (13:36) Yeah. Did you try taking that same idea to some of your other clients? Will Guyeskey (13:40) Yeah. Yes. Definitely. And it didn't work. I mean, I don't remember it working on a single other client that we had, but, you know, that was always, again, the interesting part. Yeah. When we took it to Ralph Lauren and we tried the same thing, it's like, no. Those items are more expensive. They might not be interested in more clothes from that season or or from that moment. So, yeah, it was it was unique to Barnes & Noble, which makes the insight all that much more powerful. Right? This is a unique thing for your audience visiting your site. Those are the that's gold. That's gold for them. Ashley Stirrup (14:23) Yeah. I'm curious. How would you define a winner and a loser in that test? Like, you think, okay, they came back, they bought more, winner. If they didn't buy more, it's gonna be the same as the control. Will Guyeskey (14:35) Yeah. For me, I mean, yeah. It was the difference between and and this is a really important distinction. A lot of times, there's a difference between a visit and a visitor. Right? So we could track them, especially because everything's different now with cookies. But back then, you could very easily track that person and know if they're coming back to the site and know if that worked. And many times, it was actually the same visit. So they weren't even at that point a different visitor. They clicked immediately on that recommendation, went back and purchased immediately right then. So that visit had a higher average order value because they were going back and purchasing again in the same session. Ashley Stirrup (15:24) Got it. Got it. Yeah. I think you also had a good example from your GoPro days. Yeah? Will Guyeskey (15:31) Yes. Yeah. Yeah. Yeah. Thanks. Yeah. This one was very recent. GoPro just launched our new Mission series of cameras. Everybody go check it out. Let me plug GoPro for a second. Super cool cameras. You gotta you don't get one or two or 10 of them. Just think about it. Yeah. But it was a big launch for us in a lot of ways. But one thing that our team did was we rolled out a new page type for for this for this launch. And so on the buy product detail page, for Mission, we have a configurator. And our strategy was this idea that, you know, these are high price point items, and there's a lot that can go with it. I mean, for for any GoPro camera, you have to have an SD card to make it work, and so we offer SD cards on our site. You know, we really believe that the subscription makes it all that much better. You can automatically upload your your videos and get highlight reels. And so on the new Mission PDP, we walk you through each of those choices. You choose your camera, we kind of step you through. Hey. You know, are you interested in a subscription? Here's the value behind it. Are you interested in an SD card? You know, you can choose to get it here or not, but you need one for your camera to work. Would you like to bundle this with some accessories? Things like that. And each of those steps requires a decision. So you have to either choose a bundle or say, no. Thanks. You have to either choose an SD card or say, you know, I'm not interested. And so there was some consternation internally at GoPro around whether that's too much friction for our customers. And our team's perspective on that was, well, you know, we don't think it is. Again, these are high price point items, high consideration purchases. We want to honor that by walking them through, you know, these pretty important decisions along the way. But, you know, it's a fair question to ask. And so we wanted to run the test to answer the question definitively. And the test was essentially after you choose which version of the camera you want, you we would then autofill no thanks for each subsequent option. Now, of course, you could choose instead to say, no. Actually, I do want an SD card. Right? But we would default to no thanks. And that would allow you to add to cart right away. Critically, the control in the control version, if you skipped all those questions, you could not add to cart. So you you had to make a choice for each one. And, you know, so the reason to run that test is to answer the question of whether we're adding too much friction or not. But the I think the vote for the other side for the control was, hey. You know, that might sacrifice AOV if we're just selecting no thanks for each one of these. So really interesting test to run, and it ended flat. You know, we had thought it might go one way or the other. And so, again, that is fantastic insight for us into the customer. And what it tells us is our customers, because it is a high consideration purchase, they're comfortable with a little bit of friction. They're comfortable having to make a few extra choices as long as there's value in those choices that they're making. And so we kinda take that learning and tuck it away, and that's something that informs the next PDP that we'll work on and develop. Ashley Stirrup (19:11) Yeah. Yeah. Makes a lot of sense. And you just imagine somebody a lot of times, a camera like that might be inspired by they're going on a vacation. They're gonna go snorkeling or something. And, you know, they need it in the next few days because their flight's coming up and they wanna make sure that they don't oh, I forgot the battery. I forgot the SD card, whatever it might be. Exactly. Exactly. And so they wanna take that time to make sure they've got that all right. So that's super interesting. Help me get it right. Will Guyeskey (19:40) Yeah. Yeah. Exactly. Ashley Stirrup (19:42) Yeah. Yeah. So if you're working with somebody that's saying new to experimentation, and they've got a new feature and they're all excited about it, and they're ready to go test it, but they haven't really thought about the well, what if I lose and what will I wanna learn from this if maybe the idea is good, but the implementation's not? Like, how do you help people design an experiment so they get those learnings and can get to a winner over time? Will Guyeskey (20:10) Yeah. Yeah. Yeah. That's I like that question. I think you're right. Like, you asked how do you help them design the experiment to help them learn? And I think that's the right way to frame it. Right? I think the way to frame it is you should not be worried about a loser. Like, you should not be worried about a losing test. You should not be worried about a flat test. You should absolutely be worried about constructing a test, designing a test in such a way that you will have impactful learnings regardless of the outcome. I really believe that people who or programs that run an A/B testing program and choose win rate as their North Star are missing the boat. You know, I think if if you have win rate as your North Star, you're doing something wrong. We at GoPro, losing tests for us are jet fuel. You know, that that is what helps us get to the right experience way faster than we would have otherwise. So, yeah, I think for that new person coming in, it's about kind of sitting walking them through. We need to feel successful whether the test wins, loses, or is flat, and here's how we would construct an experiment to be able to do that. Ashley Stirrup (21:40) Yeah. Yeah. Love that about the North Star metric of the program. Can't be win rate. It's gotta know, in the end, it's learning, which is a tough thing to measure. Yeah. But yeah. But you go back to the experiment you were just talking about at GoPro and, like, what do people want to where do they want their hand held, and where do they want just, like, a lot of information? They wanna take the time to go through it all. The more you can kind of unlock all those things. Right? You design an experiment like, well, they didn't click add to cart when I thought they're gonna click add to cart. And if that's all you know, you have very little information to know, okay, how do I iterate? And so Yeah. It's capturing those additional metrics. And one guest we had on, he talked about philosophy they introduced where it's like, you've got an idea for a feature, test the opposite. You think less copy will actually improve, do more copy. And if more copy hurts conversions, at least tells you you can move the metric. Right? Because there might be some things like, doesn't matter what we do. More or less, the metric's not gonna move. Will Guyeskey (22:44) I think it's... Ashley Stirrup (22:45) Go ahead. Will Guyeskey (22:46) Yeah. I was just gonna say, I think it's so true. I think it's so true. We we sometimes talk about kind of the distance between your control and your experiment. Right? And sometimes you need to look for a greater distance between them in order to ensure that you're gonna get a clear learning. Yeah. So that's something that we've talked about a lot recently, in fact, on our team. Ashley Stirrup (23:09) Yeah. Yeah. I love that. That's actually a really good way to look at it because not necessarily just do the opposite, but, you know, how do you iterate on a few different designs that kind of stretch the bounds so you can learn where which levers you've got to pull. Yeah. Precisely. Yeah. And so how do you see experimentation evolving at GoPro? Will Guyeskey (23:30) You know, I think we have some really exciting room for evolution, and and I think the ways that we will evolve will be similar to how others evolve. You know, if I feel like if I hear AI one more time, I'm just gonna go crazy. But but AI certainly will influence A/B testing and experimentation in really significant ways, and it it will at GoPro. It will elsewhere. It gives us a tremendous amount of velocity to be able to leverage AI for for the assets that we need to to run an experiment, you know, to be able to build the experiment itself. There's just a lot that it can unlock for us. I'll tell you though on the other side, the thing that I'm gonna watch really carefully for is making sure that we're not running so many tests that we start running tests without taking time to understand and implement, crucially, the learnings. And, again, that's regardless of outcome. You know, we're we will always monitor win rate. We will always try to improve our win rate over time. But if we're running so many tests that we're not really cognizant of, how those learnings are impacting our site, our customer experience, what we as a the team deploying all of this kind of understands about our visitors, we're not doing anyone any good service. Ashley Stirrup (25:06) Yeah. Yeah. I strongly believe that AI has the potential to open up experimentation to more people and to provide them with kind of a data scientist, you know, an AI data scientist to help guide them along the way to make sure they're implementing them correctly. Yeah. But I think we're a long ways from where AI can just go run the test all by itself and come to the conclusions and and all that. Yeah. And so it's finding using way finding ways to use AI to unlock that kind of new new level of speed and and productivity and kind of new level of learning. I think that's where the real opportunity is. Will Guyeskey (25:45) I think that's right. I think that's right. Somebody on my team always calls Claude we use Claude at GoPro and always calls Claude the intern. And I think that's the right way to think about it. You know? It's like, this is a really potentially capable and powerful assistant to have, but it still very much needs your guidance and your direction. And if you let it go too far without that, you know, it's it's at your own peril. Ashley Stirrup (26:12) Yeah. That's pretty funny. The intern. I'm I'm gonna share that one. Will Guyeskey (26:17) Yeah. Well, Brandon Watts. Ashley Stirrup (26:19) Yeah. Well, Will, I really appreciate you coming on the show. Love your just your your perspective on the whole market, and you've shared a number of amazing nuggets with us. So thank you so much. Will Guyeskey (26:31) Absolutely. Absolutely. It was a lot of fun. Thanks for having me. Ashley Stirrup (26:34) Yeah. Thank you. Thanks for tuning in. If you enjoyed this conversation, please support our channel by hitting the like button and subscribing. Better yet, share the episode with a friend. I'm Ashley Stirrup with GrowthBook. We'll see you next time on The Experimentation Edge.