Latest / The Experimentation Edge / Learneo on testing the opposite of every hypothesis
Transcript
- The Experimentation Edge - Rich Liebling === Rich Liebling: [00:00:00] one of the things we learned was, and we made this a general practice, is someone has a hypothesis and you wanna test it, great, but also test the opposite of that hypothesis. INTRO: Welcome to the Experimentation Edge, where product managers, data scientists, and engineers talk about how they make smarter decisions. I'm Ashley Sturrup, the chief marketing officer for GrowthBook, and in each episode, I'll sit down with an executive to unpack how they use experimentation and A/B testing to make better decisions. This show is sponsored by GrowthBook, the open source experimentation platform leader. Now let's jump in and get started with our next guest. Ashley Stirrup: Hello, and welcome to today's episode. I'm excited to have Rich Liebling, senior director of engineering at Learneo. Welcome to the show, Rich Rich Liebling: Thanks, Ashley. I'm looking forward to this discussion. Always talking about AB testing Ashley Stirrup: Yeah. Yeah, me too. Maybe we could start it off by just [00:01:00] talking a little bit about your background and when you first got into experimentation Rich Liebling: Yeah, so I got into A/B testing when I joined a startup called Shop It To Me back in 2007. I was the first engineering hire. I was the VP of engineering. And Shop It To Me operated a email subscription service that delivered people email with basically apparel items that met their preferences in terms of brands and sizes and stores. And we operated very much based on A/B testing to grow our user base Ashley Stirrup: Yeah, you had a pretty amazing experience there. You had a huge impact on the business through that, right? Rich Liebling: Yeah. When I joined the CEO founder had built the original version while he was in business school and had gotten it up to about 50,000 subscribers. But when I joined, we set our goal for the year to be getting to one million subscribers which seemed a very daunting task. But, A/B [00:02:00] testing basically led the way and we achieved it, and it was quite a ride. Ashley Stirrup: Yeah. So you actually got to the goal in nine months or something, if I remember correctly, right? Rich Liebling: Yeah it took nine months. And it was a really good experience. In particular, from the strategy perspective, it's like, okay we wanna get to a million subscribers, but the real question is how can we do that? And the strategy was we were gonna try to become viral. And okay, it's great to be viral with the refer-a-friend program, but how do you get viral? And we said we're just, we're gonna A/B test relentlessly and we believe we can get there." And so I think that's just a good example of how at a strategic level, you can translate a high-level business goal down to something that's purely execution-oriented. Because we set our goal to do 300 A/B tests by the end of the year, and we felt if we can do that, we'll get enough winners, and we can become viral. And so then everybody that we hired knew our goal and knew how we were gonna [00:03:00] get there, and there was something that they had control over. How many A/B tests could we get out there? Ashley Stirrup: Yeah, it's such a great example of where you just embedded it into the culture that you kinda hired for that and you onboarded for that and it just became part of the mentality Rich Liebling: Yeah, it was very much a core value. It was something we discussed in interviews. It was also interesting as you're recruiting engineers to a company that targets basically young women who are really into fashion, and our engineering team, let's just say, was very disjoint from that user base. But with culture we established of AB testing, everybody's ideas... Nobody was dismissed because you're not our target audience. What do you know about fashionistas?" Everybody's ideas were considered on the merits, and we decided which ones we would AB test. We had brainstorming sessions with everybody in the company. And it was often engineers' ideas that would win. And I think that helped motivate everybody [00:04:00] and was just great for the culture Ashley Stirrup: Yeah, I can imagine that being so empowering. You come up with an idea and it turns into a huge winner. And just in terms of order of magnitude you said you started at 50,000 subs and you hit a point where you were getting 30,000 subs in a day Rich Liebling: Yeah. That was like our high point, and that was, yeah, it was pretty amazing. Ashley Stirrup: Yeah. And to me, I think it's particularly interesting with your background in engineering. it's not often that it's the engineering team that's the champion for the experimentation program. Engineers often just wanna ship and move on to code the next cool thing. Rich Liebling: Exactly. And that's a key part of, we talk about it in, in interviews, and we used it to motivate the candidates that, they didn't have to be into fashion themselves. They'd have an important role here not just in terms of do what the product manager says, but you're a part of driving the product and your ideas will matter. And in fact for our new hires in engineering, while their first merge request would typically be updating [00:05:00] documentation about onboarding and getting your system running, their second merge request was very often an idea of their own to A/B test. And so that helped establish very, from the get-go, how we work and the engineers would invariably learn a lot from just running the A/B test. Ashley Stirrup: Yeah. And that's such a great way to get people started, 'cause often that first test can be a little daunting. You're not sure what to test or how to test, and once you've been through one, you're like, "Oh, okay, now I get this," kind of thing. So Rich Liebling: And we very much, we did a lot of testing and a lot of the tests were very small and very, it might just be changing the text on a button or the text and the color in different combinations. A lot were very small but you learn a lot by doing those Ashley Stirrup: Yeah. And it's been surprising. A number of guests have said that their biggest winners came from small changes in text like that, so Rich Liebling: Yeah, it's-- Once you see some very small tweak make a real difference you get sold quickly. Ashley Stirrup: [00:06:00] Yeah. Yeah, Microsoft Bing, I think, had an example where they added underlines to a certain part of the search results, and it led to I think it was $100 million in more revenue for them, so Rich Liebling: Yeah, that was a scale bigger than Shop It To Me operated at, but yeah. Ashley Stirrup: I'm sure. That's funny. Yeah, now tell us a little bit about Learneo and I guess Course Hero in particular, right? Rich Liebling: Yeah. So Learneo's the company. Course Hero is one of the websites that Learneo operates. And I've been with Learneo for six years now. And we do a fair bit of A/B testing, but not to the scale that Shop It To Me did. And there are a number of important, I think, differences in how we do it at Course Hero from different circumstances, and I'm sure many of your listeners probably have, somewhat different circumstances. Ashley Stirrup: Of course Rich Liebling: Shop It To Me was very on the refer-a-friend viral loop. And so the metrics there were pretty clear. You wanted to get users through the sign-up flow. You wanted to [00:07:00] get them to import their contacts and refer friends, and how many friends did they refer and how many of those signed up? And much of that would happen within the context of a single session. At Course Hero, things are a little bit more complicated because Course Hero basically is a platform for sharing study materials for primarily college students. And a lot of the content is free, but some of the content is blocked and you have to get access to it. And there's two ways to get access. One is you can pay and subscribe, but the other is you can upload content also to earn access. And so these two ways are somewhat in conflict, right? If people upload, we get more content, which is valuable to us, but we're not getting a paying subscriber. And so for example, one of the complications was for a while we had one team focused on getting content and one focused on getting paid subscribers, but they were essentially competing against each other [00:08:00] in many ways. And the success of one of those teams might hurt the other team. And so one of the key things for us was eventually to have a model for the lifetime value of a customer that we could evaluate early on, basically roughly at sign-up time, and use that as the metric that we were really focused on optimizing. Ashley Stirrup: Yeah, that's super interesting. And so do you kinda look at it as you have two batches of customers, paying customers and content-producing customers? Rich Liebling: Somewhat. It's also interesting because we wanna get the paying customers to upload content, and the quality of the content we get from them tends to be better. So it's a more complicated dynamic, I would say, than what we had at ShopItToMe. Ashley Stirrup: Super interesting. And how many users does Course Hero have? Rich Liebling: I don't know the current numbers, but it's been as high as a million. The AI revolution of the last few years has made business a little bit tougher for Course Hero. More students turn to [00:09:00] ChatGPT and stuff. So that's been a challenge. Ashley Stirrup: Still in the hundreds of thousands of visitors, so you have pretty good power to run tests. Rich Liebling: Yes. Ashley Stirrup: And do you have a central experimentation team? I'm just curious, like how do you spread the culture of experimentation across the team and imbue rigor and things like that? Rich Liebling: Yeah. So I would say, Course Hero never treated AB testing as much as a core value like at ShopItToMe. It was always a bit different. And as you mentioned, sometimes engineers are reluctant, they'd rather just make the change and not have to AB test it. And we've had, some of those challenges at Course Hero. We also... when I started the analysis of an AB test, we'd have a data analyst do the analysis, and there was no standard process for it. They would do a lot of ad hoc SQL queries and they might spend three days to analyze one test, which, just that level of friction tended to discourage a lot of AB test. Ashley Stirrup: Got it Rich Liebling: Yeah, [00:10:00] there's not... It was basically up to each team and each product manager, so Ashley Stirrup: Got it. It's a pretty decentralized model. Rich Liebling: Yes Ashley Stirrup: Yeah. And can you tell us about a time when you had an experiment with a lot of learnings? Rich Liebling: Yeah, I think going back to Shop It To Me, there was a page where someone had a hypothesis that, you know, this page, there's just too much text, it's too busy and if we shorten the text and just simplify, we'll increase conversion. And we initially ran that test, and actually the version with less text did worse. Not just didn't improve things, but it actually did worse. And so someone had the brilliant idea of saying maybe we should add more text, more explanation." And we did that, and it won. And so one of the things we learned was, and we made this a general practice, is someone has a hypothesis and you wanna test it, great, but also test the opposite of that hypothesis. And a couple important [00:11:00] learnings. First, it was surprising how often the opposite direction actually won. But also even just knowing that something had an effect, even if it was the opposite of the effect you wanted, it tells you that this thing, this knob matters, right? It affects what you care about. And so you should use that as an indication to, figure out how you can make it, turn it in the right direction. Ashley Stirrup: Yeah, super interesting. Love that as an idea to test the opposite and the fact that so that you had so often a time when the opposite was the winner. Just so counterintuitive. Rich Liebling: Yeah. It was, especially as new people onboarded and we'd get them into the brainstorming sessions and they'd have an idea for a test and we'd run that test, but we'd say, "Okay, but we're also gonna test the opposite of your hypothesis." Everybody felt a little weird about that, but it just became like, "Oh, yeah. Okay, that makes sense. Let's do that." Ashley Stirrup: Yeah, that's part of [00:12:00] what we do. Yeah and I love concept of sometimes the opposite, maybe it hurts the metric, but at least it shows you can move the metric in one direction or another, and that tells you spend more time on this versus something that you think will move the metric and neither direction does anything, then okay, I-- maybe I can become more confident that it's time to move on and try something else Rich Liebling: Yeah. I think the other important aspect of that is when you do very small focus tests, you can say what the opposite would be. But if you're completely redesigning a page and it does worse, you learn, okay, that version of the page doesn't work, but it's not clear what you do to try to make things better. And the more focused your tests are, I think the more clear your learnings Ashley Stirrup: Yeah. Yeah, it was interesting. We a webinar earlier today with the Philadelphia Inquirer and they really changed their sign-up flow and you looked at the A versus the B, and you're like, "Wow, [00:13:00] that A and B are different." And the data scientist is "I know. It's killing me. It's so different. We should've done this in a series of steps." But marketing was like, "No, we have to do this. We have to do it now." Rich Liebling: Right. And you can imagine that they probably had a bunch of good ideas, but then some bad ideas mixed in, and where does it balance out and how do you tell the good from the bad? Very hard to say, I imagine. Ashley Stirrup: Yeah. Although I often talk about in experimentation this idea of hill climbing, and sometimes you don't know where the next hill is. So sometimes taking those moonshots and then backing up might be a better approach, Rich Liebling: Yeah. So we at ShopItToMe we definitely had that in mind. And we ran, like I said, just in volume, tons of these very small focused A/B tests. But we'd also run-- We thought of it in terms of a portfolio. So we had the very small low investment ones, and we had medium tests and bigger tests. And bigger tests might be, as we're testing like a new product idea or something like that. Medium ones might be [00:14:00] bigger than like changing a button or email subject header or something like that. Changing more behavioral aspects and require more back-end work to go along with it or something. But I think the way we approached it was, you wanna make sure you have enough conviction about where you're going, that you're willing to say, "Okay, this didn't actually win," but we're willing to keep iterating and try to find-- We wanna get onto a bigger hill. We think that's important, but we we recognize that it's gonna take some iteration to decide whether we actually made it to a bigger hill or not. And not just, one guess and then say, "Oh no, that didn't work. We're not trying that again." Ashley Stirrup: Yeah, I think that's such a powerful mentality, 'cause I think many times, especially product managers, they'll have an idea, they fall in love with it. Maybe they don't wanna test it, but of course you should test it, and maybe the idea was great, but there's a slight tweak in the implementation you need to do. So having that conviction that this is an area [00:15:00] that I really believe in, and then go in with the expectation that I will try several variations of this. Yeah, super important. Rich Liebling: Yeah. I think related to this is also when you find a winning variation, you do an experiment and you get a winner, is not to just move on to something else, but to say, "Hey, this has promise. Can we make it even better?" 'Cause again, you found something that affects the metric you're looking to affect. There's no reason to believe you're now at the, at the optimal point. Keep going Ashley Stirrup: Yeah. Yeah, that's a really good call-out. And how often-- Like, how many iterations would you do? Let's say you got a winner. How would you think about how much to iterate? Rich Liebling: I don't know that we had any clear view there, but, if you can do another couple experiments and you find another gain, then you're probably motivated to keep going. But maybe at some point either the gains become very small or you're just not getting anywheres, and then you say, "Okay let's focus someplace else for [00:16:00] now." Ashley Stirrup: Yeah. Yeah, Ronnie Covey talked about his days at Bing and definitely like finding a vein and then like trying to go deep on that vein, so it's kinda similar Rich Liebling: Yeah, exactly. Yeah Ashley Stirrup: Yeah. Terrific. How do you see experimentation evolving at Learneo? Rich Liebling: One of the key evolutions has been that we built a multi-armed bandit framework. And the idea there I don't know how familiar your audience may be, but the idea of multi-armed bandits is that you basically constantly adjust the percentages you give to the different variations to try to minimize the regret. So you look at like what's doing the best, and you try to use that more while still balancing the explore. So it's basically dynamically adjusting the balance between exploration and exploitation. And for a company that's worried about experiments that may generate certain amount of loss because some of your variations don't do well this could make things easier. It [00:17:00] also can let you run things for a longer time and not worry about it so much. And Course Hero also our business and our traffic is very seasonal since it's timed to the school year, and there are certain times of the year, as we all know, where, exams are coming and there's a lot more interest in Course Hero at certain times. So sometimes we wanna let tests run for longer to see it across different times there. And the multi-armed bandit strategy can be a good way of doing that. Ashley Stirrup: Yeah, it makes a lot of sense. Often I'll hear people talking about wanting to open up experimentation to more people, make it more self-service, things like that. Are those areas of investment for you? Rich Liebling: We haven't done that much, but I think it is a very important thing that you want it to be as low friction as possible so that you can run, small tests and many tests. And so I think that's an important thing. We put a lot of work into that at Shop It To Me where I basically led the... we built our own AB [00:18:00] testing framework. Things were much different back in the 2000s. It wasn't as widespread, and there weren'tcommercial tools available for this. But yeah, lowering the friction is certainly important. And I mentioned, even lowering the friction to analyze the results also, of course, very important. But I think, at Course Hero our platform is generally pretty stable. I think the friction for us tends to come from interaction of multiple tests. And partly that's where a lot of the tests involve behavioral changes. And so now, when you add new things to the same area as a running test, there's just, you multiply the number of pass through the code and edge cases to consider Ashley Stirrup: Yeah. In terms of like culture and rigor, like how do you build enthusiasm inside your org for testing? How do you celebrate wins and how do you make people feel comfortable with losses? Rich Liebling: Yeah. So I think at [00:19:00] ShopItToMe that was really easy because it just, everybody in the company was an integral part of the, the testing experience. From the brainstorming to seeing the results. And we were also a very small company. And so I think that was really important. Course Hero oftentimes, like at all hands, the results of different AB tests will be presented. You'll see things there. But I think like with Course Hero, Learneo, it's not as big a part of the culture, so it's handled differently. Ashley Stirrup: Yeah. Rich Liebling: And at ShopItToMe, like I said, like as we were growing and when we would become viral, everybody was just constantly looking at the metrics and seeing our growth and so it was like, just constant excitement Ashley Stirrup: Yeah, it's interesting. It's a little bit like the snowball going downhill, right? You're just picking up speed, you're winning, and yeah, that kind of reinforces itself. Rich Liebling: Yeah. And going back to the strategy aspect, there's a flywheel effect, right? For a company like [00:20:00] ShopItToMe, as we would get more users, AB tests didn't take as long to run because we'd have more users, all the time. And like at ShopItToMe, we had, over the years, a number of essentially copycat sites pop up, and we always said, people would get a little worried, "Oh, this company's doing it now, this company..." And we always said that our big advantage was we have a headstart in our user base and our traffic, and that lets us move quicker. And that's, as you get bigger, the AB tests can be resolved more quickly, and that gives you an ongoing advantage over, someone starting smaller Ashley Stirrup: Yeah. I would also bet that just your expertise in experimentation was learning was a competitive advantage for you. Rich Liebling: Oh yes, very much yeah Ashley Stirrup: Super exciting. Rich, thank you so much for coming on the show today. This is a really fun example of how powerful experimentation can be, and I feel like we learned a lot Rich Liebling: Thanks so much for having me. I really enjoyed the discussion and yeah Ashley Stirrup: Terrific. [00:21:00] Thanks so much Rich Liebling: All right