Latest / The Experimentation Edge / Even a loss is a win: Charlie Health's approach to experiments
Transcript
- The Experimentation Edge - Joe Yevoli === Joe Yevoli: [00:00:00] whenever we're launching an experiment or thinking about something that we're gonna test, regardless of where it is in the funnel, we always try to think about, who are the teams that this is touching, and who's responsible for those areas? And before we move forward with anything, as quickly as possible, as early as possible, we are trying to communicate to them, "Okay, here's the bottleneck in the funnel that we see. Here's the hypothesis. Here's the test that we are thinking of running, like an early prototype as quickly as possible INTRO: Welcome to the Experimentation Edge, where product managers, data scientists, and engineers talk about how they make smarter decisions. I'm Ashley Stirrup, the chief marketing officer for GrowthBook, and in each episode, I'll sit down with an executive to unpack how they use experimentation and A/B testing to make better decisions. This show is sponsored by GrowthBook, the open source experimentation platform leader. Now let's jump in and get started with our [00:01:00] next guest. Ashley Stirrup: Hello and welcome to today's episode. I'm excited to have Joe Yevoli, Director of Growth at Charlie Health. Welcome to the show, Joe. Joe Yevoli: Thanks so much. I appreciate you having me. I'm excited to do it Ashley Stirrup: Yeah. Why don't we start with you telling us a little bit about Charlie Health? Joe Yevoli: Sure. So Charlie Health is a virtual intensive outpatient program. It is basically a virtual mental health company that sits in between people who might need to be hospitalized and who need a little bit more than, say, once-a-week therapy. So it is nine to 12 weeks, three sessions a week, three hours a session of intensive group and individual therapy. And since my time being here, I've been here for a little over a year, I hear almost daily talking to clients how Charlie Health has saved their life. It's incredibly rewarding. It's a great mission to be a part of. Ashley Stirrup: Yeah, this is something that's really near and dear to my heart. One of my [00:02:00] daughters went through a very similar, it was in person, but a very similar program, and it was literally life-changing for her. So Joe Yevoli: Awesome. Love to hear that. Ashley Stirrup: yeah, really excited to have you on the show and to be talking about a business that makes such a huge impact on so many people's lives. Joe Yevoli: Yeah, same here. Ashley Stirrup: So can you tell us a little bit about your role there? Joe Yevoli: Yeah, so I am the director of growth. The areas of responsibility are performance marketing, so the top of funnel. Life cycle marketing, so within the funnel and a little bit into the product. That's expanding a little more almost daily. And then obviously on-site experimentation, on-site and into the product experimentation. So any experiments that you see on any of the landing page and into the form, my team is responsible for that. And as time goes on, we're moving further and further into product, working on improving retention and things like that. Ashley Stirrup: Got it. And are you working with a number of different [00:03:00] product teams? Joe Yevoli: Yes. So we work directly with the product team, we work with the clinical team, we work with design, we work with eng. We basically work cross-functionally with almost everyone, also supporting the outreach team as well. That's one of the things that makes Charlie Health so fun, challenging, but rewarding because it is a fast-growing company. A lot of teams and responsibilities have sprung up seemingly overnight, and figuring out how we can properly communicate across teams has been a fun challenge. We don't always hit that challenge, but every day we're taking steps to get closer and closer to the ideal state, but it's fun. Ashley Stirrup: Yeah. One of the things that's so important with experimentation is, establishing the rigor, getting everybody to think experiment first, sharing lessons learned, all those types of things. How do you try to do that at Charlie Health? Joe Yevoli: Ooh so that's definitely one thing that we can always [00:04:00] get better at. But whenever we're launching an experiment or thinking about something that we're gonna test, regardless of where it is in the funnel, we always try to think about, okay, who are the teams that this is touching, and who's responsible for those areas? And before we move forward with anything, as quickly as possible, as early as possible, we are trying to communicate to them, "Okay, here's the bottleneck in the funnel that we see. Here's the hypothesis. Here's the test that we are thinking of running, like an early prototype as quickly as possible. And we just want to put this in front of you so we can get your feedback, get your thoughts, get your hesitancy, figure out what may or may not work, what we're not thinking about." And then always trying to document that conversation and communication, and then obviously documenting the results afterwards. So regardless of what we do, whether it's a win or failure, we are learning from something, we are communicating that outward, and we are also figuring [00:05:00] out what our next move is. For me, one of the funnest things about experimentation is regardless of what happens, there is usually always a next step to test even if things don't go as planned. And so as cheesy as it may sound I can feel myself eye-rolling internally. As cheesy as it may sounds, even a loss is a win Ashley Stirrup: Yeah. Yeah, that's a very common topic on the show is that so often the losses are where all the learnings come from. Joe Yevoli: 100%. Yeah, totally agree Ashley Stirrup: and I think it's so interesting. I think that the best teams are the ones that are able to kinda connect the dots with, okay, this failed, and what should I do next? And did I gather enough information on the first test to really help me know where to go next? Joe Yevoli: Totally agree. And actually I think that last point is really interesting because we launched a test in the funnel where we removed a page of it, and that page, from my point of view, essentially [00:06:00] functioned as almost a second landing page. It was the first page of the form when people clicked the Get Started button, they land on our form, and that first page of the form gave them more information. It wasn't actually relevant to their signing up. So from my point of view, it seemed like, okay, this user lands on our landing page, they click a Get Started button. They are actually looking to get started, but we are actually landing them on another landing page, right? So we removed it, and we ran that test for about three months. We saw very positive stats, big results. It was something that we wanted to move forward with. But what we also saw was that We were inflating the, unnecessarily inflating the top of the funnel because what was happening was that first page of the form was serving as a level of information that the client needed. It was giving them some information that they wouldn't have otherwise gotten. So even though the test ended up positive, what we saw was, okay, [00:07:00] there is another bottleneck being created here. We are letting too many people through. They're not getting information that they were previously getting. And okay, even though having that first page of the form on the form isn't the optimal situation, we still wanna figure out how we communicate the information that people need to know before they actually get into the form. And that led to another test that we're about to launch, which is actually a FAQ sort of chatbot on landing pages that will provide that information if people are interested in it, and hopefully we can inform people properly before they actually get into the funnel. But then once they get into the funnel, the friction is removed and they can properly get through the whole thing. And that was all because it was a win, but we didn't stop there. We looked at all the information and figured out, okay, what's actually happening from this change and what can we learn from it, and what else can we do? Ashley Stirrup: Yeah, I love that. That is such a great story. Yeah, that's one of my favorite parts of hosting this show is that you talk to people about different, [00:08:00] air quotes, buyer's journeys and understanding the user and what do they need and am I doing things that add value to that journey or not? Joe Yevoli: Yes. Yeah. The doing things that add value to that journey I really like that a lot. That is so important, not always easy to remember, but remembering that, doing that first is where I think you make the biggest improvements. Ashley Stirrup: Yeah. And what I think's so interesting about the example you just gave is you're kinda trying to balance these two things of how do you give value to the people that need value while removing friction for the people that don't need that particular value or that extra information, so Joe Yevoli: Yeah and to your point, that all came from really putting ourselves in the user's mind frame, right? When they come to a landing page, they click Get Started, the experience that happens next should be more along more or less aligned with what they're expecting. Most people, when they click Get Started, is not expecting to [00:09:00] get further information, right So we jump them right into the form, and that proved to be the right thing, but it led to some externalities that we learned from, and we're gonna adjust and see if we can learn some more and improve the funnel even more. Ashley Stirrup: Awesome. Do you have an example of another experiment where you had a lot of learnings? Joe Yevoli: Yeah. Tons. Yeah. I was thinking about this a lot in prep for this conversation. So one thing I would say that was really learning and led me down this like retention learning curve was my time at Teachers Pay Teachers. And a little bit more context, Teachers Pay Teachers is an online marketplace for teachers to be able to sell supplemental resources to other teachers. So in America, the way that it works is teachers have a curriculum that they need to teach. They are responsible for following a curriculum and giving the tests that go alongside that curriculum. But anything else in between, the quizzes or the homework or the in-class [00:10:00] dittos, things like that, teachers are required to either buy them or make them themselves. So Teachers Pay Teachers existed for teachers who are making that, the supplemental resources to be able to sell it to other teachers. That was another part of my career that was incredibly rewarding. It wasn't hard to say, "I make it easy for teachers to teach kids every day." Ashley Stirrup: Yeah. Yeah, I love that Joe Yevoli: so at the time that I was there, midway through, This was during the pandemic, and we had launched a new digital-first product. It was called Easel. And I think when people launch new products or build new companies, and I've certainly fallen prey to this and even at the time I definitely did, people's instinct are, "Let's see if we can build something for everyone, because we wanna make sure that the most amount of people use it and can get value from it." And so what we ended up doing was we built this digital-first supplemental teaching tool where teachers can teach kids online on the computer. And what we did was we basically [00:11:00] made it a PDF overlay And our thinking was this will work for every single supplemental resource the entire company. So let's just build this, roll it out, and then we can iterate. And what ended up happening was we got obviously a lot of teachers to experience it immediately, but then they weren't retaining it. And so in our mind it was like, okay, it's the middle of the pandemic, we're teaching digitally. are people not using a digital tool? How are they not utilizing the thing that we're building for the moment, right? And as we were digging in and we were talking to teachers, what we were understanding is we built something way too wide, and teachers were almost putting their own meaning into what they were experiencing, right? So they would experience Easel and then they would say something like, "Oh, this is a PDF overlay tool," or, "This is an iPad teaching tool," or whatever. They would make up their own definition, right? And that was just missing the mark for what [00:12:00] we were actually trying to teach. So we took a step back and we were like, "Okay, what's the right way to go about this?" And from the interviews that we ran, what we were hearing is what most teachers needed at the moment was a homework tracking and curriculum progress tool, essentially. I'm not saying that in the best way but that's what we were doing. They needed something to assign homework, track that people did it, and then track progress, right? Ashley Stirrup: Yeah. Joe Yevoli: And so we ran a really lightweight test positioning Easel as that. We built some lightweight features to actually have a minimum viable product and actually get it out into the market as quickly as possible. And what we saw was for that product in that minimal release, for the people that we properly positioned this tool f-for as a homework assignment tool that allows them to track progress, when they experienced that tool in that way, solving [00:13:00] that very specific need, I think we cut out probably about 80% of the teacher market for that one feature. We ended up I think probably quadrupled. , I don't remember the exact numbers, but it was an incredibly big improvement. And the lesson was, actually at the time we were reading this book called "Crossing the Chasm," and the lesson Ashley Stirrup: book Joe Yevoli: yes, fantastic book highly recommend it. But the, the lesson was the D-Day strategy, right? You find your narrow beachhead, you find your niche, you focus narrowly on how do I make this product as best as possible? And then you use the initial users of that niche product to figure out where your reference point to go next, right? And we never actually got to that point at Teachers Pay Teachers, but what we would have looked for was, okay, we have built this homework assignment tool, but where are some people using it? Are they using it in classroom? Are they using it to do tests, right? And that would've [00:14:00] informed the next product, the next iteration of the product that we would build, which would've expanded the market, but it would've expanded it in the right way. Going to everybody at first, you build something for everyone, you actually build it for nobody and you burn that first impression. you lose that first impression that is so important. Hi everyone. Just a quick interruption to today's episode to ask you all to like and subscribe. It would really help the show, so if you're enjoying this as much as I am, please do so. Thanks so much Ashley Stirrup: that's such a great story. As somebody who's done a lot of entrepreneurial things in my career, I love that when you find that beachhead and you find magic, even if it's with a smaller subset, right? If you can get that ma- it's so hard to get magic anywhere Find your beachhead. Joe Yevoli: Yeah, I remember having conversations with people internally at the company and, a bunch of people were pushing the niche approach, and the responses was we're cutting out so much of the market." And I totally get that instinct, but it in my experience anyway, it always worked better [00:15:00] to focus narrowly and expand from there. Ashley Stirrup: Yeah. Yeah. People are busy. They have lots of choices. So yeah, the more you can really resonate with the group. And I think one thing, to take that lesson and apply it to a lot of our listeners, I think it's as you're thinking about features, think about different user populations. Your most frequent users, your power users versus new users, and maybe a feature you're about to A/B test is gonna resonate really well with one group but not another Joe Yevoli: Totally. Yeah. One of the biggest things that I've learned over the years is just taking the time to understand segments, understand the problem that the the product is solving for them, and understanding the frequency that they usually experience that problem. It's so helpful in designing products that are-- allow people to properly activate, to retain, and to really get value in the proper way. And that all comes from just putting yourself [00:16:00] in the shoes of and understanding the day-to-day of the different segments that your company or product is serving. Ashley Stirrup: Yeah. I really think in the, at the end of the day, that's what A/B testing's all about is trying to understand your users and how any feature's impacting a user journey. And Joe Yevoli: 100%. Ashley Stirrup: It it can be easy to look at the average, right? When and you're actually missing out on, oh this group of users is having a very different experience than this other group. So Joe Yevoli: Totally. Yeah. We we see that at Charlie Health every day. There are people who refer to us at a high frequency, and there are other segments that are-- that refer to us at a much lower frequency, and the solves for both of those segments are entirely different. If we approached the whole thing in the same way, we would get a very muddled view of what is actually working, and we wouldn't actually be solving anything for those two particular segments. So yeah, it's really important to think that way. Ashley Stirrup: Yeah. So let's say [00:17:00] you worked with somebody, started to work with someone new on the team, and they have a feature they're all excited about, they're ready to A/B test it and they're not necessarily thinking about what if it loses. They don't realize how many features lose. How did you help them think about the experiment design so they'll get the most learnings, especially if that feature loses? Joe Yevoli: Yep. So if somebody comes to me with an experiment design the first way I think about it is there, there are two questions. It's okay, what's the user data that you're seeing that's informing the hypothesis that you're trying to prove or disprove? And then what is so what's the quantitative data, and then what's the qualitative data, right? Because you may see that a certain subset of people are behaving a certain way with data, but you'd be surprised about what you hear once you actually talk to them and understand why they're behaving in the way that they're behaving. So any new thing, I always try to think about it [00:18:00] through those two frameworks. And then let's say the data that we're seeing on both sides is supporting the experiment that the person or the new feature that the person is proposing. From there, it's basically about thinking, okay, what is the full funnel user experience for this feature? And what are we putting into place to make sure that we are measuring everything as much as we can every step of the way, and we are ultimately measuring towards a proper end goal, right? So it's one thing if you build a feature that increases opportunities, but it doesn't matter if it's not producing the end result, which is more sales, Ashley Stirrup: Yeah, right Joe Yevoli: And so you may have a good idea for increasing opportunities, but if it's not actually producing the result that the company actually needs, that provides business value it's not worth it. So what's the full funnel? What's the proper end [00:19:00] goal? Are we measuring everything in between? And then to your question, how am I prepping them for the fact that this most likely will use? I often like to say Especially with experiments, I'm very rarely launching anything that I don't feel really good about, right? But what I always say is, "I don't know anything." I've consistently been surprised. And try to remember that even if it's a loss, there is a lesson to be learned, and there is a next thing to be done, like I said earlier. So if we've set up where we're launching a test for the right reasons because we looked at the data in the right way, we are tracking everything to, to the best of our ability, and we are tracking to the end goal that drives business value, you- we've covered our bases. And then no matter what happens, we're gonna look at the data, and we're gonna see what happened, and we're gonna say, "Huh, that's interesting. What's going on here? Oh, maybe this means this. Let's talk to some users here. Let's figure out something different that we [00:20:00] can do." And eventually, more often than not, you get to that win that way. Ashley Stirrup: Yeah. Yeah. Yeah, the people always talk about, when you're kinda new in a job, you don't know what you don't know. And I think one of the most powerful things is when you have that humbling moment when you realize all the things you don't know that A/B testing can help teach you, Joe Yevoli: totally. Even recently that, that recent example of removing that first page of the form. We increased people that were coming into the top of the funnel and conversion rate was dipping. And obviously this is something I've experienced so many times over the years, but I I just forgot that it's it's almost like a law of thermodynamics right? Like a law of physics. Like you increase people in the top of the funnel, by definition, conversion is gonna get worse even if you are getting more people at the bottom of the funnel. And I really thought that first page was not giving proper information, p- not [00:21:00] information that people needed, and removing it was only gonna get more people through, and we were just gonna get more admissions. And it was just another moment where it's like, man, I had experienced this before. I had totally forgot that this was gonna happen. And another reminder of you don't know anything. There are so many things that you're not thinking about and you're forgetting that it's a surprise every time, and it's what makes experimentation really fun. Ashley Stirrup: Yeah. And you were telling me that for your more important tests, you've implemented something pretty interesting premortems? Joe Yevoli: Yeah. Yes. Yeah, so we don't do this with every experiment. It obviously depends on how big the population it's gonna test and, how risky it is or things like that. But assuming it's a large enough test, it's gonna it will eventually test a or touch a large enough segment of the user user experience, and it could have some possible negative implications if things go wrong. What I have been a really big fan of and try to implement [00:22:00] every place I've gone has been this concept of pre-mortems. And so basically what we do is we have an experiment, it's a big experiment. We try to get everybody together before the experiment launches, and we actually say to each other, we say, "Okay, we launched this experiment. The experiment wrapped," or it's day of, right? " It was a massive failure. What went wrong?" And then the room goes silent. Everybody writes down everything that they can think of that could possibly go wrong on the day of launch or at the end of the experiment. And then we just go around the room and everybody reads what they wrote down. And this does two things. It-- First, it provides an environment where people feel comfortable dissenting, right? I'm sure everybody listening to this has experienced some version of, " I think this might be a problem, but I don't know, we're pretty far [00:23:00] along and everyone seems super confident. I'm not gonna say anything. I don't wanna make people mad. I don't wanna upset people," right? And then sure enough, the thing happens and you were like, "Oh my God, that was exactly what I was worried about. I can't believe no one else was thinking about it." So it provides the floor for people to feel comfortable saying that thing. But then also it gives us a chance to be prepared and to think about what we're gonna do to mitigate the worst case scenario, right? So it allows us to say, "Oh, okay, we aren't thinking about this. What can we do to make sure that this isn't gonna be a problem?" And it's always led to some implementation of a safeguard that previously wouldn't have been there that has helped us avoid catastrophe. That doesn't mean that every test is a winner. Quite frequently they are not. But it does prepare us from-- It does make sure that we're not missing stuff, and it's been very helpful. Ashley Stirrup: Yeah, I love that. Yeah, I can definitely see that, especially if you've got, a champion in the room who's just [00:24:00] so excited about a new feature. Can be hard to bring up the here's why it might not work." Joe Yevoli: Totally. And when the experiment is proposed by, say, the CPO or the CEO, right? Who's the associate in the room that's gonna be like I don't think so." You gotta make it possible for everyone to be able to voice their concerns, and I think premortem really makes it easy to do that. Ashley Stirrup: I love that. Yeah, it just it opens up the opportunities to, just be more kind of proactive, think ahead more and yeah, it gives you a better chance to win and probably more data as well. Joe Yevoli: Yeah, definitely. Definitely. And sometimes it even leads to additional experiments that we run because just sitting there and brainstorming what is happening in this test is... It leads to some interesting ideas and connections Ashley Stirrup: Yeah. Yeah. Maybe add a C or a D to your test as well, another variant. Joe Yevoli: Yeah Exactly. Yeah, exactly. Great point. Yeah Ashley Stirrup: Yeah. So how do you [00:25:00] see experimentation evolving at Charlie Health? Joe Yevoli: Yeah. So obviously right now AI is-- everyone's thinking about AI. AI is touching everything. I think the thing that I am personally struggling with and trying to figure out how it fits into the organization and in my career in general is AI is basically flattening the organization. And so previously, in every iteration of my role, there has been some version of a pod that looks like a designer, a product manager, a growth person, a product marketer, an engineer, things like that. And everyone is working in s- in concert in their area of expertise to make sure that whatever we're working on is the best version. And that requires a lot of cross-functional collaboration, and it slows things down in a good way. But now AI is just allowing one person to be the engineer, to be the designer, to be the product marketer, to be the PM. And it [00:26:00] is speeding things up in a way that is obviously really helpful. But it's also scary. And so it is leading to situations where you can move really fast, do a lot of damage in ways that you couldn't do it before. And so how do we get the best of both worlds? How do we move as fast as we can, but how do we think more completely? That's the world that I'm sitting in daily right now. I'm not saying... I can't say that I have all the answers for it right now, but that's what I'm thinking about day to day, and that's what I think the world is-- how the world is gonna change over the next couple months to a year, five years. Ashley Stirrup: Yeah. Yeah, there's a lot of talk in the industry. Even putting experimentation aside, there's just a lot of talk about how does all this affect the designer role, the product manager role, the engineer, and, everybody's doing a little bit of everything now. Joe Yevoli: Yeah. And in [00:27:00] my experience, I know that we in tech we want or we're enamored by the, the version of AI that is going to replace a designer or a marketer or an engineer and things like that. And what I am finding more and more, even as models are getting better and better, it is just always a little off. Ashley Stirrup: Mm-hmm. Joe Yevoli: It's never quite as good as when you put work in front of a designer or when you put work in front of an engineer or when you put work in front of a PM. And I still really believe those people are essential. And what AI is leading to is this habit of thinking you have the answer when you don't fully understand it, and that is super dangerous. So I'm consistently trying to think about, okay, how do we utilize AI to make us work as fast as possible and be as impactful as possible, but also how do we make sure we are thinking things through properly and not removing the human from the loop as we're essential, obviously. Ashley Stirrup: [00:28:00] Yeah. Yeah. From my personal experience AI is at its best when it's kind of an accelerant. But that, you can't just say, "Oh, go do this," and then suddenly you get back a, a new dashboard and you're like, "This dashboard looks great, but there's something not right." Joe Yevoli: Yeah. Exactly. I have tried to force myself to write things down before I prompt AI or I ask AI anything. And sometimes I've found myself writing things down, reading what AI is saying back to me because I have fallen into this a lot. You ask AI a question, it looks at the data, it comes back with the answer, and you're like, "Okay, great, I got the answer." And you go and you report it out, and somebody asks you one follow-up question, and you're like, "Oh, damn I haven't actually thought about that," right? And that's the habit that I see a lot of people... the trap that a lot of people are falling into, and I'm really adamantly trying to force myself to not do that, to think more based on what I'm getting back from AI. Ashley Stirrup: Yeah, but [00:29:00] just also to apply what you were saying before about kind of the acceleration and the merging of roles and everything, the things that we have to be thoughtful about is how do we make sure that, as we're rolling out more features, we're actually learning that much faster as well? Joe Yevoli: Yeah. Yeah, Exactly. And that's where that sort of loop or that process that I walked you through before is instrumental, right? You, can't just say, "Hey, I set up an A/B test, and let's track if it does X, Y, or Z." You need to think through that process, and AI can be helpful thinking through that process, and AI can actually be helpful with that pre mortem process that I laid out as well. But you need to take a step back and think through holistically, "Okay, here's what we're trying to accomplish. Here's the hypothesis. Here's the test that we're gonna set up. What's the full user journey? How will this actually work? And how are we sure at the end of the test that we are getting everything we need to learn what happened or what didn't so that [00:30:00] we can make a decision about where we go next?" Ashley Stirrup: Love that. And that's a, a great note to end on. Thank Joe, so much for coming on the show. I feel like we learned a lot. It's such a powerful business that you're in, and it was great just hearing about how you're kinda helping people's lives. Joe Yevoli: Thanks so much, Ashley. I really appreciate you having me on. This is a lot of fun Ashley Stirrup: Thank you.