Latest / The Experimentation Edge / Diligent explains why moving on from an experiment might cost you
Transcript
- The Experimentation Edge - Daniel Layfield === [00:00:00] Ashley Stirrup: Hello, and welcome to today's episode. I'm excited to have Dan Layfield, Director of Product Management at Diligent, join us. Welcome to the show, Dan Daniel Layfield: Hey, Ashley. Thanks for having me Ashley Stirrup: Maybe to kick things off, you could just tell us a little bit about yourself, your background, and a little bit about Diligent Daniel Layfield: Yeah, sure. So I guess I've been working in product management for probably pushing 15 years now. So I spent most of my time in B2C products. So I was head of growth at a company called Codecademy, which was like a big mid-2010s startup, came out of Y Combinator. Then I was up came at Uber on Uber Eats, and now I'm a director of product at a company called Diligent. Diligent's one of those biggest things you've never heard of. We're probably like 2,000 people. We serve maybe 70% of the Fortune 1000, and we work on really bespoke corporate governance problems that the biggest companies in the world have Ashley Stirrup: Got it. It sounds like both your time at Code Academy and Uber, you spent a lot of time on AB testing Daniel Layfield: Lots of A/B tests. Yes Ashley Stirrup: And could you tell us a little [00:01:00] bit about the types of experiments that you ran and were you... Did you have an experimentation team that you were working with within your unit or was it more of a distributed thing where you were working with a centralized experiment team? Daniel Layfield: Yeah, I'd say it was a mix of both. So m- I'd say when I got to Codecademy, we were maybe 20 people total in the company, and when I left, we were about 250. So I ran growth from about 10 million ARR to about 50. Codecademy was like a mostly organic and SEO-based B2C subscription product. It was like one of the biggest things to learn to code in the mid-2010s. Even that sentence makes me feel old. But we kinda got to there when the free product was really well built out, and we were figuring out how to scale the paid subscription product. We eventually figured it out. It got sold two or three years ago on really good terms. But part of figuring that out was learning A/B testing. So Codecademy was small enough. Maybe we kicked this off when there was... I'm gonna say our growth team was, like, six or seven people and maybe the company's a hundred people. So it wasn't big [00:02:00] enough to really have centralized infrastructure. We ended up building most of that stuff ourselves. We worked with a really good agency which was formerly called CXL. It's now called Speero. So they did a little bit of our execution, but also really taught us the muscle of experimentation and helped us advise on tools, tracking, types of tests, et cetera. Uber was a completely different story, where Uber I got there in 2021. So Uber is a giant 25,000-person company at that point. Centralized tooling. Experimentation at Uber, at least on the Eats side, was very distributed in the teams. So pretty much every team has a data scientist. There's centralized tools. And I think for the time I was there, so 2021 to 2023 100% of the things shipped were A/B tested. So database upgrades, library upgrades, anything touching the product was A/B tested, which in most companies is probably overkill. Uber's a little tricky 'cause a, in a big consumer application like that, a 0.1% increase in conversion is hundreds of millions of dollars Ashley Stirrup: Yeah. Boy, [00:03:00] quite a contrast between those experiences. Daniel Layfield: very much so Ashley Stirrup: Yeah. Maybe we start by talking a little bit more about Code Academy. Was there an experiment that you did there where you had a lot of learnings? Daniel Layfield: Yeah, totally. We-- I think the biggest win we had there is we worked on our trial model. So like most consumer applications, like one of the biggest growth levers typically in, call it like eighty percent of scenarios, is getting the right trial functionality set up. So that means where do they see it? What are the terms? Do you need a credit card or not credit card? How long is it? What do you get exposure to it? How does it convert, et cetera? Codecademy initially went from just a direct purchase product. So like you wanna buy Pro, which is the paid product, you sign up, we charge you immediately, I don't know, twenty bucks a month or something, and then you're in it. A PM before me ran a really successful experiment implementing what's known in the industry as a reverse trial. So like everybody who signs up on sign up, you get Pro, and then we take it away after, seven or fourteen days, and then we make you make a decision. My [00:04:00] team built a trial after that where we removed the auto-enroll part and made people decide if they wanted to start the trial and put a credit card in upfront. Took a lot of balancing to get it right. We did it in probably three waves, but we eventually got to a thirty-five percent increase in conversion, which was massive for the company. Ashley Stirrup: Got it. And what kinds of things did you learn about that? Daniel Layfield: So much of what makes consumer trials and freemium models work is really like where and how they discover the product. And something I always call like paywall structure, which is really like free users bounce around the free product. How often do they come in contact with the paid product? And are they coming in contact with it at a point where the paid product relieves some pain they have? So really do you introduce it in the right way and do you sell it in the right way? The most aggressive ways companies do this typically is like you hit someone with a pop-up right after sign-up and see if they wanna start a trial. You might get decent volume like that, but you're like selling-- you're getting [00:05:00] people into an experience where they're not really sold on it. So typically you see really high churn rates and bad trial conversion rates Ashley Stirrup: And so were you giving them some of the pro features for a time before you asked them to commit? Or Daniel Layfield: No, we Ashley Stirrup: to know the free version and then showed them what was available in the pro? Daniel Layfield: Yeah, we ended up letting them bounce around the free product first and then introducing pro at the right point. But with that we ended up balancing the paywall slightly and like moving a couple gates around earlier in courses. Consumer especially B2C product management in subscriptions especially is very math heavy just 'cause you're working on giant funnels Ashley Stirrup: right. That's what I was thinking. That's what it sounds like. So sounds pretty interesting. And how do you recommend people approach, say, a losing experiment where they wanna try to extract as much learnings as possible from it? Daniel Layfield: Yeah, great question. I think the big question is always like why it lost. Most experiments when I've run them, when you [00:06:00] lose them, when they lose, you tend to not get statistically significant negative results. You tend to get like non-statistically significant positive results or non-statistically significant negative results, which really means that you don't really know what happened. Most of the times we've done that, and I think a mistake I made a lot in my career is like you move on too early. Your life as a product manager is very much like you have a roadmap of stuff to get done. And like anytime you stay in a project too long, you're not doing the other projects. But I think the big lesson that I learned in the Codecademy trial model is that probably took us four months and three to four rounds of tests to get right, but we uncovered like a massive unlock. Ashley Stirrup: Yeah. Daniel Layfield: It's like really there's not really like a clean framework on to know when to move on or not. But one of the things that we did in that experiment which was really helpful is like we saw a test not work, and then we put every screen in the user experience onto a giant Figma board and overlaid all the metrics. So whenever there was a decision point, like what [00:07:00] percent went to A option, what percent went to B option. And then you could almost see how the users are flowing through the application and figure out when you wanna move stuff around Ashley Stirrup: Got it. And what told you, "Oh, this is an area we should really stick with for a few months and just keep iterating?" Daniel Layfield: Part of it is common sense. So like we're doing this in probably 2019, like late 2019. every good consumer product is mostly running a trial of some form at this point. So like trials are not like a crazy concept in the industry. Like we're seeing like Headspace does it, Calm does it, Spotify did it for a little bit. Like pretty much all the big B2C guys do this. So there's like a way this would work for us. Also, like the payoff is really big. So if we can get the main monetization engine to be even 15% more efficient, consumer businesses compound, so it like really starts to add up fast. And then when you go through the data deep enough, you realize like we're making a lot of subtle mistakes that if you fix, we could see another one. We could see [00:08:00] a win Ashley Stirrup: Yeah. Yeah, that makes a lot of sense. That's super interesting. Just looking ve- so closely at every step of the process for a business like yours is just so critical Daniel Layfield: Yeah, exactly. I think there's to me, there's kind of two flavors of experimentation generally. There's more like the traditional CRO, let's do a little bit of research and let's launch a high volume of tests to see if we can get 25% of those to be small wins. The other one is like we're trying to do something big and we're trying to de-risk it and understand it, and then those, I think it makes a lot more sense to take multiple shots at. There's only so many email subject line heading tests you could run, and like all of them are when they trickle all the way down to purchase from the top of the funnel, like none of them are super impactful. It's really the volume that matters Ashley Stirrup: Yeah. And so then you move on to Uber where everything's a test. Daniel Layfield: Everything's tested. Ashley Stirrup: and I can imagine that was just a-- required a bit of a mindset shift for you Daniel Layfield: Yeah, definitely. So [00:09:00] specifically I was working on the home feed ranking system of Uber Eats. So it's like different than Code Academy where I'm doing it more of a product growth job. Uber Eats, I'm a very much like backend focused PM, which is working on a really vital system. So I'm probably gonna butcher these numbers slightly, but at the time something like 50% of Uber's revenues Eats something like 50% of all Eats orders initiate from the home feed. So it's a really big surface. The trick with the home feed, I had a previous PM describe it to me like this, is like you it's almost like the Middle Eastern oil. Anywhere you drill, you hit something. So it's much trickier to balance the trade-offs. So in a three-sided marketplace like Uber Eats, you have the initial bookings you capture, you have profit, but you also have quality of restaurant delivery times. You have courier avail-availability, you have consumer retention, you have merchant retention. So it's you're-- the Uber AB testing systems are very sophisticated. So any test you run, one, you have a [00:10:00] ton of traffic. Maybe we had again, I could be getting this number wrong, but something like 40 million people on the platform a month. So you can test down to like very small increments. But also in a test, the testing systems would spit out 40 metrics. So in almost every test, some go up, some go down. So like how do you have the frameworks to know if this is a good idea or a bad idea? Ashley Stirrup: Yeah. And if you're testing every single thing, I would imagine that a lot of them are more like, "Okay, I just wanna understand what did, what impact did this have? Did it do no harm? Did it have a slight up, a slight down?" And then there's the other experiments where you're like, "No, I really am trying to understand the buyer, their journey, and whether what I did helped them or hurt them," or things like that. Daniel Layfield: Yeah, I think what makes those things easier is if you should be like You should go into projects with an intent in mind. So like when you write PRDs, like e- pretty much every feature you build at Uber [00:11:00] has a set of metrics goals. So it's like we're upgrading our conversion model from this deep learning framework to this deep learning framework. We think this should improve conversion, and we think it should have the following other secondary impacts. And here are the guardrail metrics we don't wanna break. Then you launch the test. Ideally, you like see what you were hoping for, but almost always there is some complication. And then are you being intellectually honest with the results of the test? So maybe we thought conversion would go up, and then we see order defects go down. And it's is this actually like explainable Via what we did? Or is this one just a false positive? Even if we run to, stat sig level of, .99, you do get 1% of false positives in-- across thousands of tests, they do show up. Ashley Stirrup: Yeah Daniel Layfield: then even if you're causing like a secondary impact you like but may not have intended, can you like... Is this really explainable from what we did? In which case you take some of the wins Ashley Stirrup: [00:12:00] Yeah, and obviously for anything that's more substantial, then it's probably worth it to say, "Okay, if I drove this behavior, I should see these other metrics also move, and can I go find Daniel Layfield: does this story make sense? If we do something that unintentionally makes the delivery time go up a lot, and we also see like customer satisfaction go up a lot, like that doesn't make a ton of sense. Something's probably wrong Ashley Stirrup: Yeah. That's a great example. And now that you've moved into more of a B2B type of model how would you recommend product managers typically think about that? 'Cause I think it's just an interesting question because, in, in B2B you're adding this new feature and maybe you're measuring adoption. Often I think B2B product managers aren't even doing that. They're just cranking stuff out. And so how do you bring that experimentation mindset to a B2B world? Daniel Layfield: Yeah, I think the first question is normally like, how does your B2B company like conceive of product management? So one of the reasons I [00:13:00] got brought to Diligent with my boss who was my boss at Code Academy, is like Diligent's leadership wanted to adopt more of a product first mentality. So a lot of B2B companies, the PM teams are more like feature factories, where like you have 1,000 clients, the top 5% of paying ones request stuff in every QBR, and it's the PM's job to like ship that stuff. And that does a good job of making clients happy in the short term, but you end up in these like weird disjointed products in the long term. So I think the first step is like a foundation of what I would call like good almost like Silicon Valley style product management, where you run some sort of top-down OKR system. The teams and the areas that work on have North Star metrics that are thoughtfully chosen that like somewhat ladder up to the things that the businesses care about. And then the teams and the features are picking metrics that like ladder up to the North Star metrics. Typically, when I see... I guess the times that I've run things like a planning process that goes wrong, it's usually like each layer is working correctly, but [00:14:00] it's the connections between the layers. So like business, the leadership chooses meaningful business goals, but the North Star metrics of the teams only like loosely relate to those Ashley Stirrup: Yeah. Yeah, it is so interesting. Everything you're saying makes total sense, and I love that kind of framework of, okay, maybe our top-line metric is revenue, but for these individual products, we're gonna more optimize on adoption and usage 'cause that drives stickiness and renewals. And then, like you said, you go down the the funnel from there. So often in an enterprise business, you're just saying, "Is sales selling more?" And then you're not really focused on any of those adoption metrics Daniel Layfield: Yeah, exactly. I think Diligent's leadership took like a really, what I think is like smart investment in doing a lot of hiring in product and engineering design probably two years ago, and we're like really seeing the benefits now. So B2B, the trickiest thing about the, I guess the world of AB testing is if you're a good B2B company, a [00:15:00] lot of your com- your clients are on long-term contracts. So there's a very like delayed effect between like product usage and actual like dollar retention. Whereas B2C it's almost immediate, like the second they're unhappy they leave, which makes B2C like typically you have worse retention, which makes the businesses harder to scale, but also like it's a much faster feedback loop for the PMs to optimize Ashley Stirrup: Yeah. And I think that is, is so critical Daniel Layfield: Yeah Ashley Stirrup: E-even in a business like that where the long-term revenue, is harder to adjust, really understanding your usage and adoption could be so critical. Long ago, I was at Siebel Systems, and at the time we were super successful in CRM. This was before Salesforce. And our buyers loved us, and the users hated us. Daniel Layfield: Yeah Ashley Stirrup: eventually that came back and bit us. Daniel Layfield: Yep Ashley Stirrup: I think if we had been better at tracking those metrics, it could have been a very different outcome for the company Daniel Layfield: Yeah. Yeah, it's a classic B2B problem Ashley Stirrup: And [00:16:00] so at Diligent, how do you think about what's your North Star metric? Daniel Layfield: Yeah, I think we think of them differently by product. I'd say the biggest product at Diligent is a board of director suite. So if you were on like the board of Apple you can't upload all your non-public financial information to like Google Drive 'cause it's not secure enough, it doesn't have the permissions, doesn't have all the enterprise bells and whistles, doesn't have two-factor in some situations. So Diligent's main product is like a core collaboration suite for the biggest companies in the world. So name probably any famous board director, I would almost guarantee they have a Diligent account of some kind. We anchor the current boards product mostly on director side usage. So if you think of the core value of Diligent, the way this worked before Diligent is people got board decks FedExed to them in their house in the mail. Diligent is not really helping users if they just use us to receive and print a document, in which case we're a way more expensive version of just like [00:17:00] transmitting documents securely. We're much more valuable if directors like use it as a real collaboration suite, and like comment take edits, use our AI features prepare on the plane, et cetera. So we anchor most of that board on the usage. The trickiest thing about the use case we're in is it's infrequent. So if you're on one board, you probably have a meeting once a quarter. So you use Diligent like the week before and a couple days after. And if we like try to push more usage than that, it starts to feel spammy. 'Cause unless someone sends you a board deck, there's nothing for you to do. You might have committee activity, but it's like a... Retention and engagement should always ride whatever the natural use case is. Ashley Stirrup: Yeah. And do you have guardrail metrics that you look at? Daniel Layfield: Yeah. I think we look at probably I'd say the normal complaint level metrics. We look at session length, we look at abandonment. B2B also because we're in a more regulated space, we, because of our contracts, can't track as much as the consumer companies. Versus like Uber, we know [00:18:00] like literally everything. Like where your mouse is hovering, like your session lengths, everything Ashley Stirrup: Makes a lot of sense. And how do you see experimentation evolving? Daniel Layfield: Yeah, we're in the, I'd say the early steps of experimentation here. We're leaning more into the product style experiments. So like we are working on a problem, we legitimately don't know the right path. Can we de-risk this a little bit with experiments? Diligent is probably not the right, I guess within the pro- core product suite, the right surface to run CRO style experiments because if people come here for the most stressful meeting of their quarter they don't want like the nav moving around all the time So we're doing more of the CRO style on like notifications and emails and things like that. But within a product that's much more are we using this as one of the tools in a PM's toolbox to de-risk projects? Ashley Stirrup: Yeah. Yeah, another thing that we're seeing is just obviously the impact of AI in experimentation and we're [00:19:00] seeing people interested in a pretty wide variety of things, whether that's help me come up with new ideas using AI and looking at past Daniel Layfield: Yeah Ashley Stirrup: or I've built a AI-powered, I don't know, tutor or chatbot or something, and how do I know if the prompts I'm using are the right ones? Is this new prompt better than the old prompt? That type of thing. Yeah. Daniel Layfield: Yeah, totally. Yeah, I think I guess the place we've leveraged AI the most so far besides the, the core, engineering loops and things like that, is like data synthesis and also research. So the way I would've done this at Codecademy if I wanna work on a problem is like you find a hundred users, you email all of them. It takes two to three weeks to get them all on the phone. You write up all the user interviews, and then you synthesize manually. Now it's like we use Gong's MCP and like you can get like reasonably good synthesis from sales calls in probably like an hour or two to feed Ashley Stirrup: Yeah. Daniel Layfield: As well as the results [00:20:00] themselves of AB tests. So Claude is a pretty good data scientist, at least for simple AB test analysis Ashley Stirrup: Makes a lot of sense. Yeah I personally get so much value out of being able to use AI to scan all the calls. We use Fireflies and you can just get insight into trends in the market, Daniel Layfield: Yeah, it's so much easier Ashley Stirrup: so yeah. Dan, thank you so much for coming on today's episode. I feel like we, we learned a ton. Especially loved kind of your viewpoint on how to apply A/B testing to more of a B2B type of world, so really appreciate you coming on Daniel Layfield: Yeah, of course. Thanks for having me Ashley Stirrup: Thanks so much