Latest / The Experimentation Edge / Ancestry on AI Storytelling, Paywalls, and Decision Quality
Transcript
- Ashley Stirrup (00:02.76) Welcome to this episode of experimentation edge. I'm excited to have Suresh Teckchandani with us Suresh is VP of product and technology at ancestry and He has a rich background Working at other great companies like PayPal and eBay as well. So Suresh welcome to the show Suresh Teckchandani (00:26.542) Thanks Ashley for having me on this show and it's good to be here and discuss a topic that's close to my heart. Ashley Stirrup (00:33.734) Well, that's great that it's close to your heart, it's definitely close to mine as well. Maybe to kick things off, I'd love to hear a little bit about where you would say ancestry is in its experimentation journey. Suresh Teckchandani (00:46.752) Yeah, I would say in terms of the maturity curve, we are operating at scale now, actually. And the biggest unlock for us was when we started the experimentation, and we not just started to run new tests, but we focused on creating new capabilities. And these were not just incremental wins, as you would expect any organization to go through. And a good example is our metering infrastructure. So this was the first experiment that we ran a while ago called growth metering and pH widget. And that had flat primary matrix such as the registration rate, signup rate, and so on. But we saw a very healthy lift of 5.3 % in the new person hints review rate. And then you can think of hints as alerts that we send to our customers through email. and they can also see them in the login. So on its own, that's a modest result, but what it unlocked was much bigger. It gave us the ability to meter resources like records and hints, and that allowed us to invest in that area at a much deeper level, and that led to some new capabilities that have become part of our platform now. Ashley Stirrup (02:07.868) Interesting. And so I want to make sure I understand this. So you were kind of measuring the number of times you were sending like suggestions to your customers on things they could be doing. Is that right? Suresh Teckchandani (02:20.142) Yeah, so this was like especially geared towards our existing customers who were on our platform or even the new ones who were coming on. And we would kind of show them some of the information. If you are not signed up yet, we show you some information to make sure that you understand what products and services we offer, right? And the goal there is to give them a taste of the product before they become a paid customer. And we tried to sort of... Ashley Stirrup (02:46.979) I see. Suresh Teckchandani (02:49.806) put a meter on those as to how many hints you can receive, what kind of content you can view, those kinds of things. So that once you reach that limit, then we offer you a paywall and then you go through that experience and you become a paid customer. And we didn't have some of those abilities to meter these different dimensions that I referred to earlier. And with this experiment, we saw that there was value to be had. Ashley Stirrup (03:05.852) Right. Ashley Stirrup (03:12.859) Yeah. Ashley Stirrup (03:16.942) Yeah. And so the results that allow you to send more hints or were you getting more engagement? Suresh Teckchandani (03:24.526) Yeah, so that improved the number of customers who were coming in through the paywall and they were becoming paid subscribers. So, yeah, so the registration rate, sign up rate, all those things trended in the right direction and so that we could engage with them more, right? So we improved the experience for the customers. Ashley Stirrup (03:32.539) that's terrific. Okay. Ashley Stirrup (03:38.149) Yeah. Right. Ashley Stirrup (03:44.756) yeah, that's a great win and 5%. That's a lot. That's pretty meaningful for the business. And so how did that change how you thought about experimentation there? Suresh Teckchandani (03:48.098) Yeah. Suresh Teckchandani (03:55.086) So the way I think about experimentation after that experience, that was fairly early on, it kind of, it was a real shift going from isolated tests that you run to building capabilities and where every experiment expands what you could do next. So at that point, experimentation stopped being a tool and it becomes how you essentially Run your business. Ashley Stirrup (04:25.562) Yeah. Yeah, I mean, think that's what there's kind of two dimensions to experimentation. I think are really exciting. One is that, A, you can just measure the impact of your feature investments and like, are they actually impacting the business? But probably the more important one is increasing learning, understanding what your customers, you know, get more value out of what they respond to, what they don't respond to that can then help you guide future investments. When you think about it on those two dimensions, is there one that you've gotten more results on than the other? Suresh Teckchandani (05:02.04) Sorry, can you repeat the question again? Ashley Stirrup (05:04.154) Yeah, so like one side of it is just, okay, we built 10 features. How many were winners? How much lift did we get in the business through those? And okay, maybe we roll back some. But the second side of it is, what did we learn? would you say that at Ancestry, it's more about kind of measuring winners and losers and deciding what to ship and what to iterate on? or are you getting a lot of learning from just running the tests and learning about your user base? Suresh Teckchandani (05:34.574) Absolutely. So I would say both, right? So you do want to have wins along the way, right? As you run these different experiments, you want to make sure that you have results to demonstrate that you are moving the right needle, right? And you are moving in the right direction in terms of how you are evolving your product, right? That's important. But what often gets overlooked actually is Ashley Stirrup (05:42.011) Yes. Ashley Stirrup (05:56.635) Yeah. Suresh Teckchandani (06:03.414) you can get too focused on these smaller wins and you lose sight of what you could be going after or achieving in terms of bringing on entirely new experiences, investing in new capabilities that weren't realized or seen before you ran those experiments. So it's very important. Ashley Stirrup (06:25.35) Yes. Suresh Teckchandani (06:32.488) for you to sort of zoom out as often as you can and see where are things going, how are things trending, what kinds of features are customers liking, where you see deeper engagement, where you're seeing drop off or more friction where you're just losing customers engagement rate. And I think those are the area that could be Ashley Stirrup (06:38.341) Yeah. Suresh Teckchandani (07:01.422) important for you to realize and you can alter your product direction so that you can continue to make it enticing, continue to make it interesting for our customers to come onto your site, especially for a subscription business like ours, right? Engagement is key for us, right? The NPS is what we always think about, right? In all of our business metrics that we review, NPS is pretty high up. So we care deeply about our customers, how they are feeling about the product that we're offering to them. Ashley Stirrup (07:16.251) Yes. Ashley Stirrup (07:25.403) Yeah. Suresh Teckchandani (07:30.742) Are they engaging in a meaningful way? Are they spending time at the right places? On what side? Are they engaging on the areas where we are making deep investments in? So think those are very important for us as well. In addition to what I said earlier, right? It's important to have those wins along the way for sure. Ashley Stirrup (07:49.309) Yeah. Yeah, I just think it's so interesting, the trade-offs, if let's say you're a product manager and of course you want to keep getting the incremental improvements on stuff that's already out there. How do you make it a little better and a little better? And then how do you apply experimentation to whole new capabilities? we had Cameron from Fixer AI there, an email AI tool. And one of the things he talked about is they would release a new feature and it would almost always fail at first. But if they believed in the feature, they would just keep iterating on it and then figure out how to turn that into a winner. And I think that kind of mindset for innovation is really important. Yes, you want to have some bets that are the incremental wins, and then you want to have some bets that are much bigger, and then recognize that you're going to test and learn from those in a very different way than you might be doing on the things where you're trying to get a 1 % additional lift. So how are you organizing experimentation at Ancestry? Do you have one centralized team, or have you distributed it across the organization? Suresh Teckchandani (08:49.784) Yeah, absolutely. Well said. Suresh Teckchandani (09:04.494) So the way we have structured the whole experimentation effort is we have a core team, centralized team, that is responsible for the upkeep, maintenance of the experimentation platform. So this is the team that you work with in case if you have issues. We have made it pretty like, you know, Smooth at this point where somebody can go in and you know Log in navigate to the right place Schedule a new experiment even product managers can do it. You don't need to have technical background So that's you know, it's a dashboard you can go and it's very easy to follow through So that's the central team that supports the experimentation platform but Ashley Stirrup (09:41.67) Create an experiment. Suresh Teckchandani (10:03.854) Anybody in the company, like I said before, right, product managers, engineering leaders or engineering people can go in and schedule their own test on their own without necessarily having to coordinate with us. There are some things that we do coordinate as we do upgrades and form new baselines every six months or so. We reset the baseline so that you have, you know, your test running against a specific baseline so you can draw, you know, good meaningful conclusions from your test. But by and large, people are fairly educated at this point. Initially there was some ramp up and some learning, but now I would say we are, like I said before right at the opening, we are fairly mature in terms of how we run our tests now in the company. Ashley Stirrup (10:47.93) Yeah, yeah, that's great. And do you have a framework or like if a feature looks like this, we need to test it, that type of thing. Suresh Teckchandani (11:00.59) So generally, our approach is to guide people, especially those who work on the front end, right? If you are going to change the customer experience in any meaningful way, we strongly recommend you run the test first, right? Because we've all been there where we had the strong gut feeling, right? We thought this thing is going to win. There was no question in their minds. that this was going to win. But when you run the test, you see exactly opposite results. I can give you an example of that. We had some tests that we ran where we tried to improve customer experience by removing a friction. So this was like a content that represented the presenting to the customers that forced that page to reload. That was causing the page to get refreshed. Now we thought, hey, Ashley Stirrup (11:35.185) Yeah. Suresh Teckchandani (11:59.37) Why lose the context, right? Let's just introduce a new widget where customers can go in and click and that opens up the new widget and that content is contained in that widget. So it's, you know, much cleaner experience. Customers can just close the widget, go back to the page without having to do a full reload. And actually, to our surprise, we saw a huge drop in engagement on that page. The number of people who are trying to click, click on that, you know, widget or the click through rate, it just fell. by a huge percentage, I think it was about 20-25 percentage. So there was like huge drop in traffic, right, coming through from that page. We said, what happened? And we realized that we thought removing friction was a good thing, but what it was doing was it was giving, it was like changing people's mental model, right? They were expecting a page refresh because that is how sort of they were thinking of that page. Ashley Stirrup (12:33.723) Wow. Ashley Stirrup (12:54.854) Mm-hmm. Suresh Teckchandani (12:58.958) and it provided them with a clean way to think about, I'm now going to a new experience. And we took that ability away from them. So the learning there actually was not all friction is necessarily bad. Sometimes it helps to set the context. It gives people a chance to get reset in their minds as to what they're about to do. And it can make them a little bit more thoughtful about how they interact with certain. Ashley Stirrup (13:05.318) Yeah. Ashley Stirrup (13:14.545) Yeah. Suresh Teckchandani (13:27.97) parts of your site or your product. So it's something that doesn't come up in your UX discussions because whenever we talk about revamping the UX or experience, we don't often think in terms of introducing friction by choice so that it gives a chance to the customers to understand what's going on, right? Because if you move them too fast, they might not want to do that. They might not enjoy that experience. Ashley Stirrup (13:30.32) Yeah. Ashley Stirrup (13:57.425) Yeah. Yeah. No, that's a great, great story. You could see how if you were a UX designer though, you might feel a little let down in a situation like that, that you've come up with a design that you believe is clearly better and then you test it and you see the negative results. Did you, you know... Suresh Teckchandani (13:58.104) Hopefully that answered. Ashley Stirrup (14:18.853) Were you able to turn that from like a negative experience into a learning experience for the team so people didn't feel like they were undermined by the results of the test? Suresh Teckchandani (14:27.49) No, look, I mean, we encourage, you know, folks to run tests and even if a test fails, but if you got some good learning from that, it's a win in my mind. No question about it, right? So tests are not necessarily for them to like, you know, those tests don't always have to win, right? The win could be a fail test too, as long as you get the learnings from that test. And that's important for you as you... Ashley Stirrup (14:40.347) Yeah. Ashley Stirrup (14:47.942) Yeah. Ashley Stirrup (14:54.182) Yes. Suresh Teckchandani (14:56.598) move your product roadmap forward. Ashley Stirrup (14:59.898) Yes, I couldn't agree more. So is there an example of a time where you use data to uncover like a new lever for growth in the business? Suresh Teckchandani (15:10.658) Yeah, I think I mentioned some of that already earlier. But I would say the one that comes to mind was there was a test that we ran, I think some time ago, which was around adding new events to our feed. As customers come in, they do discoveries on our site. So this was a test that was designed to impact how we told stories to our customers. So the core concept involved using AI to compare two records of the same ancestor by presenting this information as a story in their feed. So this approach actually aimed at driving deeper engagement by offering users a story-like experience. rather than just looking at the raw records, right? And the results were highly successful, revealing like AI-powered storytelling from record comparisons. That turned out to be a significant new growth labor for us. And just to kind of give you a sense of the results, we saw very strong performance across the board, across all of our metrics. There was over 30 % lift in storytelling click-through rate, CTR, over 100 % increase in storytelling success rate. Ashley Stirrup (16:11.996) Right. Suresh Teckchandani (16:36.844) and over 500 % increase in AI story view rate. So that told us that there was appetite, there was a need from the customer side to tell them these stories in a different way than just kind of throwing these raw records at them. And that's a very strong signal, right? And that became sort of a major growth lever for us. And it kind of shifted how we thought about engagement actually also. So we... Ashley Stirrup (16:52.913) Yeah. Yeah. Suresh Teckchandani (17:05.804) That kind of shifted our mindset to go from delivering information to helping users interpret and connect with that information. Ashley Stirrup (17:14.49) Yeah, yeah, as you were telling that story, was thinking like, boy, I guess that's really what ancestry is all about is helping people understand the story of their ancestors. And so anything you can do to kind of bring simple facts to life is pretty exciting. Suresh Teckchandani (17:32.834) Yeah, for sure. look, we thought it was right thing to do and we experimented it and customers agreed. It's like, yeah, we love it, you know. Ashley Stirrup (17:41.634) Yeah, and so I would imagine that's an area that you've just continued to mine into and are there ways you can go even further around the storytelling? Suresh Teckchandani (17:51.865) Yeah, you can definitely kind of think along those lines and see where you want to take your roadmap, product roadmap, to see if you can bring more value for the customers, right? Because these are some of the cues that you have to drive from the customer engagement and the behavior and see if there is value. So it actually works both ways. It delivers value to the customers, but it also tells us where to invest. Ashley Stirrup (18:15.376) Yes, yeah, exactly. And how do you think about what metrics you track? And do you have kind of one North Star metric, or is it more kind of a group of metrics? Suresh Teckchandani (18:27.704) So metrics, I would say there are like two kinds. There's input and there's output metrics, right? As you would imagine, not necessarily talk about EBITDA and those kinds of financial metrics. Those are important to any business. In this context, we're talking about product, right? So most of our kind of metrics kind of focus around the core metrics that we have, how many people are staying on the platform, cancellation rate, retention rate. how many registrations we're getting, sign-up rates, things like that. Those kind of standard set of code metrics that we always track. But at the individual feature level, actually, you have to come up with the matrix that you think you need to put in place as part of your experimentation, test that you schedule to run for a couple of weeks or however long you want that test to run for. Ashley Stirrup (19:06.769) Yeah. Ashley Stirrup (19:20.282) Right. Suresh Teckchandani (19:24.078) So you can provide those input matrix saying that if you have changed, let's say, how the content is viewed or how the hints are displayed or how the storytelling is occurring, right? So you would go in and put those kinds of matrix in the system. So we track those metrics as part of your feature, but we very closely monitor some of these output matrix that I talked about, right? If your feature somehow improves, Ashley Stirrup (19:49.724) Yeah. Suresh Teckchandani (19:52.874) your metrics, right? You know, let's say the customers are viewing your widget more, they are spending more time clicking this, clicking that, right? Whatever it is that the feature does. But if it hurts the core metrics that I talked about, then you want to take a step back and pause and see whether this is really actually helping or it is, you know, it's becoming a train for the product. So you just have to kind of step back and Ashley Stirrup (20:19.408) Yeah. Suresh Teckchandani (20:22.914) put it all together and see what makes the most sense. And we have a number of tests that we ran that showed really strong performance at the experiment level, but our core metrics started to move in the wrong direction. So we had to kind of take a pause there and we sort of rolled back and then changed the experience so that we were not doing that. Ashley Stirrup (20:43.632) Yeah, yeah, that's one really interesting challenge with AAB testing is that sometimes the short-term metric like did the engagement go up is what you can measure, but what you really care about is long-term retention. so being able to kind of have that right mix of kind of. understanding what you're measuring now, understanding how it relates to your long-term metric, and then kind of understanding those trends. Super important for making sure you're getting the best out of every investment that you make. So out of curiosity, as you bring new people into the organization, do you ever get people kind of pushing back and saying, I don't need to A-B test this. I know it's a winner. Suresh Teckchandani (21:31.308) Yeah, I mean, that's a good point that you're bringing up. It does happen because when you hire people, they come in very strong product background. They have spent years and decades in some cases building these customer-facing experiences in highly successful and high growth companies. So I understand that part. But the thing is, like I said before, I think we talked about a couple of experiments. best way to validate your theories or your sort of instincts or gut feeling is to make sure that those are backed by real data, right? And the best thing to do would be Ashley Stirrup (22:20.657) Yeah. Suresh Teckchandani (22:27.19) you know, for them to try it out, right? So you encourage them to run these ideas as a test first, right? Especially if you are going to touch something on the front end that would have direct impact on the customer experience. You'll be surprised, like I mentioned, right? In some cases that we saw, who would have thought, you know, removing friction would start to hurt your, you know... business, right, or your engagement metrics and such, right. But it does happen. It did happen with us in some cases, not always, but in some cases. the way I communicate and talk to the folks is those obvious changes are often the most risky changes. We think that intuition can lead you to the decision, but they may not be actually supported by the actual data underneath, right. Ashley Stirrup (22:58.139) Yes. Ashley Stirrup (23:03.516) Yeah. Yeah. Ashley Stirrup (23:26.918) Yeah. Suresh Teckchandani (23:26.99) So those features like that feel like they are guaranteed wins, and usually the ones that we insist on experimenting more to put in country. Ashley Stirrup (23:38.19) Yeah. Yeah. Do you ever kind of try to put things through a lens of, okay, well, these features, the likely impact is, you know, bigger. So therefore we must test these versus something you're like, okay, that's probably pretty minor. It's unlikely to impact things. And so maybe you prioritize the bigger swings. Suresh Teckchandani (23:58.862) Yeah, mean, look, you know, one thing I would say caution, I mean, so far we have talked about, you know, how good great experimentation is and we should experiment anything and everything under the sun. And, you know, that may or may not be true, right? In all cases. I'll tell you why. Look, sometimes I've seen this and I think we have made this mistake too, actually, early on when we considered, you know, maximizing the number of tests, right? That meant win for us. I would, I remember I would go into you know, the review with our senior leadership, like say, oh, you know, look at our graph, how, you know, well, we are running experimentation, right? You know, platform, because I was the one responsible for it. It's like, yeah, you know, I'm doing a great job here, right? So we are boosting the number of experiments we are running day after day, week after week, month after month. Feels good, right? But once you get into that, you know, mindset of, my goal is to run tests. My goal is to just run experiments. Ashley Stirrup (24:48.624) Yeah. Suresh Teckchandani (24:57.518) So you kind of lose sight of what you're trying to accomplish there, right? And that did happen to us for a certain period of time. So we said, okay, we don't want our success metric for this experimentation platform to be just the number of tests, right? So you give yourself a pause a little bit and see where does it add the most value, right? We felt that investing more on the core customer experience would certainly help move the needle for the business by making sure that we are trying to test our theories and how we want to evolve the experience in a smaller group of users before we roll it out 100%. What's the right thing to do? But at same time, are some, like, if you're doing bug fixes or... you know, bringing some consistency in your experiences, right? Because things can go out of sync, right? If you're large enough company, I mean, you know, if you have talked to people who come from larger companies, right? You have these, you know, domain ownerships, right? Somebody owns these 15 pages, other person, other team owns these other ones, right? And over time, those can go out of sync very quickly. The experience just, you know, feels a little fractured, right? We all... Ashley Stirrup (26:24.218) Yeah. Suresh Teckchandani (26:26.368) seen those sites also when you go in and you say, you why does it look different than what I just saw on the previous page? So if you're trying to bring consistency, right, you know, there is, you know, good argument to be made that, hey, why bother with experimenting that, right? So you can go and make those changes and see if that negatively hurts your metric over a period of time. But I wouldn't necessarily hold ourselves back from doing that. Ashley Stirrup (26:32.209) Yeah. Ashley Stirrup (26:53.807) Yeah. Suresh Teckchandani (26:54.67) But yeah, absolutely anything like customer facing, I strongly recommend we go through the experimentation cycle. Ashley Stirrup (27:02.436) Yeah, and what I really heard you say there is that it's not about doing testing for the sake of testing. It's about doing testing for the sake of learning and making the product better. so making sure you kind of keep your focus on that, I think, is super important. Suresh Teckchandani (27:18.69) Yeah, and it's very important to not beat yourself up too much if the test doesn't win because that's an important learning, right? As long as, and I cannot say this often enough, right? I tell this to my team as well. It's like, as long as we are learning, we are winning. Ashley Stirrup (27:36.432) Yep, I couldn't agree. I couldn't agree more. What percent of experiments that ancestry, you know, increased lift? Suresh Teckchandani (27:37.367) end Suresh Teckchandani (27:47.374) Sorry, what kind of experiment or sorry? Ashley Stirrup (27:49.785) What percent of your tests actually are winners? Suresh Teckchandani (27:55.982) That's a hard metric to code. You know, I would say we have usually winners in most of the tests that we do, but there have been some high profile, you know, projects where we put substantial amount of effort and those did not turn out to be a win. I honestly have not done that kind of analysis to see in terms of like Ashley Stirrup (28:23.953) Yeah. Suresh Teckchandani (28:25.652) raw numbers, how many tests are the winners because these things like get done at the leader teams level. And some of the tests that I quoted to you earlier were just where I was closing in all and my team was coordinating some of that action there. So I knew those things because those were like important learnings us. But we have a very healthy rate in terms of the winners and Ashley Stirrup (28:29.093) Right. Ashley Stirrup (28:35.43) Sure. Ashley Stirrup (28:44.378) Right. Yes. Suresh Teckchandani (28:52.982) Once you have that kind of mindset, what to test and what to do, what not to do, things like that, you have a good sense of which test to run and how to impact. Ashley Stirrup (28:57.265) Yes. Ashley Stirrup (29:02.748) Yeah, it's so interesting because it's so many of our customers. The win rate, like very rarely is it even as high as 30%. Usually, 20%, 25 % win rates, sometimes as low as 10%. And I think for a lot of product managers that if they, in their head, maybe they've never really put a number on it, but in their head, they basically think that all of what they ship are winners. and you suddenly realize that, my goodness, only 25 % are winners. It really changes your whole mindset. the experimentation just becomes so much more important. Suresh Teckchandani (29:40.27) Yeah, mean, now that you've asked me that question, I'm going to go back and check. I'm curious now. Ashley Stirrup (29:43.289) Yeah, yeah, I think you'll find it pretty interesting. So yeah, so we're almost out of time here. So one last question. Do you have any parting advice for a product or engineering leader who wants to use experimentation as a competitive edge? Suresh Teckchandani (30:00.312) Yeah, I would say the biggest shift is moving from testing to validate to experimenting to learn. So the best teams aren't just looking for wins, I would say. They're optimizing for decision quality. And that's where the difference between good and great comes in. So that means getting the fundamentals right. I always tell people, start with clean baselines, proper audience selection, enough size. of your sample that you're trying to test against and ensuring that you reach a statistical significance before you draw a conclusion from the test. Without that, even strong-looking results can mislead you, as you have seen in some of the examples that I talked about earlier. So to bring the point home, at scale, experimentation is not just about trying to test individual features. It's about building a culture where you value learning. And that's a way. That's the way I would say that. That's the way where you can win in the long Ashley Stirrup (31:15.888) Yeah, I really do believe that is the single most important thing. Because if you're only shipping 25 % winners, the faster you learn, the faster you move up that curve. then the fewer losers you're shipping, the more winners, the more you understand your customer, the more you're able to align your product roadmap with the things that matter most to customers. Well, Suresh, thank you so much for joining us today. This has been a great show. I feel like I've learned so much and I'm sure our listeners have too. Thank you so much. Suresh Teckchandani (31:47.084) Thanks actually for having me on