Latest / The Experimentation Edge / Typeform on how to stop running experiments and start earning them
Transcript
- Ashley Stirrup (00:02.156) We're excited today to have Alex Bass, the Chief Product and Technology Officer for Typeform with us. And I'm especially excited about having Typeform on because they work with over 150,000 companies to collect over 500 million responses every year. And that information is just so valuable in terms of learning about customers and their usage of products and all that, that it's really complimentary. Plus I see that Typeform's doing a lot with AI and the intersection between AI software development and experimentation is a really hot topic these days. So thanks so much for having, so joining us today, Alex. So Alex, great, it's okay. And so for our, just to start it off, I'd love to hear you talk a little bit about just how Typeform's approaching experimentation. Aleksandra Bass (00:43.618) Thank you Aleksandra Bass (00:58.808) Yeah, absolutely. So Typeform is taking experimentation to a whole another level in terms of rigor. Experimentation, and I know people typically in the software development lifecycle will think about experimentation just as almost an EB test within the product experience that you're basically turning on or off for different customers you have in the platform. But we think of it more broadly. So it would be more akin to kind of medical experimentation, right? you're doing literature reviews for what is known or not known in the tech development space, and you're doing sort of in vitro testing. So user tests, understanding other pain points, needs to figure out whether there's real problem spaces or opportunities that exist. And then you're doing simulated research, right? So this is where you're going to go into animal trials in a similar type of structure where you're leveraging populations that look like yours, but reduce risk dramatically rather than going straight to your own customers. to see if there's something there in terms of the stimuli that you're going to put in front of those. consumers and then only once you've had that whole flow move through the intentional project Do you a be test the knowledge you have earned and gained through that process with your own customers, right? Because I think Oftentimes people tend to look at the experimentation as low risk But oftentimes at least a type form I mean in order for us to get a statistically significant outcome in some of these scenarios we have to put things out for six weeks to 50 % of our users and that is a big impact, right, if things are not going well and could have a revenue-impacting quarter if we really missed the mark on some of these things. And so it's really important to us to have extreme rigor in the end-to-end process. And you almost have to earn the right to A-B test within the customer environment. It's not just something that any old idea is going to get thrown into the mix. Ashley Stirrup (03:05.483) Wow, that's pretty amazing. Yeah, that all makes a lot of sense. It was really transformational for me when I learned that 80 % of the things that people A-B test fail. And so the more you can do beforehand, it makes a ton of sense to go ahead and do that. Maybe just to start off the questions, what was one of the most challenging product decisions your team faced in the last year? Aleksandra Bass (03:31.39) This is a great one. We've had a couple. One of the hardest ones was adjusting pricing and packaging while repositioning type form from just a forms tool, which is what most people know us as, to an AI engagement platform. And so one of the internal taglines we like to use is from forms to flows. People come for the forms and then they stay for the flows, right? And those are workflows that they can put in to place. So we knew that we had to raise prices given all the new capabilities. that we're putting in the platform to really continue to push forward that innovation on behalf of our customers, but it would likely reduce new business conversion. And so the real question was whether that higher revenue quality would offset the lower volume. And so we modeled it and we did A-B testing within the pricing and packaging page as well. And in the first 17 days after launch, we saw a 15 % drop in new business count, but a 32 % increase in ASP. And that was really So overall, from a revenue standpoint positive. The other piece that was really interesting is the way that we had structured the pricing change was we increased the monthly prices, but we also increased the annual discount so that people could keep their pricing if they commit to a longer term engagement with Typeform. We don't want to artificially punish folks for getting long term value at a Typeform. And so we were able to drive a 25 % increase in our annual attach rate, which was one of the most stubborn metrics. to move, at least for us it has been. And so that has been a really, really interesting experiment that, you know... really drove those insights around, okay, we're starting to get the hang of if we move these things in intentional ways, is it going to have the impact in production that we would expect or that we have learned from some of our simulated tests? And this was definitely one we simulated way before we ever put it into production for sure. So it was one of those great examples. The other one I would say is Typeform AI. we... Aleksandra Bass (05:34.632) We built Typeform AI to help people go from spending hours to days building a form, getting it reviewed and approved into a co-pilot of sorts that could work alongside you. And if it understands your context and who you are and what you're trying to get out of the platform, it should generate a ready to go form for you within minutes. And then it's your choice to decide, does it meet all of your needs? Do you have certain tweaks and changes that you need to make? And this was, we were deliberating internally, to charge more for this or is this something that we should use as an activation lever to help people get kind of across that hump and get to a published form faster? And we put it in, we've been watching it for the last few months, hoping that the activation and the conversion would increase and testing it out pretty dramatically across our cohorts. And we're seeing increases both in activation speed, I mean, it doubled our activation metrics just by putting that tool on place. And it has had a material impact on conversion as well. Increasing speech or conversions or one day conversion increased dramatically for people who are exposed to. But we still only have 40 % of people using it, right? And so there's still a decent population that's not necessarily using it. And so there's adoption and enablement tools that we can put in place that we're working on next. But those would be two that we really wrestled with over the last few months for sure. Ashley Stirrup (07:03.132) Yeah, wow. Those are both amazing examples. Pricing is obviously one that is so strategic that you have to be super careful about it. And obviously there's a short-term impact and then there's the long-term impact. As you look back on the work you did there, are there any lessons learned, things you would have done differently if you could go back and do it again? Aleksandra Bass (07:27.822) I think that we would have probably put in place on the Type-RMAI side, on the pricing and packaging side, I don't think so, because pricing and packaging is really just, there's some flexibility you have on the R &D side, right? But ultimately, there's real constraints around business need and operations, et cetera. And so, you know, it's harder. to maintain profitability of the product line and sustain an innovative R &D system without increasing prices every once in a while. And we haven't taken a price increase at Typeform for the last five years. So we've been avoiding this for a long time. So it was well overdue, especially with the volume of product we have shipped into the core experiences. So I feel good. I feel good about the way that we handled that one. I think Typeform AI, When we first released it, we gave it the same level of altitude as the other creation models, right? So was like, was three different buckets of kind of CTAs that you could go down. You could start with Typeform.ai, but you could also go straight into the builder and build your form from scratch yourself. And one of the things that we learned is that when it's at the same altitude as the other building modules, people are less likely to opt for it. when we changed our structure and helped people realize that they could just type in what their goals were and we would generate a pretty decent version off of the first attempt and put the create from scratch or duplicate another form as second order CTAs. That actually drove usage, activation, and adoption up significantly. And so if I could go back in time, I would probably have pulled that lever sooner and gotten us an even faster of adoption curve. But overall, I think the move, the team moves really quickly and looks at the data consistently and makes decisions fast, right, as fast as they possibly can from opportunities like this. so, you know, hindsight is 20-20. Ashley Stirrup (09:40.5) Yeah, right. Yeah, that makes a lot of sense. Yeah, one interesting thing with AI is that we're seeing a number of our customers go pretty deep on testing the LLMs and like which version should they use? Do they want something faster or more powerful? Which prompt is going to generate the best results? Have you been doing A-B testing at that level? Aleksandra Bass (10:02.766) We have a really powerful data engineering organization that configures the models that we use within the platform with intense oversight and a lot of testing on their end in terms of the output, the observability, the validation, the latency, the speed, the quality, et cetera. And so they are constantly testing all of the models. think at some point, once we feel like we have a good enough baseline of high quality levers within a series of models. We want to make sure that the customer output comes in as well. But right now our data engineering team is really rigorously testing the quality and the output to make sure that our customers get the best possible experience from Typeform AI. Because I'm sure as you know, Ashley, the tricky part about new capabilities like this is you lose momentum if somebody has a negative experience early I mean, I think about my own customer journey in several domains. And if the first experience is not great, which is, think, one of the risks of following the MVP approach, especially if you define MVP as minimum viable. Maybe it gets the job done OK, but you're not really fully satisfied. It didn't blow you out of the water, and you think you can still do better. The likelihood you're going to try that again in three to six months is low, unless you hear somebody else raving about it, or you have a materially different different experience, it's going to make you second guess the decisions that you ultimately made. And so for me, the quality bar of what we push out into live product is extremely high. It needs to be extremely high because sometimes you get a chance to influence people's perspectives and opinions once. And then once you've lost that, you're probably six to eight months away from them trying it again, maybe longer in some instances. And you have to have some really compelling evidence that they should. And I think the tech industry as a whole, would probably benefit from taking a more judicious approach to that. Ashley Stirrup (12:07.84) Yeah, well the crazy thing with AI is that it's just so unpredictable. mean, have customers telling me that from day to day they see different results and then slight variations in prompts can generate different results. And so you really have to test at scale to know what's going to work and what's not. So it's a fascinating part of the experimentation world. has there been an experiment that you've done recently that caused you and your team to kind of change direction on something? Aleksandra Bass (12:40.782) Absolutely. We've had several. We've had quite a few. So one of the experiments that we've done is we were gating video to one of our higher priced tiers and leveraging paywalls to help people kind of discover what the capabilities were. They could try it, but they can't necessarily deploy it within a forum with any real intention and realize that there was quite a bit of demand in lower tiers for video and that people were needing that capability even in use cases where the features of our highest tier weren't necessarily relevant to them. And so in instances like that, we would absolutely, we moved that capability down into the other plans and test that out, right? And see what that does for our conversion and our customer satisfaction on that other plan type. And are we cannibalizing any of that revenue, et cetera? And in this case, was a success story because the people that still needed that most premium plan there were enough features there that were differentiating that plan from the others that still, you know, maintained the interest and the conversions into that plan. But it actually expanded our talent plan and our business plans, right? Because in those instances, you could use video in ways that maybe weren't just customer use case focused and could be more internal or more talent oriented. And so we were able to roll that value down intentionally with feedback from customers for sure. Ashley Stirrup (14:17.77) Yeah, that's really interesting with the product manager hat on. I would imagine that there are certain use cases where it's like, I would love to have video. So for sure in the whole talent process, the more you can seem human and all that's very logical. can see in others where people might be a little nervous about, I really want to put video in? Is this going to hurt the response rates and all that? Is that something that you tested out with customers? Is whether video, including video, increased the number of the percentage of people that responded? Aleksandra Bass (14:52.27) Absolutely, and not only just the percentage of people that respond because I think our recommendations based on helping our customers, A-B test forms even, is to... create optionality because sometimes people are not in a place where they can realistically respond with video, maybe just voice is enough for them, you get more context but you don't get the entire context, et cetera. It also really depends on the audience, Because if you're paying for the sample, can set a higher bar for quality responses, right, that you're targeting versus if you are... relying on the goodwill of internal team members or others, right? There is no carrot at the end of it. They're just being generous and giving you feedback or helping you by giving you context in terms of what you're trying to understand. So that piece of it, I think, definitely plays into it. But I'll give you another example. So we actually just did an experiment around video versus standard forms, right? So like a video interview almost with AI versus a standard form. And we sent identical form in terms of the context. And it's just whether you're getting that AI interviewer experience or whether you're clicking through and selecting and typing in through a standard form. And what was really interesting is we randomly assigned people internally within our own company in terms of who was going to get which type of form and asked them to fill it out, give them a week, et cetera. The completion rates were the same, but the volume of response depth and quality that you get from the video experience through the interviewer is unprecedented. mean, 14 times more words per response, reduced skipped questions by 10x. Aleksandra Bass (16:40.484) the stuff that you can do with the interviewer experience, I've never seen anything like it in a comparable experience. It just really was impressive to see that volume difference because in regular forms you're getting like, loved the, loved the gala, right? If you're talking about an internal event, whereas in the video interviewer, you're able to just freestyle whatever it is that you were feeling or thinking in that moment. And so you just get a ton more richness for a relatively similar time to complete I would say as well. So not a dramatic difference there either. Ashley Stirrup (17:18.818) That's great. That's a great example. So was there a test that you ran that you were sure was going to be a winner or you were kind of skeptical of and it surprised you and kind of outperformed? Aleksandra Bass (17:33.155) That is a good question. think tied from AI when we released that to our alpha group, even honestly, when we went into beta and we were rolling it out systematically to subsets of the population to see what the differences were between people who had used it or hadn't used it. I mean, I had expected it would perform well, right? It was a decent experience. We made sure that the output quality was high, but it just... blew my mind in terms of just doubling activation, which are activation for a product like ours is, people sign up is one starting point, then you have to actually build the form, then you have to publish the form, and then you send the form out and collect data for it. And a lot of that is driven by intent and whether people are kicking the tires on the product understanding, could it be used for their use case or whether they actually have one ready to go. But I can tell you from, out of the 50 or so experiments we ran in 2025 before we launched the Typeform AI experiment. None of those came even close to performing as well or having even remotely the impact on our activation rate as high form AI did. And so while I expected it to do well, I had no clue how much of a incredible experience it was going to be and how much value it was going to provide to customers. So that one really blew me out of the water. Ashley Stirrup (19:00.086) Yeah, yeah. And so when you say activation, you mean all the way through to when they sent out the survey? Yeah, right. and click. Aleksandra Bass (19:05.826) Yes, and collect data. And no other experiment had had that much impact on our activation rate for like the last few years. And we would move it. Like it's one of those stubborn metrics, right? That's really hard. That's really hard to move. Ashley Stirrup (19:18.698) Yeah. Right. Did that lead you to running additional experiments around AI and how you use it in the product? Aleksandra Bass (19:30.132) Absolutely. it, because that was such a runaway success, we pivoted our strategy because of it, right? So. If AI is really accelerating the forms experience, imagine what it could do if you enable it to generate emails for you as a thank you for filling out this form. Imagine what it could do if it's analyzing the data for you, right, for any feedback, surveys or questions that you might be putting in there. And so now we're taking the AI strategy that we applied to the form creation space and essentially for all intents and purposes, dragging it across the rest of the platform and Ashley Stirrup (20:07.884) Yeah. Aleksandra Bass (20:08.112) everything else is connected into it so that, you know, we can really help our customers leverage that AI experience to drive their projects forward and take things that would take weeks and reduce it down into hours. Ashley Stirrup (20:22.338) That's great. One thing that you hear about so often with experimentation is just resistance. People feel like, get paid to know what to ship and what not to ship. We don't need to test it. I've built it. It's ready to go. Have you had that kind of pushback at Typeform? And how do you deal with that? Aleksandra Bass (20:42.358) Yes, absolutely. I think TypeFirm is a very unique place where there's two simultaneous things happening. One is a desire for being data-driven, regardless of the source that the data-drivenness is coming from, right? So we leverage research a ton, we leverage experimentation, we leverage competitive intel and things like that to help inform and drive our strategy, usage data, et cetera. but there was also a resistance to being confined to a process. And unfortunately with, or fortunately depending on how you look at it, experimentation is a very important skill set, right? You can do an experiment incorrectly. And if you do an experiment incorrectly, your learnings are essentially invalidated. You don't actually have learnings. You have errors in the experiment that you have to run again. And so I would say there was a moment in typeform's history where the experiments were not efficient. They were not set up with the right quality. And I would honestly suspect that lots of tech companies are in this space, right? Because the domain expertise for running a good experiment are not applied equally well everywhere, right? And there's a lot of assumptions that come out of, this is what I can say about the data versus not. And I see people leaning over that all the time. And it takes a lot of intention and care to make sure that you set the tone for the team that's like you actually don't have data that supports this point. You can't make this claim. You can say this is an opinion and that's totally fine, but the data is not supportive of the point that you're trying to make here because you're leaning over where the data and experiment landed you. Even simple things like, I'll give you good example of something that puts in. Aleksandra Bass (22:41.934) to put into question certain things. So like we would have experiments where we were manipulating multiple variables. in two executions and so then if one wins, you actually don't know why because you changed three things, not one thing, right? And so from experimentation best practices, that's not ideal. And so getting the team to a clear understanding of you cannot change more than one thing at a time if you want to understand what works or not. And you have to provide the right time to get, you know, it takes longer for smaller effect sizes than it does for larger effect sizes, right? So you need to understand that the time for an experiment to be live. Ashley Stirrup (22:55.906) Right. Aleksandra Bass (23:19.098) is variable depending on what you're trying to observe. And I'll give you another example of where I think there were opportunities for refinement of the standards of experimentation. When you think about what variable, dependent variable you're manipulating, right? If it's a rare occurrence, like in our instance, a conversion, Applying it at the user level and not the account level is not always super helpful, right? Because then you have two different users who may both be involved in the conversion decision seeing two different stimuli. So now you've washed out any effect that you might actually have on that experiment. So it's things like that that I think over time were more the issues. And I think people, know, people in product management, in design, in data, in engineering, who have worked in R &D organizations, you know, where experimentation is valuable and important, feel like they have earned skills that allow them to do experimentation. And I love that and I encourage that, but I think there are always new things on the horizon in terms of experimentation and what works and what doesn't. And in this kind of an applied space, it is a little bit different. And sometimes the level of detail and the level of rigor that you have to have is higher than I think in certain domains may be the case. And so it's important to just... make sure you're sanity checking and validating. And so for our experimentation process, we have a cross-functional team. It's not just PMs and designers that are setting up these experiments, right, on their own. There is a fully fleshed out process. They are submitting experimental designs for a review by our research team and our data team, who then provide feedback, ask questions, et cetera. And then we'll help set that experiment up so that we can both make sure that the Aleksandra Bass (25:20.272) experiment is clean and we will trust the results at the end of it, but also they get support in being able to interpret it so that they don't lean over what they can actually say or what they have learned out of that experiment with the data that they're generating. Ashley Stirrup (25:34.23) Yeah, wow, you covered a lot there. One of the things that I see with a lot of companies sounds like you're addressing is the fact that everybody's doing experimentation slightly differently. So having that kind of standard process hopefully means that you're measuring things the same way. And so then you can actually look across experiments and start to get that kind you know, kind of build on learning from one test to the next to the next. So that's exciting. Aleksandra Bass (26:00.833) I love that point Ashley, can I tell you my pet peeve about what I've noticed about experimentation culture and most, maybe I shouldn't say most, in some R &D organizations? My pet peeve is that... Experiments are run to your point differently, team by team. And then, so there's questions about accuracy across the board, right? Because some are right and some are wrong. And you're mixing the two of them together when you're looking at it in aggregate. Then the next thing that happens is, especially if they're leaning over the data, right? And kind of making interpretations that are not necessarily supported. Those things then become codified within the company culture and the learnings. which then adds to this series of misinterpretation and misalignment of knowledge. was like anti-knowledge almost within the organization. But this understanding is meant to be actually part of the decision framework within the organization. And so you're adding to what already quote unquote exists. But the problem is that it's wrong in a lot of instances, right? It's either unsubstantiated by the data or the execution was compromised, right? So my whole thing. And if you talk to my team, they would probably tell you the same is you cannot try one execution of something and say you learned something about that category or that variable. Unless there are multiple iterations of that that you have tested in multiple scenarios, then you can aggregate those into learnings, but you also still have to limit them to what the data actually show. And so we had to clean out a lot of those assumptions and that anti-knowledge that existed within the organization to say, Aleksandra Bass (27:47.957) These are not known facts. These are beliefs. These are hypotheses that we still have not validated or invalidated fully within the tests that we have done. And so we have to remove them from our base awareness or our operating rhythm. And I think too many companies stack too many of these non-facts into beliefs within their organization. And that prevents actual progress and growth just as much as a lack of ambition or speed or execution. or something else might. And so I think it's a really important thing to be aware of and to invest in as far as experimental culture, to both just not make sure that your processes and your systems are aligned and clear and consistent, but also to make sure that you're checking yourself and your team in terms of what the learnings are, right? And that people aren't just running away with. maybe an one-off kind of experiment outcome that they have now taken as fact but only works in a very small subset of the space. Ashley Stirrup (28:54.146) Yeah, you covered a lot there. I particularly, you know, I think a key point you were making there is really around humility. That, you know, it's such a mindset shift to say eight out of the 10 things I'm about to ship are going to fail. And if you go in with that mindset, then you're much more humble. You don't know what's going to be the winner or the loser. And you're really looking for the data to lead you. Aleksandra Bass (29:03.118) us. Ashley Stirrup (29:19.976) And it's so easy to have this concept of anti-knowledge. love those. I'm going to use that because I love that term. There was a great example with Microsoft Bing where in the UK, they ran a test where they opened up additional pop-up windows. And they were really opposed to the idea, but they eventually tested it. It turned out to be a huge home run. But it took another year and a half before the team in the US actually ran the same test. were like, no, Europe must be different. This can't possibly lead to this kind of benefit. And so it's such a great example of where anti-knowledge blocked a huge win for a long period of time for a company. Well, this has been a great conversation. Maybe as your of your parting advice, is there any piece of advice you'd give to other product and technology leaders looking to leverage experimentation to get a competitive edge? Aleksandra Bass (30:16.94) I think absolutely do it, but invest in the process, the tools, and clear understanding of what experimentation needs to look like at your organization. I think changing culture in an organization is hard, right? So like type form is very pro experimentation. It's just anti process, right? Trying to change the process of experimentation or experimentation is valued with a population of folks that think that they shouldn't have to follow a process, that they know what they're doing, you know, bringing that awareness about that actually there's better ways to do this and it will support them and their, you know, learnings and executions and not just slow them down, I think is a, is an opportunity. Ashley Stirrup (30:35.648) Yeah. Aleksandra Bass (30:58.008) Others probably will face similar resistance as they try to put in place infrastructure or processes to drive additional experimentation. The one thing that I would say is visibility. is important, right? Shining a positive light on the things you want to see repeat in your system is critical and important. So having those examples of folks that are doing it the way that you want, that are leaning in, that are investing in that quality and be able to spotlight them in front of the organization so that they can feel motivation to do similar things, I think is definitely a lever to overcome some of that resistance. And I would say the other other lever to overcome the resistance is I think often, and I do this too. think it's hard not to do this from time to time in leadership is you make assumptions about why, that person just hates process, right? Or they just don't want to do this experimentation. And it's often something different than what you assume. so going into that situation with curiosity to understand what are the real barriers here? What is the real blocker? And focusing on that rather than just coming in with force and assuming you understand why they don't want to do the thing. to do is definitely an opportunity. And so those would be the two levers I would say to lean into is reward the behaviors you want to see repeat and listen with empathy to the challenges of the blockers that folks are experiencing as you try to bring your organization closer to the learnings and the benefits that experimentation programs can have for a business. Ashley Stirrup (32:36.404) Yeah, those are both excellent. Especially love that when you kind of layer it on top of this, what we're just talking about, the fact that 80 % lose. So if you can shine a positive light on losers and say, okay, how do we learn from these? How do we make this additive to the company and the culture? That can be a huge impact. So, well, thank you so much for joining us today. This has been a fabulous podcast and I'm really looking forward to sharing this with the world. Aleksandra Bass (33:04.323) Thank you so much, Ashley. Appreciate the time.