Latest / The Experimentation Edge / How Fin went from weeks to hours of analysis using AI
Transcript
- The Experimentation Edge - Raunak Kumar === Raunak Kumar: [00:00:00] just in the last four or five months, especially since the advent of Claude Code, things have changed a lot. What used to take weeks and days of like data analysis work now can be done in a matter of hours. Ashley Stirrup: Welcome to the Experimentation Edge, where product managers, data scientists, and engineers talk about how they make smarter decisions. I'm Ashley Stirrup, the chief marketing officer for GrowthBook, and in each episode, I'll sit down with an executive to unpack how they use experimentation and A/B testing to make better decisions. This show is sponsored by GrowthBook, the open source experimentation platform leader. Now let's jump in and get started with our next guest Hello, and welcome to today's episode. I'm excited to have Raunak Kumar, senior manager of go-to-market analytics from Fin, and Fin is formerly Intercom. Welcome to the show, Raunak. Raunak Kumar: Thanks, Ashley. Really excited to be here and chat a little bit [00:01:00] more about my background and talk all things experiments. Ashley Stirrup: Sounds great. Could you tell us a little bit about your role at Fin? Raunak Kumar: Yeah. I currently am the senior manager of GTM Analytics at Fin. My team charter is to work with our demand gen and brand marketing team and provide them with all the analytics and data needs. So my team is responsible for building all things attribution, measurement frameworks for all the different marketing campaigns and channel, and really make sure that our stakeholders and marketers are, like, taking data-informed decisions. I also support the team run experiments which I feel like we'll talk a lot about in this chat and really make sure that marketing at Fin is kinda optimizing for the best thing in terms of acquisition and, driving our funnel forward. Ashley Stirrup: And you have a really great background with time at Stripe and Atlassian as well, yeah? Raunak Kumar: I did, yeah. So before Intercom and Fin, I was at Stripe where I was there for four years doing similar stuff in B2B [00:02:00] marketing analytics. Different industry obviously fintech. And before that I was at Atlassian, where I was there for five years. Started , as an IC and then bits and pieces, but essentially I was doing a lot of product marketing and like brand marketing analytics. So that's been my journey. I've been in like marketing analytics space for close to 12 years now. Ashley Stirrup: Yeah. And I bet the job's changed quite a bit with the rise of ChatGPT and all the other LLMs. Raunak Kumar: Yes. I feel just in the last four or five months, especially since the advent of Claude Code, things have changed a lot. Mainly like I'll just try to keep it short, but, what used to take weeks and days of like data analysis work now can be done in a matter of hours. But what that means is we also need to be differentiated as data analysts. So it has freed up a little bit of time on our side to do things which we might not have done otherwise. And one of those things are actually experiments. Usually we would have a lot of backlogs of experiment we just couldn't get to, but now we are able to run these more [00:03:00] experiments and also the the time to insight is improved a lot. If let's say an, a question came out from a stakeholder like what's happening in the funnel back in the day, it would take us like write a SQL, create a pipeline, et cetera. Now, like with the advent of Claude and a bit of work, and I can talk to that if you want to, like we are able to like speed up those insights very rapidly. So things have changed quite a bit. Ashley Stirrup: Yeah. That side is super exciting. I hear from a lot of folks that, organic traffic is down attribution has gotten a lot harder to track 'cause so much of the buyer's journey is now going on inside of the LLMs. Are you seeing that as well? Raunak Kumar: We are. It's so interesting that what's happening industry-wide is reflecting on pretty much everyone I talk to, including us. So typically, organic traffic represented like the demand for your product and like people coming to your website was a good representation of that. But with the advent of, LLMs and ChatGPT and like Claude, etc., [00:04:00] Gemini and AI mode actually is the biggest one. The information-- the informational queries and all the information about your product is being served right then and there. So it removes the need for folks to click through and land on your website. And hence we are starting to see like a decline in what we used to call organic search traffic. But that's been compensated by other form of traffic, right? Like direct traffic or people coming in from other sources. Like we actually saw a 5X increase in our ChatGPT referred traffic. So I think it's compensating. But I think we also lost a little bit of tracking in the process because these LLMs don't necessarily like tag the traffic coming in. So-- And that's one of the areas I've been doing a little bit of experimentation to prove that what we are putting out in the LLM world from a content is actually working, just because we don't have straight forward tracking on it. Ashley Stirrup: Yeah. Yeah. Super interesting. We could have a whole episode just on that. Could you tell me a little bit about experimentation at Fin? Raunak Kumar: Sure. So as I mentioned, I support the marketing [00:05:00] side of the world here. So typically we also have a very decent amount of product analytics and ML team at Fin. So they've done a lot of sophisticated experiment which I am personally not qualified to talk about, so I'll stick to the marketing side. At marketing we run two types of experiment. One is I call them on-page, on-site experiments. So let's say we wanna redesign our contact sales page form or, we wanna move things around, like we usually run A/B test on the webpage. So that's one side and then the other side is pre-website. So let's say we have campaigns different ad copies out there which we want to test, like what's working, what's not working, and I support the team run those experiments as well. So those are like how I categorize them. and it all depends on like how many we are running and what's the priority at any given time. But those are two areas I am involved in. Ashley Stirrup: Got it. And how does it typically work? if you're having a team run a, an experiment is it you? Is it a marketer? Is there a data scientist [00:06:00] involved? Like, What are the different roles? Raunak Kumar: Yeah. I think that's also one of the things which have changed with AI. Traditionally, we would have a data analyst, a data scientist, and what we called as a web engineer who would actually configure things. But I think right now at FIN, like I am the data person, and then I just work with one other web engineer to work out, what we want to test as an example or like I work with a marketer. And then I am able to do a lot of the sizing of the audience, creating the queries, et cetera, with the help of Claude. So that's the setup, like one, one data person and one stakeholder, like a web engineer or a marketer. And then we have a lot of tools, right? Like we have, I can go into the depth of that, but essentially we use a lot of third-party tools which helps us like orchestrate these experiments. Ashley Stirrup: Got it. It sounds like you're very lean and nimble on these, so that's terrific. Raunak Kumar: Yeah Ashley Stirrup: Yeah. Could you tell us of a time when you had an experiment where you had a lot of learnings? Raunak Kumar: Sure. So I'd like to talk about a [00:07:00] prior role experience, which was at, back at Atlassian where I was supporting our product team namely the Jira Service Desk team. And for folks who don't know about Jira Service Desk, like Atlassian typically is a well-known brand for project management, which had like leading products such as Jira Software and Confluence. But during my time supporting this product, which is the Jira Service Desk, which is suited for the IT products, and it was actually one of the fastest growing product at Atlassian, and this is back in twenty seventeen, eighteen timeframe. The experiment which we wanted to do is we wanted to supercharge the the trajectory of signups for Jira Service Desk. And what the hypothesis we had was if we bundled the product together with our core offering, which is the Jira Software and Confluence, we might see an increase in signups just because we would tap into the existing existing traffic, right? Like we don't-- we didn't necessarily had to go and like generate traffic because Jira Software is [00:08:00] already well-established at that point. Fifteen years of brand equity, Ashley Stirrup: and so just so I'm clear, there's the traditional Jira product and then there's the Jira Service Desk product as well. Is that right? Raunak Kumar: That's right. And the distinguishing was like software versus service desk Ashley Stirrup: Got it. Got it. Okay. And so you wanted to bundle those two together Raunak Kumar: Correct. Ashley Stirrup: And what happened? Raunak Kumar: Yeah. And so the hypothesis was it'll lead to an increase in sign-ups. And then the um, the counter-metric we put in place was that retention should not drop on the core offerings, right? So , we shipped this we ran it for three, three-plus weeks. And I did a lot of work at the time in concert with data scientists to figure out how do we roll it out because it was, the main thing where people would sign up. So we actually designed it carefully where we did a measured rollout where we started exposing the test group of the bundled offering to twenty percent of the traffic and then gradually increase it to fifty percent. So that was like the whole layout. Okay. So then what we found [00:09:00] out three weeks later is like our sign-ups for Service Desk increased dramatically just because anyone and everyone who would sign up on the bundle would get a instance for Service Desk. So like it, it-- I'm not able to recollect the exact figures, but I remember very vividly that there was a step change increase in s-in Service Desk trials more than fifty percent or something to that tune. But the main learning at that point was the activation drop. And like our counter-metric which we designed was a re-- the activation and the retention. That took a real solid hit, especially for our software product because what happened was folks were getting lost essentially. The short of it is because we were like, bundling things together, people were just not sure what are they actually trying in the product. It became clear when we started looking at our activation rate metrics which is, the metric we used was like week two active, are they actually coming in and logging back in the product two weeks from their sign-up. That dramatically was down compared to our control [00:10:00] group. The major learning there was that like onboarding and activation messaging matters a lot and just by bundling things together, you might see an increase in trials, but like it takes a hit on your activation. And then I also followed up with paid conversion rate, like of the sign-ups, how many actually convert to a customer. Like at the time , they had to pay us like twenty dollar a month. That rate also dropped between the control and experiment. Essentially that was the major learning. So we kind of-- Even though we It was a winner because my goal, primary objective in this experiment was to see an increase in Service Desk trials, which clearly I did. I mentioned fifty percent. My minimum detectable was ten percent, so it was a winner. But like the counter-metric was significant enough that we actually had to roll back this experiment just because it would impact our revenue targets, which was a sensitive thing at the time. So that, that's the high level o-of the whole Ashley Stirrup: Got it. Sounds like some important [00:11:00] learnings that your buyers getting confused, probably not sure when they're in the core product, when they're in the service desk product. Did you try to iterate on that? What were the kind of the follow-on next steps? Raunak Kumar: Yes. So the immediate next step we did was we were using a tool called Amplitude back then, was to try and learn through some session and heat mapping just to understand, what was the journey of these bundled trial people, like what were they clicking. And like after some qualitative and studying the sessions, doing some more in-product logging and tracking, we figured that the onboarding flow was a little bit confusing. Folks were just not sure how to navigate to service desk because we were bundling the product, but it was like hidden on the left sidebar. So th-that insight was-- led to us working our way and iterating on the onboarding flow. Essentially, guide them a little bit better. So I'm sure like you might have come across at some point where when you log into a product, called like guided pro-- guided onboarding, where like it'll-- something will pop up and it'll tell you [00:12:00] exactly what it is. So was the iteration. And then we relaunched the experiment with the revised onboarding. And then other thing I also want to call out, which was cool at the time, is we worked with our engineering team to do something called as contextual onboarding, where we actually took the keywords coming in like folks would type in "I'm interested in project management," or, "I'm interested in Agile Scrum management." So we were able to detect the keywords and then serve up those exact keyword-based template of onboarding. Because, in project management, like either people are interested in Agile or like Scrum. So once we know the intent of the audience, we could actually come up with the exact landing experience. So it's called cont-contextual onboarding. So these are the two main things we were able to dig up through the data and then redo the experiment with the corrected or I would say the enhanced onboarding flow. And I can talk to the results as well if you Ashley Stirrup: Yeah. Yeah, please. Raunak Kumar: Yeah. Ashley Stirrup: how'd it go? Raunak Kumar: It actually the biggest winner of these two [00:13:00] was our contextual one. Even though both of them, we did see increase in service desk trials the contextual one was l- was very good for activation rate because we kinda matched the intent of the audience like for and one of the other, pain points of Jira product was like there's a lot to do. You can do many things. So like this experiment helped us also identify that we should go after a use case for our target acquisition of customers. And in this case, our activation rate again, I'm not able to recollect the exact numbers, but doubled or something. And we also saw steady increase in Jira Service Desk customer count and and as a result of that we made it the default production variant. So that was exciting that, after a failed experiment and then following up with some qualitative and learning together with the product team we didn't give up, came back at it, and found something pretty unique for our onboarding experience. Ashley Stirrup: Yeah, I love that. It's such a great example of [00:14:00] unlocking the, a business and the learning and, taking a loser and turning it into a winner. Just think how many teams would say, "Oh well, we rolled out the new thing," and maybe they're, looking at some of the short-term metrics and saying, "Look, this is a huge win," and then they move on, and then they missed the true winner that you're able to uncover. So that, that's a fabulous story. I- Raunak Kumar: Thank you Ashley Stirrup: In general how do you approach experiments that lose? Like, when do you decide, "Okay, this is what I really wanna dig in on," and when do you decide, "Okay, this is a loser, I'm just gonna move on"? Raunak Kumar: Yeah, that's a hard one. So essentially, a well-designed experiment honestly is a one where you are able to detect something quickly enough. Again in B2B especially where I play, like the biggest headwind we have is the lack of traffic volume. If, had I been in B2C, you'd see millions of visitors, but in B2B it's like usually thou-thousands as a reference point. I think designing the experiment in a way where we have some leading indicators and like if we can see some meaningful [00:15:00] shift in those metrics at least even if they're not stat sig, but let's say we saw three percent increase in visit, and we were hoping for five, like close enough. That I feel like I always look for something more in the data, even though like we call it a loser just because we didn't meet our threshold which we decided to. But I like go and look at some of, again, the things I mentioned. Like I'll try to... So we log everything in our warehouse, like at a user level, like obviously anonymized, but like at a user level. So I'll try to see some engagement metrics. I like to look at some of the sample session data just to see if there's anything interesting happening there. But and on the flip side I've had experiments where for example, I can share one with you where this was at Stripe, where we were trying to redo our inbound contact sales form and make it, more in line with the industry practice where, typically you don't ask a lot, where you just simplify the form of on name, email address, et cetera. So over there, we [00:16:00] actually made the like smaller, but still we were like not seeing an increase in the contacts and submission. So clearly something was still not working. Like we thought by just simplifying the form, we'll see an increase in contact sales, but it was not working. So that was like an obvious one for us to okay this variation is not gonna work. So like, how do we like not waste cycles and stop it and then like re-attack the problem? So I guess my answer to your question is try to study the experiment data. I know like people might call on me that, "Oh, you're doing early peaking and stuff." Like typically you're not supposed to like peak ahead of time. But in, in traditional, like in real world when you know you are against the the quarter timelines and like executives are asking you like, "Hey, what's going on?" You wanna have some intuition, and that's how I approach it. Ashley Stirrup: Yeah. Yeah, it makes sense. Yeah, the whole peaking thing is tricky, but as long as you're not doing anything, as long as you're not ending the experiment early peaking isn't terrible. Yeah, and I really liked your point on the hypothesis definition, [00:17:00] so that putting a little extra time in as you define the experiment to understand like really what are you expecting to come out of it and what are all the different data points that you might look at so that maybe the overall thing's not a winner, but maybe it's a winner for a sub-segment or maybe it's a winner on certain metrics but not others and that leads you to new ideas on how to iterate to move more of the metrics at the same time. Raunak Kumar: Yeah, so great that you call that out. So the example I was sharing with you was and the contact sales one. What we did there was like we made, We only allowed business email address to submit. Like to a lot of companies you would notice they, they only accept business email. But turns out we have this whole slew of small businesses who like still use their Gmails and stuff, which is legitimately they are like trying to contact us, but we're just blocking them. So that was an insight where, okay, it works for a select segment, which is the enterprise, but like for small business, like we need to re-rethink how do we collect their intent. [00:18:00] Like we don't wanna miss out on them as an example, like just because they don't have a formal email address. Ashley Stirrup: Yeah, I was I ran marketing for Algolia, which is a e-commerce type search engine. And one of the largest customers originally came through with a Gmail and, Raunak Kumar: yeah. Ashley Stirrup: He still owned the admin account under his personal Gmail. And so we always kind of reminded ourselves of that when we were tempted to say business emails only. So it's funny what you learn and what sticks with you. So how do you see experimentation evolving at Fin? Kind of what's next there? Raunak Kumar: Yeah. At Fin, where we are is a, the, I would say at at a pretty interesting spot where our product is definitely maturing, but also not that nascent decent mind share in the market. So like one of our biggest goal at hand is to further increase the mind share of our product, Fin. Anyone who's interested in customer service AI agent should think about Fin amongst the competitors. So that's a space I deal with and essentially that's [00:19:00] what we are doing a lot more of and running more experiments in this space. So to answer your question, how do I see it evolving? We'll continue to do a lot more off-page when what I explained earlier, like on-page and off-page. And off-page meaning, we'll do a lot of incrementality experiments because, we wanna figure out like what are the channels which helps us drive this mind share. So for example, I just concluded a full funnel geo lift study in, in US where we did a match market with five states with control in five states as test where we, held back e-any and every media in the control states and then compared the incrementality, incremental lift in our traffic and like aided awareness. So that was like a winner for us. Similarly, we wanna know like which other marketing channels helps us drive awareness. So the next I'm tackling is connected TV because it's a hot thing right now where, people are getting ads served on like Hulu, YouTube, et cetera, et cetera, through Trade Desk through Bombora, through LinkedIn, et cetera. So that's one. And then I'll tackle like other [00:20:00] channels. So pretty much I'll be tackling a lot of like off-page experiments and also a lot of creative I, I forgot to mention. That's one thing where I feel like with the advent of AI, earlier if you would wait on a agency to give you a copy, right? Like now anyone can go and spin up a copy. So like same thing is hap-happening at Fin. Like we are getting like tens of copies of different things. How do we like, have a measurement framework to figure out like what's the winner quickly? That's what we are also gonna do, is run a bulk of copies through the audience we have. Do ABC test kind of a thing, and then figure out like what's working, what's not working. So those are the things I'm interested in and like I'm staffing and I'm really excited to like support at Fin. Ashley Stirrup: , That's a pretty impressive list. As a marketer myself, you're making me feel like, ah, I've got so much I need to go do. I need to find my own version of you to have on my team to do that. you know, I think it's a really interesting point about the ability to test more iterations. I don't, I still don't think AI is at the point [00:21:00] where you can create more visuals that are at the same level, caliber as if you were using an agency or something. But it does give you the opportunity to rapidly test a whole bunch of things and then maybe find a winner and then say, "Okay, great. Now let's go turn that into an even more professional version that we wanna, push out on more channels." So it gives you a lot of opportunities to get more information and know where to invest. Raunak Kumar: Yeah, for sure. And same thing is happening on the landing page. I think one of my aha moment at Fin has been like our marketing team, especially the demand gen team, is like fully enabled to spin up a landing page. Like what used to take like weeks and weeks of working, coordinating with PMM and web team to spin up a landing page. Like they have end-to-end tooling at their disposal. And then, they're launching intent-based landing page. Let's say they identified a keyword, and then they'll spin up a landing page. So just like there's so much more data coming our way to like track. So how do we actually keep up with that? And I think that's where [00:22:00] AI is helping us a lot. Like it's speeding up things, but then I can tell you that of all the fifty landing page, only ten works. So it's not everything is working. So like we're also figuring out quickly what's working, what's not working, and then we can shut those off, like not index them. Like we don't wanna index things which are not working, as an Ashley Stirrup: Yeah. Yeah, but that's a great example of the power of AI is just the ability to go test so many more things and maybe you find a handful of winners you never would've known were out there if you didn't have the ability to test a lot of things, Raunak Kumar: Yeah, for sure Ashley Stirrup: Raunakk, thank you so much for joining today's show. You've shared a lot of great examples of things that I'm sure a lot of our listeners will wanna go apply to their businesses. So thank you so much Raunak Kumar: Thank you, Ashley, for the opportunity, and hope the listeners found something valuable. Thank you. Ashley Stirrup: Yes. Thank you [00:23:00]