Latest / The Experimentation Edge / How Cogniteer Built an Experimentation Engine From Scratch

Transcript

How Cogniteer Built an Experimentation Engine From Scratch · The Experimentation Edge

• The Experimentation Edge - Fabian Hans === Fabian Hans: [00:00:00] Navigating the website is always an issue. If you have too many options, it's an issue to find the right solution for your problem Welcome to the Experimentation Edge, where product managers, data scientists, and engineers talk about how they make smarter decisions. I'm Ashley Stirrup, the chief marketing officer for GrowthBook, and in each episode, I'll sit down with an executive to unpack how they use experimentation and A/B testing to make better decisions. This show is sponsored by GrowthBook, the open source experimentation platform leader. Now let's jump in and get started with our next guest Ashley Stirrup: Hello, and welcome to today's episode. Today we have Fabian Hans, founder and behavioral psychologist at Cognitier. Welcome to the show, Fabian. Fabian Hans: Thank you, Ashley. Thanks for having me. It's a great pleasure.. Ashley Stirrup: Yeah. Really excited to have you on the show. Most of our guests tend to be practitioners at individual companies, and so it's fun to have [00:01:00] somebody who works with a lot of different companies. So maybe you could kick things off by telling us a little bit about Cognitier. Fabian Hans: Yeah, I would start at the really beginning. Maybe my journey in CRO started like 15 years ago, when I applied in a company as a online marketing manager, and they asked me as a little test how I like their website. And I told them that I really don't like the website, and I gave them really good reasons why I don't like it. And then they hired me as a CRO manager, um, and I had no idea what a CRO at that point or experimentation. And yeah, it took off. I built a team and a company or an agency, and then at that point, I actually realized that in the agency we do it quickly, not really the deep dive. We have a lot of clients, right? And It ended up actually that all of the clients got the same tests just because we have tested them somewhere and we think since we have tested them somewhere, they will be successful [00:02:00] somewhere else. So it's a mass production of tests, right? And I did not like this approach. I wanted to do the deep dive and all that, and that's when I went in-house to actually have the time to really deal with one company and do the deep dives, really understand the user behavior, Also the psychology, since I have a psychology background behind it. And then previous clients approached me and said: "Hey, Fabian, can you not help us build our in-house setup, our team since we work with you in the company, in the agency?" So that's how it started with Cognitier, and now we're focusing on helping companies, enterprises, for instance to build the in-house setup and scale their program and increase the test velocity and also the win rate, for instance. Yeah, that's what we do now with Cognitier. That was four years ago I founded Cognitier, and yeah, it has been a great journey so far with a lot of different clients with different situations, I would call it. Ashley Stirrup: Yeah. Yeah, sounds exciting. W-w-what types of clients do [00:03:00] you have? Are they more e-commerce or applications or Fabian Hans: We do have lead gen, but mostly actually e-commerce and we also had a lot of fashion e-commerce because one of my first references that we had was a fashion e-commerce brand. we do all of them because in the end of the day, for us, it's really about understanding the business, understanding the user, and what's in the way on the website, in the platform for the user to actually perform to perform the action that we want them to perform. And so it's really interesting that we have all the different fields, all the different verticals, different companies. I would say it almost just depends on the traffic, that they have enough test traffic for testing and then we think about, okay, how's the user behavior? How the website, for instance, not fit into the user's buying behavior? Let's say for instance, if you're buying a fridge, you care about the features, right? the loading [00:04:00] volume, you care about the water that it's using and all those features. But then if you look at the typical e-commerce, which is selling washing machines for clothes, they all show you a picture and all the washing machines look alike. So it really makes sense that you're using pictures for fashion, for instance, where the style and the design makes a difference in the decision-making process. But for a washing machine, the pictures itself don't really make a difference in the buying behavior, and every business model has issues like that when it comes to e-commerce. Let's say for instance, perfume, you cannot sell online because you need to smell it first. Other products, you need to touch them first. So all the user journeys for all the products are different, yeah? And that's where we come in to make our user, our customer understand how is the user journey, how do they actually buy, what needs to be done in interface. I will call it [00:05:00] interface because it's not really a shop. I don't consider it to be a shop or anything. Just say it's the interface and how does the interface need to be, so the users can actually buy online in the best way and make the best decisions, Ashley Stirrup: yeah, I love that. That's definitely a recurring pattern across the guests on the show is that every guest, their customers have a different journey and, sometimes it's buying, sometimes, chess.com is about the experience playing chess. And so of course, that's gonna be a very different type of experience. I thought it was interesting when you talked about as you first started in the agency world, you found yourself running the same types of experiments over and over again. I would imagine that there's a lot of kind of common lessons learned, especially if you're working with a variety of, companies in the same industry with a similar, user journey. Did you find yourself kind of identifying these best practices and then helping a lot of those customers implement those? Fabian Hans: Basically that was how it is because let's say for instance, [00:06:00] in the e-commerce, yeah, the basket, is always an issue. You usually have a high drop-off in the basket. That's something that happens on all the e-commerce, but that is not because of the company or of the product. It's just shopping and people not secure yet if they should really do it, and all these kind of things that happens in almost all the e-commerce shops that the basket has a high drop-off rate. So that happens, right? That is an e-commerce problem, I would say. And that's the problem, again, because your product might not be meant to be fully sold online, right? It needs the offline experience to be sold, and that makes the user insecure if they're really buying the right thing right now. I would say online shopping started with Amazon selling books right? That's my earliest Recall, right? Amazon selling books. Books are easy to sell online because you can understand the content. you understand if you like the book by reading the cover, right? You know if you're making a [00:07:00] right decision online. But all the other products are really difficult to judge online fully, right? So that's the common thing for me that I witnessed, that if you do all the same tests or running all the same tests without really fully understanding the user behavior of your product, not even the user behavior of your users, but what does your product cause, what kind of behavior does your product cause? What do they need? And if you start optimizing for that you can find the biggest levers. But then, yeah, carts for instance have a high drop-off. People don't like signing up. It's the same product, right? It's the same issue all the time. Navigating the website is always an issue. If you have too many options, it's an issue to find the right solution for your problem and all that. You see the same patterns all the time over and over again, and then you start testing the same things over and over again. Our biggest test in the agency was, like, the first test that we always did in the basket was, like, [00:08:00] your items are not reserved. That's one line. It's super easy to develop. It's psychology. It sells well to the clients. And we had a 50% win rate with that, And it's okay, we did it over and over again. Ashley Stirrup: Yeah Fabian Hans: You're like Losing the passion for it if you always over and over do the same things because e-commerce, also the product detail page usually has a really high drop-off. If I have so many shops every time when I look into it, we're losing people on the product detail page and in the basket. And why do we lose them on the product detail page? Because people are entering the shop on the product detail page, and the product detail page is not made to be a re-landing page to introduce your brand or your product to the client. It's just directly product. So all these brand questions are not answered yet. So same problem coming over and over again for all the e-commerce clients. Ashley Stirrup: Yeah. The nice thing is then at least you can take those best practices to somebody who doesn't have a lot of [00:09:00] traffic, and you can at least still go apply those same principles with a high degree of confidence. Fabian Hans: Yeah. Of course. You have to test it and test it, but it is, I would say in 80% of the e-commerce shops I see the same problems because it's e-commerce. It's simply the problem comes from it is an e-commerce shop. Ashley Stirrup: And as you're helping companies set up their own internal experimentation, what types of challenges do you typically see that companies are experiencing in doing that? Fabian Hans: I would say resources, now is disappearing because of AI. Especially a long time before, the biggest issue was developer resources. so this is disappearing now with prompt-based experimentations and all that. But then communication and alignment on the goals and the problems. Because if you asked in a company five people what's the problem on the website, yeah, four might not know it, and one might assume a [00:10:00] problem on the website. Most of the people look at the website and say, "Ah, I don't like this. I think this should be improved." But If you asked a lot of people what should be improved, you get five different opinions when you ask four people something like that. And then the bigger the company, the bigger the team, the more opinions you have, the more misalignments you have what actually has to be done. So one of our major challenges actually was always to bring people together, to align them on the same principles when it comes to testing, to align them on one roadmap, to align them on the same problem understanding on the website. So this is where we actually come in to find out why users are really not converting and give them from the outside perspective. Because in-house, you're losing the outside perspective. The client knows everything about their website. They know everything about their product. They know everything about their campaigns. They know their navigation inside out. They might even [00:11:00] know the SKUs on top of their head, right? User comes to the website and says, "Huh, am I right here? Can I trust this? And what do I get out of this?" And in-house, you don't ask these kind of questions anymore because you're working every day, eight hours on the shop, so you don't see that anymore. So our major thing is always to align on the problem first. What is the major problem? Where do we have the highest drop-off? And why do we start optimizing here? And also, I would say we align on which problem has to be tackled first. Because if you have four different problems, you cannot solve one problem and expect more conversions to come out if the second problem, the third problem are still existing. It's if let's say for instance, in a more dramatic way to say your last button in the checkout is not working, you don't need to stop optimizing the basket, your last button in the checkout. So these kind of things on a more like a psychological level, [00:12:00] we analyze and tell the clients, "Hey, this is the major problem and this problem is related to this problem," so we can build a hierarchy of problems that we ha- need to solve in the right order. Ashley Stirrup: Yeah, makes total sense. Can you share with us an example where you were working with a client and you uncovered a lot of learnings from an experiment? Fabian Hans: Yeah. It actually all the experiments have a lot of learnings, right? But there's one prob- one experiment. It's not one experiment, I would say. It's more like we identified one problem. Yeah. And then we figured out in-house that the teams or how they were approaching this problem was simply a misalignment between the teams. Let me give you an example. Let's say they had a 80% drop-off, yeah, in the checkout. And one team's target was to increase the order revenue, the average order volume, right? The other team was supposed to increase conversion rate. And if you [00:13:00] increase the average order vo-volume, you usually decrease the conversion rate because the higher the basket, the harder it is for the user to make a decision, right? And so two teams had two different goals, and they had a 80% drop-off in the checkout. So which goal to focus on is obviously not the average order value, right? Because drop-off is the problem. But then what they try to do is an upsale to sell to increase upsale or to implement upsales during the checkout process already. And we said, "Okay, now we have the problem here. We lose 80% during the checkout process," but also during the checkout process, you're trying to implement the upsales already. Yeah. So at that point, users don't expect the upsale because all these questions that were asked are basically post-purchase things that the user would consider post-purchase. So what we did is we start [00:14:00] testing the upsale elements on a different order in the checkout, right? So we said, "Okay, if we have more people seeing the upsale element, they will more likely be, maybe be more to actually convert. Or if we take out the elements from the checkout, they will be more likely to convert afterwards and do for the average order value," right? In this case, we had major problem, we defined the major problem, and we said, "Okay, the major problem is high drop-off rate in the checkout, and the high drop-off rate from the checkout is coming from the positioning of the upsale elements." So now our aim is to find the right position for the upsale elements. So how do we position the upsale elements? Where do we position them? Which order do we do position them? Do we take, put them into the checkout, outside the checkout, before the checkout? If we find a better position, then what order of the upsale elements? So we had a clear roadmap, but the clear roadmap [00:15:00] was basically just questions. Yeah. the first question is where to put the upsale elements, right? And yeah, we had a lot of learnings on that because it was always like average order value versus conversion rate. In all the tests, we had to weigh them out, and sometimes we lost in, in average order value, but we gained in conversion, which overall made more revenue. But sometimes we lost in average order value and so it's like always balancing these two metrics was really interesting had a really fun environment actually, because before the teams were always like fighting, " No, it's about the average order value. It's about the conversion rate. No, average order value." And I said, "Guys, it's not your decision." The problem was basically that the teams got different goals from the management. They have different responsibilities. But- We helped them to align and to actually understand, hey, these go together. If you don't balance them together, it's not one or another, it's they go together, They all start [00:16:00] working on the same aim to actually find the best solution for both of them. So there was, for the team, a great learning that testing actually gave like a fun environment to align again on something or build something together. But also on the other hand side that we answered a lot of questions during this process and found out during the execution, it's always like you test something, it's not working, then you basically have to iterate. And then we always had the analysis afterwards to find out why did the campaign not win, right? Why, what was wrong? What happened actually? What was the user behavior? What happened here? So yeah. Ashley Stirrup: Yeah. Yeah, so it sounds like there are a couple of really important things you did there. One is align the team, I'm assuming around like total revenue or something like that, so like a combined metric. And then y- you set out to understand the buyer's journey more by running lots of different tests across that whole journey to really understand, what [00:17:00] was the best way to structure it all. Fabian Hans: Exactly. Yeah Ashley Stirrup: Yeah. Yeah. Super interesting, and I would imagine that even within that, maybe you have different buyers with different preferences. Like some maybe are more likely to, get them on a smaller initial sale and then post-sale you upsell them versus others maybe if you show you have a total solution then, they're more likely to convert, that type of thing. So yeah. I think you also had another example with the selling water solutions. Is that right? Fabian Hans: Yes. The solution there is basically that we found out that the major problem-- It's a B2B case, actually. Yeah. And we found out that in B2B, the shopping behavior is obviously super different than B2C, right? In B2B, it's usually like you get a budget, and then someone in the company has to find a solution for the water dispensers in the company for that budget. And in B2B, it's more like, "Okay, I like this product. I will buy [00:18:00] it. It's in my budget," right? One-person decision versus three to two to three people decision. So you get a budget, and it was actually we found out during the analysis that the shop or the website applies to a decision-making process. So they show a lot of products, then they tell you, "Oh, this is a great product. It has this and this features," similar to what I just said with the washing machines. Similar B2B, it's more like, "Okay, I have a budget. me a solution for that." Then I need three offers, and with these three offers, I will go to my manager and show him these three offers. He will look at it and pick the nicest one. So we don't really need to show products on our website. That was the assumption, right? And that's what we validated then with the customer journey map and all that to actually find out that, yeah the B2B doesn't need to show that many products or not that much details of the products because they just wanna come to the website, get an offer, know that [00:19:00] this offer fits their situation in-house, in the company, like how many people are working in my company, what kind of company I am, et cetera, all these kind of things. So we implemented like a finder logic, and that increased the conversion rate by a lot and let them not navigate through the website anymore and find a product by themself, they don't know if it's the right product for their company even, And then it tells you a lot of features like filters, et cetera, how to filter the water. But all these kind of questions are not really relevant at that point when you make the decision. You just want to know-- Maybe you don't even have a budget yet. Maybe you're just getting offers to get an understanding how the budget should look like, you Happens a lot in B2B. So you go to a company, you ask them how much it is. With that, you go to the management and tell them, "This is how much. Can we get this?" And then you actually start looking for the product, We adjusted it to the logic, and then we did like several iterations with a finder to actually [00:20:00] implement the right logic for them to find or to actually do the lead on the website instead- Instead of giving them products, we show them a finder. Not a finder, it's more like a survey that you walk through, and by walking through the survey, afterwards we tell you, "Hey, we have a solution for you. It fits to your company. It fits your needs basically. Request an offer, and then we get in contact with you." So it's like a easy lead gen mechanism where we just ask a lot of question and tell them afterwards we have a solution for you, and we can give you an offer. But away from B2C basically, yeah. We moved away. So we don't have a shop interface anymore. We had like then we had a yeah, we had a lead gen form basically, which was the main focus of the page. Ashley Stirrup: Yeah. And so it sounds like what you were doing there was like A, moving more to a solution sale than just a, "Here's what the actual water dispenser looks like." And also setting people's expectations [00:21:00] that, this isn't a thing where you're gonna actually just go and buy, the individual water solution. You actually need to go talk to a salesperson, and you're gonna get a packaged offering that fits your business. And so it's much more of a B2B type of mentality. Fabian Hans: Yeah. Because in business you always, could be if you're a big enterprise, you need to serve 1,000 people maybe, right? And maybe you have a factory that needs to be supplied with water, or you have office spaces, so you need to have maybe different solution inside the company. As a B2C, you go and say, "Oh, okay, I like this product. It's black. It fits in my apartment. Looks beautiful. It gives me even hot water for my tea. I love it." That's a different shopping behavior. Ashley Stirrup: Yeah. Yeah, absolutely. One thing another guest recently said is that that they do is that they work with different teams, they're always encouraging them to design the experiment assuming they're going to lose, and that it just changes the mentality and you realize you are gonna ask for different data. Like how do you do that type of thing with your clients? Fabian Hans: [00:22:00] Yeah, we also actually have to educate our clients about it because they doing A/B testing for winning, right? We all go for the win. We want to get more revenue, we want to get more sales, more average all these kind of things. But when they start, especially when they start, they have a different expectation to A/B testing. If for instance, Ron Kohavi, he's the former VP of experimentation for Microsoft and Airbnb, he published the numbers and he said one out of eight A/B tests is successful, right? One out of eight A/B tests. That means seven out of eight are not successful, so the possibility that it's negative is higher than positive. And we educate our clients in a way that we say the process or the way you set up the A/B tests has to make sure that afterwards you know why the A/B test wasn't successful. Because most of the tests are not successful. First of all, A/B testing [00:23:00] is not just about winning. A/B testing is mostly about avoiding breaking things, right? And then the other part is that we set up the process in a way that they understand how to afterwards analyze. That means if I run an AB test, right? Or all the experimentation managers do all the tests with the best intention. They sit there, they build the variation, they develop it, they do the QA to make sure this is going to be a winning variation. But then the test is not winning, and that's really frustrating obviously, because it costs a lot of effort and also, everyone likes their ideas when they're executing their own ideas and have business impact and all that, but then it's not successful. But if you have done everything possible to make this a winning test and then it's not a winning test, right? So why would you then say I'm stopping here, right? Because usually they say, "Okay, the hypothesis is wrong." They say, " [00:24:00] My testing idea wasn't the right testing idea." But what we say, if we understand the problem first, we have a clear problem understanding and say, "Okay, the hypothesis or the test was not a winning variation, but the problem is still there," right? We haven't solved the problem yet. And then we start iterating the variations, and we need to set up metrics that makes us understand how the user behavior changes in our variation. Yeah. Like for instance, if I implement an extra button on the product detail page. Yeah. And let's say we have a carousel in the first view, and in our control variations, more users are engaging with the carousel in the first view than in our test variation, where they're more interacting with our new button, which leads them to the bottom of the page. Then we know if we have a losing variation, then we know maybe afterwards that it might have lost because less people were interacting with the [00:25:00] carousel. So the carousel is something really important. So in our iteration, we need to make sure that our new element that we created doesn't take away the attention from the carousel, which is really important for the decision-making process of the users, right? So what we usually do, we set up, I don't know, it depends on the test, but sometimes we even have 20 events that we are tracking, sometimes even more. Of course, we also do the heat map setups and all that. So we afterwards, we clearly, basically before the test goes live, we already ask ourselves Why will this be a loser? Or what could possibly happen that this test is a loser, and then we implement tracking to basically validate all the follow-up hypothesis, We already have one hypothesis that we're testing right now, but we already have an idea why it's probably not a winner, and then we try with the data that we're gathering through the test to validate. I mean, if it's a winner, okay, but if it's a loser, then our follow-up [00:26:00] hypothesis might be true, right? And we want to validate through the testing if our follow-up hypothesis could become true, basically Ashley Stirrup: Yeah. Yeah, super interesting to, to approach it with that mindset 'cause A, suddenly now you wanna learn about, how did I affect user behavior and am I tracking the right things to really understand that? And then maybe there's also a difference based on your power users versus your first time users. And Fabian Hans: exactly. We just we actually just had a case where a client, he came to us and said, "Hey, we want to optimize for our new users. We want to convince more new customers for us." And it was a business where it's a habit buying. So that means customers come regularly to the page and rebuy or restock their products. It's like buying food, basically. Every week, we're buying the same food for our fridge to restock. what we found out that the page is perfectly for restocking, but as a new buyer, in this case, you need a lot of information. You need to understand if it's the right product, et cetera, and all that. So you have a lot [00:27:00] of questions, but the page was optimized for restocking. And then we said, "Okay, the page is ideal for restocking, but really bad as a new customer because y- all the information is not accessible." And then we said, "We have to personalize this page because otherwise we cannot tackle the problem that we have restocking users and we have new users, and new users don't convert because they are confronted with the transactional element way too early, and they don't get all the information they actually need before they're buying." Then we start optimizing to make the information accessible for new users and for returning users. We took out these elements that make this information available because they don't need it. They have read it in the first session, but in the second and third sessions, they don't need all this information anymore because they're restocking, obviously. They just want to come in, press a button, and leave again, basically. Yeah. Ashley Stirrup: Yeah, that's such a powerful example. I know it. I just think about my own shopping experiences [00:28:00] on Amazon, and I'm frequently like, "Oh, what was that thing I bought six months ago? I need it again." And yeah, it's a very different shopping experience Fabian Hans: And if you know the product already, you're not going to read through all the informations again. But if you're buying it the first time, you want to know, does it fit my use case? Is it the right measurement? Et cetera, et cetera, right? Ashley Stirrup: Yeah. Yeah, that's very powerful. So as we wrap up how do you see experimentation evolving over the next couple of years? Fabian Hans: I think I would say it equals learning, right? And for me, learning, I think the humankind, I would say, has been learning all the time. But now I think in times of AI, we learn faster since now having access to information, for everyone basically. It was already before with Google, right? That we had inf- access to information. But now, nowadays, it's even faster, and it's easier to access the [00:29:00] information and also to analyze informations, to gather informations is a lot easier now. So I think like this learning will become faster, I would say, yeah, because of the AI. So we learn faster, but I also expect that we make better predictions And because we have faster learning and all the information is available now, we will have better impact with the test. It's not about not knowing what you're doing anymore and guessing if the test will win. I think now we will find great ways to actually make better predictions if this test is actually possible or if this test is actually going to be a winner because we will have more informations available. We can do research faster now, user research faster. We can gather faster feedback from the users. But also I think now since they're working with synthetic users and all these kind of things would say I would not optimize for [00:30:00] quantity my testing program anymore. I would really go for quality and dig deeper because now we have the opportunity and the informations are available. And yeah, Humankind has always been learning through experimenting and failing basically, right? The failing is the learning part. That's just learning. And if this doesn't work, do it differently and learn till it works, right? It has been always been there, but this will be in companies especially become way faster. So Ashley Stirrup: Yeah. Yeah it's gonna be fascinating to see. Obviously n- none of us knows, just how AI will evolve and what it'll be able to take over. But I'm a big believer that it's an accelerant for creativity and accelerant for learning but that humans will still be a very important part of the process, and that if AI can remove the friction, allow you to move faster it'll be very powerful. But that humans will still be the ones that need to provide that [00:31:00] additional level of context and what to test and where to go and where are the biggest opportunities. I think the humans will still have a very important role there. Fabian Hans: I'm really looking forward. I really believe in this utopia, I would say, that we might not be working anymore or I don't know. But I'm really curious about the future, where it's going to be and what are the outcomes of all that. Obviously, my job as a consultant will also change a lot because now we're also more focusing on enabling companies how to do it with AI and how to use AI in the process. And it's interesting this, especially these big enterprises, they're especially in Germany, it's all, all about data protection and security and, we put too many barriers in our head, I would say, in Germany. Like other countries, they say, "Okay, let's just do it. Let's move forward. Let's do it, let's do it." And Germany is more like, "Ah, okay what if this happens?" And always trying to predict the worst case which holds us back a little [00:32:00] bit. Yeah, and then we come into companies, and then we see that they not really working with AI yet because they're still trying to make strategy for AI. don't you just start? Ashley Stirrup: Yeah. It is so interesting with AI that you... It's really hard to know which things AI will be good at or not, and so you have to go try a bunch of things, and then you find out it's great at coding, but maybe not so good at predicting, what the user's gonna wanna do next, for example. Fabian, thank you so much for joining today's episode. I think you covered some really interesting topics, and it's so interesting to think about how experimentation programs vary across companies. Really appreciate your perspective on all that. Fabian Hans: Yeah. Thank you so much for having me, and also for the audience. If anyone wants to reach out on LinkedIn, I'm always happy to have a chat about experimentation. Would be great if you guys reach out on LinkedIn, Fabian Hans. Let me know if you want to have a chat. Ashley Stirrup: And of course, we'll be linking to Fabian's profile in the show notes. So thank you again [00:33:00] so much Fabian Hans: Thank you.