Latest / Code Story | Startup Podcast for CTOs, CEOs and Technical Founders / S12 Bonus: The Predictive Model Trap: Moving Beyond Static LTV Forecasting to Unleash Causal AI and Dynamic Multi-Armed Bandits with Tobias "Tobi" Konitzer, PhD, VP of AI at GrowthLoop

Transcript

S12 Bonus: The Predictive Model Trap: Moving Beyond Static LTV Forecasting to Unleash Causal AI and Dynamic Multi-Armed Bandits with Tobias "Tobi" Konitzer, PhD, VP of AI at GrowthLoop · Code Story | Startup Podcast for CTOs, CEOs and Technical Founders

• 0:00
  [SPEAKER_00]: But now I'll always face the second big question, which is, okay, you'll have this thing that you want to build, maybe you just build a new product and you call it Toby's brilliant autonomous marketing decisioning agent.
• 0:13
  [SPEAKER_00]: Now know it as you can tell, I'm not a product marketer obviously because hopefully nobody calls it that.
• 0:17
  [SPEAKER_00]: But you build this product and you go to a dark room and you bring all the smart people, hire the PhDs, and you gravitate towards building this thing and writing the Bayesian reinforcement learning models on the whiteboard, you feel really good about yourself.
• 0:32
  [SPEAKER_00]: Nobody wanted what we built.
• 0:33
  [SPEAKER_00]: So, I've really reworked the way that I'm thinking about tackling these big problems.
• 0:39
  [SPEAKER_00]: My name is Toby Connitzer, I am VP of AI for Grossing.
• 0:46
  [SPEAKER_01]: This is code story, a podcast bringing you interviews with tech visionaries, six months moonlighting goes.
• 0:54
  [SPEAKER_01]: I'm lost in all the backgrounds, who share what it takes to change an industry.
• 0:58
  [SPEAKER_01]: I don't exactly know what to do.
• 1:00
  [SPEAKER_01]: It doesn't go as to get right.
• 1:02
  [SPEAKER_01]: who built the teams that have their bad company is its team's help each other, which is proud of our team.
• 1:09
  [SPEAKER_01]: Keeping scalability top of mind, all that infrastructure was up there.
• 1:13
  [SPEAKER_01]: Yes, we've been fighting it as we grew up.
• 1:14
  [SPEAKER_01]: Total waste of time.
• 1:15
  [SPEAKER_01]: The stories you don't read in the headlines.
• 1:18
  [SPEAKER_01]: It's not an easy thing to achieve.
• 1:20
  [SPEAKER_01]: To get yourself a deficit of off, try to begin to ride the ups and downs of the start-up line.
• 1:26
  [SPEAKER_01]: To really want it.
• 1:27
  [SPEAKER_01]: Not just about technology.
• 1:29
  [SPEAKER_01]: All this and more on code story.
• 1:32
  [SPEAKER_01]: your host Noil Abhart, and today, how Toby Condenser, is driving compound growth by introducing customer data and agentic intelligence.
• 1:49
  [SPEAKER_01]: Toby Connitzer was born in Germany, studied cultural studies as an undergraduate student.
• 1:54
  [SPEAKER_01]: Eventually he went to Duke to get a PhD in political science, and then eventually changed to become a PhD in computational social science at Stanford, which is basically writing code to answer a social science questions.
• 2:07
  [SPEAKER_01]: After graduating in 2017, he joined Facebook research for a year, then founded two AI startups.
• 2:13
  [SPEAKER_01]: But outside of Tech, he has two young daughters who he likes to spend time with and take to the part.
• 2:18
  [SPEAKER_01]: He used to be an avid trail runner, but his favorite thing to do is think, and do so as often as possible.
• 2:27
  [SPEAKER_01]: For the last 10 years of his career, Toby has been chasing optimized decisioning and outcomes using AI.
• 2:33
  [SPEAKER_01]: Five months ago, he decided to join his current venture and use AI to shift the conversation from tooling for marketers to using AI to build an autonomous decisioning system that learns and improves over time.
• 2:48
  [SPEAKER_01]: This is Toby's creation story at Groofloop.
• 2:54
  [SPEAKER_00]: So growth loop, before I joined, and still is what is called a composable customer data platform.
• 3:00
  [SPEAKER_00]: So essentially what that means is in the marketing world without having to pull any data out, so zero copy data.
• 3:08
  [SPEAKER_00]: Customers can use growth loop, which unifies data automatically on the customer data warehouse, and then serves two marketing applications on top of that.
• 3:18
  [SPEAKER_00]: One is audience building, the other one is journey canvas or journey building.
• 3:22
  [SPEAKER_00]: So imagine your marketing person and you say, look, I really want, I want to build an audience of people that have a very high transaction volume and they open my last email.
• 3:33
  [SPEAKER_00]: Usually, that data is siloed.
• 3:35
  [SPEAKER_00]: The email engagement data lives over here and the transaction data lives over there.
• 3:39
  [SPEAKER_00]: And so the workflow would be you ask your data engineer to pull that data for you to unify it and write the SQL code to build the audience.
• 3:48
  [SPEAKER_00]: Essentially, what growth loop does is provide a UI that does not require any of these as one of my colleagues would say these bread lines, U.S. a market are having to wait for the data engineer to satisfy the ticket, and you can do the UI build these audiences directly, and it automatically merges these tables together from different parts of your data warehouse, and then also exports the audience to a destination, no?
• 4:12
  [SPEAKER_00]: Like, gray is like, innumerable, like Facebook, like Google Ads.
• 4:15
  [SPEAKER_00]: So that is a traditional world of composable CDP.
• 4:19
  [SPEAKER_00]: Essentially, if you think about it, that's tooling for the marketer, right?
• 4:22
  [SPEAKER_00]: It's tools to make your job more efficient, and a whole gain here is efficiency.
• 4:28
  [SPEAKER_00]: When I started talking to Tomino's RCTOCPO and Chris O'Neill's RCL, these guys had laid out a vision of closing the loop.
• 4:36
  [SPEAKER_00]: So essentially, of using AI,
• 4:39
  [SPEAKER_00]: to build almost like an autonomous decisioning system that doesn't ask the marketer to build the journey, but actually builds the journey in such a way that the ROI is maximized, right?
• 4:51
  [SPEAKER_00]: And shift the whole debate from tooling, tooling for the marketer, to an opinionated,
• 5:01
  [SPEAKER_00]: network that maximizes whatever return to pre-specify.
• 5:05
  [SPEAKER_00]: And after tell you, these sort of closing the loop plans are not only ambitious, but I've been chasing them for the last ten years of my career, both in Attek and in Mardtech.
• 5:15
  [SPEAKER_00]: And it's why I signed on, we can talk a little bit about where Grothel is now, what we're building towards, how we bring this vision to life, I'll stop here for a second.
• 5:26
  [SPEAKER_01]: And that's actually a perfect segue.
• 5:27
  [SPEAKER_01]: That's kind of the next place I want to go.
• 5:29
  [SPEAKER_01]: So what is being built now?
• 5:32
  [SPEAKER_01]: What is the next step with growth loop now that you've joined in and you're going to tackle these ambitious problems?
• 5:40
  [SPEAKER_00]: It's an interesting problem because many most companies have this problem.
• 5:43
  [SPEAKER_00]: Oh, let me infuse AI into existing products and or somehow into our offering and make our offering strong.
• 5:51
  [SPEAKER_00]: I think growth loop was a little bit ahead of that before I joined because it was a specific vision about the compounding marketing engine also this can, this autonomous, self-optimizing thing, weaving organism that automatically maximizes the outcome.
• 6:07
  [SPEAKER_00]: But in general,
• 6:08
  [SPEAKER_00]: And I think it's a high question to answer, and if you look at most companies out there, particularly with the emergence of Genotyfei, one of the ways very early things that most companies many companies build is something like a chatbot.
• 6:23
  [SPEAKER_00]: because it's so obvious, right?
• 6:24
  [SPEAKER_00]: You can build a chatbot that is powered by Gen AI.
• 6:26
  [SPEAKER_00]: And I always, I scoff at that a little bit.
• 6:29
  [SPEAKER_00]: Why?
• 6:29
  [SPEAKER_00]: Because for most companies, it's actually completely orthogonal to the product offering, right?
• 6:36
  [SPEAKER_00]: So I'm a car market place.
• 6:38
  [SPEAKER_00]: And my first investment in AI is to build something that is entirely orthogonal for my product.
• 6:43
  [SPEAKER_00]: So that's not the mistake that we wanted to repeat here.
• 6:46
  [SPEAKER_00]: But now we face the second big question, which is, okay, you'll have this thing that you want to build.
• 6:52
  [SPEAKER_00]: Maybe you just build a new product and you call it Toby's brilliant autonomous marketing decisioning agent.
• 6:59
  [SPEAKER_00]: Now, now, as you can tell, I'm not a product marketer, obviously, because hopefully nobody calls it that.
• 7:04
  [SPEAKER_00]: But you build this product and you go to a dark room and you bring all the smart people, hire the PhDs, not just me, but hire more PhDs.
• 7:13
  [SPEAKER_00]: And you gravitate towards building this thing and writing the, I'm gonna use one mathematical term, writing the Bayesian reinforcement learning models.
• 7:23
  [SPEAKER_00]: I'm the whiteboard, you feel really good about yourself.
• 7:25
  [SPEAKER_00]: And I've been there in my career, by the way.
• 7:27
  [SPEAKER_00]: I've raised my last venture, raised about six million dollars, hired all the smart people, built product new vacuum.
• 7:33
  [SPEAKER_00]: And then we did, we off course, where smart people, so we knew that the market wanted exactly what we had built, right?
• 7:40
  [SPEAKER_00]: Now, nobody wanted what we built.
• 7:42
  [SPEAKER_00]: So, I've really reworked the way that I'm thinking about tackling these big problems, how to infuse AI into product.
• 7:49
  [SPEAKER_00]: right?
• 7:50
  [SPEAKER_00]: So what we did is we looked very carefully.
• 7:53
  [SPEAKER_00]: Where is product usage?
• 7:55
  [SPEAKER_00]: Which are the product elements today that are used and how can we basically upgrade these properties, these primitives, one by one, with elements towards that vision such that we basically battle test all the components, think about it as a car.
• 8:12
  [SPEAKER_00]: You want to build this luxury car instead of building it in a vacuum, you build the components.
• 8:18
  [SPEAKER_00]: You infuse the existing primitives that have a guaranteed and known usage with these primitives or with these elements towards the vision that you build.
• 8:27
  [SPEAKER_00]: You get customer feedback and by the time all these components are validated, then maybe
• 8:33
  [SPEAKER_00]: You put it all into the big, beautiful Ferrari that is waiting there, the hull, the shell, and then you go.
• 8:38
  [SPEAKER_00]: So it's a very different way to think about infusing AI into product and building new AI product.
• 8:46
  [SPEAKER_00]: And now, of course, the big question is, can you go top down?
• 8:50
  [SPEAKER_00]: in terms of your thinking and prioritization.
• 8:52
  [SPEAKER_00]: And can you take such a vision as this compounding marketing engine?
• 8:57
  [SPEAKER_00]: This autonomous outcomes optimized decisioning system and break it down into components that are small enough that makes sense to infuse and to exist in products and to also ensure that you get usage out of these, you can iterate over them, you can make them better.
• 9:13
  [SPEAKER_00]: And I think that's probably the biggest challenge in this on the pick.
• 9:18
  [SPEAKER_01]: I'm curious, you know, where you started, right?
• 9:21
  [SPEAKER_01]: What would you call your quote unquote MVP?
• 9:23
  [SPEAKER_01]: Maybe not the company MVP, but your MVP when you joined.
• 9:27
  [SPEAKER_01]: Like, where did you start with this new way of thinking?
• 9:29
  [SPEAKER_01]: You know, what sort of tools are you using to bring it to life?
• 9:34
  [SPEAKER_00]: So here, I probably spend a month to try to turn this vision into a sequential order of elements that can be built and infuse the existing products most notably audiences and drinks.
• 9:47
  [SPEAKER_00]: And I can walk you through this process, this process is still higher level, but essentially taking the vision which we haven't quite talked about yet.
• 9:55
  [SPEAKER_00]: but taking the vision and then saying, okay, what is naturally the first step and it turned out that the first step was building experimentation.
• 10:04
  [SPEAKER_00]: Infusing capabilities into particularly audiences, which are most used product, that polls experimentation back from the downstream systems.
• 10:13
  [SPEAKER_00]: right, positions grow through as a central hub of experimentation and intelligence, so I can lock that data and I can understand causal relationships and the effect, the causal effect of different interventions.
• 10:28
  [SPEAKER_00]: So that to me was the foundation.
• 10:30
  [SPEAKER_00]: I cannot build an autonomous decision system or decisioning system.
• 10:34
  [SPEAKER_00]: If I don't give the system a prior,
• 10:37
  [SPEAKER_00]: As to, here's a track record of encoded causal relationships or causal statements and the results.
• 10:45
  [SPEAKER_00]: So for that to happen, I need an experimentation.
• 10:48
  [SPEAKER_00]: I actually need an experimentation to sit in the product primitive that our customers use the mouse.
• 10:54
  [SPEAKER_00]: Now we have a different problem which is you've got to go to the market and you've got to convince the market that you better use my experimentation service as opposed to the experimentation service at least drop down stream so in the customer engagement platforms in the marketing clouds.
• 11:09
  [SPEAKER_00]: And so we had to build features into the experimentation platform to make sure that this poll happens that we can be the gravity, the gravity of intelligence for experimentation.
• 11:18
  [SPEAKER_00]: And that was two features.
• 11:19
  [SPEAKER_00]: One of them was
• 11:20
  [SPEAKER_00]: Yeah, we sit on top of all your data, so we can forecast exactly how can you run the most efficient experiment.
• 11:29
  [SPEAKER_00]: So our argument all of a sudden was, okay, you run experimentation in your marketing cloud, and you usually say, you know what, I'm going to dedicate 50% of the traffic to the control group.
• 11:37
  [SPEAKER_00]: 50% to the treatment group.
• 11:39
  [SPEAKER_00]: That's all you can do, but imagine if you move experimentation closer to the data where we sit, right, where you have much more intelligence.
• 11:47
  [SPEAKER_00]: Now, we can use machine learning or AI to forecast how to run the experiment most efficiently.
• 11:54
  [SPEAKER_00]: So, imagine if there is a 10% gain in the experiment, right?
• 11:59
  [SPEAKER_00]: You're paying real money with this 5050 split.
• 12:02
  [SPEAKER_00]: because you're withholding the good treatment that you know is going to lift the damping by 10%, you're withholding that from 50% of your people in perpetuity.
• 12:09
  [SPEAKER_00]: So how can you get away with the split that still gets your suit to a stat's sake?
• 12:13
  [SPEAKER_00]: While also maximizing the ROI from the experiment, maybe it's a 1090 split, maybe it's a 980, 991 split.
• 12:21
  [SPEAKER_00]: But in order to do that forecasting, you can see all the data.
• 12:24
  [SPEAKER_00]: Guess what?
• 12:25
  [SPEAKER_00]: Where the only player that sees all the data, that was one pull.
• 12:32
  [SPEAKER_00]: the marketing conundrum.
• 12:33
  [SPEAKER_00]: It's related but different, right?
• 12:35
  [SPEAKER_00]: So you run an experiment, you observe the winner, right?
• 12:38
  [SPEAKER_00]: One condition is clearly winning on the metric that you care about.
• 12:42
  [SPEAKER_00]: Scale it up, right?
• 12:43
  [SPEAKER_00]: Bam, 100% of the traffic to this thing.
• 12:45
  [SPEAKER_00]: Okay, but now you can measure ROI anymore.
• 12:47
  [SPEAKER_00]: So now you knew boss comes in two years later and says, you know what?
• 12:50
  [SPEAKER_00]: No, you've done this thing two years ago.
• 12:52
  [SPEAKER_00]: You told everybody, I found your PowerPoint slides, right?
• 12:55
  [SPEAKER_00]: Said there was a 20% cause of lift, 10% increase in the LTV.
• 12:59
  [SPEAKER_00]: Very impressive.
• 13:00
  [SPEAKER_00]: How much revenue did we make because of that in the last quarter?
• 13:03
  [SPEAKER_00]: What are you going to say?
• 13:04
  [SPEAKER_00]: You don't know.
• 13:05
  [SPEAKER_00]: Said, oh, we scaled the experiment.
• 13:06
  [SPEAKER_00]: There's nothing else to be done.
• 13:08
  [SPEAKER_00]: What if you could still measure the returns of any experimental design that you ever did in the company, even if they were scaled up to 100%.
• 13:17
  [SPEAKER_00]: That is our concept of always on measurement, and I think it's quite a unique concept.
• 13:22
  [SPEAKER_00]: But you get the idea, so there is a list of sequential components they require some buy-in from the customer, so when you built these components, you better develop
• 13:33
  [SPEAKER_00]: value proposition back to the customer that actually entices them to do what I ask you to do, which is not easy.
• 13:40
  [SPEAKER_00]: Right?
• 13:40
  [SPEAKER_00]: I ask you to do a damnit stop experimentation all the downstream platforms.
• 13:44
  [SPEAKER_00]: Don't do it.
• 13:45
  [SPEAKER_00]: We can offer you an experimentation service which is better.
• 13:47
  [SPEAKER_00]: I better have good arguments for what better is and that's how we think about building the product.
• 13:54
  [SPEAKER_01]: So then, how do you go about figuring out what to do next?
• 13:58
  [SPEAKER_01]: Right, you figured out those two really important components and it makes sense to me how you're explaining that.
• 14:04
  [SPEAKER_01]: How did you, you know, get to wrap it in a box?
• 14:06
  [SPEAKER_01]: So how do you come up with your roadmap?
• 14:08
  [SPEAKER_01]: What sort of criteria do you decide or who do you talk to or how do you go about putting together that?
• 14:13
  [SPEAKER_01]: Okay, this is the next most important thing to build.
• 14:15
  [SPEAKER_01]: And that I assume that it flows really nicely from what you just said.
• 14:20
  [SPEAKER_00]: it's a top-down process.
• 14:21
  [SPEAKER_00]: So we came with a vision first.
• 14:24
  [SPEAKER_00]: In the vision I probably should spend a couple of more sentences describing the vision.
• 14:28
  [SPEAKER_00]: I've talked a lot now about this autonomous system, but basically the vision was that ultimately, we let the agent decide what marketing treatments we will consider in your quest for optimization.
• 14:42
  [SPEAKER_00]: So we ask you what's your goal?
• 14:43
  [SPEAKER_00]: Okay, your goal is maximizing LTV.
• 14:45
  [SPEAKER_00]: Like any other lifecycle marketer, there's only one goal.
• 14:48
  [SPEAKER_00]: I can already tell you them.
• 14:49
  [SPEAKER_00]: Every life cycle marketer, head of CRM, is paid for exactly that, right?
• 14:54
  [SPEAKER_00]: How can you maximize the squeeze once you have a customer in your CRM?
• 14:57
  [SPEAKER_00]: There's nothing else to do.
• 14:58
  [SPEAKER_00]: It's only gold.
• 15:00
  [SPEAKER_00]: Using a genetic AI to define the universe of treatments and then using decisioning to essentially or one-to-one personalization and decisioning to essentially allocate the traffic or allocate each user to the marketing treatment that will maximize his or her returns.
• 15:18
  [SPEAKER_00]: So that was the vision.
• 15:20
  [SPEAKER_00]: Now, how do we validate that vision?
• 15:22
  [SPEAKER_00]: One, it's my vision.
• 15:23
  [SPEAKER_00]: So I really like it.
• 15:24
  [SPEAKER_00]: Second, you have a lot of conversations.
• 15:26
  [SPEAKER_00]: And to make sure that you are aligned as to where that market goes.
• 15:29
  [SPEAKER_00]: And that's really freaking difficult.
• 15:31
  [SPEAKER_00]: I'm not sure if I'm allowed to swear.
• 15:32
  [SPEAKER_00]: So I'll say freaking difficult.
• 15:34
  [SPEAKER_00]: Why is that so difficult?
• 15:36
  [SPEAKER_00]: Because, look, I've been a founder before for the last seven years before doing this, you cannot build the product without the customer.
• 15:43
  [SPEAKER_00]: That's true.
• 15:44
  [SPEAKER_00]: But also, if you just do everything the customer tells you, you're not a product company.
• 15:49
  [SPEAKER_00]: You live under the dictate of the status quo and you can't grow.
• 15:53
  [SPEAKER_00]: That's conundrum of companies, that's a conundrum of product building.
• 15:56
  [SPEAKER_00]: There is a latitude there.
• 15:57
  [SPEAKER_00]: You gotta have some conviction, you gotta have some pull from the market, you gotta bring it to.
• 16:01
  [SPEAKER_00]: We did our best to verify that this vision is the right mix of both of these extremes.
• 16:06
  [SPEAKER_00]: We talked about agente AI building the universe of treatments at your disposal at any given time.
• 16:13
  [SPEAKER_00]: If I do that, I need what we call causality data.
• 16:16
  [SPEAKER_00]: So I need to tell the agent, you know what?
• 16:18
  [SPEAKER_00]: Here is a large history of causal relationships.
• 16:22
  [SPEAKER_00]: Or here is a large history, a large data set of encoded causal statements that you can learn from, that you can learn from how to initialize.
• 16:30
  [SPEAKER_00]: So that comes first, that has to come first, just to sequential order.
• 16:34
  [SPEAKER_00]: And then from there, once you have that, you built this into an agentic context graph.
• 16:39
  [SPEAKER_00]: And that includes a lot of machine learning.
• 16:40
  [SPEAKER_00]: You got to basically imagine that.
• 16:42
  [SPEAKER_00]: You have an audience that is exported to a target destination that gets exposed to
• 16:47
  [SPEAKER_00]: At content, I got to pull that at content back and use the LM of the authentic infrastructure to embed it, to store it, to log it, so that comes next, and then obviously the last part is once I have the LM that initializes or specifies the universe of possible treatment options, then I need a smart decisioning engine that allocates the traffic to each of these options such that returns from maximized.
• 17:11
  [SPEAKER_00]: So basically, once you are an alignment in terms of the high-level vision, then it's easier to do a sequential list of things that naturally have to come after one another and then you have a product roadmap.
• 17:24
  [SPEAKER_00]: I will also say just operationally, what I did is I wrote an AI619180, this is not a surprise, every VP does this, and shape it in sales to partnership, to CTO, to CL, and basically forced
• 17:41
  [SPEAKER_00]: So we have that on the record and that's one thing that I learned, I think it's important to have that buy-in, right?
• 17:46
  [SPEAKER_00]: If you do that vision and sales has very different perspectives, you're in trouble, right?
• 17:52
  [SPEAKER_00]: Because think about it, now that we're building this thing, it requires a large amount of discipline.
• 17:58
  [SPEAKER_00]: Because we're building components one after another, stoically.
• 18:02
  [SPEAKER_00]: If that starts to be outside, what is desired and outside drives revenue, you're in trouble very quickly as a company.
• 18:10
  [SPEAKER_01]: Let's talk about team.
• 18:11
  [SPEAKER_01]: To do something like this, you gotta have the right people.
• 18:14
  [SPEAKER_01]: Obviously you're one of those right people.
• 18:16
  [SPEAKER_01]: How do you grow your team and how do you build your team?
• 18:18
  [SPEAKER_01]: What do you look for in those people to indicate that they're the wedding horses to join you?
• 18:23
  [SPEAKER_00]: The number one thing that I'll say is do not work as recruiters.
• 18:26
  [SPEAKER_00]: That to me, if learned as the hard way, look what we're building is highly specialized.
• 18:31
  [SPEAKER_00]: Recruiters can work for entry-level positions, they can work for commoditized positions, but not for what I'm building.
• 18:37
  [SPEAKER_00]: How do you do it?
• 18:38
  [SPEAKER_00]: There is something that cannot be avoided, which is painful community building.
• 18:42
  [SPEAKER_00]: You know, I looked at, on decisioning science, basically that is a fancy word for reinforcement learning, Bayesian multi-harm bandits, things like that.
• 18:50
  [SPEAKER_00]: Where is the community live?
• 18:52
  [SPEAKER_00]: Lives in Europe.
• 18:53
  [SPEAKER_00]: Those are companies like Expedia, their companies like Zalando, the German multi-billion dollar marketplace.
• 18:59
  [SPEAKER_00]: They've built these systems at scale.
• 19:02
  [SPEAKER_00]: So you start an AI decisioning working group, a reading group, and you back these people to show up.
• 19:07
  [SPEAKER_00]: that have the right background, and at some point they show up and you observe how they behave in these reading groups so they get the cross pressures of revenue and product building, which we have to face as is still a fairly early stage startup.
• 19:18
  [SPEAKER_00]: Do that can they build product under pressure?
• 19:21
  [SPEAKER_00]: Are they pureists that are PhDs that just want to prove their riot or their practical?
• 19:25
  [SPEAKER_00]: And then you go from there.
• 19:26
  [SPEAKER_00]: And then as last, I can give you two other pieces of practical advice that's probably going to get cut.
• 19:31
  [SPEAKER_00]: But I don't hire from business schools, one.
• 19:33
  [SPEAKER_00]: And then two, my preference is to hire a PhDs.
• 19:36
  [SPEAKER_00]: Why?
• 19:36
  [SPEAKER_00]: Because I tackle product problems, the way I use to tackle research problems.
• 19:41
  [SPEAKER_00]: So I'm very familiar with that thinking.
• 19:43
  [SPEAKER_00]: But here's a crux.
• 19:44
  [SPEAKER_00]: There is nothing that replaces two or three years of industry experience.
• 19:47
  [SPEAKER_00]: You don't have that.
• 19:48
  [SPEAKER_00]: I think it's a very risky hire.
• 19:50
  [SPEAKER_00]: I've done these hire as before when I was really cash-constrained.
• 19:53
  [SPEAKER_00]: There's just a much higher risk factor to it, right?
• 19:55
  [SPEAKER_00]: Here I didn't, so that people that we brought have have industry experience because I had the flexibility to do it.
• 20:01
  [SPEAKER_00]: But that gives you a little bit of an idea.
• 20:03
  [SPEAKER_00]: If you build something as specialized, that in my opinion is, I don't want to be presumptuous, but you know what, probably, the only way to build a team
• 20:17
  [SPEAKER_01]: I'm curious about scalability, and this will be interesting, right?
• 20:21
  [SPEAKER_01]: Where scale was considered in the beginning?
• 20:24
  [SPEAKER_01]: Obviously, there's going to be some elements that are part of your bread and butter, but I'm also curious about where you've had to fight scale as you've grown, and that could be people who could be technology, and primarily interested in the technology aspect.
• 20:36
  [SPEAKER_00]: One thing that I've found is very rarely do you tackle problems that are really not scalable from a technology perspective?
• 20:43
  [SPEAKER_00]: I'll give you a concrete example of that, by the way.
• 20:46
  [SPEAKER_00]: My first company, the back end code, was written in a coding language called R, and it used an MCMC's, a Monte Carlo,
• 20:55
  [SPEAKER_00]: processor that was called Stan.
• 20:57
  [SPEAKER_00]: It was a C++ Brepper.
• 20:58
  [SPEAKER_00]: So that thing fantastically didn't scale.
• 21:00
  [SPEAKER_00]: It was meant for a sample size of 100 people.
• 21:04
  [SPEAKER_00]: We tried to apply it to sample size of millions.
• 21:06
  [SPEAKER_00]: And that was a hard constraint.
• 21:08
  [SPEAKER_00]: And what we did back then is we somehow managed to get this thing to the graphical processing unit when these guys came up on AWS and made it scale.
• 21:16
  [SPEAKER_00]: That was probably the wrong way to do that because, born a lot of resources, I was way too stubborn to let go of this beautiful, Bayesian purest Bayesian implementation, and even then there was a way it's things scaled.
• 21:28
  [SPEAKER_00]: So, I think it's very rare that you face hard constraints where technology really doesn't scale.
• 21:33
  [SPEAKER_00]: There's obviously exceptions to the rule, if you're trying to do something in session or your latencies, something like 30 milliseconds, and you build complex APIs, there's scale is important, but usually it can be solved one way or another.
• 21:46
  [SPEAKER_00]: What cannot be solved to easily, scale when it comes to people.
• 21:49
  [SPEAKER_00]: And my conviction is this, I don't want to say what I said elsewhere, customers are smart, and sometimes you go to force customers a little bit to their own luck.
• 21:56
  [SPEAKER_00]: Right?
• 21:57
  [SPEAKER_00]: So when you build something like experimentation, I always will buy this towards defaulting new releases to on and still turn it off.
• 22:06
  [SPEAKER_00]: If you have a validated vision that has a lot of ingredients, I take a more radical approach where I don't do that notion of going to every customer and trying to convince that customer that it's the right thing and then hoping for the opt-in,
• 22:24
  [SPEAKER_00]: Makes it a little bit harder if you really don't want to do experimentation on the platform.
• 22:28
  [SPEAKER_00]: It makes it a little bit harder, but it's not prohibitive.
• 22:30
  [SPEAKER_00]: By what you get, as you get is an important usage.
• 22:32
  [SPEAKER_00]: And it goes towards sometimes showing the customer the way a little bit.
• 22:36
  [SPEAKER_00]: Again, if you do too much of that, you're in trouble.
• 22:39
  [SPEAKER_00]: Right?
• 22:40
  [SPEAKER_00]: That's the beginning of the statement.
• 22:41
  [SPEAKER_00]: You cannot build product without the customer in buy-in.
• 22:44
  [SPEAKER_00]: That doesn't work.
• 22:45
  [SPEAKER_00]: But again, if you flip that statement around completely, it doesn't work either.
• 22:49
  [SPEAKER_00]: So there is some tricks to deal with people management, which I certainly have come to understand to be the much harder element of scale.
• 22:58
  [SPEAKER_01]: I'm curious, Toby, as you step out on the balcony, look across all that you've built thus far at growth, what do you most proud of?
• 23:07
  [SPEAKER_00]: What I'm definitely proud of is taking that ambitious vision and distilling it into a product, roadmap that it's tangible.
• 23:16
  [SPEAKER_00]: It's takeable.
• 23:17
  [SPEAKER_00]: So I think that translation we have done.
• 23:21
  [SPEAKER_00]: But that's one thing that I said early on to is on the product side, what we're building some ambitious, but I think it can be done.
• 23:28
  [SPEAKER_00]: And I certainly think can be done by me, but I also, there's other smart people, and I'm not even sure if I'm smart.
• 23:33
  [SPEAKER_00]: Product marketing is hard, and that relates to change management.
• 23:36
  [SPEAKER_00]: I think we've started to set a good cadence and a good foundation of getting marketers to rethink some of the big topics that we need them to rethink.
• 23:46
  [SPEAKER_00]: The biggest one, and we touch on that, is that correlative decision-making does not get you anywhere.
• 23:51
  [SPEAKER_00]: So basically, if you ask the LM, hey, look at all my data, and Susie is a new customer, just designed the intervention that maximizes her returns, right?
• 24:01
  [SPEAKER_00]: It'll do so based on correlational thinking.
• 24:02
  [SPEAKER_00]: It'll crunch all the data that got you to the status quo, and then it'll apply that to Susie with Adnon effects.
• 24:08
  [SPEAKER_00]: right?
• 24:08
  [SPEAKER_00]: We need marketers to think causally, confectually, right?
• 24:13
  [SPEAKER_00]: To think, okay, I doesn't matter that I have all this correlational data.
• 24:17
  [SPEAKER_00]: Really what I need to know is what is the effect of X on Susie meaning, what is the outcome for her?
• 24:25
  [SPEAKER_00]: with seeing this ad, and what is the outcome for her without seeing this ad?
• 24:30
  [SPEAKER_00]: This is called the fundamental problem of causal inference because it's not observable.
• 24:34
  [SPEAKER_00]: You can't clone Susie and give her both at the same time, but at least there is answers that get you closer to that reality and decisioning as one of them.
• 24:43
  [SPEAKER_00]: If we get marketers to understand that reality,
• 24:45
  [SPEAKER_00]: We're one step further along our continuum, and I think we've done a good job both laying the foundations in terms of content, but also now cadence of how we speak about it in the market and how we're heard by the buyer.
• 25:06
  [SPEAKER_00]: First of all, Noah, I made the planning of mistakes and continue to make planning of mistakes.
• 25:12
  [SPEAKER_00]: And I'm not even sure I used to think that the only thing that you can do is learn from mistakes.
• 25:17
  [SPEAKER_00]: But I think unfortunately our capability to learn as humans is quite low, so I'm not even sure if that's still, that's too ambitious of a statement from me even.
• 25:25
  [SPEAKER_00]: But that's just the intro, I've made plenty of mistakes.
• 25:29
  [SPEAKER_00]: I think what I really learned the hard way
• 25:32
  [SPEAKER_00]: is that using machine learning to build predictive models does not do anything, and does not win anybody over.
• 25:39
  [SPEAKER_00]: It does not change outcomes, and I think ultimately marketers will pick up on that.
• 25:43
  [SPEAKER_00]: So, example being here, my second venture is a company that used machine learning to predict customer lifetime value.
• 25:49
  [SPEAKER_00]: Cool, you had to use much more data, better algorithms, a framework that we called Piantil you die, which is a little bit of an inside joke, as the modeling framework up to this point was called Piantil you die and you brought it by torch.
• 26:02
  [SPEAKER_00]: Anyways, imagine you had a magic wand and you could predict the revenue future of every customer.
• 26:07
  [SPEAKER_00]: Cool, right?
• 26:08
  [SPEAKER_00]: And the company grew because we could predict the accuracy of these predictions or we could show we could prove the accuracy of these predictions.
• 26:14
  [SPEAKER_00]: But what do you do with descriptive ML?
• 26:17
  [SPEAKER_00]: How do you turn that into revenue growth, right?
• 26:20
  [SPEAKER_00]: And it was a very hard question to answer.
• 26:22
  [SPEAKER_00]: This is all about closing the loop, right?
• 26:24
  [SPEAKER_00]: If you know this future, remember, this is the future based on your status quo.
• 26:27
  [SPEAKER_00]: The whole idea of marketing is you want to do something different for customers.
• 26:32
  [SPEAKER_00]: You want to change something, otherwise you don't need to be there.
• 26:34
  [SPEAKER_00]: And it turned out that sort of translating these predictive scores or predictive models into this outcomes-based thinking was really hard.
• 26:43
  [SPEAKER_00]: I lost a company over that because I saw that too late.
• 26:46
  [SPEAKER_00]: We crafted, at all, by the way, we crafted by piping these predictions back into the add options to add bidding systems on Google Ads.
• 26:55
  [SPEAKER_00]: You can see that was the only way to turn this descriptive machine learning model into an end-to-end pipeline that generated measurable returns.
• 27:03
  [SPEAKER_00]: But so, one thing I've learned the hard way is don't put too much stock into accuracy of machine learning.
• 27:09
  [SPEAKER_00]: It's the wrong question to ask.
• 27:11
  [SPEAKER_00]: The right question to ask is, how much business impact does technology or AI generate and how easily is it provable and how early is it provable?
• 27:24
  [SPEAKER_01]: You've touched on this a bit, but I kind of want to give space for a full, full answer and some things that you see in where you're going as growth loop, but also where the industry is going and all the things I'm curious about the future.
• 27:36
  [SPEAKER_01]: What does the future look like for growth loop for what you're building, for your team, for the industry, all the things?
• 27:43
  [SPEAKER_00]: Future for the industry certainly will look like machines making more and more decisions.
• 27:48
  [SPEAKER_00]: And by the way, I think that's a good thing.
• 27:50
  [SPEAKER_00]: Machines making more and more decisions of traditional marketing workflows are the traditional job to be done at marketers right now all, and that's by the way, that's not unique to marketing, you'll see the same in diagnosis, you see the same in sentencing, or my favorite examples, and you can see that it's not a bad thing, sentencing, but if you think about human decisioning in sentencing, sentencing is arbitrary, you can't audit the human, there is a lot of bias in there,
• 28:19
  [SPEAKER_00]: There is a lot of noise in there, hiring, think about hiring.
• 28:22
  [SPEAKER_00]: You had a bad breakfast, you don't hire the guy, but the guy was fantastic.
• 28:25
  [SPEAKER_00]: Not even your biased against the person that just noise, which is probably the bigger source of human error, actually.
• 28:30
  [SPEAKER_00]: In general, I think that's a very good thing.
• 28:33
  [SPEAKER_00]: I think the big question is how do machines take over decisioning?
• 28:37
  [SPEAKER_00]: And what we certainly believe will be here for the foreseeable future is a hybrid system.
• 28:43
  [SPEAKER_00]: where you have manual elements, or you have human in the loops, where they say the machine comes up with 10 different variations of content.
• 28:52
  [SPEAKER_00]: You, as a human in the loop, validate that content you iterate over that content.
• 28:57
  [SPEAKER_00]: Then the machine makes the decisioning and basically decides for any individual what content gets served up when he or she comes to that point in the journey.
• 29:06
  [SPEAKER_00]: Yeah, you let the machine do that decision, but the human can audit that decision alongside quantitative parameters.
• 29:14
  [SPEAKER_00]: And we can talk exactly about what that is, visualize that decision.
• 29:18
  [SPEAKER_00]: Okay, that I think is the future for the foreseeable, that is the reality for the foreseeable future.
• 29:23
  [SPEAKER_00]: This is what we're going, so it doesn't replace the human.
• 29:27
  [SPEAKER_00]: It puts more on its decisioning side to the AI, but at least the way we build it.
• 29:34
  [SPEAKER_00]: It ensures that there is a human in the world.
• 29:36
  [SPEAKER_00]: Now, goal three years forward.
• 29:38
  [SPEAKER_00]: The question, the big question to me that I've been thinking about a lot, is there any human in the loop anymore when it comes to decisioning?
• 29:45
  [SPEAKER_00]: Now, I'll give you a very specific example.
• 29:47
  [SPEAKER_00]: Right now, a genetic AI is, and we talked about that, but it's terrible at initializing, meaning, but you have this whole thing, and you say, oh, I want to maximize LTV.
• 29:57
  [SPEAKER_00]: Now, build me 10 different uses of content that are closed to the maximum, and I can do decisioning over.
• 30:03
  [SPEAKER_00]: L and suck at that, right?
• 30:05
  [SPEAKER_00]: A Gentic AI cannot do that because it doesn't have that data.
• 30:08
  [SPEAKER_00]: We are collecting that data as we can train specific a Gentic AI on the back of this a Gentic context graph that can do that better.
• 30:16
  [SPEAKER_00]: But now, three years later, maybe we have the emergence of a Gentic commerce, and I don't think that's going to come by the way quite to the extent, but let me define a Gentic commerce.
• 30:24
  [SPEAKER_00]: The Gentic commerce, essentially, agent, and
• 30:27
  [SPEAKER_00]: or agentic AI being present in the whole checkout process, right?
• 30:31
  [SPEAKER_00]: So it's U.S. a human or maybe U.S. as another agent that is zero to zero commerce, talking to an agent and essentially the agent understanding and dictating and governing the entire checkout process.
• 30:43
  [SPEAKER_00]: At that point, you could argue that the agent collect so much data on causal relationships that right now it does not have, that it can take over the whole decision-ning process.
• 30:54
  [SPEAKER_00]: I still am skeptical whether that's going to come for two reasons.
• 30:58
  [SPEAKER_00]: One reason is think about this quantitative auditability, or if it wouldn't phrase a coin, I need to write about this, but something like radical quantitative auditability.
• 31:08
  [SPEAKER_00]: That does not work with LNs.
• 31:09
  [SPEAKER_00]: You can ask LNs, hey, why did you make the decision?
• 31:12
  [SPEAKER_00]: Why did you serve this ad to Susie?
• 31:15
  [SPEAKER_00]: And the agent can give you a textual response, but it cannot really give you quantitative response, not at least in the way that these traditional systems can.
• 31:24
  [SPEAKER_00]: And then the other thing I would say is replicability.
• 31:26
  [SPEAKER_00]: These systems are not replicable.
• 31:28
  [SPEAKER_00]: Why?
• 31:28
  [SPEAKER_00]: Because first of all, models change all the time.
• 31:31
  [SPEAKER_00]: Frontational models evolve all the time.
• 31:34
  [SPEAKER_00]: But the data on which shared train changes all the time, too.
• 31:37
  [SPEAKER_00]: So basically, if Susie comes back 10 minutes later, the agent makes a different decision as to what content it would have served her.
• 31:44
  [SPEAKER_00]: And I don't think that's what marketing wants, I think marketing desires more transparency.
• 31:49
  [SPEAKER_00]: And just to be sure where we are going is this first world, right?
• 31:52
  [SPEAKER_00]: A lot of the decisioning turned over to the AI, but the human clearly in the loop and the systems being transparent and the systems being auditable alongside cleanly defined numerical parameters.
• 32:03
  [SPEAKER_00]: Now, I think that is a question by the way for all fields, will there ever be a future where there is no human in the little described in the letter statement?
• 32:12
  [SPEAKER_00]: I think for most fields that will be a disaster, there is exceptions, text a code, I don't think anybody complains if the agents become very autonomous.
• 32:21
  [SPEAKER_00]: But I think for most fields including marketing, including sentencing, including diagnostics and the medical space, what faulted disaster, if there was such a version of autonomous identical eye, without human in the loop, and it's not what we're building towards.
• 32:36
  [SPEAKER_01]: Okay, Toby, let's switch to you, individually, who influences the way that you were?
• 32:41
  [SPEAKER_01]: Name a person or many persons or something you look up to and why.
• 32:46
  [SPEAKER_00]: On name two people, my kids always, and my daughters, my girls always influence the way I work, mostly in the sense that I work, so I can spend most of my time with them.
• 32:56
  [SPEAKER_00]: But that aside, they're certainly, I think, big influences in the way that I think about limits of human decisioning, but also limits of machine decisioning.
• 33:04
  [SPEAKER_00]: One is the Rubin causal model to praise kind by Donald Rubin, who is an economist slash statistician, but basically it's this conundrum, this fundamental problem of causal inference, right?
• 33:16
  [SPEAKER_00]: Marketing as a causal language, most other fields are causal language, think about diagnosis.
• 33:22
  [SPEAKER_00]: If you do cancer diagnosis really well, that's great, but nobody cares because you care about the treatment.
• 33:27
  [SPEAKER_00]: If you do out TV prediction really well, that's great, but nobody cares because you care about maximizing this thing, not static predictions.
• 33:34
  [SPEAKER_00]: So, all of these fields are essentially dependent on causal thinking, and the way that Ruben defying causality is very simple.
• 33:55
  [SPEAKER_00]: But that is the way that we have to think if we want to make progress in these fields.
• 33:59
  [SPEAKER_00]: So that's one, and then the other person that named Donny Kahneman and all the school of behavioral economists who put a big question mark on this ridiculous assumption of early economic theory, which was the rational voter, the rational individual humans, as entities that make rational decisions.
• 34:19
  [SPEAKER_00]: What kind of an, I think, very successfully showed, and he's a noble laureate, of course, in the first, I think, basically the godfather of behavioral economics, was that human decisioning is most of the times awful.
• 34:32
  [SPEAKER_00]: And human decisioning is awful, not because necessarily there is bias, even though there is.
• 34:36
  [SPEAKER_00]: But I think his big contribution is so show that bias is just one factor in the weakness of human decisioning.
• 34:42
  [SPEAKER_00]: The other one is noise, and we talked about this too, when you have had a bad breakfast, you may make a bad hiring decision.
• 34:50
  [SPEAKER_00]: If I'm in a bad mood, I don't play as nicely with my girls, makes no sense, it's not biased, it's noise, it's just circumstantial, it's idiosyncratic.
• 34:58
  [SPEAKER_00]: So, where does that leave me?
• 35:01
  [SPEAKER_00]: It leaves me to say that I do want the future of machine decisioning.
• 35:05
  [SPEAKER_00]: Certainly in marketing, certainly in sentencing, certainly in diagnosis and treatment, but the big question is, how does the human in the loop get positioned?
• 35:14
  [SPEAKER_00]: And how do we avoid a world in which we turn the entire decisioning process over to a black box that we call a genetic AI?
• 35:22
  [SPEAKER_00]: And I think that future will be a disaster.
• 35:25
  [SPEAKER_01]: Toby last question, so you're getting on a plane, you're sitting next to a young entrepreneur who's built the next big thing.
• 35:31
  [SPEAKER_01]: They're jazzed about it, they can't wait to show it off to the world and can't wait to show it off to you right down the plane.
• 35:35
  [SPEAKER_01]: What advice do you give that person having gone down this road a bit?
• 35:39
  [SPEAKER_00]: think more, do less, I think we're in this world of entrepreneurship where we have this bug in our ear that thanks to cursor and the emergence of a genetic eye for coding, we can build prototypes now more quickly than ever, and we can build product quicker than ever, but I think what gets wept aside a little bit is critical thinking, which I do look, I sound pessimistic, but at the end of the day I think I'm an optimist when it comes to the eternal questions and the big questions, I do think that if you want to build a successful product,
• 36:09
  [SPEAKER_00]: and hopefully have some aspiration to make the world a better place and not just make money.
• 36:13
  [SPEAKER_00]: You have to think about where you're going and I think the difficulty and the problem of the fast emergence of these tools is that it's very easy to sweep aside these elements of critical thinking, sweep aside these larger ambitions and just build something for the purpose of building something.
• 36:30
  [SPEAKER_00]: So my advice would be slow down, take a breather, think, and then you can get back to building.
• 36:35
  [SPEAKER_01]: I think that's absolutely fantastic advice, Toby, but I appreciate you being on the show today and telling the creation story of Griffloot.
• 36:43
  [SPEAKER_01]: Thank you for having me.
• 36:45
  [SPEAKER_01]: And this concludes another chapter of Coat Story.
• 36:55
  [SPEAKER_01]: code story is hosted and produced by no a lab part.
• 36:58
  [SPEAKER_01]: Be sure to subscribe on Apple podcast, Spotify, or the podcasting app at your choice.
• 37:04
  [SPEAKER_01]: And when you get a chance, leave us a review, both things help us out tremendously.
• 37:11
  [SPEAKER_01]: And thanks again for listening.