Latest / Responsibly Different™ / What the Data Misses About Being Human with Dr. Ram Singh

Transcript

What the Data Misses About Being Human with Dr. Ram Singh · Responsibly Different™

• 0:06
  SPEAKER_01: Welcome to Responsibly Human, a responsibly different podcast from Campfire Consulting.
• 0:11
  SPEAKER_01: We live in a world getting remarkably good at finding patterns in human behavior, but understanding people is something different.
• 0:19
  SPEAKER_01: On this show, we talk with researchers, technologists, marketers, creators, and leaders about what actually moves us, earns our trust, and makes us feel known instead of simply seen.
• 0:30
  SPEAKER_01: And as more of our world becomes automated, one question keeps getting more important.
• 0:35
  SPEAKER_01: What does technology still miss?
• 0:38
  SPEAKER_01: Before we get into it, one small way you can support more thoughtful work is by hitting subscribe.
• 0:43
  SPEAKER_01: That simple act helps this show reach more people who are trying to build businesses, communities, and a world that works a little better for all of us.
• 0:52
  SPEAKER_01: If something you hear in these conversations helps someone else think differently or do their work differently, that's the whole point.
• 1:00
  SPEAKER_01: I'm Chris Marine, founder and CEO of Campfire Consulting, and this is responsible to you.
• 1:49
  SPEAKER_01: ROAS is up, conversions are down.
• 1:52
  SPEAKER_01: The creative perform better than that one.
• 1:54
  SPEAKER_01: We put all that into dashboards and models, and eventually, those outputs start to feel a lot like truth.
• 2:01
  SPEAKER_01: But what if some of the most important signals are the ones we've been calling noise?
• 2:05
  SPEAKER_01: That's where my conversation with Rom Sing went.
• 2:08
  SPEAKER_01: Rom had spent his career building measurement systems and challenging the assumptions underneath them.
• 2:12
  SPEAKER_01: And what I found most interesting wasn't a new model or another way to optimize media.
• 2:17
  SPEAKER_01: It was his argument that better technology should actually make us more comfortable with uncertainty.
• 2:24
  SPEAKER_01: Not less.
• 2:25
  SPEAKER_01: Because people don't always behave the way the data says they should.
• 2:29
  SPEAKER_01: Sometimes the campaign that looked successful was going to get the sale anyway.
• 2:33
  SPEAKER_01: Sometimes the message that looks like it failed simply hasn't had enough time to become familiar.
• 2:38
  SPEAKER_01: And sometimes the most important thing a model can do is tell us something we didn't want it to tell us.
• 2:45
  SPEAKER_01: So we talked about what marketers are measuring because it's useful, what we're measuring simply because it's convenient, and why the ability to challenge the answer may be one of the most important things we still bring to the table.
• 2:59
  SPEAKER_01: Here's my conversation with Rom Singh.
• 3:06
  SPEAKER_01: So, Rom, what pulled you toward computer science, analytics, and eventually marketing to kind of just get everyone familiar with your background and how when we were talking a few minutes ago before hitting record, uh that idea that this was kind of a sounded like an a place you didn't know you were gonna end up in.
• 3:27
  SPEAKER_00: Yes.
• 3:28
  SPEAKER_00: Well, first off, thank you, Chris, for having me on the show.
• 3:31
  SPEAKER_00: I appreciate this.
• 3:32
  SPEAKER_00: Um, just like we were discussing, my journey has been more defined by the need and then jumping in to solve a problem.
• 3:41
  SPEAKER_00: I I started uh with an undergrad in computer science, I started at IEBM as a software engineer.
• 3:48
  SPEAKER_00: But very quickly, when I was in the field talking to clients, talking to partners, I started to enjoy learning about what their problems were and how to solve them.
• 4:00
  SPEAKER_00: And that led me to a lot of management consulting realm, and I started using data a lot.
• 4:07
  SPEAKER_00: I was a computer scientist by trade, not a data scientist.
• 4:10
  SPEAKER_00: In fact, the word data science didn't exist then.
• 4:12
  SPEAKER_00: And then in order to solve those questions, I often had to look at volumes and volumes of data and try to decipher what I'm going to be building.
• 4:20
  SPEAKER_00: That is what started to lead me into data science, and that to specifically marketing decision sciences, eventually followed the road of becoming a marketer, became a CMO at a very large company, started to run marketing, and then standing up in front of a board of directors and trying to answer what in marketing is working, why something happens that you can't explain in data, that kept refining my purview.
• 4:47
  SPEAKER_00: And eventually, then I spent the last decade and a half purely focused on advanced measurement.
• 4:53
  SPEAKER_00: So it was a journey.
• 4:54
  SPEAKER_00: Uh, it found me.
• 4:56
  SPEAKER_00: I didn't uh I don't know if I will I will say I started uh by that thought in mind, but I certainly enjoyed uh doing that.
• 5:05
  SPEAKER_01: It it found me.
• 5:07
  SPEAKER_01: Those are the best kind of journeys, though.
• 5:08
  SPEAKER_01: I mean, I think that's what makes like a well-rounded perspective, and that perspective is something that makes us all human.
• 5:14
  SPEAKER_01: So was there a moment though, like through that journey that you ever think back and point to where it's like there was that moment of understanding there is a number or a story that people were trusting where you're like, that just doesn't seem right to me?
• 5:30
  SPEAKER_01: Correct.
• 5:30
  SPEAKER_00: Correct.
• 5:31
  SPEAKER_00: Many, many, Chris, along the way.
• 5:34
  SPEAKER_00: Look, at the end of the day, one of the things you'll probably hear this from a lot of data professionals, and it is one of the things that I've sent a significant amount of time.
• 5:43
  SPEAKER_00: When we use data to find a pattern or prove something that you will do, you will invariably find that data.
• 5:50
  SPEAKER_00: When I first, my first foray into the marketing world where I was running marketing and I was answerable to things, I would often look at reports that would say, okay, this is an opportunity.
• 6:02
  SPEAKER_00: I whether it's a campaign, whether it's an in-market spend, what have you, right off the bat, there would be so many instances that you make a decision based on a metric that is available, you direct spend, but then it doesn't work.
• 6:15
  SPEAKER_00: So uh it doesn't produce the results that you were expecting.
• 6:17
  SPEAKER_00: My point in sharing that is there were so many instances from early on where I saw you can apply this to bids.
• 6:25
  SPEAKER_00: One of the things I did early on is build a bid engine based on a portfolio model because oftentimes when you were bidding on certain digital campaigns, whether it was ad placements, ranks, etc., you would find you do it with one purpose looking at data, but the outcome is different.
• 6:42
  SPEAKER_00: So my point in sharing all of this is there were many instances where I saw what was reported out or what was in front of my my eyes didn't produce the results.
• 6:53
  SPEAKER_00: So that is when I went on a quest of like, is there a better way to make decisions?
• 6:58
  SPEAKER_00: Is there a better way to build a portfolio model instead of just looking at a report and doing a highest bid win kind of thing?
• 7:06
  SPEAKER_00: Let me pause for a second and try to give you some examples of these are the things which really challenged my thought process, saying, am I looking at it correctly or is there an alternate way of looking at the problem?
• 7:18
  SPEAKER_01: I mean, and that's always a challenging thing going from like what you know to then building something different.
• 7:23
  SPEAKER_01: Is there in your journey?
• 7:27
  SPEAKER_01: I know in my personal journey, like there's all sorts of things that I had to unlearn going from like the working within media companies and like the ad sales side to the buying side.
• 7:37
  SPEAKER_01: Are there things in your journey that you had to unlearn to get you to the point where you could build something different?
• 7:45
  SPEAKER_01: Uh I I I yes, and I think the answer is in your question.
• 7:50
  SPEAKER_00: Um, you know, in my case, I've I've been a I've been a builder for a long time, uh very long time.
• 7:56
  SPEAKER_00: Uh almost the entirety of my work has been building out solutions, right?
• 8:00
  SPEAKER_00: One thing I learned early on is in order to improve your solution, uh, and this is just my point of view.
• 8:06
  SPEAKER_00: So I just want to make sure that I'm the best point of view.
• 8:10
  SPEAKER_00: Yeah, I I there could be differences of opinion.
• 8:13
  SPEAKER_00: In my case, the learning was if any solution that I build is not my life's work.
• 8:19
  SPEAKER_00: Because if I attach myself so hard to it, I start to see what I could do to improve it.
• 8:25
  SPEAKER_00: And I have built a lot of blockbuster solutions.
• 8:29
  SPEAKER_00: And as builders, we have a natural affinity to stand very, very, very close to what we build because we're passionate about this, right?
• 8:37
  SPEAKER_00: I build a new model.
• 8:38
  SPEAKER_00: I just told you I build a portfolio model.
• 8:40
  SPEAKER_00: At that point, second price auction was a new concept.
• 8:44
  SPEAKER_00: So, me, even building a bid engine using portfolio theory from finance was a new thing.
• 8:50
  SPEAKER_00: I was so proud of it.
• 8:51
  SPEAKER_00: But in doing so, you attach yourself to that position, even if you don't know it, your ego gets attached to what you build.
• 8:59
  SPEAKER_00: And one of the things I had to unlearn over the years is if I have to go back and make changes and improve my product, I have to be very conscious about the distance between my product and my ego.
• 9:10
  SPEAKER_00: So one of the things I consciously try to do is if I build something, I'm also the one occasionally challenging my own thinking.
• 9:18
  SPEAKER_00: That doesn't come naturally or that didn't come naturally to me.
• 9:20
  SPEAKER_00: I had to learn it because if I don't do that, I'll be stuck with this one thing that I built while the rest of the world moves on, and then I'm playing catch up.
• 9:29
  SPEAKER_01: That's interesting.
• 9:30
  SPEAKER_01: I love that framework and thing.
• 9:32
  SPEAKER_01: That's just uh self-awareness, I feel like is everything, and that's a huge indicator of it.
• 9:38
  SPEAKER_01: Are there people in your circle though?
• 9:40
  SPEAKER_01: Because I know when trying to like question yourself, it can be challenging.
• 9:44
  SPEAKER_01: Are there, do you have do you build like a close trust circle of humans where they are you go to and you're like, I'm I'm coming to you because I want you to push back?
• 9:52
  SPEAKER_01: Yes.
• 9:53
  SPEAKER_01: Or how do you kind of frame that up?
• 9:56
  SPEAKER_01: Or is it much more expansive than a small circle of people?
• 9:58
  SPEAKER_00: No, you're your your very astute, your question hit the nail on the head.
• 10:03
  SPEAKER_00: When I first came to the agency side 11 years ago, I started building sandboxes, whether it was for Publicis or any company that I work for.
• 10:12
  SPEAKER_00: In fact, my current company, Etacore, is founded off of a work that we were doing in a sandbox for multiple years.
• 10:20
  SPEAKER_00: The idea of a sandbox is to get thinkers in a room and have a productive point-counter point kind of discussion.
• 10:30
  SPEAKER_00: When we are in the world of measurement and analytics, things can get very passionate.
• 10:36
  SPEAKER_00: You are building a new algorithm, you're building a new solution, and oftentimes it is more usual than an exception, more of a known than an exception that you'll find people put something out and not vigorously stand behind that.
• 10:49
  SPEAKER_00: So I have been using sandboxes for the last 11 years.
• 10:53
  SPEAKER_00: And the purpose of a sandbox is like an incubator.
• 10:56
  SPEAKER_00: That is where you work on a solution and you give others the opportunity to challenge, or you can challenge your own thing.
• 11:02
  SPEAKER_00: One of the things I've always talked to my team members building product is identify the assumptions.
• 11:08
  SPEAKER_00: Once you take the effort of listing the assumption, you're telling your brain what are the things I may have missed to look in building the product.
• 11:17
  SPEAKER_00: So you're not only becoming a little self-aware, list that out when you go into a sandbox, a lot of then discussion will, you'll be surprised how much discuss the discussion can go in terms of improving the outcome.
• 11:28
  SPEAKER_00: You sowed a seed, but the nurturing can be done by multiple different people.
• 11:33
  SPEAKER_00: And that has always worked for me, Chris, for 11 years, because in product building, you need to be aware of what are your oversights.
• 11:40
  SPEAKER_00: Einstein said you cannot solve a problem by looking at it from the same angle when you discovered the problem.
• 11:47
  SPEAKER_00: Otherwise, you're going to keep seeing the problem.
• 11:49
  SPEAKER_00: You have to change your point of view.
• 11:51
  SPEAKER_00: The best way to change your point of view is do a little self-pretty and do a point-counter point discussion with other experts who can point out potential issues that you may have missed to look.
• 12:05
  SPEAKER_01: And you've expressed that distinction before between systems that confirm what happened and learning models that tell us what should happen next.
• 12:12
  SPEAKER_01: I mean, to everything you just said, you I think you partially answered it, but but why why is that distinction so hard for organizations to accept and actually move forward through?
• 12:24
  SPEAKER_01: Is it that attachment to ego or is it something?
• 12:27
  SPEAKER_00: I I think there is a mindset in play here.
• 12:30
  SPEAKER_00: I think when you look at data or a measurement solution to prove or justify what was done, the I'm not a neuroscientist.
• 12:42
  SPEAKER_00: I'm just putting it in the ways I understood it myself.
• 12:45
  SPEAKER_00: When you use data analytics to prove what you are doing, you're basically sending a signal to your brain saying, I did something right and I need to find evidence on why that was right.
• 12:58
  SPEAKER_00: If you take the other approach, you've just built something, and you're like, I might be surprised by the finding.
• 13:05
  SPEAKER_00: I build a data and analytic solution, I build an NL model, I build an AI native solution that might present to me an outcome I was not expecting.
• 13:14
  SPEAKER_00: It asks me to cut a certain campaign because it's detecting cannibalization that otherwise would not be detected in your standard way of just looking at tech logs.
• 13:23
  SPEAKER_00: It says, reduce the spend on this, but you're like, no, that's my hero campaign, that's my main brand campaign, or that's my main lead gen campaign.
• 13:31
  SPEAKER_00: I need to keep it ever green.
• 13:33
  SPEAKER_00: My point in giving that example is you were presented an outcome that you were not expecting.
• 13:38
  SPEAKER_00: That's a different signal to the brain.
• 13:40
  SPEAKER_00: You're essentially going back and saying, okay, what could I have missed?
• 13:44
  SPEAKER_00: So when you find data to prove what you're doing, versus when you try to look at outputs that present to you something that you were not expecting, those are two different patterns of working.
• 13:55
  SPEAKER_00: That's not just a difference in observation.
• 13:57
  SPEAKER_00: It's not report is wrong or outcome is wrong or they are not aligning.
• 14:03
  SPEAKER_00: You're sending a different signal to yourself, to the team members.
• 14:06
  SPEAKER_00: In one case, you're gonna go put a lot of effort in trying to justify what was done.
• 14:11
  SPEAKER_00: In the other one, you're going to be more forward-looking and saying, maybe there's a different way of addressing the problem.
• 14:17
  SPEAKER_00: So I that's why I feel those things are not only important from a measurement perspective, but they're also uh fundamental to defining how measurement works for you, your team members, your clients, your audiences, etc.
• 14:32
  SPEAKER_00: Distinctly different patterns of working.
• 14:35
  SPEAKER_01: Which goes to one of your core arguments, which is around that there are a lot of organizations still running AI area analytics through operating models built for retrospective BI.
• 14:44
  SPEAKER_01: Correct.
• 14:45
  SPEAKER_00: Correct.
• 14:45
  SPEAKER_00: Correct.
• 14:46
  SPEAKER_00: Correct.
• 14:47
  SPEAKER_01: Is there one metric that you find uh that marketers lean on uh most when it's not entirely useful?
• 14:56
  SPEAKER_01: Especially if they're using the models in the wrong in the wrong uh lens?
• 15:03
  SPEAKER_00: I'm sure you've heard this from others as well.
• 15:05
  SPEAKER_00: ROAS tends to be used as a as a gold standard to performance.
• 15:11
  SPEAKER_00: Um it used to be ROAS, then we pivoted to incrementality-oriented ROAS, but the whole idea of using a metric, and you can you can take your pick.
• 15:22
  SPEAKER_00: There are others like that too, right?
• 15:24
  SPEAKER_00: Conversions could be one of those.
• 15:27
  SPEAKER_00: Right.
• 15:28
  SPEAKER_00: Because the thing is, the implied assumption there is if you think, if you break down return on ad spend and how it not only how is it calculated, but how people approach it, it's essentially trying to say, I want to quantify a benefit per unit spend.
• 15:44
  SPEAKER_00: That is what it is.
• 15:45
  SPEAKER_00: What is the benefit?
• 15:46
  SPEAKER_00: That benefit could be revenue, that could be latent revenue.
• 15:48
  SPEAKER_00: You we're ex- The good news is we're expanding our consideration.
• 15:52
  SPEAKER_00: But in that metric is included a basic premise, a quantification of output.
• 15:59
  SPEAKER_00: What does that mean?
• 16:00
  SPEAKER_00: Spend is not debatable.
• 16:02
  SPEAKER_00: That that you can you you you know how much money you spend.
• 16:05
  SPEAKER_00: So there's no there's no uh gray area in there.
• 16:08
  SPEAKER_00: But the quantification of benefit itself is a very undefined space.
• 16:13
  SPEAKER_00: You can quantify the benefit as uh revenue attained in the next 30 days, you can uh quantify revenue, uh uh benefit as revenue now plus future revenue, you can uh keep thinking of engagement metrics, quality metric.
• 16:29
  SPEAKER_00: My point here is that quantification becomes subjective.
• 16:33
  SPEAKER_00: And if you don't have a good measurement basis to define that, your ROAS number will be misleading.
• 16:39
  SPEAKER_00: Um I I I'll I'll pause for a second to see if you have uh any question uh before I build up on it.
• 16:45
  SPEAKER_01: I mean, I think it uh I'm and maybe you're going here with this, but I mean, we see that quite a bit.
• 16:49
  SPEAKER_01: Media sometimes gets credit it deserves and sometimes a credit that it doesn't deserve.
• 16:54
  SPEAKER_01: So if ROAS looks great, but the casual model says the spend got credit for demand, that would have happened anyway, which we have seen where literally media hasn't even been running, but they're still trying to attach a ROAS.
• 17:06
  SPEAKER_01: We're like, media wasn't even active when that was happening.
• 17:08
  SPEAKER_01: Correct.
• 17:08
  SPEAKER_01: So I mean, how should marketers respond in those moments?
• 17:12
  SPEAKER_01: Um, if if that one metric ROAS, which is kind of this now umbrella term to all how your brand moves through the world, how how do you respond to that, to those metrics?
• 17:23
  SPEAKER_00: See, one of the things I'm trying to build, Chris, is a balance between what gets immediately measured and what are some of the latent benefits of measuring.
• 17:33
  SPEAKER_00: So one of the examples I'll use is uh and I'll bring this back to marketing in a second.
• 17:38
  SPEAKER_00: Think, for example, how stock valuation metrics work, right?
• 17:42
  SPEAKER_00: If you're buying a stock and you're looking at a PE multiple, a PE multiple of 30 for a tech start might be considered okay or low.
• 17:52
  SPEAKER_00: A PE multiple of 120 for a start like Palantir is still like, okay, still buy, even though it is super expensive.
• 17:59
  SPEAKER_00: If you think of ROAS in that kind of a consideration, until unless you have some form of a valuation metric in marketing, it's very difficult to assign interpretation to a value of ROAS.
• 18:12
  SPEAKER_00: So where I'm going with this thing is any of these, that was my premise.
• 18:17
  SPEAKER_00: The computation of a return or a return on benefit is reliant on what are all the potential ways in which marketing could spend could help.
• 18:26
  SPEAKER_00: It can increase revenue, it can increase conversion, those are the direct measures.
• 18:30
  SPEAKER_00: But are we measuring any latent lift in brand value?
• 18:35
  SPEAKER_00: Are we measuring any latent lift in uh you are in a very competitive segment and it will require a minimum set of exposures before any conquesting can happen?
• 18:45
  SPEAKER_00: It will require a minimum amount of investment before your saturation in a given market moves.
• 18:51
  SPEAKER_00: Are we measuring or have a proxy for any of those benefits that reflect in ROAS calculation?
• 18:58
  SPEAKER_00: Because that contextualizes ROAS.
• 19:00
  SPEAKER_00: And without that, you may find yourself in a trap.
• 19:05
  SPEAKER_00: Oh, my number looks weak.
• 19:07
  SPEAKER_00: My number looks great, either pull back or put more money into it, only to then realize, oh, money was already spent or wasted, and you didn't see the return.
• 19:17
  SPEAKER_01: And I think that's important too, like in regards to what's impacting that.
• 19:23
  SPEAKER_01: And one of the things that I've I've read from some of your posts is arguing that creative itself may be one of the marketing's biggest unexplained variables.
• 19:32
  SPEAKER_01: So I mean, and I mean, we feel this we solely focus on the paid media strategy and the planning side of everything, which influences brand.
• 19:42
  SPEAKER_01: But when we're given creative, there's still there are some reports that you can do in regards to creative, but it seems like the industry for so long has been so fixated on ad placement and what that unit return is versus trying to understand how that message is actually trying to make not even trying, making people feel.
• 20:04
  SPEAKER_01: Um why do you think for someone that's been in this, uh, why do you think uh got so good on veered one way and neglecting that other really important pillar of advertising and marketing?
• 20:16
  SPEAKER_00: I I I I I I might underwhelm you with this answer.
• 20:21
  SPEAKER_00: I think it's measuring what is convenient versus measuring what should be measured.
• 20:28
  SPEAKER_00: And the reason I bring that up is oftentimes we grew up looking at at tech lots.
• 20:34
  SPEAKER_00: We grew up at analyzing clicks, costs, saturation, frequency, reach, conversion, etc.
• 20:40
  SPEAKER_00: Nothing wrong with that.
• 20:42
  SPEAKER_00: Look, I think direct values and building analytics around them is absolutely the basic thing that we can do.
• 20:48
  SPEAKER_00: We, for example, now also look at interpretation variables between combination of factors, derived variables.
• 20:54
  SPEAKER_00: But before I go out, go off too far of the deep end, my key premise here is metrics that you find in any ad tech log or that can be captured, they are convenience metrics.
• 21:06
  SPEAKER_00: They are good metrics, but they are convenience metrics.
• 21:08
  SPEAKER_00: What we're trying about measuring is harder to measure because there is no ad tech log that says when I expose somebody to a message eight times and four of those happen to be about a feature or a benefit, and then follow that uh four exposures about a feature or benefit, then expose them to a brand message so that the brand becomes the answer to the problem.
• 21:33
  SPEAKER_00: There is no metric that will define that this needs to be measured for you to then go back and correlate to say, okay, when you do it in this fashion, in this creative, in this narrative order, using these tenth pole classification, that increases my foot traffic.
• 21:50
  SPEAKER_00: That increases the bike.
• 21:52
  SPEAKER_00: It's hard to measure.
• 21:53
  SPEAKER_00: One of the things we have been very passionate about is weak signal detection.
• 21:57
  SPEAKER_00: The whole idea of weak signal detection is.
• 22:00
  SPEAKER_00: What we have thrown out as noise is was because it was hard to measure.
• 22:07
  SPEAKER_00: Anything that you couldn't explain immediately, the outside temperature is low, so my thermostat must be set to a higher temperature.
• 22:15
  SPEAKER_00: Anything outside of that is noise.
• 22:18
  SPEAKER_00: I am challenging that basic premise because the thing that we are throwing out, calling noise, might be the most important factor for optimizing, she uh optimizing that is left in our hands.
• 22:29
  SPEAKER_00: So to answer your question, that is one of the reasons why you haven't necessarily seen these areas picked up as more, because they tend to be a lot more difficult.
• 22:35
  SPEAKER_00: They're not based off of convenience KPIs.
• 22:38
  SPEAKER_00: They actually do require dis different observations, different derivations, different interpretations.
• 22:44
  SPEAKER_00: It requires a lot of work to be able to say what I'm discarding as noise.
• 22:47
  SPEAKER_00: There are nuggets in that noise that I can correlate better to outcome and then push on those things.
• 22:53
  SPEAKER_00: So it is a little bit harder to measure.
• 22:55
  SPEAKER_01: Are there some of those nuggets that you that that you look for or one that stands out to you as being most important?
• 23:03
  SPEAKER_01: I know it's a collective set of signals that you're looking at across to get enough nuggets that make a meal for a really bad analogy, but like what are some of those nuggets for people to look out for that are often probably missed because they're looking at those louder ones like ROAS or those traditional media metrics.
• 23:22
  SPEAKER_00: Right, correct.
• 23:23
  SPEAKER_00: We find a lot of those in creative and narrative analysis.
• 23:26
  SPEAKER_00: That's the place where we're finding most of those.
• 23:28
  SPEAKER_00: And some of those examples could be placement of a class.
• 23:32
  SPEAKER_00: A class is let's say defined as we break the creative down.
• 23:35
  SPEAKER_00: Let's say classes are defined as feature-oriented messaging, benefit-oriented messaging, presence of a brand, uh uh a promotion, uh, what kind of uh image invokes what kind of uh recollection in your mind?
• 23:50
  SPEAKER_00: We we build out classes in creative, right?
• 23:52
  SPEAKER_00: And so oftentimes we'll we're finding non-zero weak correlation between these classes and outcomes.
• 24:00
  SPEAKER_00: So we will go back and say these classes tend to maximize click-throughs.
• 24:06
  SPEAKER_00: You on the ad text log side, I don't I mean rhetorical view, people on the ad tech log side will see, oh, this is why I have a creative with a very high hook but zero sustain.
• 24:18
  SPEAKER_00: I only see the first 10% of a creative view through, and after that, people click out.
• 24:24
  SPEAKER_00: That doesn't mean that the creative is bad.
• 24:25
  SPEAKER_00: That simply means that the sequencing of the signals that you're sending in that creative is designed for a high impulse.
• 24:33
  SPEAKER_00: And after that, there is not much left in that creative to tell.
• 24:37
  SPEAKER_00: So you shouldn't be throwing that creative out the uh out the window saying, I put creative uh that only works seven seconds out of a 30-second ad, and so toss it out.
• 24:47
  SPEAKER_00: No, that creative has, because we've done a breakdown, it could be the single biggest lever that you can couple with, let's say, say skippable buys and increase your click-throughs or view-throughs, then sequence it out with something that has a higher sustain.
• 25:02
  SPEAKER_00: Now all of a sudden you're pull through.
• 25:04
  SPEAKER_00: My point in sharing a little bit and going a little off of the deep end here is this is how we think.
• 25:08
  SPEAKER_00: These are some of the signals that we are detecting, which otherwise there's no place to go to.
• 25:14
  SPEAKER_00: When I present this outcome to you, you don't have to change your experience, you don't have to change your story.
• 25:19
  SPEAKER_00: Just knowing that this is what causes a outcome.
• 25:23
  SPEAKER_00: That outcome may not be a measurable increase in revenue or a conversion just yet, but it is a outcome.
• 25:29
  SPEAKER_00: Because when you drive this outcome, the next step is it'll show up in revenue or conversion.
• 25:34
  SPEAKER_00: So that's how we're thinking about finding these signals that otherwise you simply cannot find looking at an ad tech log.
• 25:41
  SPEAKER_01: Am I answering your quiz correctly?
• 25:43
  SPEAKER_01: You are, yeah.
• 25:44
  SPEAKER_01: I mean, in and it's kind of a natural segue into the tension of all of this and where technology and and where people that study human behavior are coming into I don't know, some friction.
• 25:56
  SPEAKER_01: But I mean, are there models that you've been able to either create or see that tell the difference between content or creative that predicts a response and content that creates real recognition?
• 26:09
  SPEAKER_01: Like I'm deep into wanting to understand.
• 26:12
  SPEAKER_01: I think trust has been looked at for a while.
• 26:14
  SPEAKER_01: I'm in what we're building is really around this idea of familiarity.
• 26:18
  SPEAKER_01: And what does familiarity mean to people and to how people consume media and how people think of their response to brands and the messaging that gets served to them?
• 26:28
  SPEAKER_01: But are there models that you that you see that can help predict a response?
• 26:35
  SPEAKER_00: There are there are quite a few models beyond your standard model.
• 26:39
  SPEAKER_00: So almost every I say this with a lot of empathy.
• 26:42
  SPEAKER_00: A lot of times the answer to these hard questions are let's just use a Bayesian MMM or a Markov chain and study attribution.
• 26:52
  SPEAKER_00: Nothing against these models.
• 26:53
  SPEAKER_00: I teach them in classrooms, we've used them for a while.
• 26:56
  SPEAKER_00: So to answer your question, this goes back to also one of the earlier discussions.
• 27:00
  SPEAKER_00: You need to think outside the box.
• 27:03
  SPEAKER_00: These models were built for a certain need at a given point in time.
• 27:09
  SPEAKER_00: Those needs have evolved.
• 27:10
  SPEAKER_00: So to answer your question, there are many more models that we are trying to expose our uh industry to that are less known but can do a better job of prediction.
• 27:23
  SPEAKER_00: Those are ensemble models, those are different.
• 27:26
  SPEAKER_00: Go ahead.
• 27:27
  SPEAKER_01: No, I you keep going.
• 27:29
  SPEAKER_01: I mean, that's I I you're you're making sparks go off in my brain.
• 27:32
  SPEAKER_00: So no, I I uh the the reason I the reason I pause is because uh just like you were saying, look, we use allow me, I'll be very brief, allow me to give you a little bit more specifics on this thing.
• 27:46
  SPEAKER_00: I I will I will not go off of the the deep end here, but we started to bring to the industry what we call USEM, uncertainty aware causal models.
• 27:55
  SPEAKER_00: A lot of people haven't heard about that.
• 27:57
  SPEAKER_00: The reason they haven't is because those models originated in the world of healthcare.
• 28:02
  SPEAKER_00: A lot of them were used for cancer cell propagation detection.
• 28:05
  SPEAKER_00: Again, in the sandbox, and now my co-founder, we've worked across industries, so we are aware of models that are aware outside of the world of marketing that marketing veterans don't have access to.
• 28:18
  SPEAKER_00: So when we bring USEM models to bear, we're not trying to predict cancer cell propagation, but we are trying to say that in marketing data, because there's so much noise, there is a better way to pick up a signal that than the standard Bayesian regressions that you've been using or uh Markov chain to try to uh uh assign attribution.
• 28:40
  SPEAKER_00: Similarly, when it comes to creative analysis, we brought to bear a lot of theories, quantitative theories from they're called NPT, narrative performance theory, from the world of anthropology.
• 28:53
  SPEAKER_00: So my point here is we go out and look at models that are outside, that are not as you as frequently used in marketing.
• 29:02
  SPEAKER_00: So we're trying to bring them here for better purposes.
• 29:05
  SPEAKER_00: That's part one of the answer of your question.
• 29:08
  SPEAKER_00: Yes, there are better models.
• 29:09
  SPEAKER_00: Part two of the answer is I think what I'm trying to affect a little bit uh uh uh uh uh uh by raising awareness is not everything has to be predicting a response.
• 29:21
  SPEAKER_00: When you use better models, it also gives you a little bit of a better power to do more informed experimentation and prediction.
• 29:30
  SPEAKER_00: In other words, if you're in the in the business of predicting a point, a stock price that will be Nvidia will be at X price on day 60 from today, it's almost a guaranteed way of failure.
• 29:43
  SPEAKER_00: And so if you think of everything I can predict as a point estimate, you're gonna be disappointed.
• 29:49
  SPEAKER_00: But if you think of using these prediction models to produce an outcome, whether that is a range of prediction or something you must be doing as a marketer to try out uh a new way of experimentation, that's the right way of using it.
• 30:03
  SPEAKER_00: So short answer to your question: yes, there are models uh that are better than the ones that we use.
• 30:09
  SPEAKER_00: Uh also the caveat I would say is how we use those models is also something that we should drive towards a little bit more maturity in our industry.
• 30:18
  SPEAKER_01: I like the stock analogy.
• 30:20
  SPEAKER_01: That's something that uh we've always tried to correlate and how people think about media as investing in in stocks in that capacity.
• 30:27
  SPEAKER_01: It's like, you know, there's there's a lot to be learned from that correlation.
• 30:31
  SPEAKER_01: But I mean, is there something one of my fears, just to put my cards on the table, is that sometimes through all of this, I think about are we engineering something that humans experience as authentic when sometimes, especially with advertising, I mean, there's advertising and content, and we can have a whole other conversation about the differences or what some people think of similarities in that.
• 30:58
  SPEAKER_01: But it's like sometimes we reach people and there's just casual moments, and it's not this thing that can be even have some sort of causal effect on or model out.
• 31:10
  SPEAKER_01: So I'm like, Correct.
• 31:12
  SPEAKER_01: Are we engineering something that's not there that's just like intrinsic to being human?
• 31:16
  SPEAKER_01: I I appreciate the question.
• 31:18
  SPEAKER_00: Uh the way the the way I look at it is we shouldn't try to engineer the expect uh uh experience because there will be moments that you will have a certain set of campaigns that will reach people, like you said, they reach them in a certain they reach them at a certain point in time that there's no response, or they reach them at a point where they have already been served a lot of ads.
• 31:41
  SPEAKER_00: So there's a relative desensitization built into that, or they could reach them in a point where it is an absolute opposite of the thought process they are in, and you could see a spike in recognition by saying, Oh, I just saw a message that just piqued me or challenged me in my thinking.
• 31:57
  SPEAKER_00: My point here is we shouldn't try to engineer a response.
• 32:01
  SPEAKER_00: One of the things that we're using analytics for is a simple way of saying is analytics should be used to try to understand, try to meet the customer on their terms.
• 32:12
  SPEAKER_00: If I have somebody who has a certain need state, and that need state, I cannot predict at what point in time that need state will be.
• 32:20
  SPEAKER_00: I have to have analytics that'll be say, that'll be able to say at what distribution should I be trying to reach them at a need state so that when they are in that need state, the response feels more natural.
• 32:34
  SPEAKER_00: Oh, I was looking for this, I was searching for this experience, I wanted to buy a certain product for a need state that's coming up in the future.
• 32:42
  SPEAKER_00: But if I try to if I try to zero in at that point in time, at that second, all we're doing is responding to a signal that may or may not be true.
• 32:53
  SPEAKER_00: Just because there's an opportunity to serve an ad on an inventory that is assigned with a certain characteristic does not mean that the user, the human sitting behind that inventory, is in that exact need state and can will respond to your ad in exactly the same way that you're trying to do.
• 33:09
  SPEAKER_01: I think that need state is really important framework to keep in mind because every platform is coming out with all these AI tools that are doing exactly to what you're speaking of.
• 33:21
  SPEAKER_01: It's like they're taking that immediacy of a data point and then trying to give you something to react to.
• 33:26
  SPEAKER_01: And it's like there's a difference between reacting and then responding to enough to be able to give you a more clear picture.
• 33:31
  SPEAKER_01: So, I mean, with that, is there any decisions that you see from your work or marketers in the field that you're just like, I would never hand that to an algorithm to be able to dictate how we're responding, not reacting to something, even if the model's statistically more accurate than the people in the room.
• 33:55
  SPEAKER_00: I'll just say my answer.
• 33:56
  SPEAKER_00: For I'm a data guy, it might sound a little strange coming in.
• 34:00
  SPEAKER_00: I'm talking about the empathy of being a human.
• 34:02
  SPEAKER_00: So the short answer is by nature, and you see this across measurement people, you see this across everything.
• 34:07
  SPEAKER_00: We tend to be, our minds tend to be critically oriented, right?
• 34:11
  SPEAKER_00: You I'll present to you a solution, you will like that solution, but naturally there'll be some questions you're gonna want to ask of that solution.
• 34:18
  SPEAKER_00: Anything that I present to you, like, what about this?
• 34:21
  SPEAKER_00: What your brain is essentially doing is it is taking a response and it is critically challenging the output.
• 34:29
  SPEAKER_00: That challenging part should always stay human.
• 34:32
  SPEAKER_00: Whether it's a measurement solution, whether it's a campaign or what have you.
• 34:35
  SPEAKER_00: In the world of measurement, we call hypotheses.
• 34:37
  SPEAKER_00: In the world of measurement, we call that context setting.
• 34:39
  SPEAKER_00: But the whole idea is naturally, as humans, you don't just give something to somebody and they take it and run with it.
• 34:46
  SPEAKER_00: There is a natural human trait of understanding or challenging the output for either better understanding or pointing out what could be done better.
• 34:54
  SPEAKER_00: That I think should stay squarely and will stay squarely human because AI models run off of prediction, run off of correlation, run off of vectors that will predict based on a certain vector what the next vector should be.
• 35:11
  SPEAKER_00: They're not at a point, or I don't think uh anywhere in the near future, they'll be at a point where they will do independent critique of all of these things.
• 35:21
  SPEAKER_00: They need an input.
• 35:22
  SPEAKER_00: They need you to present a hypothesis, they need you to go back and say, hey, um, audience can be a confounder.
• 35:30
  SPEAKER_00: Audience can affect spend as an input, and it can affect revenue or take rate as an output.
• 35:36
  SPEAKER_00: When an input affects both input and when a factor affects both input and output, it will throw your models off.
• 35:44
  SPEAKER_00: So until unless you as a human can point that out saying, this is a confounder, this is my critical thinking going to work, setting the right context, AI will not have the right context to keep automatically running models.
• 35:57
  SPEAKER_00: So this context of challenging critical analysis, hypothesis, confounder identification, no matter what you want to call it, that I squarely see sitting in the human domain.
• 36:09
  SPEAKER_01: Oh, that gives me hope.
• 36:10
  SPEAKER_01: Sometimes I wake up in a cold sweat in the middle of the night being like, they're gonna take over.
• 36:14
  SPEAKER_01: And then sometimes I'm, you know, have all the hope in the world.
• 36:18
  SPEAKER_01: It's really a roller coaster of emotions with all of this.
• 36:21
  SPEAKER_01: But I mean, on that notion, I am always trying to challenge myself, and I'd be interested to hear from you.
• 36:25
  SPEAKER_01: And the idea is like, is there are there things that we're underestimating that these machines can do that it is getting right about um kind of pinpointing and understanding human behavior?
• 36:36
  SPEAKER_00: See, one of the things that and I see this in data all the time, you you you you you've seen this forever.
• 36:45
  SPEAKER_00: The difference between when you pull somebody and say, take a survey and say, what are they going to do, and then the reality of what actually happens, we've known of the disconnect, implicit versus explicit for a long time.
• 36:57
  SPEAKER_00: I'm using that as an example to be able to go back and say that is one of the things that will always get underestimated.
• 37:05
  SPEAKER_00: If I provide a data to any model, any AI native model, ML model, it will do the best to decipher that data and anything that it finds in that data, it takes that as an instance of a fact.
• 37:22
  SPEAKER_00: But my response as a human being to an ad being served to me or to a qualification of my traits as an audience member of uh X, Y, Z audience group.
• 37:35
  SPEAKER_00: My point here is those qualification factors, the data points that define me, take those definitions as an absolute truth.
• 37:42
  SPEAKER_00: But the behavior will always vary from it because there is a big gap between what we say and what we do.
• 37:50
  SPEAKER_00: And I think if you think of that realm, there is no basis for any AI engine or any machine to know that.
• 37:57
  SPEAKER_00: Whatever they see as data, they will treat it as a fact.
• 38:00
  SPEAKER_00: When you do something different, you were served an ad, every prediction said, you are 78% likely to either be grouped into a click-through or a view-through based off of that ad being served to you seven times.
• 38:13
  SPEAKER_00: There is an absolute level of precision attached to that, but you didn't do that.
• 38:17
  SPEAKER_00: The moment the machine detects that difference, that call it calls it as a noise.
• 38:22
  SPEAKER_00: The reality is you are a human.
• 38:24
  SPEAKER_00: There is a non-zero chance that you will not take action because you may do different than what your data describes to you.
• 38:32
  SPEAKER_00: So, long way of saying that is the difference between stated behavior and predicted behavior, that cannot be model.
• 38:38
  SPEAKER_00: And a machine will always throw that out as a noise.
• 38:41
  SPEAKER_00: A human being will always be keep, will always keep that inherent to their behavior.
• 38:46
  SPEAKER_00: That's the one disconnect that I don't think machines are anywhere close to one.
• 38:52
  SPEAKER_00: How do you balance that disconnect?
• 38:54
  SPEAKER_01: Especially, I mean, as you're talking with people who aren't in the weeds like you, I mean, that is a juggernaut to kind of try to wrap your mind around if you're not used to living in this world all the time.
• 39:04
  SPEAKER_01: And most CMOs aren't.
• 39:06
  SPEAKER_01: They have a million other things that are on their plate.
• 39:08
  SPEAKER_01: So it's like when you're showing them what these models are putting out.
• 39:14
  SPEAKER_01: Are you training people like how to balance the intake of this information with what is real human behavior?
• 39:22
  SPEAKER_00: Yes.
• 39:22
  SPEAKER_00: So so we we we certainly don't always go if we're if we're in the room with um uh we we sit in the room with a lot of global analytics team across the world.
• 39:31
  SPEAKER_00: So if we're in an analytics discussion, uh this discussion can go really into the depth.
• 39:35
  SPEAKER_00: If we're sitting with our um key partners in marketing and finance, it doesn't usually go into the depth of it.
• 39:42
  SPEAKER_00: But the essence of your question is um we try to, in our measurement solution, account for uncertainty.
• 39:50
  SPEAKER_00: That is why I had mentioned something called uncertainty aware causal models, USCM.
• 39:55
  SPEAKER_00: Uncertainty aware models, the solutions we build assign a certain level of inherent risk to an outcome.
• 40:03
  SPEAKER_00: We I don't think I or anybody here at this point in time can say we've got that cover.
• 40:10
  SPEAKER_00: The difference between predicted behavior and actual behavior, if anybody says, I can totally capture that and account for that, I would call them on it.
• 40:23
  SPEAKER_00: Because that is just an area.
• 40:25
  SPEAKER_00: There's no specifics in it.
• 40:26
  SPEAKER_00: I can say, as a block of things, a block of factors, 40% of the decision is uncertain.
• 40:34
  SPEAKER_00: Like so, my point here is what we try to do, Chris, is make people aware that there is a range within which you have to operate.
• 40:41
  SPEAKER_00: There is a portfolio within which you have to operate.
• 40:44
  SPEAKER_00: We can define with reasonable confidence what the bounds of that, that uh those walls are, but we try not to make it into a point prediction because there is, it's it's impossible to, you as a human being, today, you're right now you're talking to me and you feel that you are interested in a subject X, two hours later, you may meet somebody else, you may talk to another human being, and you may say, I'm interested in subject Y.
• 41:10
  SPEAKER_00: There was a difference between X and Y, and it would be amiss for me to go back and say, I know Chris so well through my models that I could predict that he was going to change from X to Y.
• 41:21
  SPEAKER_00: That's a fallacy.
• 41:22
  SPEAKER_00: But I could say, after analyzing Chris's behavior, I can say that the chance of Chris moving from X to Y or not exactly falling on point is a plus minus 5%.
• 41:33
  SPEAKER_00: That I can say with a reasonable confidence.
• 41:36
  SPEAKER_00: So that's how we approach uncertainty.
• 41:38
  SPEAKER_00: We model a certain level of risk with every solution that we're bringing forward.
• 41:44
  SPEAKER_01: I don't want to lead the question, but I think you just answered one of mine.
• 41:47
  SPEAKER_01: But I'm interested in what you would tell someone to stop measuring and start questioning.
• 41:53
  SPEAKER_00: I I I uh I appreciate that.
• 41:56
  SPEAKER_00: I I I since you like stock uh uh analogies, I'll give you the valuation metrics.
• 42:02
  SPEAKER_00: There is no valuation equivalent of metrics in marketing.
• 42:05
  SPEAKER_00: There is no PE ratio, there is no PE floor, there is no price to book, there is no price to sales.
• 42:11
  SPEAKER_00: Because one of the things that finance has it right, that marketing needs to grow up to is understanding forward-looking metrics.
• 42:19
  SPEAKER_00: Everything, going back to the ROAS kind of conversation, revenue calculation, everything has to be immediate, everything has to be measurable, everything has to be precise for that ROAS to work.
• 42:31
  SPEAKER_00: There are very few, if at all, forward-looking valuation metrics.
• 42:35
  SPEAKER_00: So there is no equivalent of am I tracking movement in plus minus movement in any level of awareness of a product or brand?
• 42:45
  SPEAKER_00: I'm giving you simple examples, those could be measured.
• 42:48
  SPEAKER_00: There are some complex ones too.
• 42:49
  SPEAKER_00: Um, the uh level of saturation and its effect on conversion.
• 42:54
  SPEAKER_00: My point here is there's no forward-looking metrics.
• 42:56
  SPEAKER_00: So I would encourage CMOs or anybody to start.
• 42:59
  SPEAKER_00: Don't give up what you have.
• 43:00
  SPEAKER_00: You have to stand in front of board of directors, your CEO to be able to justify what you're doing.
• 43:05
  SPEAKER_00: That doesn't change.
• 43:06
  SPEAKER_00: So you need the measurement that exists today.
• 43:09
  SPEAKER_00: But actively start thinking about forward-looking metrics that you can include in your consideration, because a lot of what we do is a pay forward model.
• 43:19
  SPEAKER_00: If you are a huge brand, your incrementality is going to be low.
• 43:22
  SPEAKER_00: There's no two ways of looking at it.
• 43:24
  SPEAKER_00: If somebody just keeps coming to you and says, I've got an incrementality solution that can give you a 20% incrementality, my first question to those people is There are so many of those, by the way.
• 43:34
  SPEAKER_00: Right.
• 43:34
  SPEAKER_00: Right?
• 43:35
  SPEAKER_00: So that but but my point, it that is exactly right, Chris.
• 43:38
  SPEAKER_00: But that conversation, the first question in that conversation is what's the level of your brand presence?
• 43:43
  SPEAKER_00: If your level of your brand presence is low, incrementality will be high.
• 43:46
  SPEAKER_00: But if you're, let's say, a recognized brand, if you're an Apple or a Home Depot or any of no matter what model you run, your incrementality dimension will be low.
• 43:56
  SPEAKER_00: Now think about forward looking metrics.
• 43:58
  SPEAKER_00: If I said the only range of Incrementality you can affect is 5%.
• 44:02
  SPEAKER_00: But over a period of time, that five can go to five and a half, six percent.
• 44:05
  SPEAKER_00: I have a forward-looking metric.
• 44:07
  SPEAKER_00: And it gives people the ability to have a dialogue on what should you do now that will impact your current results, but also not give away future benefits.
• 44:17
  SPEAKER_00: So I I will wrap up by saying is I don't know if marketing currently has very many forward-looking metrics.
• 44:25
  SPEAKER_01: A lot of what we've been kind of talking about is marketing in the sense of what most marketers are looking for.
• 44:33
  SPEAKER_01: They're like, grow my business, I'm looking for that sales, that foot traffic lift, whatever.
• 44:38
  SPEAKER_01: I that clicks to convert to econ.
• 44:42
  SPEAKER_01: But I mean, at the core of that, something I believe deeply in and our and our team at Campfires constantly always trying to like wrap our arms around is like really at the core of this, though, is like we're really trying to connect with humans.
• 44:54
  SPEAKER_01: Yes.
• 44:54
  SPEAKER_01: And so it's like if the goal of marketing isn't to merely just generate a response, but create that connection and like real connection, something that mobilizes people around something more than just a product or service, but a way of thinking and doing and moving through the world.
• 45:10
  SPEAKER_01: What does measurement need to become better at seeing in order to grasp that true tie?
• 45:20
  SPEAKER_00: I I think you hit the nail on the head.
• 45:22
  SPEAKER_00: I think measurement is a vehicle, not an end.
• 45:26
  SPEAKER_00: It's a means to a delivery.
• 45:28
  SPEAKER_00: Measurement needs to try to understand how to bring what you're doing closer to the consumer over a certain period of time.
• 45:38
  SPEAKER_00: One of the things I have not seen very many people do is buy their brand or even within a brand, buy their product.
• 45:46
  SPEAKER_00: What is the momentum of your message?
• 45:50
  SPEAKER_00: And when does it start to even show up in the mind of a customer?
• 45:54
  SPEAKER_00: That journey could be three weeks.
• 45:56
  SPEAKER_00: That journey could be three months, it could be a year long.
• 45:59
  SPEAKER_00: But if you miss to account for something as simple as that, your campaigns will become mechanical.
• 46:05
  SPEAKER_00: Your campaigns will be in market, you'll run it for three weeks, four weeks, eight weeks, and say it's not performing, pull it away.
• 46:10
  SPEAKER_00: What why I bring that up is that is a genuine missed uh opportunity to make the connection with the customer.
• 46:18
  SPEAKER_00: Because it's not, you you just threw a threw something out that could actually make that connection because you didn't understand what was the time to make that connection with the customer.
• 46:26
  SPEAKER_00: You didn't account for the fact that you, I, as busy human beings, are exposed to a few thousand ads in a given day.
• 46:32
  SPEAKER_00: I might not be in the neat state when that ad was run.
• 46:35
  SPEAKER_00: By the way, I'm not advocating that everybody run unlimited budget.
• 46:39
  SPEAKER_00: This is also one of those things that we do in measurement to be able to say how do you maximize the stretch of your dollar so that you're not like my my answer isn't uh uh isn't uh advocating for the fact that you need to throw a lot of money.
• 46:52
  SPEAKER_00: My point here is an example like this is, in my opinion, a trap that people fall into because you were genuinely trying to build a connection, but you cut it off before the tipping point happened.
• 47:04
  SPEAKER_00: The converse also happens oftentimes.
• 47:07
  SPEAKER_00: You sink a lot of money right off the bat, hoping for an immediate effect, but that immediate effect was never gonna be the case because you're a new product reaching out to a new audience or saturating the market with a new message, or people who you've not connected with and you're conquesting and you're trying to get an immediate result that will not happen.
• 47:26
  SPEAKER_00: So, my point here is we should at least have some baselines based off of what measurement should be run so that you have an understanding of what does it take to resonate with the customer instead of putting absolute values on these metrics and then cutting off things that could actually work for you or over-indexing on things that are going to be a waste.
• 47:46
  SPEAKER_01: And is it fair those baselines take time to generate a baseline?
• 47:52
  SPEAKER_00: You are right.
• 47:52
  SPEAKER_00: I mean, baseline generation does require bad data, but because we are living in a high compute world today, you don't always need a massive amount of data.
• 48:01
  SPEAKER_00: The reason I bring high compute is let's say you're a new product and launching a new product and a new market, you will not have any data.
• 48:07
  SPEAKER_00: You will not have any baseline because it's a new product and a new market, it's a new message.
• 48:11
  SPEAKER_00: But here's the beauty of measurement, and this is why I get excited.
• 48:15
  SPEAKER_00: We use a lot of models that are backpropagating models.
• 48:20
  SPEAKER_00: It's like an SAT.
• 48:22
  SPEAKER_00: You, a good score in an SAT or ACT is a function of getting multiple questions right, not one question right.
• 48:29
  SPEAKER_00: So if you're in a new product in a new category in a new market, your first output, your first action, whether it's spend assigned to that campaign, whether it's audience assigned to that, whether it's creative assigned to that, your first one has a very high amount of risk.
• 48:44
  SPEAKER_00: But because we have high compute models available, I can literally take that data, feed it back to the model and say, what do I need to correct?
• 48:51
  SPEAKER_00: So in this case, I don't have a baseline, but I'm self-correcting the output at each run and kind of using a way of zeroing down to a natural state.
• 49:01
  SPEAKER_00: So to answer your question, baseline generation in a standard way does require data.
• 49:05
  SPEAKER_00: But if you are not there, if you don't have that much data to generate a baseline, rely on the power of models now available to you.
• 49:14
  SPEAKER_00: You don't need to be a data scientist.
• 49:16
  SPEAKER_00: Now you have libraries available, you can ask Claude to code it for them.
• 49:20
  SPEAKER_00: There are models that are available that can zero in on an effect, and now all of a sudden you have just filled the gap for a missing baseline.
• 49:29
  SPEAKER_00: Now you have better data to work with.
• 49:31
  SPEAKER_01: One of the traps that I've seen when it comes to measurement, and we'll kind of start start rounding out, but is that measurement seems to be, again, one of those things that's reactive to being like, we need to hit a number.
• 49:44
  SPEAKER_01: And it comes much later in the in the strategic decision making process.
• 49:49
  SPEAKER_01: Like it seems like it's almost like how people used to think of the paid media where it's like one of those last boxes to check.
• 49:56
  SPEAKER_01: It's like, cool, we've done all this creative brand, all of these things have been checked out.
• 50:01
  SPEAKER_01: Now we get to media and measurement as those last stops.
• 50:04
  SPEAKER_01: Yes.
• 50:04
  SPEAKER_01: Is that a fair assessment?
• 50:06
  SPEAKER_01: That comes with complete bias as someone that's worked in media for close to two decades, or it's like, no, that's how I've seen it.
• 50:12
  SPEAKER_00: It it is definitely a fair, it is a very fair assessment, and it is definitely something that we run into all the time.
• 50:18
  SPEAKER_00: Even in the example I was providing a new product being launched and a new market or whatever, they always invariably come with uh, I expect this to be hitting a hundred dollar cost per acquisition, and I need this to be uh 18% growth in the next six months since launch.
• 50:32
  SPEAKER_00: So there's already a lot of targets that are associated with this, and usually that's when measurement gets involved to be able to say, how do I hit those?
• 50:41
  SPEAKER_00: It's not upfront.
• 50:42
  SPEAKER_00: Uh there is a missed opportunity of including it more in the upfront process because the first part of this process should be about what your potential is, and that you can unlock through measurement and then work your way downward from there to hit the target, not the other way around.
• 51:01
  SPEAKER_00: Long way of saying is and I've also seen what you're saying, that's where we get pulled in oftentimes once the decisions have been made and the targets have been set, and then measurement is brought in to bridge the gap.
• 51:16
  SPEAKER_00: It could benefit by being up front.
• 51:18
  SPEAKER_00: It could also give you insights on product launch, product changes from the upfront itself that might put you in a better place to deliver on those goals if brought up front.
• 51:30
  SPEAKER_01: Yeah.
• 51:31
  SPEAKER_01: I look at it as a thing.
• 51:32
  SPEAKER_01: It doesn't need to be a thing that's bridging a gap.
• 51:34
  SPEAKER_01: It can be the road that leads to the bridge that leads to the opportunity on the other side.
• 51:39
  SPEAKER_01: It's like if we're talking about really trying to grow our businesses, we know we have to look outside of those traditional KPIs of business and marketing and look towards the impact on humans.
• 51:52
  SPEAKER_01: Well, the only way to do that is if you do bring that measurement strategy up to the forefront of the conversation.
• 51:58
  SPEAKER_01: Yes.
• 51:58
  SPEAKER_01: And you can start ideating around what is actually what's our path to measurement once we hit those standard namesake KPIs that everyone's aware that they need to report to their board.
• 52:10
  SPEAKER_00: I I it it's so true.
• 52:12
  SPEAKER_00: And if I were to show you empirical evidence on that, the amount of and I say this with a lot of empathy, no critique, the amount of um uh wasted spend or underperforming spend that typically happens at the beginning of any cycle tends to be the highest.
• 52:32
  SPEAKER_00: And one of the things that because if you think if you think of measurement as just any other business factor, right?
• 52:40
  SPEAKER_00: Yes, we do we do oftentimes get into the weeds of things, but if you think of measurement as an enabler, just like any other business factor, bringing it up front also saves you.
• 52:51
  SPEAKER_00: Because I I I have crunched so much data, I'm sure you have seen this too.
• 52:55
  SPEAKER_00: The level of waste that happens at the beginning of a cycle without measurement already puts you in a hole because that you're then trying to claw out of.
• 53:03
  SPEAKER_00: If brought up front, you could have avoided that.
• 53:06
  SPEAKER_01: So yes, uh, that's a big flag.
• 53:10
  SPEAKER_01: Yeah.
• 53:11
  SPEAKER_01: So, from how we end every episode here is with this uh big lofty question.
• 53:19
  SPEAKER_01: Would love your take on it.
• 53:20
  SPEAKER_01: What is one thing about human behavior that no algorithm will ever be able to replicate?
• 53:27
  SPEAKER_00: I I think I might have touched upon this.
• 53:30
  SPEAKER_00: It's the ability to critique, it's the ability to challenge.
• 53:33
  SPEAKER_00: I I do not think, and I'm saying this not just out of the empathy of my heart, but also because of an understanding of how AI, ML models, GPT transforms were the ability to critically challenge an output and introduce a new dimension for consideration, um, I don't think there is a replacement for that right now.
• 53:59
  SPEAKER_00: Um it it is don't get me wrong, it is not an easy task.
• 54:04
  SPEAKER_00: A lot of times that doesn't happen because maybe you're you're gonna find critiquing your own solution that I talked about.
• 54:11
  SPEAKER_00: Maybe there's many factors in there.
• 54:13
  SPEAKER_00: But if I were to answer that question in one way, the ability to critique and the ability to poke holes in a solution squarely sits human today, in my mind.
• 54:27
  SPEAKER_01: I think there's a great theme, one that I'm picking up from our whole conversation, which I'm super grateful for and your time and sharing all your all your insight, is like you've touched on it numerous times around empathy.
• 54:38
  SPEAKER_01: And you can tell that's like a core part of what brings you to this work, but also how we can remain that human in the loop and make sure that humans are making sure that we're responsible for each other.
• 54:52
  SPEAKER_01: So I mean, you're touching on empathy.
• 54:54
  SPEAKER_01: Our previous episode conversation ended with this idea of wonder.
• 54:59
  SPEAKER_01: We have wonder and empathy.
• 55:00
  SPEAKER_01: I can't think of like better, more human in instincts to to end on.
• 55:05
  SPEAKER_01: So thank you for reminding us of empathy today.
• 55:08
  SPEAKER_01: That's great.
• 55:10
  SPEAKER_01: I I I appreciate that.
• 55:11
  SPEAKER_00: I appreciate you sharing me about wonder.
• 55:14
  SPEAKER_00: That is a really good one as well.
• 55:17
  SPEAKER_01: Empathy is up there too.
• 55:18
  SPEAKER_01: I appreciate it.
• 55:19
  SPEAKER_01: Well, thank you so much, and uh appreciate your time.
• 55:22
  SPEAKER_01: Thank you.
• 55:25
  SPEAKER_01: One thing from this conversation keeps sticking with me.
• 55:27
  SPEAKER_01: ROMSAD measurement should be a vehicle, not the destination.
• 55:32
  SPEAKER_01: That feels particularly important right now.
• 55:34
  SPEAKER_01: We have more ways than ever to predict behavior, optimize toward outcomes, and identify patterns.
• 55:40
  SPEAKER_01: But there's a difference between predicting what someone might do and understanding why they do it.
• 55:46
  SPEAKER_01: There's also a difference between optimizing a response and building a connection.
• 55:51
  SPEAKER_01: And maybe a part of being responsible with all of this technology is being willing to leave room for the things it can't explain.
• 55:58
  SPEAKER_01: The person who doesn't click when the model says they should, the campaign that needs more time, the signal buried inside what we've dismissed as noise.
• 56:07
  SPEAKER_01: Rom's answer to our closing question was critique.
• 56:11
  SPEAKER_01: The ability to poke holes in the answer, introduce another possibility, and ask what we might have missed.
• 56:19
  SPEAKER_01: The last conversation on this show left me thinking about wonder.
• 56:23
  SPEAKER_01: This one left me thinking about empathy and the ability to critique.
• 56:27
  SPEAKER_01: And I'm starting to think those ideas are more connected than they first appear.
• 56:32
  SPEAKER_01: The machines are going to keep getting better at giving us answers.
• 56:36
  SPEAKER_01: Smart ones at that.
• 56:38
  SPEAKER_01: Maybe our job is to keep getting better at asking whether they're the right ones.
• 56:47
  SPEAKER_01: This has been Responsibly Human, a responsibly different podcast from Campfire Consulting.
• 56:52
  SPEAKER_01: At Campfire, we believe data can tell us a lot about people.
• 56:56
  SPEAKER_01: But building brands people actually connect with still takes curiosity, judgment, empathy, and a willingness to see the person behind the pattern.
• 57:05
  SPEAKER_01: If this conversation gave you something worth thinking about, subscribe and share it with someone else doing the work that could benefit from it.
• 57:11
  SPEAKER_01: Every listen, follow, and share helps these ideas reach more people trying to make business, media, and culture a little more human.
• 57:19
  SPEAKER_01: Responsibly Human is hosted by me, Chris Maureen, with Pip Colmar and Kaylee Elwell helping shape and produce the series.
• 57:26
  SPEAKER_01: And yes, our cover art was made by actual humans, illustrated by our friends at Narrative.
• 57:33
  SPEAKER_01: Until next time, keep making room for what can't be measured.