Latest / CXOTalk / Quantitative Marketing: Data-Driven Strategies and Challenges
Transcript
- 0:00Today, on episode number 787 of cxo, talk, we're discussing data
- 0:06and intuition. The role in marketing, we call
- 0:10that quantitative marketing. Our guests are two folks from
- 0:15the Columbia business school at Columbia University in New York
- 0:18City ODed. Netzer is the vice Dean for
- 0:22research and a professor and author of the new book decisions
- 0:26over decimals and Amy Jake is the chief marketing officer of
- 0:31the institution for me marketing.
- 0:34Is that part of any organization that is on charge of
- 0:38facilitating and enabling successful transaction with the
- 0:41customer successful? Whether it could mean profit if
- 0:43it's for profit could be engagement for example if it's
- 0:46non profit but the ones who are in charge of making sure that
- 0:50such transactions relationships do exist and and now you want to
- 0:56layer on it. The quantitative part of it in
- 0:58order to create this. Successful interactions.
- 1:02I really need to understand if under firm who's on the other
- 1:05side and it's not exactly the other side will have my
- 1:07partner's, my customers. And for that, I need a lot of
- 1:11data, a lot of information about my customers and who they are
- 1:15and good understanding of who I am, what am I capabilities and
- 1:19that again requires data and analytics.
- 1:23And specifically the analytics is often done in order to try
- 1:26and understand this match the met with the match between the
- 1:29customer. And and they the firm and what
- 1:32the firm has to offer and even changing what the firm has to
- 1:34offer to make sure it matches. What what customers want?
- 1:39Particularly in, in, in, in recent years where a lot of the
- 1:42effort went over Z, he's getting better and better understanding
- 1:47of the customers because the better and the more we
- 1:50understand the customers, the better, we can actually serve
- 1:53them, what it is that they want, and that requires a lot of data,
- 1:57a lot of analytics and a lot of quantitative marketing.
- 2:01Amy, what is the impact of this data driven approach on your
- 2:08work right now? There was a period of time where
- 2:12it was somewhat difficult to explain to senior leadership the
- 2:17value of marketing. The increase in the data that we
- 2:21have, the ability to show quantitatively, and to connect
- 2:24an action that marketing is taken to a bottom line impact.
- 2:28That's what a lot of the data has done.
- 2:31It has allowed us to be better marketers, we understand a lot
- 2:35more. So dead was saying about the
- 2:37audience, we can engage in different tactics and strategies
- 2:41personalization is so important. You know, we are inundated with
- 2:46material and it is very difficult for us as humans to
- 2:52have that, much flying at us and to be able to figure out what to
- 2:55pay attention to. You know, cognitively, if
- 2:58there's this influx of information, we have to pick, we
- 3:02have to sort, we have to think about what makes sense for us to
- 3:06pay attention to what limited time do we have.
- 3:08And how We react to that and in Mass marketing sometimes you can
- 3:14get that benefit with your audience but more often what
- 3:17we're seeing is that the more personalized, that the marketing
- 3:20is, the more likely that person who is otherwise inundated will
- 3:24take a moment will stop and will engage with you.
- 3:27And the only way to do that is to be able to leverage the data
- 3:31that you have to say, this is the right course, for this
- 3:35person, this is the right path. This is the right Next Step for
- 3:38person a But for person B, it's going to be a little bit
- 3:40different and every time that our audience or consumer or
- 3:44customer has that experience, the more likely, they are to
- 3:47come back because there's a benefit for them.
- 3:50They're getting what they need. They've seen that it works.
- 3:53They know it was worth their time and so I think data both
- 3:56helps us on the strategy side. Develop better strategies.
- 4:00It helps us on showing results to senior leaders to, to
- 4:06shareholders to stakeholders and it helps Us to deliver a better
- 4:10experience to the customer. So they're all wrapped up in
- 4:14sort of one larger marketing effort.
- 4:17But I think those three components and thinking about
- 4:19it, in those ways is really helpful.
- 4:22I've been teaching the marketing core for quite a few years here
- 4:25at Columbia business. School is probably the course
- 4:27that has changed the most actually em out of any course we
- 4:31teach. And the reason is that the world
- 4:33of marketing is changing tremendously.
- 4:35And if you think for example about advertising, The
- 4:39advertising. Again, many of us have watched
- 4:42madman, right? And that's that's really the old
- 4:45days of marketing. But even up to the early 2000's,
- 4:48the center of advertising was here in New York was in Madison
- 4:52Avenue. If you think about what are the
- 4:54three biggest advertising companies in the world then not
- 4:58anymore in New York and there are not any more advertising
- 5:01company companies, there are Google Facebook and I'm an
- 5:06Amazon. These are these companies in
- 5:08Silicon Valley and they sit in, in, in Seattle, which means
- 5:12there are technology companies, and with that comes the
- 5:15quantitative aspect of marketing.
- 5:18Amy made a very interesting point, she spoke about the
- 5:23enhanced ability to connect the marketing activities, back to
- 5:29impact on the organization. Can you talk about that?
- 5:33Because that's what Business Leaders ultimately care about,
- 5:35right? We're spending.
- 5:37A certain amount of money on our marketing and what is the
- 5:39ultimate impact? And how do we measure that?
- 5:42The one of the biggest impact of quantitative marketing on their
- 5:46world of marketing? Was this notion of our eye and
- 5:48there is return on investment, right?
- 5:50I mean, will the famous quote and, you know, half of my
- 5:54marketing expenses are? Well, spent, I just don't know
- 5:56which half and then came the world of online marketing,
- 5:59right? Then came the world of digital
- 6:01marketing and with this promise of we're going to see what
- 6:04people clicked on. We actually going to see whether
- 6:06someone clicked on the ad where the summit eventually even
- 6:09bought some finally, we can get our iron marketing, we know what
- 6:14is the impact on of a marketing on consumers?
- 6:18I think it's true. It we have moved a long long way
- 6:21in the ability to show return on investment on marketing.
- 6:25And we, it is much easier now to have that conversation.
- 6:28And I think that marketeers that haven't taken that step have
- 6:32truly left behind. And if still struggle with are
- 6:34within their organizations, I will.
- 6:37So the mention that that promise may have been even too strong.
- 6:41It, we now started seeing the pendulum shifting on that
- 6:44statement in the sense that the fact that I see whether customer
- 6:47clicked on an ad doesn't yet fully tell me whether it's a
- 6:51full return on investment, right?
- 6:53It could have been that this customer would have bought
- 6:55anyway, so we need some sometimes more sophisticated
- 6:59quantitative techniques in order to truly ask questions, like
- 7:02attributional in current through a mentality.
- 7:05And whether we truly have Roi on these.
- 7:08It seemed as if we solve the problem and now we realize the
- 7:11problem is, we have way way ahead, but there are still
- 7:15things to be done, which is where the world of quantitative
- 7:17marketing works on today. A few years ago, a marketer had
- 7:21to say I see a correlation but I may not be able to show
- 7:24causation and now I think you're right there.
- 7:26Are there still are numerous questions that we have there's
- 7:31still challenges with attribution models but I think
- 7:34we are much closer to being able to say It's not just happening
- 7:38at the same time in, isn't that nice?
- 7:40We're able to at least connect often some of those dots and
- 7:44say, well, I know that when my marketing spend goes up 13% 27
- 7:49percent of customers, you know, purchase more or 13, you know my
- 7:55marketing spend goes up 13 percent and 50 percent more
- 7:58individuals enter this online store.
- 8:01So we're able to do it in a way that you know, if our
- 8:05attribution models are right and we Have the right technology at
- 8:08some point in the path we can say there was a direct
- 8:11connection and we know that it's actually going to make a
- 8:14difference. And I think directionality is
- 8:16important, but directionality with, you know, some very
- 8:21specific models that show that relationship is what ultimately
- 8:25gets you farther in terms of your budget, in terms of being
- 8:30able to enter into new markets, and really buying into sort of
- 8:33the confidence of the c-suite, I think, which is super important.
- 8:37I think one of the things that truly help with that, and
- 8:39particularly, in the world of digital marketing is the ability
- 8:42to run these A/B tests, right? Yes.
- 8:44I mean it's very difficult to say, you know, how about my
- 8:46customer going to see one store and a half of my customer going
- 8:49to see a different version of my store.
- 8:51You know, my story. My story is very difficult to
- 8:54play with it in the online World it because it became very easy.
- 8:57And you know I split my time between Colombia and Amazon, I
- 9:01spend some of my time at Amazon advertising, the practice at
- 9:05Amazon advertising is running. These A/B test these, let's try
- 9:09version a version version B, and when we do that, we truly can
- 9:12get to RI. We truly can get to.
- 9:15What is the difference between offering the customer version a
- 9:18and version B? It's still easier to doing
- 9:20digital environment where we can take, you know, we have 100,000
- 9:24visitors, we can split them half or any any way.
- 9:27We split them randomly and see, throw to to different customers
- 9:31different different offerings and see how it works.
- 9:34It's still a little bit more difficult in the offline mode.
- 9:37But even there, we are making progress toward thinking in the,
- 9:40in a language of A/B tests in the language of testing, in
- 9:45order to measure Roi. And in order to measure, truly
- 9:48what customers want? I'll give you just an example of
- 9:51that. Several years ago, I was working
- 9:54at a large company and my role and my team's role was focused
- 9:59around telling investors the story of the success of the
- 10:02business. And when we had our hypothesis
- 10:05about which creative would work, The best to get them to engage
- 10:08with our financial results. I would have thought one thing
- 10:13and we a be tested over a period of about a year.
- 10:15So that's four, four quarters worth of data and we actually
- 10:19found that in telling the story with our IP, so the show's
- 10:23themselves when we use that as a creative investors, now, this
- 10:27isn't an audience that you would naturally think would pay more
- 10:30attention to, you know, a character on a show than a
- 10:34financial times headline. But when we Talked about our
- 10:38efforts. And when we use the creative,
- 10:41when we use our IP, we found that the interest in the
- 10:44engagement, in the information, the financial information about
- 10:49the company increased, you know, exponentially.
- 10:53And so I think to your point that's an a/b test that 10 years
- 10:56ago or 20 years ago, one, you wouldn't have been able to run
- 10:59but to you wouldn't have hypothesized that.
- 11:02But when we just have the ability to continue to test and
- 11:05swap in and swap out, Out and we see, very clear results in the
- 11:09data, you know, it gives us confidence to both try new
- 11:12things, but then it gives us Security in the decisions that
- 11:15we make. Because we know now that we can
- 11:18see that that happening, please subscribe to our YouTube channel
- 11:23and hit the Subscribe button at the bottom of our website.
- 11:26So you can get our newsletter and we can keep you up-to-date
- 11:30on upcoming live shows. What about the data?
- 11:34What kinds of. Of data, should marketers be
- 11:40collecting? What's the best way to get that
- 11:43data and ODed? Can you take us a little bit
- 11:46behind the scenes of the kinds of models that you're looking
- 11:51at? And again, my real interest here
- 11:55is the impact of that data. And those models on the real
- 12:02lives of marketers, hey, it sounds like a reality TV show,
- 12:05right? The real lives of ders.
- 12:08We have a fire hose of data, right?
- 12:09We have more data than we ever had before.
- 12:12It's still the situation that we don't always have the data.
- 12:15We exactly need. We always sometimes, looking for
- 12:17the right data, we need there is this something we talked about
- 12:20in the book that you mentioned earlier, then we call the the
- 12:23certain teammate. There was this belief that
- 12:25finally when we have all of these data will get to the
- 12:28certain decisions. Certainties I me that we still
- 12:31make decisions under uncertainty, but we do have much
- 12:34more data than we ever had. And Specifically, the data that
- 12:38marketeers value and cherish tend to be data that tells us
- 12:44about customer preferences, being able to measure and
- 12:47understand customer preferences and the reality of Eden.
- 12:51And the reason why it was difficult, it may be still is
- 12:54difficult in some ways is because of heterogeneity,
- 12:59because customers are so different than 0 to customers
- 13:02that are the same. And, you know, I always tell my
- 13:04students to know they and always Find within my class.
- 13:07This type of students, those who treat the Apple Store, like a
- 13:12like a called. And those who would never be
- 13:14caught dead in an Apple store, right?
- 13:16And the marketeers needs to understand this and he's trying
- 13:19to stand this often with limited data, depending who you are, if
- 13:22your Google or Amazon, you have a lot of data.
- 13:25If you just, if you are media Outlet, you may have less data
- 13:29about about the customer and who they are and we tend to see
- 13:33unlike. If you go to data for example,
- 13:35in finance, I can always Go one more year and have longer data
- 13:39about this about stocks, right? I can also just do an analysis
- 13:43of the entire Market when it comes to stocks, when it comes
- 13:48to to Consumers. A I cannot do data just on the
- 13:51entire Market because of the heterogeneity that I'm talking
- 13:54about. I do need to understand each
- 13:56customer in their own preferences and I'm limited with
- 13:59respect to the history because if I'm talking about a travel,
- 14:02for example, if I'm Expedia one of these dry only see EU, you
- 14:07know, five ten times, that's all I have.
- 14:09So I need to work with that length of history of what I
- 14:12observed about the customer, but and that's where economics helps
- 14:17so. Or statistics, there is a
- 14:19trade-off when it comes to data between how good your data is
- 14:25and how complicated your model needs to be the better data, you
- 14:29have the simpler, the model you can use.
- 14:32If we run the A/B test of a just talked about before.
- 14:35All I need to do is compare average.
- 14:37Is a customer in the control but that much customer in treatment
- 14:40but that much, that's it. I mean, level of math, that that
- 14:44is, you know, a sixth grader. Could do if I need now to build
- 14:48a model where I am again a Expedia and I'm trying to
- 14:52understand customer preferences from this visit and the previous
- 14:56visits and so on, that's where I need a much more sophisticated
- 15:00type of modeling and type of analysis.
- 15:04The other, maybe distinction I want to make about data.
- 15:07Is it extinct distinction between structure data and
- 15:09unstructured data until fairly recently, Circa 2010, most of
- 15:16our analysis was done on what is called structure data structure.
- 15:20Data are numbers data that comes in the form that we've by the
- 15:23way, generally think about data in Excel, or in a table with
- 15:27numbers in it. But if you think about it,
- 15:30majority of the data we have as business people, as marketeers
- 15:35is actually unstructured data, it can Ones in text image,
- 15:40video, Audio customer calls us to the call center.
- 15:43And we have data in terms of tens of thousands, if not
- 15:47hundreds of thousands of ads that we can, we can analyze ads
- 15:50are images and text, we have company reports, different
- 15:56estimates, depends who you ask, but 80 to 95 percent of data
- 15:59available. For business is likely in the
- 16:02form of unstructured data in text audio video and image.
- 16:07And it is only since 2010 that we actually know how to analyze
- 16:12this data at scale. I mean I'm structure that
- 16:14existed probably since the Ten Commandments right.
- 16:17We're waiting on a stone but but really they build it or to
- 16:21analyze this data at scale, came with machine learning type
- 16:24machine, learning methods, type tools and and we seeing more and
- 16:29more companies are using it. And, of course, recently, the
- 16:32whole chechi PT and generative AI is an example of Jing,
- 16:37unstructured data, Amy your marketer.
- 16:40I'm assuming that you are not and expert statistician and data
- 16:48scientist. And so given what ODed was just
- 16:53describing in terms of the quantity of data, the quality of
- 16:59data, what can marketers do to take advantage of these
- 17:04important points? But without Without having to be
- 17:10a deep level statistician. For example, don't be scared of
- 17:13data. I'm not a statistician as you
- 17:16very correctly, pointed out, I never have been.
- 17:19But I think for marketers, when we talk about quantitative
- 17:23marketing or we talked about data-driven marketing, for some
- 17:27especially individuals early on in their career, it sounds like
- 17:31they're going to have to be gathering the data sets.
- 17:34It sounds like they're going to have to be building the model.
- 17:37It sounds like they're going to have to be interpreting.
- 17:39And I think we're very lucky in that, we have a whole generation
- 17:43of individuals who understand data, who are data scientists,
- 17:48who are insights and research-driven colleagues.
- 17:51And the more that we can develop very close relationships with
- 17:55those individuals as marketers the better.
- 17:57Because most marketers, I know aren't as you said
- 18:01statisticians. But what they do know is how to
- 18:04look at data and I know we're going to talk a little bit about
- 18:06this and and say, well, based on my understanding of the
- 18:09audience, this seems odd, let's explore that or they have a
- 18:14question that they know they want to answer.
- 18:16And they know that the data is an important part of that and so
- 18:20they can ask for help in building the model.
- 18:22So the first is, you know, you don't have to be the expert.
- 18:25Don't be scared, don't not use it.
- 18:28Use it, find a partner, find a colleague, you know, find
- 18:31somebody who is the expert and the second is think about the
- 18:35data in Meaningful way. So you as a dad said, there's
- 18:41infinite data available so it really is about what are you
- 18:46looking for? And then when something and I
- 18:48know Dad's going to talk a little bit about this, when
- 18:50something doesn't feel right, ask more questions and a good
- 18:54example of using kind of numbers and qualitative elements
- 19:01happened a couple of years ago. I worked in a publishing company
- 19:05and it's very common for people to unsubscribe from
- 19:09subscription-based businesses, including magazines.
- 19:12And so, you build a model and, you know, that churn will be X
- 19:16based on historical data. And you know, that there might
- 19:20be some macro or micro conditions that will impact that
- 19:23Year's churn and you go out and you acquire customers, knowing
- 19:29this percent is going to leave and you need to have, you know,
- 19:33X percent of new people coming in and subscribe.
- 19:37But as a marketer it's not just about making sure that you can,
- 19:42you know, fix the revenue equation on both sides.
- 19:45It's also about understanding your customers.
- 19:47So you have this model we have this model, we know that X
- 19:51percent, you know, are unsubscribing.
- 19:53And so then you have to dig deeper to think about why what
- 19:56is it? That what's the reason that you
- 19:58know, they've chosen to cancel this subscription and we could
- 20:01get a little bit of insight in asking structured questions and
- 20:06our Hypothesis was it was a value situation.
- 20:10Somebody decided that that product that the magazine was no
- 20:13longer worth what they were. Paying feels pretty simple,
- 20:16right? But we actually had done some
- 20:18brand research and had some qualitative components and what
- 20:22the qualitative components told us was that actually had nothing
- 20:26to do with value, it had nothing to do with the product, it was
- 20:30about unfinished ability. So people felt that this
- 20:35particular magazine Worth the price worth, you know, worth had
- 20:40great value, but they couldn't finish it, and they would stack
- 20:43it up in a pile. And they would say, I'll come
- 20:46back to it, and they didn't. And every time they looked at
- 20:48the pile, it crew and their guilt.
- 20:51So this is an emotional reaction.
- 20:52Their guilt over not finishing, the magazine was actually, what
- 20:56was contributing to canceling subscriptions, so that Insight
- 21:02is not in sight. You were ever going to get from
- 21:04structured data. It's not even in sight, you were
- 21:06to get from some of your survey questions.
- 21:09It came exclusive these through some qualitative conversations
- 21:13and there were some solutions to that digital.
- 21:18Combated the problem a little bit because people can't finish
- 21:20the internet. So if you can't finish a digital
- 21:23issue of a magazine, you didn't feel as guilty.
- 21:26You know, you can introduce newsletters and quick quicker, B
- 21:29of content and summaries and things like that.
- 21:32But I think the other important thing is I and I want to really
- 21:36Reiterate. This is that there's the
- 21:38structured data. There's everything, you know,
- 21:40there's quantitative and then there's this other qualitative
- 21:44element. That is equally as important for
- 21:47marketers to pair together and I think it is important that will
- 21:50think about data in the most General sense of the word, a
- 21:53Tamiya conversation with a person is data or just maybe
- 21:57want to touch on one thing with respect to the, you know, when I
- 22:01build the end model statistical model and I truly mean it.
- 22:05I would I would actually Really go to a me, first to tell me
- 22:09this model is right, before I would go to a statistician to
- 22:12tell me whether it, whether they think my model is, right,
- 22:14because the True Value comes from, let's say it's a model
- 22:18that predicts click click through on an ad and the model
- 22:21showed, you know, twenty percent click-through rate.
- 22:24It's the decision wouldn't know to tell me that 20% clicks.
- 22:27All right, on an ad is unheard of, unless you're Ed truly
- 22:30offered money to customers. No one.
- 22:33There's not, there's no one in five.
- 22:34Customers click and click on a man.
- 22:36I needed. Cat ears.
- 22:37I need a me to tell me. I don't know what you've done
- 22:39wrong, but I can tell you this is wrong because it's it's
- 22:42impossible. That customers. 20% 1811 in five
- 22:45customers would click on your ad.
- 22:47We have a couple of questions that have come up on Twitter,
- 22:51actually, to folks, Chris Peterson, and our Salon Khan,
- 22:54who are both regular listeners of cxo Takin.
- 22:57So thank you guys are asking independently.
- 23:00Essentially the same question and that is, where do we draw
- 23:05the line for? Privacy in terms of data-driven
- 23:09marketing, that's from Chris Peterson.
- 23:11And arsalan, Khan says how, and who should strike a balance
- 23:15between collecting data that can affect privacy and who decides,
- 23:21what's private? What's not?
- 23:22And also, then you have the issue of biases.
- 23:26So can we talk about that? And then we'll switch over to
- 23:29quantitative intuition privacy is an extremely important topic
- 23:34to think about to discuss. To realize that that companies
- 23:40have huge responsibility when we are collecting individual data.
- 23:47I think there are, there are there is the issue of data being
- 23:50shared and then there is the issue of data being used by, by
- 23:54the company. I mean despite what actually
- 23:55most people think most, at least the large companies do not sell
- 24:00data. Now, I'm not sure it is, by the
- 24:02way, being done because and because of pure Ethical issues
- 24:07of because data is just way too expensive to sail, the use of
- 24:11data by by company. By again, the thing about the
- 24:13top companies like Facebook, Google Amazon is so important
- 24:18and useful for their own purposes that actually selling.
- 24:20It would be, would not be very wise.
- 24:23So generally, at least from these companies data generally,
- 24:26do not do not flow around. It generally does tend to stay
- 24:30Within These these companies but even there they should treat it
- 24:34with the with the Ability that these data should be treated
- 24:38with and that's where recent regulation came about first
- 24:43party data where you're allowed to use your own data.
- 24:47But you're you're not allowed to actually use someone else's
- 24:50data, which by the way in, and of itself, there is an
- 24:52interesting tweak there. It's on the one hand, it sounds
- 24:56right. It's indeed protects customers.
- 24:59In terms of their data, being being in floating around but it
- 25:04also increases, maybe the Gap between the top companies that
- 25:08have really good first-party data their own data and the
- 25:12smaller companies that don't have their own data.
- 25:14And that's what that where they could have closed the Gap by
- 25:17getting data, Maybe from, from someone else.
- 25:20I do think that we shouldn't let the cat Garden milk there.
- 25:25I think we do need regulation there, right?
- 25:27And Europe is much better than the u.s. in that science.
- 25:30And we've made that mistake previously, with social media,
- 25:34where we let the the Guard themselves.
- 25:38We've made it with the privacy and I think we need to move
- 25:40there and I'm happy to see if there is generative an RA.
- 25:43I this conversation is starting already now.
- 25:45And not once this is fully fully out there and I mean, I don't
- 25:49know if you have other selves. Yeah, I mean it going to an end
- 25:52even kind of larger question is, who owns the data?
- 25:57We're now in a world where our Behavior has been tracked for a
- 26:00very long period of time, and there are reams of data, people
- 26:04probably know that I prefer baked macaroni.
- 26:06And cheese, you know, over brussels sprouts, but who
- 26:09doesn't, you know, that, that data is gone.
- 26:12It's with somebody, I hope I get an ad for macaroni and cheese.
- 26:16I will let everybody know shortly, but I think there's a
- 26:21larger question as we think about a next-generation, right.
- 26:24Let's talk about kids for instance, for whom maybe there's
- 26:28some data, there's not a lot who owns that data, who should own
- 26:31that data is my child, the holder of that data or is it the
- 26:36platform? Who's on which they're engaging,
- 26:38and there have been these really large high-level conversations
- 26:42around, you know, should I be able to own my data and I choose
- 26:45where when, and how to monetize that?
- 26:48Or is there a trade-off? I get to go and watch cat videos
- 26:51or, you know, whatever it is videos that I like and in
- 26:55exchange, I'm giving somebody an insight and I know that because
- 27:00I understand that's, you know, the way that a lot of practices
- 27:04work because I'm getting a pop-up that says There are
- 27:07cookies and I'm willingly engaging in a behavior.
- 27:09So I think there's both the kind of what is happening now.
- 27:13And then how do we think about data the value of data and
- 27:17ownership of data for a whole new generation, little bit off
- 27:21topic. But I think an important
- 27:22conversation for any of us who are, who are using data and
- 27:26whose data is being used in traded, in some way and we need
- 27:30to move too much more of an opt-in environment, right?
- 27:33We're at least, we make the decision of whether to Charity.
- 27:37Obviously, this is a crucially important topic.
- 27:40This issue of data privacy and data ownership.
- 27:44And we're not going to solve that one now, but we can get a
- 27:49better understanding of is, oh, Dave.
- 27:52Aude of your concept of quantitative intuition.
- 27:56What is that, and why does it matter what even Division?
- 28:00I think this particular is important in the world of
- 28:03marketing. Where again, there is a, the
- 28:05more If you will, they the the unstructured data that Amy
- 28:10talked about or the the conversations that we have, we
- 28:13tend to use our intuition regularly in our private lives.
- 28:17You know, we talked about surprising thing, an interesting
- 28:20thing. Sometimes we even call them
- 28:22gossip, but we are very worried of using our intuition, when it
- 28:27comes to the world of business, to our to our work life.
- 28:32And the idea here is not to just, you know, trust your gut.
- 28:35The idea here is You are approaching data, bring in your
- 28:40your judgment into it. In fact, we have to bring in
- 28:43your judgment into it. So on on its surface
- 28:45quantitative in intuition sound like an oxymoron but in I, in
- 28:50fact, together with my co-authors for the book, Chris,
- 28:52Frank and permanently, we believe that not only that,
- 28:56they're not oxymorons, particularly the level of
- 28:58leadership. It's the only way, the only way
- 28:59to make decisions is to combine the data with a good sense of of
- 29:04judgment. And when I say judgment, I mean,
- 29:07Intuition or judgment. This could come actually, in in
- 29:11at least three stages of the decision making process in the
- 29:16questions we asked in being a very clear about what problem
- 29:19we're trying to solve more often than not, we are just going on a
- 29:23straightening. Oh, we've been collecting all of
- 29:25this data, then got to be something interesting in.
- 29:27Its right, we need to guide the process.
- 29:29To me, the one of the biggest Nono in making data-driven
- 29:32decision, making is we expect the data to provide both a
- 29:35questions and There's we should ask questions then.
- 29:39Hopefully data. Can provide answers.
- 29:41In fact, we've learned it better than ever since November of
- 29:45twenty twenty with with a 2022. Sorry with GPT.
- 29:49We need to provide good proms. If you want to get good answers,
- 29:51we need to ask good questions. Second is something you already
- 29:54talked about how do we interrogate data and, and
- 29:57exactly what I mentioned that you do need the context as
- 29:59humans. We are very good in context and
- 30:02particularly, if you are an expert in your domain, as Amy is
- 30:05in, Amy is in in, in marketing, it's much easier for her to look
- 30:09at the model of very sophisticated analysis and
- 30:13interrogated. Not from the p-value of from you
- 30:15didn't use this three-letter acronyms or another, but from
- 30:20this number doesn't make sense. I don't know what you've done
- 30:22wrong, but I can tell you that that it doesn't make sense.
- 30:25And finally, you're in a crucial place for judgment is in the
- 30:29synthesis of the information generally data and Analysis will
- 30:33tell you the what it would not tell you the server.
- 30:36We need to do it. What does it mean?
- 30:38And then now what are we going to do about it?
- 30:40And these are different components where we want to
- 30:43combine the quantitative together with intuition.
- 30:47I use the so and learn a lot all the time.
- 30:50After you taught me that it's those two questions for
- 30:53marketers. Truly will change the way that
- 30:57you approach your, your data and your strategies.
- 31:01And in fact we said that one of the reasons we went on this
- 31:03journey of quantitative intuition was we were simply
- 31:06tired of meetings of horrible meetings or meetings that go on
- 31:10the water. And so we had a meeting and
- 31:13someone shows data and now we're going to spend an hour talking
- 31:16about the water. In fact, there is a Persona, we
- 31:17talked about in the book, we call them the seymours, the
- 31:21seymours in the organization are the ones who in every meeting of
- 31:24only one comment can, I see more data?
- 31:27And then we can have meetings over meetings and postpone any
- 31:30decision Forever by asking for more data by by keep digesting.
- 31:35And, and, and slicing, and And slicing again the data moving
- 31:39from from the walk, right from what's in the data to?
- 31:43What does it mean? Right?
- 31:44What are the implications? And again this is a place where
- 31:47you know people are even fearing a lot with chat GPT and so on
- 31:50with generative AI what's left for us as human humans.
- 31:54If all you do is the what, yeah, you will be replaced by a
- 31:58machine but if your focus is on the so, what on What does it
- 32:02mean on the now? But what are we going to do?
- 32:04Not yet. I mean, I'm not saying that
- 32:06machines will Not get there at some point but not at least in
- 32:09the near future are so and calm comes back and he says, as AI
- 32:14continues to be used more and more, do we really need
- 32:18marketing people? If a I can understand the
- 32:20context around that data. Yes.
- 32:24Definitely. You do you do and I'm sure your
- 32:28dad has a lot to say as well. I mean, where we are in terms of
- 32:32chat GPT right now is I would say it is a Full assistant in
- 32:38many ways, you know, having used it myself, my team uses it, it
- 32:43gets you, I'm making this number up but let's say a third of the
- 32:46way but it's not replacing what we're doing, it's not replacing
- 32:51the knowledge that we have. And it's not doesn't always get
- 32:55the context and the nuances that are very specific to humans and
- 32:59then even more, you know, kind of granular or smaller there
- 33:06more, Subsets of nuances amongst different audiences.
- 33:10Now, you could argue all of that is data, and overtime, maybe
- 33:13those things could be put into a formula and maybe it could be
- 33:17understood. But I think we're still a bit
- 33:20far away from saying this will resonate because or from
- 33:26understanding that a word, that means something in this context
- 33:30can be taken in an entirely different way in that context.
- 33:35And that context is, Medic. And so yes, is it helpful for
- 33:39instance, first summarizing some research that you might have.
- 33:42Absolutely. Does it give you something to
- 33:45gut? Check your approach?
- 33:47Can it provide more information? Yeah, it can.
- 33:50But we're not at the point where it could run this sophisticated
- 33:54campaign for you and get equal or greater results.
- 33:58And I think Michael you you really hit it on the head with
- 34:01it within the day Ward context. I think this is the key.
- 34:04The key is truly context. Both in terms of, by the way,
- 34:08how these methods have improved. So the difference between, you
- 34:11know, GPT to which was the previous version and the GPT
- 34:15three and a half, which is Chad upto.
- 34:17Now we already GPT for is context meaning and what I mean
- 34:21by that is in order to predict the next word on door in order
- 34:25to interpret a board understand the world with these tools.
- 34:28Do they take the previous that many words to understand in the
- 34:32context of the board and the reason why and it truly was.
- 34:35I mean Church uppity Was a huge leap in Innovation.
- 34:38Over the previous versions is, literally because of context
- 34:42because they had they use usable is called 8000 tokens.
- 34:46Think about 8,000 tokens, something like 6,000 words
- 34:49because it includes periods and so on but six the previous six
- 34:54thousand words to understand a particular word, the previous
- 34:57versions, for example, GPT to use two thousand words to
- 35:01understand any particular word. Is humans.
- 35:04We are tremendous actually in doing that in in context, I'll
- 35:07give you two examples. The first example is when I say
- 35:12the word model and in fact, it Michael, when you use the word
- 35:15model, both of us understood that in the in this context, we
- 35:18are not talking about the fashion type.
- 35:19We're talking about the nerdy type, right?
- 35:21Because we have enough context to understand that in this
- 35:24conversation, unless I'm going to give you a real clue with my
- 35:28previous words that we are talking about the fashion type
- 35:31model. We are talking about the nerdy
- 35:33type. Type right?
- 35:34And and in fact, we don't even understand how good or how our
- 35:39brain actually in understand context.
- 35:42I'll give you an example for that.
- 35:44Think about a toddler at one and a half year old does not have
- 35:47the intelligence to speak yet. And we show them illustration in
- 35:51a book and the book has an illustration of a cat.
- 35:55And we tell them. This thing does meow.
- 35:57And then we show them illustration.
- 35:58The same book of a dog kind of book about animals as we often
- 36:03read. You are a one and a half year
- 36:05olds and he tells them. Well this thing does wolf and
- 36:08then we show them maybe 45 times illustration, different
- 36:11illustrations from different books and so on the next day
- 36:15we're going to take them to the street.
- 36:16They cannot speak here they're going to see a cat or dog a
- 36:19real. A real cat and a dog will first
- 36:21time ever they see a real one and they're actually going to be
- 36:24up meow and wolf without even again they can speak yet but
- 36:29they do it with five observations a few years back.
- 36:33The first time that machines went close to human level, in
- 36:37detecting cats from dogs from images and they've done it
- 36:41because the researcher made available to million
- 36:44observations take by humans million of cats and dogs.
- 36:48How can a human do it with four five observations and a machine
- 36:52needs to million. We don't understand how we do it
- 36:54with only four, five observations.
- 36:56If we did, we would have trained machines to do it because it
- 36:58will be very useful to have less data.
- 37:01And I think that's gives you the the The understanding of why
- 37:04when it comes to The Sword. And the novel words, we don't
- 37:07have enough data of similar. So what an hour maybe if you are
- 37:12a physician and we have enough decisions of the kind given
- 37:14symptoms, maybe it works. But for typical business
- 37:18decision, which is doesn't come that often.
- 37:22It's unlikely. It's unlikely that at least not
- 37:24with the current tools and where we are today or even in the near
- 37:27future that machines would have enough context is 8000 context.
- 37:32But repeatedly, Only with enough data to truly go to to that
- 37:36step. Again, may happen at some point,
- 37:38not yet. If you want one place, where it
- 37:40did happen already and happened before generative, AI online
- 37:45advertising, the allocation of Heirs of which atoms.
- 37:47And seeing when I go to Google, or when I go to Amazon or
- 37:52Facebook is almost fully automated apart from the choice
- 37:56of the creative of The Advertiser.
- 37:58Yeah. That's has been already
- 38:00automated because the rules are fairly clear there isn't a Oh so
- 38:04what does it mean? Well it means that this end is
- 38:06likely to lead to a higher click-through rate, then another
- 38:09ad, that's a predictive model machine.
- 38:10Learning can do that already, how can marketers avoid being
- 38:15data myopic, which is to say losing sight of the fact that
- 38:20business decisions involve people circumstances.
- 38:23We use that term context and not just numbers when marketing has
- 38:29a seat at the table and is involved.
- 38:33And in the Strategic decisions things like growth or new
- 38:38audiences or understand where value is derived for a company
- 38:43that's when they can put together the pieces and they're
- 38:46not only seeing their small part.
- 38:49So I think being part of having a marketer, truly, truly
- 38:53involved at the senior level, with the seat at the table,
- 38:56who's, you know, just as involved as anyone else in the
- 39:00growth of the company is really important because it then
- 39:05becomes top-down. You ask your team, not to look
- 39:09at this slice. You ask your team to think about
- 39:11how it connects to something larger.
- 39:13Oh, Dad, how can organizations strike, the right balance
- 39:17between relying on quantitative marketing techniques, versus
- 39:22trusting, their own intuition and experience bringing the two
- 39:26together. Ask yourself as you looking at
- 39:28data. Ask yourself what surprised you
- 39:32And it's amazing. How this fairly simple.
- 39:35Deceptively simple question often cut straight to the
- 39:38straight to the chase. You're either finding a mistake
- 39:42or you're finding an Insight either way, you benefit.
- 39:46And so, in other words, don't trust your gut.
- 39:49Trust your doubts, look for these days surprises.
- 39:52In the data, Amy, you're going to get the last word here.
- 39:55What advice do you have for marketers given everything?
- 39:58We've just been talking about your best friend, is the
- 40:01personal? One who can build the models,
- 40:04your you need to be embedded in business decisions to drive True
- 40:11Value. And I don't think that chat GPT,
- 40:15we'll take our jobs, but if you don't know how to use it, to
- 40:19make yourself more efficient, or more effective as a marketer, it
- 40:22will so it's not a substitute but it's an important complement
- 40:25that we all have to be using and working with them on a daily
- 40:29basis. Can I accurately Phrase that as
- 40:33saying, be part of the business, understand the tools.
- 40:37Understand the data and bring all of that together to make the
- 40:40decisions that rely on both the data and your experience with
- 40:44the business and with the context even better.
- 40:49And with that a huge thank you to ODed netzer and Amy.
- 40:55Jake from the Columbia Business. School Columbia University.
- 40:59Thank you both for taking the time.
- 41:02Time to be with us and share your expertise with us today.
- 41:05Well thank you, it's beneficial. Yeah thank you by going to thank
- 41:07you for the audience and thank you to the audience.
- 41:11You guys are awesome and your questions are great.
- 41:13Now before you go, please subscribe to our YouTube channel
- 41:18and hit the Subscribe button at the bottom of our website.
- 41:22So you can get our newsletter and we can keep you up-to-date
- 41:25on upcoming live shows. Just like this one.
- 41:28Thanks so much. Everybody, check out cxo
- 41:29talk.com and we'll see you next time.
- 41:31Have a great day.