Latest / The Marketing Meeting with Itir Eraslan / 37. The Great AI Workplace Experiment with Iavor Bojinov
Transcript
- 0:00Iavor Bojinov: Right now we're at the stage where AI can solve specific tasks.
- 0:04So what I tell companies is you should look at all the tasks that your employees
- 0:10are doing, and then basically wildly experiment across as many of those tasks
- 0:16as possible to try to identify where AI is really able to create a lot of value.
- 0:25Itir Eraslan: Hi, this is The Marketing Meeting and I'm your host, Itir Eraslan.
- 0:30Every two weeks I meet with experts and we talk about topics.
- 0:33related to brands, marketing, and businesses, and sometimes
- 0:37add random lifestyle topics too.
- 0:39I hope you enjoy the show.
- 0:46Welcome to the marketing meeting.
- 0:48And I'm happy to host Iavor Bojinov.
- 0:50He's an assistant professor of business administration at Harvard business school
- 0:55and the co PI of the AI and data science operations lab at the school as well.
- 1:00Uh, welcome to the marketing meeting.
- 1:04Iavor Bojinov: Thank you so much for having me.
- 1:05I'm excited for our conversation today.
- 1:08Itir Eraslan: I'm glad that Felix got us introduced a while ago.
- 1:12And then because I was pushing him with some questions on AI, and then he says,
- 1:17like, of course I can answer those, but there is another expert that would
- 1:21be really helpful for you to discuss.
- 1:24That's why, that's how I think we got connected through Felix.
- 1:27And I'm super excited.
- 1:27Saying hi to Felix from here.
- 1:31So I'm in, um, many of my connections are from LinkedIn and that's the main platform
- 1:36that I'm announcing these podcasts.
- 1:38And I know that you worked at LinkedIn as a data scientist.
- 1:42So I decided to start with a question from there.
- 1:45I remember that in one of your research projects, you highlighted
- 1:49how some LinkedIn connections matter more than the others.
- 1:53Uh, can you share some details about that project with us?
- 1:57Iavor Bojinov: Yeah, absolutely.
- 1:58So maybe it's just helpful to just give you a little bit more of that background.
- 2:01So I have a very non traditional background for business school professors.
- 2:05I actually have a PhD in statistics, uh, from Harvard, and there I really
- 2:10sort of studied how to handle missing data and how to do causal inference or
- 2:15trying to understand cause and effect and You know, through that a lot of
- 2:18experimentation, uh, and during my PhD, I worked with a number of different
- 2:22tech companies, Google, LinkedIn.
- 2:24Uh, and then when I finished my PhD, I actually joined LinkedIn full time.
- 2:28Uh, and I'm a huge fan of the platform.
- 2:31I still have many friends that work there, many connections.
- 2:35Uh, and you know, one of the most central, uh, ideas of LinkedIn is,
- 2:41so their vision is basically to create economic opportunity for
- 2:43every member of the global workforce.
- 2:46And the way they operationalize that is to basically build a social network,
- 2:51uh, that allows people to sort of connect to people that they know.
- 2:55Uh, and they do this through their proprietary algorithm known as
- 2:58the people you may know algorithm.
- 3:00So you've probably seen this on their website, it sort of just recommends,
- 3:04okay, you might know these people.
- 3:06Um, and, and the idea there is that if you're able to expand your network,
- 3:10you have access to more opportunities and you'll be able to get more jobs.
- 3:16Right.
- 3:16That's kind of all get better jobs.
- 3:17And this all goes back to this theory, uh, known as the theory of weak ties from
- 3:22rather better in sort of the seventies.
- 3:25Um, and what he did was he basically interviewed a lot of people here
- 3:30and actually in Massachusetts, uh, right outside of boston.
- 3:33He asked them, you know, where did you find your job?
- 3:36How did you find this job?
- 3:39Who connected you to this opportunity?
- 3:40Mhm.
- 3:41And what he found was that it wasn't sort of your best friends
- 3:44that helped you get jobs.
- 3:45It was people that were maybe one step away from you.
- 3:48So it was, you know, someone in your area that maybe you met at a conference
- 3:53and then you stayed connected with them.
- 3:55Or it's, you know, maybe your career mentor, or maybe it's, uh,
- 3:59your friend's, uh, uncle or best friend or something like that.
- 4:03And those people seem to be the ones that are much more useful in getting jobs.
- 4:07New job opportunities.
- 4:09And since the seventies, you know, this is probably one of the biggest
- 4:12theories in social science, probably one of the most highly cited one.
- 4:17And this is sort of the motivation for all these social networks, et cetera.
- 4:20Come back to this theory.
- 4:22Um, and what we wanted to do was basically try and test if this theory holds out.
- 4:27In the digital world.
- 4:29And what we did was we basically leveraged a number of historical experiments that
- 4:33LinkedIn had run, uh, that just randomly vary the algorithm that's being used under
- 4:38the hood for the people you may know.
- 4:41And sometimes that algorithm would recommend you to sort of do
- 4:44what's known as triangle closing.
- 4:45So connect you to people who, uh, you already have many mutual connections
- 4:49with, and then sometimes other versions of the algorithm will sort of connect
- 4:53you to people that were further afield.
- 4:56And this is basically an experiment because.
- 4:58You know, LinkedIn would always try many different versions of this algorithm to
- 5:01try to figure out, you know, which version works best for its customers, right?
- 5:07Um, and what we found was, and we were able to sort of, um, update this theory
- 5:12of weak ties a little bit was that it's actually a little bit of a U shaped
- 5:15relationship where people who are really close connections, you know, think of
- 5:20people you went to, to university with or people you're currently working with.
- 5:25They're not that useful because they have all the same information as you.
- 5:29They're not giving you access to something new.
- 5:31And on the other extreme, people who are super weak ties, where you just like,
- 5:36they're completely different industry.
- 5:37You have no mutual contacts.
- 5:39They're not that useful because they like the information they
- 5:42have is just too far afield.
- 5:43And what we found was that there is this like sweet spot of people
- 5:47who are kind of in between.
- 5:50The really weak ties and the really strong ties, people we call moderate ties, and
- 5:54they are an order of magnitude more likely to help you find a new opportunity and
- 5:59get a sequential job transition, which means sort of get a job after you've
- 6:04connected with them at the same company.
- 6:06And, you know, that's kind of helped us verify that LinkedIn
- 6:11is creating a ton of value.
- 6:13And they actually help them realize that actually these
- 6:17algorithms are super powerful.
- 6:19Uh, and they can tune them to really help them actually deliver on their vision.
- 6:23So there were lots and lots of learnings from it.
- 6:26Uh, and you know, I really, it was amazing.
- 6:30Actually, the linkedin let us publish this research, right?
- 6:32Because we did that internally.
- 6:34And then they let us actually share that with the broader community.
- 6:37So that's kind of what that work is about.
- 6:39. Itir Eraslan: So in that case it means that we have to, uh, make sure that we
- 6:43pay attention to people you may know.
- 6:45Uh, yeah.
- 6:46Apart, uh, in our , in our app.
- 6:50Iavor Bojinov: AB absolutely.
- 6:50I mean, not so of course we shelter for work.
- 6:53Um, but people have shown that, uh, we ties are more useful
- 6:57when it comes to innovation.
- 6:59So you sort of get new ideas from these people.
- 7:02Uh, and basically any economic outcome you can think of, uh, has pretty much
- 7:07been justified through weak ties.
- 7:10Uh, so these are super powerful and I would, I always encourage everyone to,
- 7:14you know, broaden and open up their social network and actually LinkedIn
- 7:18did a major media campaign around having open social networks so that
- 7:23you're connected with many people.
- 7:24It's not a closed social network where you just have, you know,
- 7:27a handful of really strong ties.
- 7:29And that's what's really good for the economy.
- 7:31Itir Eraslan: Perfect.
- 7:32So on podcast, I try to talk with experts that are dealing with AI that are all
- 7:39day to day in the AI world, but I'm.
- 7:42At the end, I'm tying it up to marketing because this is a podcast about marketing
- 7:47And many of the companies that I work with or interview with or talk to Their
- 7:54biggest concern is actually about Which platform, which softwares to adopt,
- 8:01which parts of the business to integrate AI, and also like many of them have
- 8:06a question mark about where to start.
- 8:09So that's why I would like to, you know, start with that and then
- 8:12link it to the marketing piece.
- 8:15So when a business considers building a AI team or AI systems in place for their
- 8:21companies, what should they consider when they evaluate new tools or softwares?
- 8:27Iavor Bojinov: Yeah, that's a great question.
- 8:29And it's one I hear all the time.
- 8:31Um, the way I like to think about it is right now we're at the stage
- 8:37where AI can solve specific tasks.
- 8:41So what I tell companies is you should look at all of the tasks that your
- 8:46employers are doing and then basically wildly experiment across as many of those
- 8:52tasks as possible to try to identify where AI is really able to create a lot of that.
- 8:59Places where I've seen companies fail is when they try to completely.
- 9:03Automate a whole function, you know, I've seen the number of companies say,
- 9:08okay, we're now going to do AI marketing.
- 9:11And we're going to use chat GPT to create our adverts, create
- 9:14our creatives, everything.
- 9:16It will be done by that.
- 9:17And they try to automate the whole process and inevitably it fails because
- 9:21Again, AI is really good at very, very niche specific tasks right now.
- 9:27And it's not very good at linking those tasks.
- 9:29Now that's going to change in time, but right now we're just not there.
- 9:32So what I say to companies is.
- 9:34Experiment wildly, try, you know, as many different task based
- 9:39solutions as possible, and then just be really scientific about it.
- 9:42I think this is one thing that's great about the marketing world over the
- 9:45past, maybe over a decade now, they've been really pushing for this sort
- 9:50of scientific approach to marketing, where you try to quantify everything.
- 9:53You try to measure, you know, the ROI on this, which ad is better, right?
- 9:58So I think you need to bring that experimentation mindset that's really
- 10:03What is really adding value, but again, it has to be at the task level and
- 10:07then for each task, you have to think about are we going to automate it?
- 10:10Or are we going to try to augment it?
- 10:12Is this a task that's going to be fully delegated to the AI?
- 10:17Or is a task where maybe just the first draft will be done by the AI,
- 10:21and then the human's going to come in and fix it up and improve it, etc.
- 10:25So, again, I'm giving you sort of very vague ideas.
- 10:28answers, but it is a really broad question and, you know, experimentation, trying
- 10:33things out and being at the task level is really the big sort of key takeaways.
- 10:38Itir Eraslan: I think your point about like, seeing it as tasks rather than,
- 10:43you know, trying to, Um, combine it as a whole, uh, is an important one because,
- 10:48you know, even in marketing, for example, there are places that you can automate
- 10:52with AI, but there are really bits of the pieces that you cannot really automate.
- 10:58And it's like, it's a big orchestration work.
- 11:00Uh, so, You know, that's, that's the dilemma that I'm facing right
- 11:04now, for example, for companies that I'm trying to integrate AI to.
- 11:08So, you know, you automate email or newsletters or those type of things.
- 11:12But, uh, when you think about as a consumer behavior, for example, how do
- 11:16we integrate AI to consumer behavior, to understanding consumer behavior,
- 11:20which has nothing to do with automation, email, uh, automating emails is,
- 11:25is a very, uh, Basic task of that.
- 11:28You can integrate AI actually.
- 11:30So in that sense, you talked about return on investment, which is like
- 11:35a hot topic for him for a marketer.
- 11:36It has always been the case.
- 11:38Okay, what's the return of investment for marketing?
- 11:42And then on the business side, there is one other question is that is AI
- 11:47economically used, meaning that does it Bring any economical value, uh, to
- 11:52the business since you see everything as a whole from the business side,
- 11:57how can you assess the economic impact of AI within the business?
- 12:01Iavor Bojinov: Yeah.
- 12:01So there was sort of a few questions within that broader question.
- 12:05Um, maybe just to quickly comment on what you're saying about
- 12:09sort of the automating, like the email and the consumer behavior.
- 12:14I think that's exactly right.
- 12:17That is the strategy of going task by task.
- 12:20And sometimes, as you said, AI is really, really good at, and we
- 12:24know that, for example, generating first draft of mood boards.
- 12:29It's really good at that because You know, previously a marketer who spent
- 12:32a lot of time on Pinterest, et cetera, trying to find the right images, but
- 12:36they know what they're looking for.
- 12:38So now they can just type it out.
- 12:40It gets them the draft, but that's not the final output, right?
- 12:43The final output still requires taking that.
- 12:47Uh, idea.
- 12:48That's that's where, you know, this little bit of augmentation
- 12:52could save huge amounts of time.
- 12:53So connecting that to the ROI, uh, comes back to this experimentation mindset.
- 12:58So I'm working with a number of companies where we're trying to bring rigorous
- 13:03experimentation to the deployment process.
- 13:06Of AI and you can do this either through like mini hackathons where you have
- 13:11people tasks Using the tool versus not using it Uh, there's a very famous now
- 13:18of a year ago bceg and hbs run this joint experiment where they had basically
- 13:23consultants Uh, do a number of different tasks, uh, and they measured, you know,
- 13:28the quality of the output, the time it took for that and many other metrics
- 13:33to get a sense of how, um, these tools sort of impact quality and productivity.
- 13:38Now, of course, that's not the ROI, but those are inputs into the ROI equation.
- 13:44So sort of building that up to your broader question of like, what is
- 13:46the impact within an organization?
- 13:49Again, it comes back to experimentation and just being as careful as you
- 13:53can be measuring as many different outcomes around productivity quality.
- 13:59And then at the end of the day, you probably will have to do a little bit
- 14:02of a back of the envelope calculation to map that to, uh, ROI because Internally,
- 14:08it's really, I mean, it is really hard to say, okay, what's the ROI on
- 14:12this one particular employee, right?
- 14:14It's, that's a really hard equation.
- 14:16If we're deploying things to our consumers, as tech companies
- 14:20do, it's really easy to measure revenue and things like that.
- 14:23But when it's sort of internal uses of AI, That becomes really tricky.
- 14:27So that's why companies usually focus on that quality.
- 14:31Um, work satisfaction is another really big measure, uh, productivity, et cetera.
- 14:35Those are the types of metrics that companies are really measuring and
- 14:39then trying to map it back to that.
- 14:42Itir Eraslan: Because at the end of the day for a business
- 14:44to adopt something new or a new technology, it will all tie up to.
- 14:50The economic value that it creates, right?
- 14:52Like if you are able to solve a simple problem in your workflow, but just
- 14:58hiring like a part time employee, rather than adopting a whole technology to the
- 15:04whole company, then it will probably be for the time being sitting, sticking to
- 15:09hiring that part time employee and not adopting a whole new technology around it.
- 15:14I think that's something that.
- 15:15A lot of people are missing out because at the end of the day, AI
- 15:19feels quite democratized and it feels, um, you know, it has no price.
- 15:24I mean, you don't think it's like for free, but it's not for free actually.
- 15:28Iavor Bojinov: It's super expensive and super hard to do.
- 15:30And I see this all the time in the IT world.
- 15:33Where a lot of companies have outsourced a lot of their IT, uh, work and it's so
- 15:41cheap that the sort of the calculation of trying to automate any parts
- 15:47of it is just astronomically high.
- 15:50Um, And, and this is a, you know, you're kind of setting yourself up for failure.
- 15:56If that's the mindset that you have, because we also know that there is
- 15:59always with new technology, there's a little bit of a J curve where
- 16:03productivity quality, everything sort of drops initially and then it grows.
- 16:08So initially the adoption of AI is going to be painful.
- 16:13You're going to have to deal with change management, which is much harder when
- 16:16it comes to AI than traditional IT that involves a lot of retraining, et cetera.
- 16:21And you're going to take a huge cost of setting this up and then
- 16:25initial drop in productivity.
- 16:26But for me, it's still a no brainer that you need to get this
- 16:31done because The reality is this is where every company is going.
- 16:35And the question I ask executives all the time is if you were to redesign your
- 16:40company today, forget about all your tech debt, forget about everything, you
- 16:44know, you just have, you know, your, your value proposition, let's say you
- 16:48keep that, but you can redesign your operating model, would it look different?
- 16:52And the answer that I inevitably get is it will look fundamentally
- 16:57different from how it looks like.
- 16:59Hmm.
- 17:00Then they say, okay, but we can't get there because we have tech there.
- 17:03We have all these other things, but what I say to them is if you're not going to
- 17:07get there, someone else will, and someone else is going to come and be a competitor.
- 17:11And we've seen this time and time again in the rise of the internet, right?
- 17:15Like companies like Amazon were able to completely destroy,
- 17:20uh, the book retail business.
- 17:23Barely exists now because of companies like Amazon and you, you've just
- 17:27seen this time and time again, and it's going to happen again.
- 17:30So if companies are not ready to really integrate AI and accept the
- 17:34fact that it's going to be costly and painful right now, Um, they're going
- 17:39to run into trouble in the future.
- 17:40And I do think a lot of these companies are just going to go out
- 17:43of business in the next 10, 15 years.
- 17:45Itir Eraslan: Um, AI started like a, comes like a magic stick, right?
- 17:50Like it just, everything happened like a year ago and then we
- 17:53expect everything from it.
- 17:54But although, uh, some people may not like it, but I would like to talk about
- 17:59the challenges that it comes with.
- 18:02Uh, what are some of the key challenges that you see in these models?
- 18:06Absolutely.
- 18:06So
- 18:06Iavor Bojinov: first of all, AI did not come a year ago, right?
- 18:10AI has been around for a very, very long time.
- 18:12It's had many different names over the years.
- 18:15Um, and this is I mean, if we look at companies like Google, that is what
- 18:20gave them a competitive advantage over Yahoo and everyone else is that they
- 18:24were able to build predictive machines that was so powerful, uh, that they
- 18:29were able to do personalization like Netflix, for example, their recommendation
- 18:34algorithm is far superior to anything that Disney and anyone else has, and
- 18:39that's why they're worth so much money.
- 18:42Uh, so just to, you know, make that clear,
- 18:44Itir Eraslan: yeah,
- 18:45Iavor Bojinov: IGPT came out and now when I teach about AI, I don't
- 18:48need to spend the first five minutes saying AI is important and AI is cool.
- 18:52You should be paying attention.
- 18:53It's sort of for granted, but it has been around for a while.
- 18:56Now.
- 18:57When it comes to the challenges, I think there's many, many different challenges.
- 19:01The way I like to think about it is through the lens of how you
- 19:05develop a specific AI project.
- 19:08And the way I think about that is essentially it's having four steps,
- 19:12which is the selection, and that's kind of picking what projects you're
- 19:15going to work on, the development, which is the actual building of the
- 19:19AI, the evaluation, which is the experimentation and measuring the ROI.
- 19:24And if it works, it's adding value.
- 19:26And then.
- 19:27The adoption piece of it, which is sort of the change management piece of it.
- 19:31Uh, and of course, there's sort of a fifth step, which is managing the
- 19:34overall architecture across these.
- 19:36But if I spend just a few minutes, just zooming into each of them, I can tell
- 19:39you a little bit more about what goes on.
- 19:41Um, so in the selection, uh, we're all used to prioritizing our projects, right?
- 19:46We usually think about the impact of those projects and the
- 19:49feasibility of those projects.
- 19:50Um, but when it comes to AI, there's a few nuances that I
- 19:54think are important to understand.
- 19:56So When it comes to the impact, uh, very often, uh, the project selection
- 20:01of like AI projects is left to data scientists who usually lack a deep
- 20:07understanding, uh, of the business.
- 20:09So what they end up picking are basically projects which allow them to
- 20:13use the latest technology, uh, in a not necessarily strategically aligned with
- 20:18where the business is trying to go.
- 20:21So that's a major, major pitfall is just the impact is not there because
- 20:25they're just not picking projects are going to be, uh, significantly meaningful
- 20:29for the business, which is why you need, um, you know, let's say you're
- 20:33trying to build an AI for marketers.
- 20:35You need those marketers in the room to really.
- 20:37Be part of the conversation and be part of the innovation process.
- 20:41That's one big failure.
- 20:43The second big when it comes to the selection is really
- 20:47around the feasibility of this.
- 20:49Um, when it comes to feasibility, we're often used to thinking about,
- 20:54you know, what's the timeline?
- 20:55What are the costs?
- 20:57But when it comes to AI, there's a number of other questions, which
- 20:59are, you know, what's the data?
- 21:01Do we have the right data to do this?
- 21:03Uh, are there any ethical considerations, right?
- 21:06Is there going to be any issues around privacy, around fairness?
- 21:12Um, is the modeling going to be transparent and sort of interpretable
- 21:15to the people who are going to use it?
- 21:18Uh, and then the question that I always, uh, like to remind people of
- 21:21is, You know, just because you can do it doesn't mean you should do it, right?
- 21:27So when it comes to that feasibility with AI, you really have to ask
- 21:31yourself, Should we really be doing this?
- 21:34Is this really the thing to do for our employees and for our customers?
- 21:40Uh, and then the other thing, and I just always like to give a shout out
- 21:43to some of my colleagues who do some amazing research in this area, like, uh,
- 21:47Jack Lane, uh, and Karim Makhani, they have this beautiful paper that shows,
- 21:51uh, Uh, the humans are really bad at disentangling impact and feasibility.
- 21:55They tend to correlate those two things.
- 21:58Uh, so they would say if something's high impact, they would inevitably say, Oh, it
- 22:02has to be highly feasible and vice versa.
- 22:05And these are like field experiments with experts, everyone
- 22:09who's like, Oh, I never do this.
- 22:11You absolutely do.
- 22:12I do it all the time.
- 22:14Uh, and the way to get around from that is just literally writing down
- 22:17and being really careful to first focus on the impact of the project.
- 22:21And then to focus on the feasibility,
- 22:25Itir Eraslan: because I have a question about the data piece, because as a
- 22:29company, for example, I have lots of data that I collected over the
- 22:32years from my customers and so on.
- 22:35How do I know that that data is enough or is not biased?
- 22:40Iavor Bojinov: It's a great question.
- 22:41Very often.
- 22:42Um, one of the things that I've been quite frustrated about is there's been
- 22:47a number of companies going around saying, Oh, we're generative AI.
- 22:51You don't need to get your data and all that.
- 22:53Just shove it all in and magic will happen.
- 22:55And that's just so far from reality that it's almost comical.
- 23:01Uh, so what I've been saying to a lot of companies is, of course,
- 23:05you need your data strategy.
- 23:06You need to have your data lake.
- 23:08You need to have your data warehouses in order, um, because having
- 23:12clean curated data is actually super helpful and super powerful.
- 23:18But having said all that, the reality is sometimes you don't know if that
- 23:23data is enough until you start, which actually beautifully moves
- 23:26us into the development process.
- 23:29And I have usually when I give a talk on this, I have this beautiful slide
- 23:33that sort of builds and it sort of walks through all the different stages,
- 23:36which is The different stages in the development, you know, so it starts off,
- 23:40data scientists needs to go and find the data, they find the data eventually, they
- 23:45clean it, uh, they curate it, they realize that's not the right data, they go back to
- 23:49get some more data, they clean it again, then they build some model, they realize
- 23:54that's not quite right, it's not enough data, go back, try to get more data, and
- 23:59you end up having these like super long loops that can take months and months.
- 24:03Eventually they build a model, uh, they, you know, evaluate
- 24:07it and it looks like it's good.
- 24:09Uh, they're getting ready to deploy it, but even that's not enough because
- 24:13usually data scientists work in these sort of notebooks, which are
- 24:17not really built for scalable AI.
- 24:19So then they basically have to pass it on to like an AI engineer or a software
- 24:24engineer that restarts the whole process, builds these whole data pipelines.
- 24:29Retrains the model, figures out the data is not being refreshed, goes
- 24:33back, builds data pipelines, etc.
- 24:35So, so the whole development process ends up taking six, nine months, uh, etc.
- 24:40And it's a huge, huge, uh, bottleneck.
- 24:43And, and the way I like to think about it for most companies, Uh,
- 24:46forget about the sort of leading tech companies, but for most companies,
- 24:50I'm going to be really nice to them.
- 24:51It's like building a Rolls Royce, right?
- 24:53It's really white glove service.
- 24:55Every little detail is handmade.
- 24:58It's beautifully crafted.
- 25:00And this is me being nice because.
- 25:02It could also just be like, you know, pre industrialization
- 25:05people building, you know, pre 4T just building terrible things.
- 25:10But I'm going to be nice and say they're building gross voices.
- 25:13And that's kind of this current state of the world for most companies.
- 25:17Uh, and, and, and really what they need to do in the development process is move
- 25:20away from this Rolls Royce model and move to more of like a Mercedes or even, you
- 25:25know, a Maybach Mercedes where a lot of it is actually automated and then you
- 25:30have people coming in and doing just the bits where people really add value.
- 25:34So like the building of the models itself, right?
- 25:37Because if you have your data.
- 25:38In order, you have your data pipelines that just beautifully
- 25:41should feed into a model.
- 25:43And that's the bit that the data scientists works on.
- 25:46So that's kind of how I think about the development process.
- 25:49Uh, the data piece, even if you have, uh, this like AI factory for automating
- 25:54the development of AI, if you have that in place, you still have to worry about
- 25:59having all the right data, uh, et cetera.
- 26:02Itir Eraslan: What I find it quite useful, especially, uh, on the marketing
- 26:06side is that So you have the data, you assess the data, and then for example,
- 26:11you automate like in the simplest way, you automate your emails based on
- 26:15the data and your, you know, customer profile that you have in that data.
- 26:20And then for example, you target people, uh, that are always buying from you.
- 26:26But the thing is that, If, for example, you are talking to a specific demographics
- 26:31and you are selling only to a specific city and specific demographics where
- 26:35actually you want to grow and you want to target a new demographics, then that
- 26:40data and the outcome of that data will not be helpful for you, because if I'm gonna
- 26:45target another city or a female consumer base rather than a male consumer base,
- 26:51then you will have to make sure that you also create that data, which is actually
- 26:56is not available in your database.
- 26:58Um, so that's why it's always like important to make a match of, okay,
- 27:03what's our data and what is, does it say, uh, is everyone, uh, interested in it?
- 27:08It's in black boots and they are living in New York and they are so used to, you
- 27:12know, buying black boots around September and October when it starts straight, but
- 27:17whereas, I mean, maybe you're, you're going to start selling in more in Los
- 27:20Angeles, which will not be the case.
- 27:22So, you know, I always say, okay, I mean, what's the data saying to us,
- 27:27but our data at the end of the day, and what are we trying to achieve here?
- 27:30Uh, is a good story to catch our words.
- 27:33And then, you know, after the development parts.
- 27:37Or are we at the development part now?
- 27:39Iavor Bojinov: Yes, so exactly.
- 27:40So that's the development we've been talking about.
- 27:42And I think, again, this is as if you're sort of predicting
- 27:45where this is all going, right?
- 27:46Which is, uh, that's where the experimentation comes in.
- 27:50Uh, where you have to really, uh, try out whatever AI you're doing,
- 27:55you have to go and try it out.
- 27:56So if, if your data is on New York, uh, black boots and you want to start
- 28:02selling in LA, uh, you build your model.
- 28:05That's going to try to pick some people.
- 28:07You run an experiment and maybe compare it to just, let's say you're
- 28:12not going to do any personalization.
- 28:14You're just going to, you know, let's say maybe you have some mailing lists.
- 28:18And you're just going to send it to half of the people, and
- 28:19then for the other half, you're going to do some personalization.
- 28:22And you can use that to try to learn how generalizable
- 28:25this AI is in other contexts.
- 28:27And through that process, you start to gather more data, which you can then
- 28:31incorporate back into the development.
- 28:33So that's why I always say, once you've built it, whatever AI you've built,
- 28:38that's why you need that experimentation, because What it does on your laptop
- 28:43isn't necessarily what it's going to do when you actually put it into the wild.
- 28:46There's many reasons why, when you take AI and you put it in the wild, its
- 28:51behavior is fundamentally different from how it behaved in your little closed
- 28:56environment when you were developing.
- 28:58So that's where the experimentation comes in.
- 29:00Again, it doesn't fully get rid of that challenge of, well, if
- 29:03we have no data, what do we do?
- 29:05Well, you know, you got to start somewhere.
- 29:08You got to try to collect that data.
- 29:10Buy it
- 29:11Itir Eraslan: from third parties.
- 29:12Iavor Bojinov: Buy it from third parties.
- 29:15Or you just realize you're going to take a hit and you're not
- 29:18going to have personalization.
- 29:19You're just going to deploy things and try to see what
- 29:23happens, try to gather some data.
- 29:25Um, but maybe just to quickly just touch on that final piece, which
- 29:28I think is really important, which is the actual adoption of AI.
- 29:32Itir Eraslan: Yeah.
- 29:33Now,
- 29:33Iavor Bojinov: this is not as important if you're just, you know, Uh, you know, using
- 29:37it to to do some personalized emails, but very often right now, especially
- 29:41with GNI and in marketing with a lot of the tools that I'm seeing, they're
- 29:45really supposed to be used by marketers, um, and other professionals to really
- 29:50automate or augment specific tasks.
- 29:53And.
- 29:54Very often, this is where companies really stumble.
- 29:57Like I've lost count of the number of companies that I personally know and
- 30:01have spoken to sort of their leadership team that's in charge of, say, their
- 30:05gen AI deployment, and they say things like, we bought product X and we spent
- 30:11this many million getting a contract.
- 30:13You know, we did our pilot study.
- 30:15The productivity gain was.
- 30:17200%, you know, we could take tasks that took two weeks and do it in half
- 30:21an hour, you know, we launched it and none of our employees downloaded it or
- 30:27none of the employees actually used it.
- 30:28And then I'm like, yeah, of course.
- 30:30Did you like speak to them?
- 30:30They're like, well, no, we spoke to like the five that did the pilot, but
- 30:34like, we didn't speak to anyone else.
- 30:35Like, why would we?
- 30:36It's like, Oh, my God.
- 30:37You're all used to building products for your customers, but most
- 30:40companies are not used to building products for their employees.
- 30:43So the second they look at, they flip things around, they forget, like,
- 30:48how to interact with their employees.
- 30:50Like, there's a whole team for marketing external products, but very
- 30:54few companies have actual marketing teams for their internal products.
- 30:59But you kind of need to do that because if you want people to use it, you need to
- 31:03put that same, it's the exact same thing.
- 31:05You need to sell that product to your employees and with AI, the sort of the
- 31:10framework I like to think about is really the basically need to build trust between.
- 31:15The human and the AI.
- 31:16And for me, trust sort of falls into three buckets.
- 31:20You have the trust in the AI itself, which is, is it accurate?
- 31:24Is it fair?
- 31:25Do I sort of understand its limitations and when it's going to work well,
- 31:30when it's not going to work well.
- 31:31So, you know, have I been trained on AI piece of it?
- 31:37The second layer is, do I trust the people who built it?
- 31:40Uh, and this is super important because very often we are reluctant if I gave
- 31:46you some really fancy tool And I just said oh, it's really really accurate Go
- 31:51ahead and use it and you have no idea who built it or why they built it You're
- 31:57not going to want to use it, right?
- 31:58You're just going to be like I I don't know where my data is going.
- 32:02I don't know.
- 32:03Are they going to use this to just automate me and then
- 32:05just fire me in two weeks?
- 32:06Like, I don't trust these people.
- 32:08Uh, and that's why trust in the developers is so important.
- 32:11Very often I actually find it completely trumps the trust in the AI.
- 32:15For example, if you have a team that's building these Gen AI tools, and they
- 32:19work really close with the marketing team, they have many meetings with them,
- 32:22they explain to them what they're doing.
- 32:24Even if the AI itself is not that interpretable, you know, it has
- 32:28all these other things, you're just willing to trust it because
- 32:30you trust the people who built it.
- 32:33So that's why that layer is super, super important.
- 32:35And then the final layer, which like everyone just forgets about this
- 32:40is actually trust in the processes.
- 32:42Uh, and what I mean by that is how do you handle when things go wrong?
- 32:45Who's at fault?
- 32:47How do you handle disagreements?
- 32:49Now, what happens if the AI is saying, do this, but the person is like, I
- 32:53actually want to do something opposite.
- 32:54How do you handle that?
- 32:55Who's at fault if things go wrong, right?
- 32:58If I follow the AI and some bad decision gets made, am I on the line?
- 33:03Because if I'm on the line, I'm not going to want to use this.
- 33:06Like, why would I use something where I don't fully understand that?
- 33:09I don't know who built it.
- 33:11Um, and if it goes wrong, it's my fault.
- 33:14Right.
- 33:14So, so those are some of the three layers of trust, trust in the AI,
- 33:18trust in the developer and trust in the processes that you need to
- 33:22really focus on to get that adoption.
- 33:25Itir Eraslan: And I think like for marketing, there are two
- 33:28big layers, uh, that we need to take care into concentration.
- 33:31One of them is trust because Brand is all about trust and brand building is
- 33:36all about building trust about your brand, about your business, business,
- 33:41uh, in the eye of the consumer, but like in your case, also it's in the
- 33:44eye of the internal teams as well.
- 33:47So that is adopted by the teams.
- 33:49But the second one is the, um, consumer piece, because you have to understand
- 33:54the consumer and its behavior and how to deal with that consumer behavior, which
- 33:58trust is a big pillar in that section.
- 34:01How do you think AI might transform, uh, the way we understand the
- 34:05consumer behavior or is it too early to say something or is there
- 34:10areas that are already useful to understand the consumer behavior?
- 34:14Iavor Bojinov: Yeah, so it's a really interesting question and
- 34:18I'm gonna try and take it in a few different ways and you can tell me
- 34:21if I'm going completely off track.
- 34:23On the one piece of it, AI is now getting really good at trying
- 34:29to understand our consumers.
- 34:31Uh, and trying to understand what they really want.
- 34:34Um, you know, we're able to analyze text data.
- 34:38I know examples.
- 34:40Uh, where, uh, companies have their frontline workers with, you know,
- 34:47like little audios in stores, and they basically can say things like,
- 34:51Oh, we seem to be out of this.
- 34:52Or, you know, the customer said that this product wasn't as good.
- 34:56And that gets fed into basically a model that's able to extract lessons.
- 35:02So it's giving you the ability to learn about your customer in, In a
- 35:07much more nuanced and detailed way from a much broader set of inputs,
- 35:15as opposed to just typical surveys or typical consumer panels as we're
- 35:20sort of used to doing in the past, so it's sort of opening up that door.
- 35:25The other thing that's happening is it's also changing how we
- 35:28communicate with our consumers, right?
- 35:31Um, In the past, of course, we chatbots have been around forever, but they're
- 35:36getting really good now, and that's kind of changing things because, and
- 35:40this is my biggest question and sort of, uh, worry, um, with really good A.
- 35:47I, um, let's just focus on, say, say, sort of retailers or even high end luxury.
- 35:53We're talking about like shoes, right?
- 35:55Um, yeah.
- 35:56If you go to, say, Prada, Gucci, in the store, you have this amazing experience
- 36:03where someone comes, they talk to you, they help you pick out your product, and
- 36:07they're like, okay, this is the right thing for you, you try it on, it's great.
- 36:11Now that AI is getting so good, in the very, very near future, we're going
- 36:15to be able to have online experiences that are just as personalized.
- 36:19So we're all going to have our own sort of digital shopper that's like, hey, here is
- 36:24Uh, the product that's just right for you.
- 36:26Here's the thing.
- 36:27And then my big question is like, how do you continue to sort of
- 36:32differentiate yourself and sort of build that customer trust and customer
- 36:36understanding when everyone is moving to this super hyper personalized.
- 36:41Uh, digital experiences.
- 36:43So I don't know what the answer to that question is, but I think a lot
- 36:46of us should be thinking about it, uh, because that's going to happen very,
- 36:51Itir Eraslan: very soon.
- 36:52That was one example about, uh, AI is really good at finding the right consumer
- 36:57or right products to offer to the consumer, but it's not great at closing
- 37:02the deal, especially the business deals.
- 37:05Uh, I don't know if it was.
- 37:06on one of your papers or so on.
- 37:08But I think this is a very nice example of, okay, there are parts that we
- 37:12can really get the high value of.
- 37:15Um, but when it comes to, especially, I guess, emotional intelligence
- 37:19and how to tap into the, uh, Yeah.
- 37:23Iavor Bojinov: So, so this is a great question.
- 37:24So, first of all, that was definitely, that wasn't my
- 37:26research, but I, I think I know it.
- 37:28I'm trying to remember whose it was, but it's very interesting research.
- 37:32A couple of comments, right?
- 37:32This comes back to what we were talking about where it's
- 37:35really good at specific tasks.
- 37:36So you need to figure out what those tasks are and give it those tasks.
- 37:40But it's changing and it's getting better.
- 37:42So, uh, in terms of the emotional intelligence, I don't think there's
- 37:46too many studies right now in the emotional intelligence, but there's
- 37:48a really interesting one on empathy.
- 37:51Uh, that was done by, I'm forgetting the team, but it was done on doctors and
- 37:56they basically had doctors write notes.
- 37:58Uh, and then they had chat GPT write notes, uh, and they compared
- 38:02it and people rated chat GPT as scoring higher on empathy, uh, which
- 38:08is a very human emotion, right?
- 38:09So it's able to mimic empathy way better than, uh, the average physician can.
- 38:15Uh, so I think these things are changing very, very quickly.
- 38:23And I wouldn't be surprised if it gets better at closing deals because.
- 38:27Frankly, it has way more data on how to close and how to sell.
- 38:31It's read every single sales book ever written, right?
- 38:36That's better than, than sort of your sales team can ever do.
- 38:39So I think that's going to change.
- 38:41I know a number of companies who are working on a sale co pilots
- 38:45for the sales team to help them.
- 38:48Handle those conversations and navigate sort of how to pitch the
- 38:52product and how to personalize it.
- 38:55Itir Eraslan: And how about the trends?
- 38:56For example, when you are working in fashion, you know that there's a trend
- 38:59coming in and you see it on the high street of New York, London and so on.
- 39:03How you think AI is right now at the moment good at spotting trends?
- 39:09Upcoming trends in business.
- 39:11Yeah.
- 39:12Iavor Bojinov: Yeah.
- 39:12And it sort of comes to data, right?
- 39:14Like, um, there's the Chinese company.
- 39:17I think it's a sheer fast fashion retail, right?
- 39:19They're bigger than H& M, Zara and several others combined.
- 39:23Um, what they do is they basically even put up ghost products that
- 39:29don't even exist, uh, to test.
- 39:32To test.
- 39:32Yeah.
- 39:33They'll put up products or they'll do a limited run of 10 items.
- 39:37Uh, just to see what consumer demand is for those products.
- 39:41Uh, and then if it looks good and promising, they very,
- 39:45very quickly scale that up.
- 39:46And this is, you know, this is not individuals trying to
- 39:49figure out what the trends are.
- 39:51This is, uh, A.
- 39:52I.
- 39:53helping gather data and then efficiently figure out what those trends are.
- 39:58Uh, so I think you need to again, it comes back to what we were saying
- 40:01earlier, which is If you are a high end fashion retailer that has 12 products
- 40:08and you know, you have a whole creative team that's Generating this is what
- 40:13the future of fashion is going to work look like you don't really have the
- 40:16data to do this but if you're a company That is able to that is a fast fashion
- 40:22retailer and can gather data really really quickly You know, you can actually
- 40:27try to delegate that to your consumers
- 40:31Itir Eraslan: I have one question because uh since we talked about trust and trust
- 40:35is usually linked to You The strength of your brand as well, you know, you
- 40:41mentioned about how if you don't trust the process or the company, you don't share
- 40:46your data or you don't, um, you know, rely on the, um, recommendations of that data.
- 40:53Um, in that sense, I feel that.
- 40:56Bigger companies, because I, I'm okay to share my data with Apple
- 41:01because Apple is a big company.
- 41:03And then I rely that they are using the best systems in place
- 41:07to protect my data and so on.
- 41:09Whereas, I mean I'm reluctant to share my information or to rely
- 41:14on the data of a smaller company.
- 41:16And I feel like, is it like the bigger gets bigger or, uh, like
- 41:22it's a very interesting thing that I'm quite interested in nowadays.
- 41:26So I wanted to ask you a question about, uh, this going forward for companies
- 41:31that are, especially the startups.
- 41:33Iavor Bojinov: Yeah, it's super interesting.
- 41:35So I think it comes down to value.
- 41:38Um, None of us question sharing our data if it adds value to whatever it is, right?
- 41:47Like OpenAI when I made my OpenAI account, they weren't that big
- 41:53Uh, but they were giving me value.
- 41:55So I gave them their data.
- 41:56Google back in the day when They were starting out their first email.
- 42:02I remember getting the email with, you know, started out, um,
- 42:06almost like as April Fool's joke.
- 42:09Um, well, they they proposed joke was that you have unlimited storage.
- 42:13But I remember just like being like, yeah, there's gonna add value to me.
- 42:18So I'm gonna give my data.
- 42:19So I think a lot of consumers are happy to share their data as long as
- 42:24they're getting something in return.
- 42:25So for startups, if you're adding real value to your consumer, everyone
- 42:30is happy to share their data.
- 42:32And there's a lot of other sort of trust factors that they can do.
- 42:36You know, for example, I remember five, 10 years ago, you would go to this like small
- 42:40website and they would have their own like custom way of taking your payment.
- 42:44And And you're typing in your credit card and you're like, this, there's
- 42:47no way I will ever get this product.
- 42:51There's no way that my data is not going to be sold on the dark web.
- 42:55Uh, now everyone has like Apple and Google pay integrated and an Amazon pay.
- 43:00So, so all of that is now like, there's just so many ways of leveraging the
- 43:05trust that those large companies have.
- 43:08Build around payment processing, the cloud infrastructure.
- 43:11Everyone's on the cloud now.
- 43:12That's way more secure than having a server at home.
- 43:17I used to remember having my own server that I was renting out and that was not
- 43:22at all anywhere near as secure as AWS.
- 43:24So I think, you know, if you're able to create value and
- 43:27you're using sort of the right infrastructure, there's no reason why.
- 43:30Uh, people should be worried about sharing their data, uh, with small companies.
- 43:34Mm hmm.
- 43:34Itir Eraslan: Mm hmm.
- 43:36One final question about marketing and AI, uh, what important trend do
- 43:40you see for the future of marketing?
- 43:43Iavor Bojinov: I think marketing and every other role is going to fundamentally
- 43:47change, and I think marketing is one of the first places it's already
- 43:51starting to really, really change.
- 43:54And the biggest shift I see is really this transition from being, uh, content
- 43:59creators to being content in marketing.
- 44:03Uh, in the past, every marketer was a content creator, right?
- 44:06They were making, uh, the adverts.
- 44:08They were making the text.
- 44:10Every piece of it was being created from scratch by marketers.
- 44:15I think the transition we're seeing now is that they're all shifting to
- 44:18content managers where the first draft.
- 44:22It's being created by AI, and then they're managing it, they're tweaking it, they're
- 44:27owning the interaction with the AI.
- 44:29It's still the final product that's, you know, it's really their
- 44:32creativity, um, but it's like the tools they're using are changing.
- 44:37And I'm sure there was a huge change when first people were hand drawing
- 44:42adverts, and then they moved to computer.
- 44:44Drone AdWords, and then they moved to sort of Adobe automating parts of that.
- 44:48And now it's even like that shift has already been happening for many, many
- 44:51years, but I think it's gonna speed up and it's gonna be even more exponential.
- 44:55So that's kind of the biggest area.
- 44:57Uh, and then the other piece of it, so, of course, the content creation
- 45:00and content management, the other piece of it is just Personalization
- 45:04is basically free now, right?
- 45:06The cost of variation has dropped to zero.
- 45:09So now you no longer need to create just one advert, you can generate thousands
- 45:13of adverts and personalize them, uh, and then use experimentation to figure out,
- 45:20you know, marketers are already used to A B testing, you know, many, many
- 45:23adverts, but now that's like, on steroids.
- 45:26You can do thousands of adverts and do this in a dynamic way where you're
- 45:31learning about your customers and you're offering personalization.
- 45:35So all of that to me means that marketing is fundamentally changing.
- 45:40And it's a really, I think it's a really, really exciting time for marketers.
- 45:44Itir Eraslan: It's a great take.
- 45:45And, uh, two last questions to wrap up with.
- 45:48If you could recommend one book to the listeners, what would it be?
- 45:53Iavor Bojinov: This is not a marketing book.
- 45:55It's a book that two of my colleagues, um, road marker in the city, Karim
- 45:58Lakhani on meeting in the age of AI.
- 46:02Um, so the book was actually written before Gen AI blew up.
- 46:06I mean, they talk a little bit about it, uh, but I think the book will give you
- 46:09the foundational understanding around how AI is impacting both companies,
- 46:14business models and operating models.
- 46:16Uh, and I think it's a fantastic book and the book came out
- 46:19literally a couple of years ago.
- 46:20But I'm hoping for an updated version that has.
- 46:24You know, all the extra things that are happening.
- 46:26So that would be my one.
- 46:29Itir Eraslan: I will check that out for sure.
- 46:30Because I always have this dilemma of, uh, okay, if that book is written
- 46:34like a couple of years back, okay.
- 46:36Would it be still relevant, but it's good to hear it from you.
- 46:39And the final, my favorite question is your, what's your
- 46:42favorite coffee place in Boston?
- 46:45Iavor Bojinov: Um, probably Intelligentsia, which
- 46:47Itir Eraslan: is a
- 46:48Iavor Bojinov: chain, but I love their coffee.
- 46:50Do
- 46:50Itir Eraslan: they have Intelligentsia in Boston?
- 46:52I didn't know that.
- 46:54Iavor Bojinov: They have it in Boston.
- 46:55I think they have maybe a couple of spots now.
- 46:57Itir Eraslan: Okay.
- 46:58Okay.
- 46:58I missed that probably.
- 46:59Uh, I'm trying to remember.
- 47:00Yep.
- 47:01Iavor Bojinov: But my favorite coffee chain will have to
- 47:04be Phil's in California.
- 47:06Itir Eraslan: Oh, okay.
- 47:07Iavor Bojinov: Not Phil's coffee.
- 47:09Itir Eraslan: Okay.
- 47:09I, I was in LA last week.
- 47:11I should have tried that.
- 47:12Uh, if, if, if you have taught a week earlier, actually.
- 47:17Iavor Bojinov: They do pour overs and if anyone from that company is watching,
- 47:21please open up a Boston location.
- 47:22I know they have some DC and sort of, um, West Coast
- 47:27locations, but nothing in Boston.
- 47:29I'm waiting.
- 47:30Itir Eraslan: Yeah.
- 47:31Okay.
- 47:31I'm going to try it when I go there next time to have my
- 47:34pour over there in the morning.
- 47:36Uh, yeah.
- 47:37Thanks so much for joining me.
- 47:38And, uh, I'm really happy that FedEx introduced us, uh, so
- 47:42that we can have this chat.
- 47:44I already have so many other questions because I'm gonna just stop it here.
- 47:48Uh, thank you so much.
- 47:49Iavor Bojinov: so much.