Latest / The Road to Accountable AI / Eric Bradlow: AI Goes to Business School

Transcript

Eric Bradlow: AI Goes to Business School · The Road to Accountable AI

• 0:00
  Eric, really a pleasure to have you with me on the road to accountable AI.
• 0:05
  Thanks Kevin, it's great to be here with you.
• 0:08
  You have been leading what was the Wharton Analytics Initiative for a long time.
• 0:13
  How long has it been actually?
• 0:16
  Well, it depends how far back you want to go.
• 0:18
  mean, we started 20 years ago.
• 0:20
  My colleague, Pete Fader, and I started the first ever center in any business school in the application of, if you'd like, and data science to business problems.
• 0:29
  I've been vice dean of, well, then analytics for eight years.
• 0:32
  And now for the last two years, since everything's about AI today, I'm now the vice dean of AI and analytics.
• 0:39
  So it depends how you want to count.
• 0:40
  I've been a researcher for 30 years.
• 0:42
  Center of Wharton for 20 years, guess officially at Wharton for eight years.
• 0:47
  So what then is new and different about AI from your perspective?
• 0:51
  Yeah, so I think one has to look, the way I've always liked to bifurcate it is AI has been around for 50 plus years, but that's more of what I'll call traditional AI, which is, you
• 1:03
  know, I have a video file, I have an audio file, I have text data, or if you like, have non-numeric data, and I want to encode that non-numeric data.
• 1:14
  I mean, obviously in my home field of marketing, that's a big deal because we have advertisements, we have online ratings and reviews and
• 1:21
  All of that stuff is non-numeric, but we want to code that non-numeric data.
• 1:26
  That's not the new part.
• 1:27
  That's the traditional AI that computer scientists, statisticians like myself have been working on for years.
• 1:33
  The new part is, of course, the word generative, generative AI, where if you like, the large language model engine can actually generate content for you.
• 1:45
  And that could be generating text, audio, video, pictures.
• 1:51
  PowerPoint slides, all kinds of things.
• 1:53
  So the new part is the ability of these models to generate new content.
• 1:58
  But also I think the new part is its facility to handle what we call multimodality data.
• 2:04
  So maybe you have a text data file, you have a PDF file, and a picture, and an audio file, and you want these all simultaneously ingested.
• 2:13
  So I think there's new stuff, what I'll call, on the ingestion side.
• 2:16
  And I think there's new stuff, especially on the content generation side.
• 2:21
  So what does that mean in terms of our businesses that you interact with?
• 2:25
  And also for those of us in academia as scholars, so many of the basic techniques have been around for a long time.
• 2:33
  And if we broaden it out to all of analytics, it's anyone using Excel is doing analytics at some level.
• 2:39
  You talked about what's new with this concept of generative AI, but what's really changing?
• 2:45
  Yeah, the way I view it, and think it's a good term to use on a podcast on the road to accountable AI, is it's really a form of democratization.
• 2:54
  Which is, know, I've been programming in what do you want to call it, S and then R and then free versions, online versions, then Python for like 30 years.
• 3:04
  But that's because I'm a statistician.
• 3:05
  Most people don't do programming.
• 3:08
  But if you actually think about what large language models can do,
• 3:11
  they transform business because they put the power of analysis, data science, data cleaning, querying data in interesting ways, collecting data, cleaning data into the hands
• 3:26
  of so many people today.
• 3:28
  So the reason I think it's so transformative to business is there's twofold.
• 3:32
  One is I can't imagine anybody listening to your podcast or my show on SiriusXM, Wharton Moneyball.
• 3:39
  I can't imagine anybody that says,
• 3:41
  wow, there are parts of my job that are rote and mechanical, I really don't enjoy doing them.
• 3:45
  But a large language model could do it for me.
• 3:48
  Everybody has that.
• 3:50
  And then there's the second part of it, which is there are just people that want to be able to analyze data and do data-driven decision making.
• 3:59
  And now large language models allow a much broader set of people to do this.
• 4:05
  Okay, and you mentioned you were the Vice Dean of Analytics, we had the Analytics Award, now it's AI and Analytics.
• 4:11
  So just in terms of what you're leading at the Wharton School, what has changed?
• 4:15
  Yeah, so I think it's important that, you the way I view it is there's kind of at least, I probably will skip a few, but there's at least four audiences we're trying to impact
• 4:25
  through, you know, we think we have a pretty cool acronym, WAIAI, which is the word in AI and analytics initiative.
• 4:32
  We think there's at least four audiences.
• 4:34
  One, of course, is not surprisingly researchers.
• 4:37
  So we're trying to provide resources, both financial data sets, connections to companies.
• 4:43
  so that the scholars at the Wharton School and around the world can do better research because of access to data.
• 4:50
  The second thing, of course, is students.
• 4:52
  So one of the things I'm trying to innovate a lot on, including in my own class that starts tomorrow, which is I'm trying to bring AI and data science into the classroom.
• 5:01
  But my course is, course, in data-driven marketing strategy.
• 5:04
  I'm the chair of the marketing department.
• 5:06
  It's not a statistics course, but statistical methods can be used.
• 5:10
  to understand marketing strategy.
• 5:12
  So how do we use AI to do that?
• 5:15
  So first, researchers, second, educators.
• 5:18
  Third part, of course, is we're the Wharton School, so we're trying to impact businesses.
• 5:22
  So how can we partner with companies, which has always been our secret sauce?
• 5:26
  The way I like to describe it, at least it helps me sleep well at night, is other schools hold conferences on AI and data science.
• 5:34
  we partner with companies and ingest data.
• 5:37
  So we actually provide students with experiential learning opportunities where instead of interviewing Eric Bradlow at a company and saying, so what have you done in AI and data
• 5:47
  science?
• 5:47
  And like, well, I took a bunch of courses and they're like, next?
• 5:50
  Well, or how about I worked with, I'll make this up, I worked with Google on a project doing X, oh really, tell me about that project.
• 5:57
  You're hired Mr.
• 5:57
  Bradlow.
• 5:58
  And then the fourth part of course is,
• 6:00
  what we'd like to call, I don't know, hashtag analytics or AI for good, we want to have a positive impact on society.
• 6:07
  And so Wharton has the opportunity to take a leadership role, whether it's on things like responsibility, accountability, or the application of AI for public good, whether it's for
• 6:18
  nonprofits, issues like policing reform.
• 6:21
  There's tons of educational benefits where Wharton can play a big role.
• 6:26
  So really, WAIAI's mission, again, is fourfold.
• 6:30
  businesses, students, business, and society.
• 6:36
  Those are the four areas we're trying to impact.
• 6:39
  And as you know, the way we do it is think of me as the vice dean as being the CEO of a parent holding company, where we have 10 centers that sit underneath the parent, WAIAI,
• 6:49
  one of which is the center we just launched that you're running on Accountable AI.
• 6:54
  But there's 10 centers that make up the heartbeat of what we do at Wharton around AI and data science.
• 7:00
  Great, and we'll get into that in a minute, but I just want to get you to paint the picture a little more broadly about the overall initiative.
• 7:09
  When most people think about AI research centers in academia, their probably first thought is computer science or some of the more technical fields.
• 7:18
  So can you say a little bit more, give some more examples about in the business context, how is this relevant?
• 7:24
  Sure.
• 7:25
  So you could even find this for those listeners of yours that want to go to just go to your favorite browser and type in WAI AI.
• 7:33
  You'll see our mission statement is actually very clear.
• 7:36
  We're the business school.
• 7:38
  We're the applied group.
• 7:39
  So if you're looking to develop new statistical methods, whether it's machine learning, transformer models, et cetera, around AI, that's exactly the kind of research that is done
• 7:51
  at the engineering school and is very well done at
• 7:54
  whether it's pen engineering, et cetera.
• 7:56
  We're an applied school.
• 7:57
  And so our mission statement is about the application of AI to a bunch of different problems.
• 8:04
  And so while we borrow the methods of statisticians and computer scientists, and while there are many of us, like myself, and maybe I'm making a number up, Kevin, 15 to 20 of us
• 8:13
  at the Wharton School that are applied or methodological statisticians that do develop new methods, most of the Wharton faculty in their research want to use state-of-the-art
• 8:23
  methods
• 8:23
  to solve a research problem in their domain.
• 8:26
  So that's really the difference is that most people at Wharton aren't developing new methods around AI, although some are.
• 8:34
  Most of us, most people, are actually using AI and data science to solve either substantive questions, policy questions, statistical optimization questions, things like
• 8:47
  that.
• 8:49
  What's been the biggest challenge since we made this pivot to have more of an AI focus?
• 8:54
  Yeah, it's a good question.
• 8:57
  I would say that one challenge, which is not going to be that surprising, I'm sure many of your guests have said this, it's a constantly moving target.
• 9:05
  So what are you building for?
• 9:06
  Are you building for what exists today, like the capabilities of large language models today?
• 9:13
  Are we building for five years from now, 10 years from now, even though we have no real ability to forecast what that's going to be?
• 9:21
  So I would say that's the number one challenge, which is in many fields, statistics, machine learning, optimization.
• 9:28
  It's not that there aren't breakthroughs, but they're not at the rapid pace that we see happening today.
• 9:34
  So my concern as the Vice Dean, and when I think about the investments we make at the school level, I almost like, you know, it's kind of like as someone that's written a lot
• 9:43
  of code in his lifetime.
• 9:44
  I almost don't want anything hard coded.
• 9:47
  Like any parameter I put into something that said,
• 9:50
  Well, assume chat GPT4, well, that's not necessarily true.
• 9:54
  Or assume the large language model can handle this type of data, but not in this type.
• 9:58
  Well, that's not true.
• 10:00
  Assuming it can only handle a data set of this many tokens, well, that's not going to be true forever.
• 10:05
  So it's almost like the way to stop yourself is any time something literally gets hard-coded, like you're typing in a number, or a specific company, or a specific large
• 10:15
  language model, you probably aren't building something for the long run.
• 10:20
  So this is challenge that many companies have as well with AI.
• 10:25
  And so what do you do in your organization to be adaptable in that way, given how fast technology is moving?
• 10:33
  Yeah, that's the great question.
• 10:35
  So we're doing stuff as quickly as we can, which means we're actually utilizing large language models.
• 10:44
  We're noting what they're good at, what they're not good at.
• 10:47
  We're learning how to train them better so that they can become better at what they're doing.
• 10:52
  And so, and we're doing our best to speak to thought leaders about, what is coming next?
• 10:58
  Is the next great benefit quantum computing?
• 11:02
  which will allow for even smarter trained models.
• 11:05
  And so it would still be of the same format, just the models will do better or more types of data or larger data.
• 11:12
  Or our large language models going to become, this is the classic agentic problem, our large language models are just going to become embedded in everything we do.
• 11:20
  And therefore it's not about using large language models.
• 11:24
  It's like you will have no choice because whether, you know, whether you're a Microsoft office user or Salesforce or
• 11:31
  speaking into your phone or whatever, large language models are going to get built into them anyway.
• 11:39
  And so my guess is we're in the very active phase now, what I would call active learning, where you actively have to do things to use large language models.
• 11:47
  But my guess is, Kevin, I don't know if it's two years, three years, five years, but I think probably most of us agree we will be in the passive phase sooner than later.
• 11:57
  where large language models are just, just like other types of algorithms, are embedded in all the things that we do.
• 12:05
  And we just won't know it because, and then we could talk about whether that's an ethical thing, but they will just become embedded in the software and other programs we use.
• 12:16
  If that happens or when that happens, how do you think that's going to change what we do as academics and how do think it's going to change what business people do?
• 12:26
  Yeah, I think one of the questions we have to ask ourselves and you know, this type of, I'll call it revolution.
• 12:32
  Look, when COVID happened and you know, everybody went virtual for a while.
• 12:39
  The question that was asked then is, is brick and mortar university based education dead?
• 12:44
  And then we figured out that actually not only do learners learn better in a collaborative environment, they're more engaged.
• 12:51
  And as matter of fact,
• 12:52
  How many times, Kevin, did you as a scholar, me as a scholar, me as an educator, you as an educator say pre-COVID, wow, it would be great to teach online.
• 13:00
  And I don't know anybody afterwards that didn't say, wow, this is the worst.
• 13:03
  There's no way I'd want to spend the rest of my career teaching online.
• 13:07
  I'm not saying some.
• 13:09
  I'm actually the counter example on both of those but that's a separate question.
• 13:13
  But your point is well taken.
• 13:14
  I also know because one of my sons took your course, you led at Penn, certainly at Wharton, one of the largest online courses that existed during COVID.
• 13:24
  And it was highly, my son told me, it was highly, highly successful.
• 13:29
  But most people still prefer in-person teaching.
• 13:34
  And so I think one of the questions we have to ask ourselves as educators is,
• 13:39
  What are we educating students in a business school to do?
• 13:42
  Now, I may be, you know, I'll call it, well, this way, given my political leanings, I would never describe myself far to the right on any scale.
• 13:49
  So I'll say far to the left.
• 13:51
  What does far to the left mean for me?
• 13:53
  It means I want to teach students how to use large language models to address the kind of business questions I want them to learn from my course.
• 14:03
  Other faculty might want to prohibit the use of large language models.
• 14:07
  To me,
• 14:08
  Again, I think of this as an applied school.
• 14:11
  I know for a fact that most of our students are going to use large language models in their jobs.
• 14:18
  Therefore, my view is I should train them how to use them in a responsible, ethical way in the type of work that they're doing.
• 14:27
  And so that's my goal.
• 14:28
  And that's why my job as an educator, as I mentioned, I'm starting to teach tomorrow, has changed dramatically because I have to think about if this were me,
• 14:37
  taking Professor Bradlow's stuff and actually trying to apply it, what would I actually do with this now that I have this, if you'd like, AI, large language model assistant that can
• 14:49
  help support me?
• 14:50
  I think all of us as educators have to answer that question.
• 14:54
  So you mentioned the ethical dimension to it a couple of times, which I appreciate.
• 14:59
  And you've always been very supportive of the work that I and others are doing on legal and ethical aspects.
• 15:06
  And so I'm curious just to start, and then we do want to talk about the Accountable AI Lab that we've just launched.
• 15:13
  As a statistician and a marketing expert, why does that seem important to you to have lawyers, ethicists?
• 15:22
  philosophers, behavioral scientists, people like that working in this space.
• 15:27
  Yeah, people ask me this all the time, like, why are you so, it's not what I do, why are you so interested in this?
• 15:33
  I think there's a couple reasons.
• 15:35
  One is more, I don't want to call it, there's nothing, there's no subterfuge or anything about it.
• 15:41
  One is just, it's a unique differentiator of, I have multiple reasons, let me start with the first one.
• 15:47
  If I was trying to build the most differentiated product at Wharton around AI and data science,
• 15:55
  that would make Wharton unique and that would have an impact on society.
• 15:59
  Well, we have something almost no other business school, certainly any competitor business school has.
• 16:04
  I don't have to tell you, you're the chair.
• 16:06
  We have a legal studies and business ethics department.
• 16:08
  So why wouldn't I leverage that as an asset?
• 16:12
  So that's number one.
• 16:13
  Number two, I spend half my day talking to companies.
• 16:17
  They're asking questions about ethical, transparency, responsible AI.
• 16:24
  The third,
• 16:25
  purely as a statistician, one can view problems in ethics, let's call it accountability or privacy, one can treat those as mathematically constrained problems.
• 16:39
  mean, there is ways to, it's not like actually, and that's why it's interesting when you talk about all the different people that you bring together in accountable AI.
• 16:49
  Computer scientists have a lot to say around accountability, transparency, responsible use of data, privacy, etc.
• 16:57
  And their job and my job is to translate those types of things into mathematical algorithms that solve certain properties.
• 17:06
  Now we can debate about what it means to be ethical or fair or responsible, but if you tell me what that definition is,
• 17:14
  I can then translate that into something that I'll incorporate into my algorithm.
• 17:19
  I do it really, again, you asked me a question for three reasons.
• 17:22
  One is, why wouldn't we leverage one of the most prized assets of Wharton, which is our Legal Studies and Business Ethics Department?
• 17:29
  Two, companies are demanding it.
• 17:31
  And three, it's actually a set of fascical statistical and mathematical problems to think about how to bring these fairness, responsible, accountability principles into algorithms.
• 17:44
  what are the constraints of that algorithm before someone can actually build it.
• 17:48
  I leave that to you.
• 17:49
  And this is why I joke actually in my own research.
• 17:53
  You know, for years people since my background is statistics, computer science, but if you look at my CV, I've written tons of papers with quote unquote consumer behavior colleagues
• 18:02
  of mine.
• 18:03
  And people say, why do you do that?
• 18:04
  I said, because I know enough math and computing.
• 18:07
  I need people that know something I don't know.
• 18:10
  And that's why, you know, I try to stay in my lane.
• 18:13
  I'll let you.
• 18:14
  and the ethicists and the linguists and the philosophers, all of them, you know, hash out all kinds of interesting definitions and then we'll work together on it and then I'll say,
• 18:24
  okay, well, here's how we would implement that in some sort of algorithm.
• 18:27
  And so, one of my favorite things in life is I try to stay in my lane, that type of philosophical or, you know, legal or, because there's difference between philosophically
• 18:38
  and legally.
• 18:39
  Those discussions are had by others and I love being part of them, but I don't see myself as a contributor to them.
• 18:46
  You mentioned that companies are asking for this.
• 18:49
  Can you say a little bit more?
• 18:50
  I agree.
• 18:51
  I think people have this misnomer that businesses are only concerned about profits and therefore anything where you say, you know, do it responsibly, they run away.
• 18:59
  But what has been your experience with the organizations you talked to for Warden AI and Analytics?
• 19:05
  Yeah, one is just due to policy.
• 19:07
  mean, lots of companies in the tech space, for example, that you and I both deal with a lot, they actually, there's all kinds of policy restrictions.
• 19:16
  Now, under the new administration, which started yesterday, those might change.
• 19:20
  We don't know.
• 19:21
  But I mean, there's literally, whether you want to call it gates that are put up between this type of data cannot be used for this type of, let's call it, whether it's targeting
• 19:31
  or identification, et cetera.
• 19:34
  That's one thing.
• 19:35
  It has to do with companies that care about policy.
• 19:37
  Second is consumers care.
• 19:40
  Consumers care about the respo- not everyone, there's heterogeneity, but many consumers care about the responsible use of their data, the ethical use of their data.
• 19:50
  So if in some sense you have to, you know, I could make more profit as a company, but on the other hand, the backlash could come if consumers know that I'm using the data in this
• 20:01
  way.
• 20:02
  I think every company, it's kind of like a risk reward trade-off.
• 20:05
  I why can't we just view it as what it is?
• 20:08
  Yeah, I mean, and by the way, I do not agree with the premise, and I'm sure you, I'm guessing you agree with what I'm about to say.
• 20:15
  I do not agree with the premise that one has to trade off profits to be ethical or responsible.
• 20:21
  I think that's a, whatever they call it, it's a false dichotomy.
• 20:24
  I don't think it's the right way to think about it.
• 20:27
  And people will say, well, you're constraining profit making to be this.
• 20:31
  And I don't think that's the right way to look at it at all.
• 20:34
  Yeah, no, I totally agree.
• 20:36
  you know, going back to your earlier point, no one is entirely sure how we're going to maximize the profitability and the economic benefits of this technology anyway, given how
• 20:46
  fast it's changing and the uncertainty.
• 20:48
  So yeah, it's just, it's another important piece of it.
• 20:51
  So, so what do you think?
• 20:52
  So we've just, you know, now as we're recording this, recently launched the Accountable AI Lab, as you mentioned, I'm leading as part of this larger Warden AI and Ambulance
• 21:00
  initiative.
• 21:02
  What's your hope for this activity?
• 21:06
  Yeah, so there's different, let's call it a pyramid, there's different pyramids of hope.
• 21:11
  So one hope is I'll call it just purely from the brand perspective, when companies and other universities think about what school or business schools are leading in accountable,
• 21:27
  responsible, AI, they think of Wharton.
• 21:30
  So that's number one.
• 21:31
  And that helps us make connections with companies.
• 21:34
  I can imagine many alumni interested in supporting us.
• 21:37
  I can imagine it making easier for us to hire.
• 21:40
  There's all kinds of benefits that come with that brand value.
• 21:44
  So that's number one.
• 21:46
  Two, at the heart of what sits at us, we're a research and educational center.
• 21:51
  So I'm hoping that by forming this lab, whether it's post-docs, doctoral students, whether it's a
• 21:58
  You can hire better faculty in legal studies and business ethics because of the existence of this.
• 22:04
  The way I describe it, Kevin, is that I'm on an intellectual capital campaign.
• 22:09
  My job is to help Wharton acquire, whether it's faculty, doctoral students, postdocs, the best students, RAs, et cetera, around AI and data science.
• 22:21
  And I think there are many, many faculty, students, et cetera.
• 22:26
  that will be more excited about Wharton's work because of the accountable AI lab.
• 22:31
  And so it could be the brand level, the research level.
• 22:35
  I also think, you know, I get questions all the time.
• 22:39
  of fact, I'm predicting starting tomorrow when I'm teaching the Wharton MBAs, someone's going to ask me a question and I will say, well, you should go to Kevin Werbach and you
• 22:49
  should go to the legal studies because this is an ethical question to which I'm not sure of the answer.
• 22:53
  Students all the time ask me questions about like, you think it's okay for companies to do this?
• 23:00
  And I think students are the eyes of the industry.
• 23:03
  They reflect what everyone's thinking.
• 23:06
  No, absolutely.
• 23:07
  I mean, it works the other way too, that there are people who are legal scholars or business ethicists who, if you're too unmoored from both practice as well as the
• 23:19
  technical, I mean, I can say, this is what I think the law should say, but if companies can't actually implement it, if you can't actually build a statistical model that does
• 23:29
  what the law says, then it's not very helpful.
• 23:32
  So I do think this is a partnership that really is essential and doesn't happen enough.
• 23:37
  Yeah, no, I mean, I'm excited about, mean, obviously you've been teaching a course on this and related topics for a long time.
• 23:45
  And now the Economal AI Lab, and I know you're trying to grow the faculty around this and hire scholars around this.
• 23:53
  And this is what WAIAI does where, you know, I joke with people, more cocktail party kind of stuff, which is they ask me, so what do you actually do as the vice dean?
• 24:03
  And I tell them what I do, but I say, one of the things I do,
• 24:06
  is I'm a venture capitalist.
• 24:08
  invest in what I think, it's my opinion, I invest, the school has entrusted me with this, I invest in what I think is the next big idea in AI and data science.
• 24:17
  And I think Wharton having a leadership position in accountable AI will put us in a unique and differentially advantaged way.
• 24:27
  And that's why I'm doing it.
• 24:29
  It's my opinion for all the reasons I said, the brand, the connection with companies, student interest, research.
• 24:35
  teaching, think it will affect all.
• 24:37
  And admissions, I think people will come to Wharton because of the work we're doing in Accountable AI.
• 24:45
  Sounds good to me, not surprisingly, but no, it's great to hear.
• 24:49
  And I think, I mean, for me, I also keep coming back to when I talk to companies, these are issues they're struggling with.
• 24:56
  And it's also, there are lots of conversations about regulation, but I find a lot of companies that I talk to, I'm curious if you have a similar perspective, they are really
• 25:08
  worried about being in an environment where the law is not clear, where the government says anything goes,
• 25:14
  from a legal standpoint, from a regulatory standpoint, because then something's going to go wrong.
• 25:18
  You're going to deploy some system and someone's going to say, look, this is causing terrible bias or look, a user of this AI companion service killed himself and we're going
• 25:29
  to sue you and we're going to go to the press.
• 25:31
  And I talked to a lot of companies that say, well, we want to be able to say that's terrible.
• 25:36
  We didn't want to see that happen.
• 25:38
  Here are all the things we did and we did everything that we are expected and required to do and the best practices to mitigate and prevent those harms.
• 25:47
  mean, cars crash and blow up and sometimes things go wrong.
• 25:50
  And I hear from lot of companies saying, well, actually we are more worried about being in a situation where we can't say the government has told us this is the standard.
• 26:00
  We can say we're here, we're above it, or at least we're there.
• 26:03
  Now we're on our own.
• 26:04
  So I'm curious if that reflects what you're hearing from the organizations that you're
• 26:08
  It is, but I think part of it is, I'm sure you know this, it's part of who are you talking to in the organization.
• 26:17
  So probably not surprisingly, the organizations I'm talking to, I'm talking to either the usually, maybe it's the CMO as a marketing professor, but not necessarily.
• 26:26
  It could be the chief financial or chief revenue officer.
• 26:29
  It could be the chief information officer, the person that runs data science.
• 26:33
  It's rarer.
• 26:34
  that I would speak to somebody that might be in legal or compliance, et cetera.
• 26:39
  The reason I get involved with people in legal and compliance has to do when it comes to sharing data or access to data.
• 26:47
  But in some sense, as you said, I like the way you framed it, which is let's suppose we have students or faculty work on some algorithm based on AI or data science to solve some
• 26:59
  sort of business problem.
• 27:02
  How much are we thinking about, okay, so we've provided some algorithm.
• 27:06
  What are the downstream consequences of this?
• 27:09
  Like suppose it leads to some, as you mentioned, horrific type of outcome, which can be measured in all kinds of ways.
• 27:16
  Well, how coupleable, how responsible is the firm for deploying the algorithm that led whether it's to someone's death or to some loss of job or to something else?
• 27:28
  The answer is we probably
• 27:31
  don't think about it enough when we're developing the algorithms.
• 27:35
  I'm just speaking for WAI.
• 27:36
  I'm not saying, well, maybe no one does.
• 27:39
  But I think it's a fair question to ask.
• 27:42
  And partly also maybe because the people we speak to aren't really telling us or they don't know necessarily what those regulations are.
• 27:50
  That's possible as well.
• 27:52
  Yeah, no, everyone is really struggling with this and the point that you make about, you know, responsibility and liability is a really crucial one and a really uncertain one
• 28:03
  because it's in the nature of these technologies that there is a gap between, you know, especially talking about something like an LLM.
• 28:12
  it's, you know, it's pre-trained and it's trained and then it's, you you get some.
• 28:17
  intermediate process at the inference stage and so forth, and then some other company is deploying it somewhere and then something bad happens.
• 28:24
  It's a really hard question about what we just the right thing to do or even the formal legal thing to do about redress for those harms.
• 28:33
  So those are definitely things that we hope to look to.
• 28:36
  And I think there are companies that think about it more actively and there definitely are companies in regulated industries where they have existing.
• 28:44
  structures and finance and healthcare and so forth that they have built for data governance or otherwise But no one really knows the answer to those questions
• 28:52
  Yeah, I think the two areas that I think are the most mature in the kind of the lane that I sit in are, would call it privacy and data governance.
• 29:01
  Those are the two that are a little bit more mature.
• 29:04
  Like I think people have some sense about, know, whether it's, you know, de-identifying data or what can be shared, et cetera.
• 29:14
  I think people have some sense about
• 29:16
  know, data governance and security and some sense.
• 29:19
  I'd say, I'm not saying those are fully mature, but I'd say those are more mature than I would call it, you know, ethical and other kinds of moral types of investigations, at
• 29:30
  least from the lane that I sit in.
• 29:32
  And matter of fact, a lot of the algorithms I've been working on now, it's literally been for 15 or 20 years, is most of the work that I do in my research career is about using the
• 29:42
  most granular data.
• 29:44
  possible.
• 29:45
  But I also did forecast, it's no Nobel Prize for this, that there were going to be limitations on data availability.
• 29:54
  And so a lot of the algorithms I've been working on for the last 15 or 20 years are not assuming that you have as granular data at the individual level.
• 30:01
  And what can you do as a firm, know, given I'm in the target marketing or, you know, business.
• 30:07
  What can you still do when you lose access to data that's maybe it's captured at that level, but you don't have access to it?
• 30:15
  Or it's not even captured at that level because the firm's not even allowed to store it at that level.
• 30:19
  So there are many of us in marketing and other places that have been working on these, you know, what happens if you're not living in the land of granular data plenty and what do we
• 30:30
  do in those cases?
• 30:31
  Matter of fact, I will say.
• 30:33
  my colleague peter and i wrote a paper maybe ten years ago which we presented to a legal studies conference which was it's not obvious in the in many situations that profits from
• 30:45
  a purely from perspective have to be compromised
• 30:48
  by using more aggregated data.
• 30:51
  of course, this audience was eating that up because they're like, wait a second, the classic argument is, how do you expect us for-profit businesses to operate if we can't?
• 30:59
  And we actually showed that that's not actually true in many cases.
• 31:03
  So that's a problem that's purely as a marketer, as a statistician that I'm interested in.
• 31:08
  No, absolutely.
• 31:08
  This is a really great example you're giving of this conversation that we described earlier that really needs to continue happening between different disciplines because we
• 31:17
  identify a problem like privacy.
• 31:19
  It's true about bias and explainability, any of these things.
• 31:22
  Typically, there's a simple answer.
• 31:24
  If you're giving me all the data, here's what I'll do.
• 31:28
  But that doesn't necessarily mean if you don't give me all the data or even if I have all the data, I'm not going to that.
• 31:32
  That doesn't mean that necessarily I'm inherently disadvantaged, but I need to
• 31:36
  think differently about how I'm going to use it.
• 31:38
  No doubt about it.
• 31:40
  So, we're almost out of time.
• 31:44
  What do you see going forward as some of the most crucial issues?
• 31:48
  We can talk about the accountable AI piece or generally in terms of, you mentioned agentic systems and so forth.
• 31:54
  What do you see coming that is going to be most important to think about in this realm of AI more broadly?
• 32:00
  Yeah, I mean, it relates to a meeting that you and I were just at earlier today.
• 32:03
  I think the thing I'm trying to do both as an educator, but also as a researcher, as a department chair, as a vice dean, is I'm trying to reduce barriers.
• 32:15
  And I think the big challenge right now is, you know, we have a large fraction of the population that are actually sitting there that haven't used large language models,
• 32:27
  haven't tried them.
• 32:29
  don't know where to get started, don't know necessarily what their limitations are, don't realize that, you know, I was working on something today using large language models and I
• 32:39
  had to have identified in just two hours of using them 15 to 20 errors.
• 32:43
  Like the answers were just plain wrong and I know they're wrong.
• 32:46
  There's no ambiguity.
• 32:47
  These weren't subjective.
• 32:48
  Like Eric thinks it's wrong but it could be right.
• 32:50
  No, it's just.
• 32:51
  but you know they're wrong, you know to ask whether they're wrong, because they might be.
• 32:55
  asked it and then it said, you're right.
• 32:57
  I'm like, well, if you knew I was right, mean, it's like I'm having a conversation with a person.
• 33:01
  I mean, once I pointed out it was wrong, it knew it.
• 33:04
  But then of course, the next time it gets it right.
• 33:07
  See, that's the thing about training and stuff.
• 33:09
  But I think one of the challenges I'm trying to do is I would like it to me, it's a tool that people should explore.
• 33:18
  I don't know how anybody today can't see it as, you know, in some sense, you
• 33:24
  If we think about it, we've all been doing search for a long time.
• 33:28
  Google's made a lot of money off a box, typing stuff into a box, learning about you, and selling advertising based on that.
• 33:37
  Well, now we have something much more powerful than search, because it's not just retrieving information, it's generating information.
• 33:47
  And by the way, we could argue...
• 33:50
  If you think about Google, think Google's Gemini, right?
• 33:53
  That's Google's product, right?
• 33:54
  If you think about it, they're building a future where they're going to put their own Google search out of business.
• 34:01
  I they have to be realizing that search in a box is not necessarily, and just retrieval is not the future.
• 34:10
  And AI has made it so.
• 34:12
  So I think that's what I'm thinking about going forward is
• 34:16
  How do I use it more in my own research?
• 34:18
  How do I get more of my colleagues to use it?
• 34:21
  How do we teach people to ask the right questions?
• 34:24
  I I'm sure you've had people on that have talked about prompt engineering.
• 34:28
  And also, how do we train these models to do better in the future, but in a way that secures the data and privacy?
• 34:37
  And the way I describe it to people is they're like, well, chat GPT is free.
• 34:41
  How are they going to make money?
• 34:42
  I'm like, well.
• 34:43
  Some versions are free, but not the enterprise version.
• 34:46
  As a matter of fact, that's the same with Google.
• 34:48
  I get Google for free, but the enterprise version's not for free, and it's not free to advertisers.
• 34:55
  And that's the same thing that's going to happen here.
• 34:57
  Let's imagine in your area of the world, let's imagine I'm a law firm and I want to build a large language model.
• 35:04
  don't want all my, matter of fact, I'm not even allowed to have all my content sucked up into the cloud and used for training.
• 35:10
  But localized, so localized large language models,
• 35:13
  are going to be a big, big part of the future, not just for privacy and regulatory reasons, but because they will perform better.
• 35:21
  Localized models perform better.
• 35:23
  So that's what I'm thinking about.
• 35:25
  And how do we do this at scale?
• 35:27
  Because again, I'd love to not only impact Wharton's researchers and educators, but I'd love to do stuff that impacts hundreds of thousands of learners around the world.
• 35:37
  But that means we have to figure out how to do things at scale.
• 35:40
  All right, but it's a great challenge to have.
• 35:42
  We could keep talking for a while.
• 35:44
  We will be talking about many other things outside of this podcast, but Eric, it's really, really great to have you.
• 35:49
  It's exciting for me to be part of an institution that thinks in such a creative and innovative way about this area and the opportunity to integrate the accountability and
• 36:01
  legal and ethical piece with the other aspects of AI.
• 36:04
  I really think there's a lot of exciting things coming.
• 36:06
  So thank you so much for being part of this.
• 36:08
  you're welcome.
• 36:09
  And thank you for having me on your podcast.