Latest / The Road to Accountable AI / Rumman Chowdhury (Humane Intelligence): The Need for Discernment

Transcript

Rumman Chowdhury (Humane Intelligence): The Need for Discernment · The Road to Accountable AI

0:09
Kevin: Hi, I'm Kevin Werbach, Professor of Legal Studies and
0:11
Business Ethics at the Wharton School of the University of Pennsylvania.
0:15
For decades, I've studied emerging technologies, from
0:17
broadband to blockchain.
0:20
Today, AI is promising to transform our world.
0:24
But AI needs accountability, mechanisms to ensure it's developed and deployed in
0:29
responsible, safe, and trustworthy ways.
0:32
On this podcast, I speak
0:33
Rumman: with the experts leading the charge for accountable AI.
0:40
Kevin: My guest is Rumman Chowdhury, for many years one of the most influential
0:44
voices working in responsible AI.
0:46
She co-founded Humane Intelligence, a nonprofit that pioneered community-driven
0:50
evaluations and red teaming of AI models, where she served as CEO
0:54
before stepping down to focus on a new public benefit corporation bringing
0:59
forward this AI evaluation work.
1:01
She previously served as Director of Machine Learning Ethics, Transparency,
1:05
and Accountability at Twitter, founder of Parity, an algorithmic
1:08
auditing platform, and Global Lead for Responsible AI at Accenture.
1:12
Our conversation ranges from the specifics of AI assessments and governance processes
1:18
in organizations to broader questions about AI's impact on jobs and education,
1:23
and ultimately why we need to reframe our perspective on thinking itself.
1:29
I enjoyed speaking with Rumman, and I'm sure you'll enjoy listening.
1:35
Rumman, thanks so much for joining me on "The Road to Accountable AI."
1:39
Rumman: Thanks for having me, Kevin.
1:40
Kevin: Uh, you were really one of the, the pioneers of this field of responsible
1:44
AI, and, and obviously, you know, a lot has changed since those early days.
1:49
We're, we're in a period right now where there's been a lot of pushback
1:52
and pullback from some of the at least regulatory activities around, uh, AI.
1:57
Uh, just in general, how do you see the environment that we're in
2:00
right now in terms of these issues?
2:03
Rumman: Yeah, I, I'm, I'm glad you're framing it as a, a push/pull.
2:06
I think the, uh, maybe superficial narrative is we are in an era
2:10
of pure AI acceleration, and everyone's just all in all the time.
2:14
But if you scratch beneath the surface, you do see actually a lot of very
2:18
sensible regulations, standards, groups, and bodies being set up, uh, that are
2:23
trending in a really positive direction.
2:25
When I say positive direction, to me that means very applied, as in like
2:30
how can we implement reasonable audits?
2:33
How do we create good governance?
2:34
How do we set standards that we can adhere to?
2:37
And then how do we start giving some teeth around all of these things?
2:40
And I think a lot of people would be surprised to hear that, yeah, it's
2:43
happening in the US as well as in other parts of the world, so I'm-- what I'm not
2:46
talking about right now is necessarily the EU AI Act, but actually, you know,
2:50
things that I'm working on that, that we're seeing in the United States as well.
2:54
Kevin: What are you seeing in the United States?
2:56
Rumman: Yeah, so there, there's a few things.
2:57
Um, one is I'm part of an AI assurance industry trade group that
3:01
was established by this group called Fathom, as well as some others.
3:05
Um, independent verification organizations, IVOs.
3:08
There's a legislation that just passed in Virginia.
3:12
Uh, so what that would be is like what I've been advocating for, I
3:15
don't know, 10 years at this point.
3:17
Sometimes I forget that I've been in this field for 10 years.
3:19
But I would say probably the first time I testified to Congress calling
3:22
for independent bodies and, and that actually has a very specific
3:26
meaning, was probably about...
3:28
It was pre-CoVet, so like five years ago at this point.
3:31
Now we are actually seeing Virginia as the first state passing, uh, the need
3:35
for an IVO, which is an independent body.
3:37
Now, my colleague Dr. Gabriella Waters, who runs the Virginia State
3:41
University Center for Responsible AI, is, uh, is actually tasked by
3:46
their Joint Commission on Technology to put some structure around that.
3:50
They can claim things.
3:50
Uh, what does that look like?
3:52
What is the feasibility?
3:53
What is the responsibility?
3:54
Most importantly, where does the liability lie when you find issues?
3:58
Kevin: Is it realistic to rely on these, uh, private sector or, or
4:03
public-private types of governance mechanisms without having mandates?
4:08
Rumman: Well, an IVO would be a mandate.
4:10
That's kind of-- That's what I think is so appealing about it to so many people.
4:14
I think you're, you're right to sound a bit skeptical.
4:16
We've, we've had voluntary commitments and, you know, all
4:19
sorts of nice glossy media moments for, for quite some time now.
4:23
Uh, and I think we are very much past the point of that.
4:27
I do think that, you know, we do need standards.
4:29
We do need, um, some guidance.
4:31
But at the same time, work- the work of people like us will
4:34
never really be done, right?
4:36
When a standard is set, that means the, the water level is higher, and then our
4:39
work moves up higher from that, right?
4:42
Like most of us are now in the business of compliance.
4:44
Uh, compliance then becomes the job of legal teams and the associated
4:49
technical parties working with legal teams for people working on
4:52
responsible AI, AI safety, et cetera, then the bar just gets lifted, right?
4:55
Then now we can think of the next stage of, of big problems
4:59
and, and issues to deal with.
5:01
All that is to say, you know, we will not magically go from
5:04
a world of zero regulation to a world of perfect regulation.
5:07
We will go from a world of zero regulation to some regulation, some
5:10
standards, some norms, some public pressure, a wide range of things,
5:14
and our job is to uplift all of them.
5:18
Kevin: Mm-hmm.
5:18
Uh, and I guess part of why I'm asking these questions is just trying to get
5:22
y- your sense of the environment, uh, because th- there is that pressure to
5:26
go fast, and, uh, some of it is just the, you know, the, the acceleration
5:30
is, but some of it is companies- Mm-hmm
5:31
saying, you know, this is, we see as an imperative.
5:35
In that kind of environment, uh, you know, how do people like you
5:39
and others make the case to need to focus- Mm-hmm ... on these issues?
5:43
Rumman: Well, I'm, I'm glad you're bringing up industry, 'cause I see
5:45
industry kind of in two parts, right?
5:47
Mm-hmm.
5:47
Th- there's the tech industry, frontier model companies for whom exactly it's
5:51
AI at all costs, and, you know- Mm-hmm
5:52
there's kind of some language around safety.
5:55
It's usually very much indexed on catastrophic risk, uh, and
5:59
little else, while we're seeing explicitly these companies
6:02
funding anti-regulation movements- Mm-hmm ... you know, all around the US.
6:06
On the other end, a lot of the companies I work with are actually,
6:10
uh, not AI native companies.
6:12
They're companies that are giants in their own industry- Mm-hmm ... and
6:15
it could be, you know, retail, auto manufacturing, insurance, finance.
6:19
They're all interested in AI, but the way I put it is, you know, it's not
6:23
their baby, so you can say it's ugly.
6:25
You're not challenging their core product if you say, "You should
6:29
do this differently," or- Mm-hmm
6:30
you know, "You need for better governance," when
6:32
actually they're seeking it.
6:33
So with most of the companies that I work with today, what we see is
6:37
that they've spent the last few years building out some sort of an
6:40
AI governance or oversight body.
6:42
That body is usually consi- consists of widely, like, six to eight
6:45
people, you know, ranging from senior leadership and everything
6:48
from, like, legal to product- Mm-hmm
6:51
AI, cybersecurity.
6:52
The new one I'm seeing is there's a lot more HR in the
6:54
room because of so much AI being- Mm-hmm ... used in, in the workplace.
6:58
Uh, that being said, you know, that, what that group has been doing is reviewing
7:03
every AI im- internal AI implementation one at a time, and while it is very loved
7:10
in the organization, everybody feels comfortable with that modality, right?
7:14
They feel that they're getting the input they need.
7:17
They're improving their products.
7:18
It's just not sustainable or scalable.
7:20
So the tension that I'm helping companies navigate is we
7:23
love the system that we have.
7:24
We've been governing AI, and we're happy with it, but it's not gonna
7:28
work for a world in which we want to have hundreds of AI systems.
7:32
Mm-hmm.
7:32
It's not answering the questions of third-party procurement and liability,
7:36
especially as- Mm-hmm ... we start to see a little bit more shape around these laws,
7:41
and importantly, we don't know what to do with agentic systems, so- Mm-hmm ... it,
7:45
it's a completely different creature.
7:47
So for them, they wanna scale, but they wanna scale their governance as well.
7:50
Kevin: And what are you telling them about how to do that?
7:53
Rumman: Uh, well, I'm specifically working on some products and
7:56
tooling, so it's Human Intelligence Public Benefit Corporation.
7:59
Um, so two things.
8:00
One is specifically it's a public benefit corporation because I do see
8:03
our mission while being a for-profit to, you know, to serve individuals
8:07
and organizations that want to see AI being used better in the real world.
8:11
And second is, one thing I'm working on is developing like
8:13
a modular tooling environment.
8:15
Mm-hmm.
8:15
So it's an orchestration environment that has a wide library
8:18
of things that people can use.
8:20
Think of it more as like a GitHub or a Hugging Face environment,
8:22
so it's not like a SaaS platform.
8:24
Uh, interestingly, like obviously started building the public
8:27
benefit corporation last year.
8:29
Now, you know, everyone's talking about SaaS is dead, and I'm like, "Well, good
8:32
thing my instinct was I'm not building a SaaS platform because everybody in
8:36
responsible AI and AI safety has a SaaS platform." Like, w- and I saw this coming
8:41
and, 'cause the big thing is this field moves so quickly, companies are not
8:44
willing to lock into a specific vendor- Mm
8:47
'cause they worry that the tools of that vendor will be
8:49
outdated as technology updates.
8:51
Like I'm tool agnostic, so we've, we've built out a way of doing at scale
8:55
human-in-the-loop testing, running evaluations, and also giving feedback.
9:00
One thing I've been doing a lot is customized evaluations and,
9:03
and customizing evaluation output.
9:05
So what I mean by that is, you know, as I, a- again, like I said,
9:08
there's like the Frontier Labs world, and then there's everybody else.
9:11
Yeah.
9:11
In Frontier Labs world, MMLU and SimpleBench and all those benchmarks
9:16
are amazing, and everyone just talks about them all the time.
9:18
I have never had a single client outside of tech say that those
9:22
benchmarks are helpful for them, 'cause it's not, it's not, right?
9:24
If you are an auto manufacturer, some other person's definition
9:29
of contextual is not yours.
9:32
Um, so I'm helping companies with using, doing testing
9:35
and creating reusable assets.
9:36
So all that is to say- Mm ... like the thing I'm trying to
9:38
balance is context and scale.
9:40
Kevin: Mm-hmm.
9:41
Yeah, and I'm, for these purposes, uh, I agree with you.
9:44
We should focus on the, the enterprise side, you know, those who are, you
9:47
know, maybe doing some development, but really more deploying these
9:50
foundation models and other tools.
9:52
Um, and the question I have there is how well matured is this ecosystem of
9:59
assessment, where if they understand that there's a, a set of issues they may
10:02
have, whether it's, you know, bias or accuracy or, or whatever else, uh, are,
10:06
are we at the point where the, the tools are there, uh, and it's a matter of them
10:10
just developing and implementing them?
10:12
Or, or is there still a lot of customization or a lot of figuring
10:16
out how you can effectively assess?
10:19
Rumman: G- great question.
10:20
It is an incredibly immature market.
10:22
Uh, I've been calling it vibes-based evals, right?
10:25
Like, you know, we, we... And, you know, uh, and I think most people
10:28
listening to, to, to this show will, will know the difference, right?
10:32
Generative AI introduced probabilistic output, not statistical output.
10:35
Yeah.
10:35
So pretty much roughly three years ago we were thrown this brand-new problem-
10:40
Mm-hmm ... for which none of the old tools- Mm-hmm ... really worked.
10:42
So we have been scrambling to do evaluations while figuring out how to
10:47
do evaluations, and I, you know, I would say we spent the first few years just
10:51
scrambling behind the company saying- Mm
10:53
"Hey, hey, hey, we're gonna do these evaluations.
10:55
There are things you've missed or things you're not focusing on." And now, not
10:58
to say we have breathing room, but we can now take the time and say, "Okay,
11:02
we also have to design better methods." So, you know, there are... It, it's
11:06
immature in every sense of the term.
11:08
Mm.
11:08
Like, one, we don't have the tools, two, we don't have the methods, three,
11:12
we don't have the model access, four, we don't have the legal protections.
11:15
Mm. Five, we don't have the institutions, and six, and maybe most importantly
11:18
to me, which is why it create- started the nonprofit, is- Mm ... we just don't
11:22
have enough people doing this work.
11:24
It's interesting, I, I testified to the State of New York on the need
11:28
for e- e- evaluations and audits, et cetera, and some of the pushback from
11:31
companies is that, "Oh, there isn't really enough of a need," and I, I
11:34
think I am now on, on record saying, "Really?" 'Cause I'm really tired, and
11:39
I need more people doing this work.
11:42
Kevin: Um, well, it's great you're doing it, but, but I, I agree with you
11:45
and, uh, you know, it's certainly the reason that, that people like me are
11:48
working to, you know, not necessarily train professionals, but, but at least
11:51
raise the overall, uh, ambient level of knowledge among people that, that we're
11:55
sending out from a place like Wharton.
11:57
Can we overcome some of those, those barriers, though?
11:59
So, so for example, in terms of the, you know, the non-deterministic
12:02
nature of these systems, i- is it fundamentally possible to, uh, understand
12:08
things like bias and, a- when we're in this environment where you don't
12:12
necessarily know what's going to happen?
12:14
Rumman: Yeah.
12:15
Y- you know, we- We always like to use, you know, the term in context has
12:19
become- Mm-hmm ... very, very popular.
12:20
And- Mm-hmm ... and that's always been the case, right?
12:22
So we can create lots of guidance and rubrics and measurement tools-
12:26
Mm-hmm ... but it is up to the evaluator to have to understand the
12:29
context in which it's gonna be used.
12:31
Mm-hmm.
12:31
And again, kind of why, why I'm, I'm often actually very excited
12:34
to work with non-frontier labs.
12:35
The frontier labs are making general tools, and it's hard to,
12:38
like, narrow it down to a vertical.
12:40
I love seeing the thing in action.
12:41
Mm-hmm.
12:42
So if I'm working with, for example, an insurance client, you know, they're
12:45
specifically testing an insurance tool, and I know exactly who the stakeholders
12:50
are, who the end user is, and what the situation is, then I can craft, you
12:53
know, what, what the testing should be and what I should look out for.
12:57
Which is, you know, very different, uh, from the task that the frontier
13:01
labs have at hand, which is, like, think of every bad situation-
13:05
Yeah ... ever in any setting of anybody, and I don't envy them that.
13:08
But I do think then there's almost this handoff from what... A- and there needs
13:12
to be, again, this, like, industry understanding, that there is a certain
13:15
amount of work the frontier labs will do and actually can do, and there's
13:18
actually some work that they cannot do.
13:20
So what the frontier labs cannot do is, again, for this insurance
13:23
client, figure out, okay, you're gonna be using whatever, Claude-
13:27
Mm-hmm ... in this insurance setting.
13:30
We need to make sure as Anthropic that it's gonna work.
13:32
But that, that I would not place that as- Mm-hmm ... a- as their responsibility.
13:36
So, you know, these things are, are very in-depth, they're very contextual,
13:39
and the struggle that all of us have right now, everybody's grappling with
13:42
this, again, like, how do we scale?
13:44
How do we scale our testing, right?
13:46
Mm-hmm.
13:46
There is this industry push, and I don't think it's a bad push to integrate
13:50
AI into all these different things.
13:51
It really can improve efficiencies.
13:53
Like, I've been using a lot of agents for all sorts of things,
13:56
uh, and I, I really like it.
13:58
But then h- how do we take this good work and not, not stifle it by
14:04
not having good, good governance?
14:06
And- Mm-hmm ... you know, you've, you've probably heard me say this, I've said this
14:08
a whole bunch of times, it's a fallacy that regulation stifles innovation.
14:12
Every company I've ever worked with is actually looking for guidance on what
14:15
to do- Yeah ... and how to do it right.
14:17
Otherwise, they're reinventing the wheel with every new project and
14:20
every single time, and it's really not, not a helpful place to be.
14:23
Kevin: But in this environment where, especially in the US, there, there's
14:26
not a lot of regulation, what, what is it that motivates companies to
14:31
do this investment, especially given that uncertainty you've talked about?
14:33
Rumman: Yeah.
14:34
And it's really just assurance.
14:35
You know, and the easiest definition of AI assurance is, is your AI model
14:39
doing what you promised it's doing?
14:41
So for them it's a, it's a customer- Uh, it's a customer risk.
14:45
What I've noticed i- what I've noticed for the past few years is that AI
14:49
company ... Uh, sorry, a lot of enterprise AI rollout is internal, so
14:54
they're using LLMs for backroom, back office processes, you know, for internal
14:59
stuff, and they're very, very wary to use anything customer facing, and this
15:03
is a reliability and assurance problem.
15:05
They're okay if the model says or does something wrong in the context
15:09
of, you know, reading some paperwork.
15:11
You know, they can absorb that liability, and there's probably very
15:15
low liability because it's happening in the context of their workplace, but
15:18
they will not put it in front of a, a customer because for them, again,
15:21
like, uh, their, their baby is their tool, their product, their service.
15:26
AI, as long as AI can help advance that-
15:30
Kevin: Mm-hmm
15:30
... Rumman: they're happy to use it, but if there is undue risk being introduced,
15:34
they're not, and that's why we have seen very, very careful rollout.
15:37
And I think last year everybody was talking about how, you know,
15:39
whatever, some, some very high percent of pilots never moved to scale, and
15:43
that's really a lot of what it is.
15:45
We cannot, we don't have good ways of monitoring these
15:48
systems at scale quite yet.
15:49
Kevin: And I know you've also been looking at, at broader issues.
15:52
Um, you were part of a debate with Chris Hughes, who's actually doing
15:55
a PhD in my department at Wharton.
15:57
Ooh, really?
15:58
Where, where ... Yes, where the two of you were arguing, as I understand it, a-
16:01
against the proposition that there would be this tremendous AI job apocalypse.
16:04
And so I'm, I'm curious your, your thoughts, because obviously
16:07
th- this concern about, uh, AI completely disrupting labor markets
16:12
is top of mind for a lot of people.
16:15
Rumman: Yeah.
16:15
Yes, so that was a, an immensely fun debate.
16:18
I, I really loved it, and actually the ... like, couple of weeks later
16:21
I was up at MIT, and I- Mm ... and I visited Professor Johnson's class.
16:24
Mm. And so it's funny.
16:25
Uh, I also showed up late, and I'm like, "Oh, this reminds me of undergrad."
16:31
Um, so he teaches this amazing class with Gary Gensler, who used to lead
16:35
the SEC- Mm ... and, and it's really about, like, AI and AI innovation.
16:38
Um, but yeah, I ... and I think, you know, a lot of us were kind of making
16:41
the same argument that it's not ... W- work is not a finite pie of things,
16:45
and AI is not coming to eat more of the pie than we can eat, right?
16:48
It is an ever-expanding- Set of tasks and new technology
16:52
creates new jobs and new tasks.
16:55
The, the example I used to give is like, hey, remember in the '90s there
16:58
used to be this job called a webmind.
17:00
It was kind of one person who knew some HTML, and now you not only have entire
17:05
web teams, you hire external firms for your branding and your photography
17:11
and all this other work that you do.
17:14
Again, you know, 1998, if you had told somebody that one day this will be, you
17:19
know, a very significant expense your company spends and they hire dozens of
17:24
people and external firms to do this, they would just kind of think it's funny.
17:28
And, and that's, that's really what I'm getting at.
17:29
You know, some jobs are- Mm
17:31
are going to be automated away, and it is our responsibility from a
17:34
policy perspective to ensure, this is what Chris is arguing, I know Chris
17:38
is working on this, from a policy perspective- Mm-hmm ... to create the
17:41
right kinds of safety nets for people.
17:43
Mm-hmm.
17:43
The right kinds of safeguards, and in particular, there are a
17:46
couple of audiences that we have identified, and one in particular
17:48
is young people graduating with- Mm
17:50
degrees and entering the market.
17:52
Why?
17:52
Because AI does the, uh, a, a level of work that does
17:57
not require domain expertise.
17:58
Mm. So what, the, my 2026 word of the year is discernment.
18:03
A lot of people have been using critical thinking.
18:04
Mine is actually discernment.
18:06
Mm. And what do I mean by that?
18:07
What I mean is, like, I use an AI agent to design a website.
18:12
I have no design skills.
18:15
That website looks amazing to me.
18:16
If I showed it to a web design professional, I can guarantee you they
18:20
would immediately point out little issues, problems, et cetera, right?
18:24
And we're all in, in, those of us who have been in the job market, been in our-
18:28
Mm-hmm ... our professions for a bit- Yep
18:29
have that, we have that discernment.
18:31
Kevin: Yeah.
18:31
Rumman: That, again, to the median person who does not have that expertise, the
18:34
output of AI looks really interesting, it looks great, it looks wonderful,
18:38
and maybe, you know, I'm fine with my super generic AI agent produced website.
18:42
But if I were trying to go for something more innovative, deeper, et cetera, I
18:46
would still have to call a professional.
18:47
Now, if we're not hiring young people into entry level jobs, we're
18:51
gonna have to do something to ensure that they're getting upskilled and
18:55
able to be the next generation with that discernment, 'cause I am not
18:58
planning on having my job in 40 years.
19:01
I certainly hope that there will be other people who are better than me doing it.
19:05
Mm-hmm.
19:06
But who are those people, and, and what are, what are we doing to
19:09
ensure that- Mm ... you know, they're safely entering the job market?
19:12
Kevin: So there's a piece of that, as you say, that's, that's a policy question,
19:15
but a- again, parallel to what we were talking about before, now companies need
19:20
to make their decisions about what to do in this environment, and they're,
19:23
they're under pressure to cut costs and they've got these technologies.
19:26
So how, how should these, these enterprises sh- think about what to
19:30
do in this environment, whereas, as you suggest, if they just don't hire
19:33
young people, then ultimately they're shooting themselves in the foot?
19:36
Rumman: Yeah.
19:36
Yeah, and, and it's almost this, like, age-old conversation about
19:40
short-term benefit and, and- Mm
19:41
long-term impact.
19:43
Uh, and- You know, for the past few years, a lot of the AI in the workforce
19:47
conversations started off with like AI literacy, which- Mm-hmm ... I
19:50
don't think we ever very well defined.
19:52
And it's like, "Oh, everyone needs to know, know how to use AI." Again, I'm
19:55
not quite sure what that means, but sure.
19:57
And then it became, hey, I'm a, you know, my... I have an AI policy, but
20:01
my employees are still using AI for things, you know, and became a little
20:05
bit more of a harder governance thing.
20:07
Like we should not be taking sensitive company documents and dumping it into
20:10
your version of ChatGPT and getting email outputs or whatever it is, right?
20:15
And, and we did see shadow AI use.
20:17
Mm-hmm.
20:17
Now the big question is in workforce integration.
20:20
This is actually like very... I think it's a very exciting.
20:23
It's exciting for a few reasons.
20:24
One is I think the job of good leadership is to mitigate any fears about AI
20:29
replacing everybody, and that is not just your job as a leader, it is the smart,
20:34
sensible, strategic decision to make.
20:36
Because you should know how you want to roll out AI in your organization, and
20:40
part of that is workforce integration.
20:42
What do I mean by that?
20:43
Mm-hmm.
20:43
There's this great paper that just came out of MIT, Sinan Aral and, and
20:47
his grad student, and I loved it.
20:48
I was, I was there at the Big@MIT conference talking to Sinan,
20:51
and I saw his grad student, you know, uh, Michael present.
20:54
It's called the Augmentation Trap.
20:56
I love that framing, 'cause basically what they're saying is the short-term
20:59
efficiency gains of putting an AI system to do work actually has the long-term
21:03
impact of cognitive decline for the worker because they're not incentivized.
21:08
Mm-hmm.
21:08
Like, they're sort of doing the cognitive offloading.
21:11
They're just dumping it on the AI system.
21:12
Yeah.
21:12
So why is that bad?
21:14
One is you're gonna have a workforce that actually doesn't have discernment.
21:17
People avoid that muscle will atrophy- Mm-hmm ... 'cause you have
21:19
to use your discernment skills to retain your discernment skills.
21:23
Uh, second is for new employees, they're call- it's cognitive death.
21:26
It's a muscle they've never built.
21:28
Mm-hmm.
21:28
Like for us it would atrophy, a- atrophy, but for them it never
21:31
existed, so that's an even worse case.
21:33
So maybe in the short term you're getting quote unquote cost efficiencies.
21:37
In the long term, you have a workforce that does not know how to make a decision,
21:41
or worse, does not know the difference between a good and a bad decision, right?
21:44
And that's really when we talk of all the language about human in the loop-
21:47
Kevin: Mm-hmm
21:47
... Rumman: back to discernment, we need people who are able to tell if something
21:51
was a good or a bad decision, 'cause an AI actually can give you input into it,
21:55
but ultimately it's humans that best understand the system and the environment.
22:00
Kevin: Mm-hmm.
22:00
No, it's a, it's a fascinating problem, and it, it strikes me similar to
22:03
what we're seeing in other areas.
22:05
People like, uh, my colleague, Hansun Bastani, have done research on education
22:08
where- Mm-hmm ... similar phenomenon.
22:10
You, you give people an L- students an LLM, they use it, they get higher
22:13
scores, then you take the LLM away, and they get lower scores- And they struggle
22:16
than people who never used it, right?
22:17
So- Yep.
22:18
Rumman: Yep
22:18
... Kevin: so how do we escape from that trap?
22:21
Rumman: Yeah.
22:22
I fell down-- I'm doing some work on AI and education as well.
22:25
Mm-hmm.
22:25
Last year I presented to the California School Boards Association and, you
22:28
know, one thing I had forgotten and was pleasantly reminded is that we've
22:32
been studying education for decades and decades and decades, right?
22:35
Mm-hmm.
22:35
Lots of brilliant people, um, have been statistically analyzing education.
22:40
We have some of the best longitudinal data.
22:42
So it's, it was a field that was ready, right?
22:44
They had the people, they had the means- Mm-hmm ... they've
22:46
done this before to test AI.
22:48
One of the most important things that I learned, and this is about
22:51
the pedagogy of AI, and not just for kids, but I think it's for all of us.
22:54
Mm-hmm.
22:54
We're all learning how to use AI.
22:57
The difference is in your framing of what AI is for.
23:00
So AI is often presented as a tool of production, and
23:03
that is the shorthand, right?
23:05
Use AI to write an essay, use AI to draft an email, use AI
23:08
to make the slide look pretty.
23:09
It is producing a thing.
23:11
Instead, a- and that's actually the bad place, and this is where, you know, like
23:14
you said, you stick kids in front of an LLM, and then they're just like, "Write me
23:17
a 300-word essay about Romeo and Juliet."
23:20
They got it, you know, but then if you ask them who is Romeo,
23:23
they could not possibly tell you.
23:25
A- and not just that.
23:26
There is this deeper meaning of why we read these stories and books.
23:29
It's not just to torture us and, you know, make us read old
23:32
English and write stupid essays.
23:34
It's to ex- to learn how to communicate and explain things,
23:37
and we've lost all of that when we think of AI as a tool of production.
23:40
Now instead, if we think of AI as a tool of mastery, so in education, it's often
23:45
called the zone of proximal learning.
23:47
So when kids are learning, they're going to the next stage.
23:49
If we integrate, and this is about human AI configuration, right?
23:53
Mm-hmm.
23:53
This is like the HCI crowd should be extremely excited about this, right?
23:57
Mm-hmm.
23:57
'Cause this is about how do we integrate this tool and how are people using it,
24:02
and how are they taught how to use it?
24:04
Mm-hmm.
24:04
Again, this goes back to this idea of workforce integration, AI
24:07
literacy, but like in the classroom.
24:09
So AI as a tool of mastery means what are the ways in which they're using an AI to
24:15
achieve the next level of achievement?
24:16
What that requires from the teacher and, and from the person designing the
24:19
curriculum is that, again, the point of writing a 300-word essay about
24:23
Romeo and Juliet was not to write it.
24:25
It was to, you know, read this complex te- te- test, uh,
24:28
sorry, this, uh, complex text.
24:30
Mm-hmm.
24:30
Um, you know, atomize themes and viewpoints, string them all
24:34
together, make a coherent argument.
24:35
All of these things AI can help you do, but should not do for you.
24:40
Kevin: Mm-hmm There seems to be a, a challenge though in, in, in both of these
24:44
domains, and I, I'm, I'm thinking back to the, the enterprise governance kind
24:48
of situation that we've been talking about, that, you know, the, the processes
24:52
that are about, as you say, assurance.
24:55
You've got this AI tool, you want it to do what it's supposed to do.
24:57
We don't want it to fail in these different ways.
24:59
We need to govern it and restrain it.
25:02
That seems to me to be a, a different kind of process and typically a different group
25:07
of people than the ones who are going to look at these questions about integration,
25:11
about how do we have that discernment about what are we using these tools for?
25:15
So is there a way to bridge those gaps?
25:17
Am I, am I identifying a real problem?
25:20
Rumman: Yeah.
25:20
No, you, you are, and I think it, it goes back to what we've been talking
25:24
about with the use of really any technology is it have to- Yeah ... has
25:26
to be solving a real problem.
25:28
Uh, a lot of the AI hype was just excitement around the
25:31
tool and its existence- Mm-hmm
25:33
and not so much an assessment of whether or not it's useful.
25:36
I do like to think that the hype is dying down a bit in kind of a positive way.
25:40
I think everyone, a lot of people have gotten very smart about AI.
25:43
It's actually quite easy to learn, you know, how to integrate with large
25:47
language models, how to use them, and specifically agentic AI, like every user
25:52
journey becomes more and more natural.
25:55
Kevin: Mm-hmm.
25:56
Rumman: Um, so my, you know, and I'll give you a slightly spicy take.
25:59
I was not particularly... Not that I wasn't excited.
26:03
I mean, I actually wasn't particularly excited about LLMs.
26:04
I mean, they were- Mm-hmm.
26:05
Gen AI was like a cool- Yeah ... achievement.
26:08
Kevin: Yeah.
26:08
Rumman: I didn't really see, I'm like, "This is not revolutionizing
26:11
the workforce because it's this like call and response thing."
26:15
Mm-hmm.
26:15
Uh, agentic AI however, I- Mm-hmm ... for me has been, like I
26:19
did not use gen AI stuff for anything.
26:21
I didn't use LLMs for anything.
26:22
With agentic AI- Yeah ... I'm doing lots of stuff.
26:25
Yeah.
26:25
And again, it's because I didn't want to replace my discernment.
26:28
I'm like, no, the output of this is very, very superficial
26:30
and nothing I would ever say.
26:32
But the things that agentic AI can do, and this is where it becomes very powerful,
26:37
like all the different connectors, you can ask it to do certain tasks for you.
26:42
And, you know, again, I still have to be the one at the end of
26:44
it with the discernment and, you know- Yeah ... giving it feedback.
26:47
Like for anybody who's a builder, it's a very, very exciting time.
26:50
I kind of feel like I did as, you know, in high school when I first started
26:54
learning about the internet and all the things you could do- Yeah ... with it.
26:57
It kind of has that same vibe, you know?
27:00
So, so to your point, you know, in terms of solving a problem, you have
27:04
much more tools at your disposal, and I hope that there's a lot less
27:08
hype around the capabilities of these tools, and we just see them
27:11
as technological artifacts- Mm-hmm
27:13
and things we piece together.
27:15
Kevin: But let's get real.
27:16
There's a lot of hype around these tools, right?
27:18
And, and, and more capabilities of agentic AI even more.
27:22
I, I guess what I'm wondering is- Yeah ... you're right.
27:24
It's, it's surprisingly easy to most people to figure out how to
27:28
use it, at least for some things.
27:29
But as you say- Mm-hmm
27:30
the discernment is not can I build a website?
27:34
'Cause now I can.
27:35
The discernment is should I build a website or when, when can my,
27:39
you know, simple website do what I need, or when do I still need-
27:42
Rumman: Mm-hmm
27:42
... Kevin: the human?
27:43
So, so I guess I, I, I come back to how, how do we get to that point in this era?
27:51
Rumman: Yeah, you're right.
27:52
There is a lot of hype, and I think the hype kind of goes both ways.
27:54
There's a lot of AI safety hype or, like, crit- crita hype is
27:59
one term I, I had, I had heard.
28:02
Yeah.
28:02
And then there's, like, you know, sort of the, the accelerationist hype as well.
28:06
And like, what is the way around it?
28:08
I don't know.
28:09
Yeah.
28:09
Um, because I think for anybody who's new to the technology, you're being bombarded
28:13
with frankly misinformation often- Mm-hmm
28:17
um, of what these systems can do.
28:19
A lot of it's very orchestrated.
28:21
It's very staged.
28:22
Even capabilities testing is designed in these ideal world scenarios.
28:27
Mm-hmm.
28:27
So back to that open to debate thing that we were doing, my, i- in my,
28:31
like, intro I, I said something like, "Anybody who thinks AI is coming
28:35
tomorrow to replace our jobs has never tried sharing a SharePoint
28:38
file outside of their organization."
28:40
And like half the, like m- like the room just like broke out.
28:43
Yeah.
28:43
'Cause... And it's like, and it's dumb, but it's true.
28:45
Yeah.
28:45
Like this is what AI integration means.
28:47
It means your databases have to be connected.
28:49
It's all the work that, like, no one says they wanna do.
28:51
Everyone wants to do the cool, exciting work of, like, the prototype building,
28:55
but wow, is scaling and integrating and having all the hardness and security done
29:01
right and done well the ultimate beast.
29:04
So, you know, there is a lot of hype, yes, but hype doesn't
29:07
translate to anything once you're trying to do something at scale.
29:10
And, and, you know, and again, this is another split between
29:12
Silicon Valley and everybody else.
29:14
Mm-hmm.
29:14
And in Silicon Valley there's lots of bold proclamation about
29:18
AI replacing w- you know, workers.
29:20
Mm-hmm.
29:20
And in literally every case it has either been rolled back quietly or not
29:25
so quietly or, you know, in the case of let's say like Block Inject or I see
29:29
a lot of people pointed out that, "I don't think this was AI. I think this
29:32
was a- Yeah ... a series of bad business decisions plus a Bitcoin tumbling, sir."
29:36
Mm-hmm.
29:37
And not really AI.
29:39
You know, i- it's, i- it's easier to hide behind that and sound like
29:42
a good business leader than we, we made some bad decisions on hiring
29:46
and, and product design last quarter of- Mm-hmm ... you know, last year.
29:50
But yeah, the, I, I'm not seeing as much of that hype in the non-tech sector.
29:55
Mm-hmm.
29:55
There is enthusiasm, there is interest- Yeah ... but I think
29:58
people are, are quite wise to it.
30:00
Kevin: And lastly, I wanna ak you, ask you about, I know that you've, uh,
30:03
you're working on a book and a podcast, uh, even more broadly on, on the nature
30:07
of thinking, so it'd be great if you could talk a little bit about that.
30:11
Rumman: Yeah.
30:11
So my podcast launches next week if I get my act together and, and
30:15
get everything posted, et cetera.
30:16
And I did use an AI agent to make the website for that.
30:20
Um, it did a really good job, I think, from my naive perspective.
30:23
That's why I was using that example.
30:24
It did- Thinking about thinking.
30:27
There's also a companion substack, and it, it's about intelligence as
30:30
a social, political, scientific, and economic construct.
30:33
So, you know, just taking a step back, a lot of the fear about AI
30:38
taking jobs or making us dumb or whatev- whatever it is, right?
30:41
A lot of the cognition and economic-based fears are actually
30:45
rooted in this idea of intelligence, the measurement of intelligence.
30:49
Mm. So I went back and said, "Okay, well, where does this idea
30:52
of intelligence even come from?"
30:53
Mm. So, you know, we use these terms like superhuman AI.
30:56
These benchmarks of intelligence were something that we as humans fabricated.
31:00
We fabricated them in the first Industrial Revolution, because at
31:03
that time, we had to assess people for their, their ability to tally numbers-
31:09
Mm ... and remember lists and, you know, have correspondence and act in
31:13
a workforce, wake up on time, get to
31:15
You know, things that today we au- will automate with AI.
31:19
We don't need to tally numbers.
31:20
We have, you know, our calculators and our AI systems for that.
31:23
We don't need to do a lot of these tasks.
31:25
So, uh, I argue that notion of intelligence is actually antiquated
31:30
'cause it's built- Mm ... for the Industrial Revolution that we
31:32
have now automated away, or we're increasingly automating away.
31:35
Mm-hmm.
31:36
Great, so then what ... So that does not give me fear about humankind.
31:39
What that tells me is that we have an imperfect measurement.
31:41
Mm. So how do we then arrive at a better and different measurement?
31:45
And one of the, the main things I, I talk about are other fields
31:48
of science and how they eval- how they evaluate intelligence, right?
31:51
Mycelial intelligence, insect intelligence, even
31:54
extraterrestrial intelligence.
31:55
Mm. The biggest mistake we made was we indexed artificial intelligence on humans.
32:00
There was really no need to do that, and I'm not necessarily arguing for a new
32:04
form of machine intelligence or whatever.
32:06
What I'm saying is our measurement of human intelligence just
32:08
needs to be recalibrated.
32:10
So the book explores that.
32:11
Mm. The podcast is a little bit different in its setup.
32:13
It's, you know, it's really not an interview-based podcast.
32:15
It was actually just a recorded seminar- Mm ... that I did at Harvard, and I
32:19
purposely gathered, you know, from the Harvard community, people in bioethics,
32:23
education, law, computer science, policy.
32:26
So a wide range of really, really smart people and, you know, we
32:30
had just weekly discussions and we just- Mm ... live recorded them.
32:34
Um, so it's kind of a, it's a little, it's a bit of a master class and kind of
32:37
a very different idea for a podcast, and maybe people will love it or maybe they'll
32:41
hate it, and they'll think it's very confusing when 10 people are all talking.
32:44
I personally think it's really fascinating because I learned a lot
32:48
from the students in the class, right?
32:49
Mm. You know, I'm not a bioethicist, so learning about how, you know, they think
32:53
about animal intelligence- Mm ... and the rights that animals get as a function
32:58
of that intelligence from a bioethical perspective is very, very different from
33:01
any world that I would ever be in, right?
33:04
Um, so, you know, it's, like I said, coming out next week
33:07
if I get my act together.
33:08
Kevin: For people who are listening to this who are interested in these topics
33:13
more broadly, any other advice to, you know, where they should look, either,
33:18
either to better understand the, you know, the more immediate practical
33:21
issues about in organizations, how to appropriately manage AI, or to think more
33:26
coherently about these broader questions of workforce integration and so forth?
33:30
Rumman: Yeah.
33:31
Um, I mean, first I'm always happy to talk to folks.
33:33
Yeah.
33:33
My website is just my name, so that's easy enough, and I'm on, really my only social
33:37
media at this point kind of is LinkedIn, but I'm not really that great at that.
33:40
You know, I, I would very closely follow a lot of the legislation that's
33:44
going on, and, and while I do agree that at the federal level there's a lot
33:47
of, um, pushback on it, on regulation on around AI, it, it is interesting
33:53
to kind of follow the narratives in certain states, Illinois, California,
33:57
Virginia, and then to follow certain fields in which we're seeing it pop
34:01
up most and, and a lot of it's, you know, like education and thinking about
34:05
A- and it's sort of corollary is sort of online child safety or, you know,
34:09
like safety for kids with AI systems.
34:11
Those are the things that I personally would keep an eye on, you know, a-
34:14
and, and I don't think that we're gonna have any hard and heavy regulation
34:18
in, in the very, very near future, but I am optimistic about, you know,
34:22
independent bodies, verification becoming very, very important, uh,
34:26
and, you know, and what does that
34:27
But, uh, the nuts and bolts of what that means and what that entails.
34:31
Kevin: Absolutely.
34:31
Rumman, thanks so much
34:32
Rumman: for your time.
34:34
Yeah, thank you.
34:36
Kevin: This has been The Road to
34:37
Rumman: Accountable AI.
34:38
If you like what you're hearing, please give us a good review and
34:41
check out my Substack for more insights on AI accountability.
34:45
Thank you for listening.
34:49
Kevin: If you wanna go deeper on AI governance, trust, and responsibility
34:53
with me and other distinguished faculty of the world's top business
34:56
school, sign up for the next cohort of Wharton's Strategies for Accountable
35:00
AI online executive education program, featuring live interaction
35:05
with faculty, expert interviews, and custom designed asynchronous content.
35:09
Join fellow business leaders to learn valuable skills you can
35:12
put to work in your organization.
35:15
Visit execed.wharton.upen.edu/acai for full details.
35:22
I hope to see you
35:23
there.