Latest / Agentic Conversations (formally mlops.community) / Trustworthy Machine Learning // Kush Varshney // Coffee Sessions #124

Transcript

Trustworthy Machine Learning // Kush Varshney // Coffee Sessions #124 · Agentic Conversations (formally mlops.community)

0:00
Hey, everyone, my name is Cushman.
0:02
I'm a research scientist at IBM research working at our IBM.
0:06
T.j. Watson Research Center.
0:08
I work on trustworthy area and I like my coffee to be
0:12
trustworthy. Welcome.
0:15
Krishna ROM. This is a very exciting episode
0:18
because you are probably the only other person in the world
0:22
that I think can be in the same room and hold, maybe more merits
0:28
than our guest Chris, Today. Could be being too kind of
0:33
Demetrius. So let's talk about a who you
0:38
are, what you're doing here because people are probably
0:40
like, who's this guy in case you missed it, Kristen ROM has
0:43
graduated from being a guest on the podcast to now being a
0:47
co-host, I wanted to get you in the same internet room as Kush
0:52
because I feel like you two have very, very aligned goals and
0:58
you've been doing all kinds of cool stuff around fairness.
1:02
Around responsible AI as Crush has been doing.
1:05
And we'll hear about in a sec trustworthy machine learning.
1:08
I mean, it's killer, I know that you are the chief AI scientists
1:12
at Fiddler's Fiddler doing incredible things around,
1:15
explain ability. And so let's talk a minute about
1:22
who kush's. And then, I want to hear your
1:24
takeaways from the conversation. We just had with Kush, so kush.
1:28
For those who do not know, you heard that he is doing.
1:31
He's a researcher at IBM. We talked a little bit about
1:35
what IBM is doing in the field, but Kush himself has been
1:41
recognized for so many different things.
1:44
The best paper award in. The fusion 2009, the solely
1:50
2013, kdd 2014 and SDM, 2015. I mean, the accolades just keep
1:59
going. There's a long list of them.
2:02
It would take me probably an hour to read out all of them,
2:05
but more recently, one thing that I want to highlight is this
2:09
year, he released a book called trustworthy machine learning.
2:12
And that's really what our conversations centered around
2:15
today. He also open sourced with a
2:19
bunch of other Incredible Minds. Some trustworthy, AI packages
2:26
and I want to hear from you Krishna room, what were some of
2:29
the key takeaways that you had in this This conversation.
2:34
Thanks to juice. One of the takeaways we
2:37
discussed was why should we as Miculek practitioners or other
2:42
stakeholders care about trustworthy machine learning,
2:46
who better to discuss this. Then the author of the book of
2:50
boo contest with the machine learning.
2:52
Yeah. Then we discussed about.
2:54
How do we apply these principles?
2:56
As part of the machine learning operations or ml Ops pipeline.
3:01
How do we apply this during In say data, preparation or model
3:05
validation, is that enough? We will discuss that it's not
3:09
even after the models are diploid, we need we need to
3:12
monitor these models, not just for accuracy, but also for bias,
3:17
robustness, explain ability, and other dimensions.
3:21
So we discussed how holistically we need to look at just what the
3:27
machine learning. Not just confining to model
3:30
developers or missionary. She has but also looping in all
3:34
the other relevant stakeholders, this could include people who
3:38
are impacted by the model people with background in policy, law
3:44
product management, business and so forth.
3:48
I'll let you two discuss about the packages and the other
3:51
things we discussed. Yeah well there is a great point
3:54
that he brought up and it was almost like it reminded me of
3:58
all of those different diagrams that you see on what is mlr ABS
4:02
and The different phases of a moab's.
4:05
He basically broke down the different phases of
4:08
trustworthiness. And what you need to be aware
4:11
of, on each different step of the ml Ops journey.
4:15
And so I appreciated that whether it was just figuring out
4:19
what numbers are important to you.
4:20
What are the metrics that you are looking at, and how you can
4:24
make sure that those are trustworthy all the way to that
4:28
monitoring, and how to make sure that the ground truth, which is
4:31
delayed. You're And how do you know when
4:34
to retrain something? If you're getting that ground
4:36
truth, 30 days later. So that was awesome.
4:40
Like you said the open source packages, he has been releasing
4:43
have been incredible, it's really cool to see what they're
4:46
doing and how they're helping. Just regular people regular
4:50
practitioners start to implement this in their workflow and last
4:55
but not least, I also enjoyed the conversation, almost more of
4:59
like a philosophical conversation around these large
5:01
language models. And I'll let people discover
5:04
what we got into their when it comes to trustworthiness.
5:08
So, that's it. I want to mention something.
5:12
Before we jump into this conversation, which is, if
5:15
you're not subscribed to our ml, Ops Community, newsletters, jump
5:18
into that. Like, you're definitely going to
5:20
enjoy it. We've got three different
5:22
newsletters that are being pumped out.
5:25
One is around the weekly ml, Ops roundups one is a monthly and
5:29
then the other is the best of slack, everything that has
5:32
happened. Slack.
5:33
All the best threads get put into a newsletter, so you can
5:37
engage with our very active slack community in that way.
5:41
That's all for. Now, let's get into this
5:43
conversation. So, Chris, I'm excited to be
5:52
talking with you, Krishna ROM. I am very thankful that you came
5:56
along, and join me decided that you could do this.
5:59
It feels like I've got to juggernauts in the Belay I
6:03
trustworthy AI space and I want to start with this kis.
6:08
You pursued a degree in electrical engineering, what
6:13
made you decide that? It's a good question.
6:19
Um, so I think there's a few different things.
6:22
So one, interesting fact, both my grandfathers and my dad,
6:27
they're all electrical engineers actually.
6:29
All were professors of electrical engineering, so I'm
6:32
sure that had a huge role but beyond that, I think, when I was
6:38
going to college, I wanted to do something where I would be
6:42
thinking about what new could be done in the world things that
6:45
have not been done before. And that seems like one
6:49
discipline where that could happen.
6:52
And also, it's kind of, I think the, in some ways, the liberal
6:56
arts of Engineering in a way, there's so many different things
7:00
that come together with, in electrical engineering, you
7:02
could do very physics oriented things, very sort of Matthew
7:07
things, and not some kind of a lot of things in between.
7:10
So I think that was really developed into this career, that
7:15
you've undertaken. Akin, how can we connect those
7:19
two dots like looking backwards? What was it and that retrospect
7:24
now? Yeah.
7:28
So I think yeah well I was kind of studying.
7:32
I mean, I kind of gravitated towards the more mathy side of
7:35
electrical engineering and then worked on Signal processing and
7:40
other topics and that kind of morphed into doing more machine
7:45
learning, which is actually very closely connected to
7:48
technically. So then, yeah, I became a
7:51
machine learning researcher in some capacity and then I joined
7:56
IBM. Research and the group that I
7:58
joined led by sashka Mozilla, which they were doing a lot of
8:03
interesting things. So this was back in 2010 and it
8:07
was very new for me to even think that machine learning
8:10
could be used in real-world applications, so especially ones
8:15
that affect people's lives. So, at the time, the group was
8:18
working on for making predictions to better manage
8:22
employees doing some things in healthcare as well.
8:25
And that was all. New to me because I didn't
8:28
realize that this could be possible and then that
8:30
eventually led to kind of further considerations more
8:34
broadly. I know Krishna ROM has he knows
8:38
this already but for people like myself that aren't so Versed and
8:43
well-versed in the space. Can you break down real quick?
8:46
What exactly trustworthy AI is and trustworthy machine
8:50
learning. Yeah, so the way I like to start
8:57
to answer that question is just ask you, I mean what would it
9:01
take for you to trust another person?
9:03
What makes them worthy of trust? So if you were hiring like a
9:07
carpenter to work on your house or like any of any sort of
9:11
thing, like what would it take for that other person to be
9:13
trustworthy for you. I'm usually not the one doing
9:19
the answering on this questions here.
9:25
What are you doing? Well, what is it?
9:30
Take something I think trust is a feeling first and foremost,
9:34
right? And it comes from me, feeling
9:40
like, I know the person and I know that their intentions are
9:46
good and Krishna Rao. Maybe you have a different
9:49
answer though. I think you had that said
9:54
there's a great angle and as Kush was mentioning this as kind
9:58
of limited of this counter argument.
10:01
We have heard sometimes read. Like see we have we have learned
10:07
to trust fairly complex objects, complex missions, like airplanes
10:13
or internal combustion engines and several other things read
10:16
even medicines without understanding how they exactly
10:19
work. Why is this any different?
10:22
Yeah, so no. I mean, I think everything that
10:24
you guys under the things that I also think about.
10:27
So it turns out that when you talk about trustworthy people,
10:33
there's been a lot of research on what it takes and there's
10:36
kind of these four attributes that come about from the
10:40
research. So one of them is like
10:42
competence. So can the other person do what
10:45
they're supposed to do. Second is some sort of
10:48
reliability so that that competence sticks around.
10:51
Around in different settings, different conditions.
10:54
The third one is kind of going more towards with Demetrius said
10:59
Tom. So can you communicate back and
11:01
forth with the other person is there's some level of intimacy
11:04
that you have with them and then the last also touches on I think
11:08
Demetrius is point. So do they have goals some
11:11
Beyond just their own goals? So are they doing something
11:14
more? Broadly, something for good.
11:16
So all of those are kind of, I think just as applicable for
11:20
machine. And to Krishna rooms point I
11:23
mean it doesn't necessarily mean that we need to understand every
11:28
single thing but we do need to have some level of understanding
11:31
but also just being competent just I mean the medicine does it
11:38
actually treat me or not? Does the airplane fly or not.
11:41
Right. So all of those things kind of
11:44
come together and they have kind of mappings to precise things
11:47
that we want out of machine learning systems.
11:49
But before even getting to To that.
11:51
I think that the worthiness of the trust is that feeling and I
11:56
think all of these attributes contribute to it also looks like
11:59
another perhaps maybe difference might be in the case of say
12:04
internal combustion engines that we we expect the behavior to be
12:08
somewhat deterministic like we don't, whereas with the machine
12:11
learning models Rea systems, broadly often their behavior is
12:16
some sometimes the emergent they behave in ways that we may not
12:20
anticipate. And maybe that's why there is an
12:22
additional layer or additional need for an understanding how
12:27
this work. Yeah, no, I think that's right.
12:32
And I mean, the internal combustion engines, are a good
12:35
example because they are general purpose, technology, just like
12:40
machine Learning System machine learning as a technology is
12:42
rates. So you can have a engine in a
12:45
car and a tractor in a pump, I mean all sorts of things, right?
12:49
So, the same way you can use machine learning and hiring
12:53
decisions in lending decisions in criminal justice decision so
12:57
forth. So, I mean, there's some Aspect
13:00
of what is the machine being used for that helps determine
13:04
what's going on and then there's some things that are just
13:07
naturally inherent to it. So I think there's a mix of
13:13
what's involved. The reason that we trust
13:16
something is basically the track record and we've seen like
13:21
whether it is a plane that we get on and we've seen thousands
13:26
of planes fly before so we trust that this one is going to be No
13:30
different or if it is like a different kind of trust with a
13:35
person. Maybe that's why it's harder for
13:38
us to trust someone that we don't really know because there
13:41
is no track record at the beginning.
13:43
And so it's fascinating to think about that when it applies to
13:48
machine learning and how whether or not we're working on this
13:54
machine learning model. Do we trust it?
13:57
Or is it if we're the end user of it?
14:00
And we are being fed, whatever the machine learning is telling
14:03
us, do we trust it? Because do we have that track
14:06
record? We understand that we we can see
14:09
the lineage or we can see the what it's it's inferred before.
14:15
And so thinking about all of that, like I know there's
14:19
probably another layer that we can go deeper, which is
14:23
trustworthy AI. Who is it for?
14:26
Is it like the end user? Is that what you're talking?
14:30
What about when you're talking about trustworthy machine
14:32
learning and AI? Or are you talking about this
14:35
for a machine-learning engineer? Yeah, I think that's a great
14:41
point. So I think eventually we do want
14:44
the end users and even if they're not users as in, they're
14:48
touching the system but they might be the affected parties
14:51
about whom decisions are being made to really have that sort of
14:56
trust. But part of that I think is that
15:00
the the engineers themselves also kind of need to have that
15:04
because otherwise, how are they building?
15:08
The rest of us can also have some trust in.
15:11
And there's also this concept, I mean, you kind of were
15:16
mentioning. It's a difference between kind
15:18
of institutional, trust and interpersonal trust.
15:21
So, like when you think about an institution, like a bank or the
15:25
post office or something like that, I mean, it's the reason
15:29
you're trusted is very different than why, or how you would trust
15:33
another person. So those institutions you kind
15:35
of just trust them because Someone else who's kind of like
15:41
backing it in a way. Whereas, with another person you
15:43
kind of have to develop over time.
15:45
So all those are kind of different nuances to it.
15:49
And yeah, I mean the engineer has certain needs the end user
15:56
so that could be the doctor or the lending officer they have
16:00
different needs. The patient has different needs
16:03
some the loan, applicant so forth.
16:05
So I mean everyone kind of has different Instead we need to
16:09
appeal to would eventually, I think we want to go to a point
16:13
not only where on these systems are trusted but they're worthy
16:17
to be trusted because there's I mean situations where you trust
16:21
things that you shouldn't really be trusting but we want to work
16:25
towards getting to the point where they're I mean just good
16:29
enough to deserve that. That's a great Point.
16:33
Like sometimes I My worry is that machine learning with
16:36
machine learning models? And a systems people, people
16:40
start trusting them more than they should read.
16:43
Like it's very important for if you think of a doctor and a
16:46
doctor using, let's say missionary model for medical
16:50
diagnosis or treatment recommendation, they should know
16:53
when to trust the model that what are the limitations of the
16:56
model? When should they use their own
16:59
judgment or other resources when should work?
17:03
What are the failure modes of the model?
17:05
And even when they think that the model is Correct.
17:08
How is that model making the decision?
17:10
Is that perhaps the taking the right inputs is that just
17:13
relying on maybe some false indicators.
17:16
I think all those things become, very, very important and I
17:22
agree. I mean, I think there needs to
17:24
be some level of calibration of that trust you can't under or
17:27
over trust things and I think the system's themselves can help
17:32
put that using uncertainty quantification in other
17:36
approaches to Present their own limitations to be humbled
17:40
themselves. So, yeah, I think there's work
17:43
to be done. We can, you can also these
17:45
different systems a little bit. Sorry to cut you afternoon room.
17:49
But can you dig into what exactly do you mean by it these
17:53
different systems and when they can come into play?
17:59
Yeah, so I mean I think krishnam was alluding to, for example,
18:03
explain ability, which is a way for a machine for us as people
18:07
to understand how the machines make their decisions.
18:11
Then when I was saying uncertainty quantification,
18:14
that's basically the idea that the machine gives its own
18:20
confidence. So it will tell you.
18:22
I mean, I'm really like 99% sure.
18:25
I'm just guessing and neural. For example, which is the most
18:31
popular machine learning approach today is notoriously
18:34
bad at its own calibration, so it'll say 99% even when it
18:39
really shouldn't. So when we are talking about
18:43
uncertainty quantification, what we want, is the system to
18:48
actually be true to its word. So so that's kind of the
18:52
important part of that should make your shirt.
18:54
This is never trust a neural network.
19:00
So the additional challenge is like see if you think of other
19:03
types of stakeholders read like in domains which are regulated.
19:07
We may all too often have say a complaint steam or a regulatory
19:11
agency. They want to also know whether
19:14
the models are free of biases and whether the models are
19:19
making Fair decisions for different stakeholders that s
19:22
like a little bit different type of trust or understanding.
19:27
So it God, I consider that also as part of trust with the well,
19:34
I certainly would here. So we maybe can you shed some
19:37
light on, what are the challenges red?
19:39
When it comes to see, something, like fairness of a missionary
19:43
model, what is famous even being and how should we go about
19:47
thinking about that? Yeah.
19:50
I mean I'm sure you know even more than I do on this topic but
19:55
I think some of the challenges are that fairness itself is a
19:59
contested concept. It comes from lots of different
20:02
Traditions whether philosophically or
20:05
sociologically or otherwise. And I mean when we're talking
20:10
about machine learning and fairness were often very limited
20:14
in the scope that were considering it's usually
20:17
allocation decisions of some kind.
20:20
Which Related to a concept called distributive justice and
20:24
even then there's so many mathematicians of this concept
20:30
and even if we try to mathematize it, that already
20:33
could be fraught with issues. But let's say we do, there's a
20:38
lot of conflicting sort of definitions.
20:41
Some of them with one of them is called disparate impact or issue
20:46
or somewhat. Similarly, a statistical parody
20:49
difference. So, those Those that kind of is
20:51
related to, is there an equal selection rate across groups?
20:56
So do men and women have an equal chance of getting that
21:01
loan or something like that. Getting the job.
21:04
Whereas there's other definitions, which are more
21:07
related to the accuracy of the model.
21:09
So things like average, odds difference is one of those
21:13
examples in there. You don't need to have the same
21:16
rate of approval but the ACT underlying accuracy, Among the
21:20
applicants should be the same and they're very different
21:24
philosophically because in one case you're kind of in the first
21:29
case you're assuming that there's some sort of social bias
21:32
already that the things that you're measuring their already
21:35
flawed. And so the only common
21:38
denominator is the selection rate that that should be the
21:41
same. Whereas in the other case,
21:44
you're assuming that the features are good, the stuff
21:49
that you're basing. The Asian on makes sense,
21:51
doesn't already have its own bias.
21:53
He's so then kind of an accuracy measurement being the same
21:57
across groups would make a lot more sense.
22:00
Yes, I think that's a really, really nice subtle point.
22:03
Like I think fairness there are even tensions between different
22:07
Notions of fairness as you alluded to and that makes it
22:10
even harder a lot more challenging.
22:13
So talking of safe fairness or the challenge of adopting these
22:18
practices worship. Say companies care about
22:23
trustworthy machine learning or why should they adopt this
22:26
fractal? Just wrote The Machine learning.
22:31
Yeah, so as we talked to different companies in different
22:34
Industries, there are several factors that are coming forward
22:39
as reasons. So one of you, one of them you
22:42
already mentioned, which is regulation.
22:44
So certain industries always have been regulated and with AI
22:49
involved in the decision-making process.
22:51
That doesn't mean that they get to kind of disregard their
22:55
existing responsibilities so vending be a great example on
23:00
fair housing. Equal Employment and so forth.
23:02
So even if AI system is involved, they still have to
23:07
uphold their responsibilities. And there's also a lot of new
23:11
regulations coming about that are specific to a.
23:14
I so in the European Community, for example, there is this new a
23:20
I act and that one. I mean he's going to regulate
23:25
things quite quite a bit. So insisting, on fairness to
23:28
explain ability for so-called highwomen Risk applications of
23:31
AI. So that's on the regulation side
23:35
and it's not just in Europe, I mean everywhere, I mean this is
23:38
happening but there's other reasons, right?
23:41
So there's the issue of kind of the reputation of the company as
23:47
well. So they want to do things that
23:50
are in line with what all of their stakeholders want, not
23:55
just their shareholders may be. So there's a lot of reputation.
24:00
Morris. Good found a company is doing
24:03
something with AI and it's discriminating or doing other
24:06
bad stuff. Then I mean that's going to come
24:09
back on the the organization and kind of be like this isn't
24:14
something that this company should be doing and a lot of
24:18
that is also stemming from A Renewed focus on social justice
24:22
and in several societies around the world.
24:25
So in the last few years especially there's been a
24:28
renewed Reckoning there and then another factor that we see a lot
24:33
is, I mean, just companies are now using operationalizing AI
24:40
systems using a mellow in tons and tons of applications.
24:44
So now you'll have a company that'll have hundreds or even
24:48
thousands of models in operation and just making sure that
24:52
they're all well tested. Well cover and is its own
24:54
Challenge and that's another place where I'm trustworthy
24:58
machine learning is important. So a lot of different Things.
25:01
It's not one or the other so that it remains me of something.
25:05
I recall reading any Kathy, O'Neal's book corn, the weapons
25:10
of mass destruction, she kind of gives this example if your judge
25:15
is was biased in the past maybe that judge effects.
25:19
Just that constituency read that District or that County but now
25:24
if machine learning model becomes systematically biased
25:28
and and that model has the potential to To be widely
25:31
adopted across the entire country, or maybe even multiple
25:34
States multiple countries. Then there's a potential for
25:37
that bias to get exercise a bit late to be scaled and to
25:43
propagate everywhere. I think that's like humans are
25:46
of the real. All the always implicit or
25:49
explicit bias is amongst human decision makers as well, but
25:54
with the missionary models that is aside additional dimension.
25:59
Yeah, I know. I fully agree.
26:01
I mean, the scaling is the bigger issue because when all of
26:05
us is people, we have our own different biases.
26:09
They kind of in aggregate, kind of cancel out in some ways.
26:12
It's not always, but I mean, you can think of it that way but
26:16
then I mean, a single machine learning model, that's supplied
26:20
at a very large scale, it has exactly one bias and that's
26:24
going to apply across. So yeah, I think that's one big
26:28
big. Speaking in things and one way
26:33
to think about it. I mean, for people who have
26:36
thought about machine learning in other contexts.
26:39
So there's this concept of Ensemble models where you bring
26:43
together lots of different models and kind of average them
26:46
together. So that actually reduces certain
26:50
biases when you do that Ensemble ink.
26:52
So same thing with lots of human decision makers, you kind of
26:56
average out some of the bias. But if it's a single Sort of
26:59
model than you don't get that event and this is this great
27:04
Point like a lot of attendees our listeners to this podcast on
27:10
machine learning practitioners safe as a missionary
27:13
practitioner. So I have to decide yet
27:18
trade-off between say standard metrics like accuracy and
27:23
ensuring fairness or other restore.
27:26
The machine learning Dimensions. How should Could have got, what
27:30
is the way to think about this? How should I finish?
27:32
Should I even start if am additionally practitioner Yeah,
27:39
that's a very good question. And I don't think we have a
27:42
solution right now necessarily. So there's even the question of
27:47
are there trade-offs or not? Because your question assumes
27:51
that there is a trade-off. So I think we're still
27:55
understanding what all these different dimensions.
27:57
So between like, adversarial robustness and explain ability,
28:01
is there are trade off or not? This one help.
28:05
Can you do both at the same time?
28:07
Same with Things like privacy fairness and basic accuracy.
28:12
So I mean, first of all I think we need to have a better
28:15
understanding of what even the characterization is but in
28:20
practice right now. I mean you don't need that full
28:23
picture necessarily. So just like you do cross
28:27
validation to kind of optimize over accuracy.
28:32
I'm just I mean look at a bunch of these metrics for
28:37
interpretability or For robustness or fairness,
28:40
alongside accuracy and just work in that direction and there's a
28:46
few things that you need to think about.
28:48
I think. So, there's the values
28:51
themselves. So what do you value?
28:54
Do you only care about that? One of the metrics are?
28:57
Do you care to balance out a bunch of these different pillars
29:02
of trustworthiness? And that's some of the question
29:05
that you have to ask yourself or I mean the practitioner maybe
29:09
not but you have to go and talk to your organization's sort of
29:14
policy makers. So your compliance folks here Co
29:18
your I mean in general, whatever the sort of moral compass of
29:24
your company is has to influence kind of how you think about the
29:28
balance among these different different aspects of your
29:32
system. So think that's the starting
29:35
point and then one thing I will mention is that A lot of people
29:40
do assume that there is a trade-off between fairness and
29:44
accuracy, and that's not always the case.
29:47
There's lots of empirical work, and lots of consequential
29:51
decisions that show that there is really no trade-off
29:56
empirically. And then theoretically, we had a
29:59
paper, a couple of years ago, the first author.
30:03
She just started as a professor to Maryland.
30:05
Some time with her Delta and, um, and that pain It kind of
30:10
illustrates that in a bias free world.
30:15
There is no trade-off, so-called construct space.
30:20
So the reason we see accuracy dropping sometimes in practice
30:25
is because the accuracy itself is measured in a biased way.
30:29
Because when you get rid of biases, then the accuracy is
30:34
now, being kind of reflected in a different bias.
30:38
So called and that is flawed in many ways.
30:42
So I mean, this is not a easy sort of thing to think about but
30:49
a drop in a couple percent of accuracy in the traditional way
30:52
of measuring a c is actually maybe not truly a drop in
30:56
actors. So yeah a lot of different
30:58
things to be afraid. This isn't a nice Point like
31:01
yeah. Sometimes as a missionary
31:03
practitioners that they may be a tendency for us to think of like
31:08
The mission earning as a closed system but but it's a good idea
31:12
to go beyond that. Like if you think of facial
31:15
analysis and the challenges that says various companies had with
31:20
that the the solution real world not just say optimizing the
31:24
missionary pipeline. They went about collecting more
31:27
data that further, they went about realizing there is not
31:31
adequate representation of certain groups and then they
31:33
collected more data. And in the process they not only
31:37
improved and the accuracy or other metrics for that group,
31:41
but also improved overall. So it's not not always a
31:45
trade-off between accuracy and fairness or other dimensions.
31:53
Yeah, I agree. And I mean this podcast is about
31:56
mL of straight and so the the whole life cycle has I mean
32:02
issues where we can influence the mitigation of various
32:07
aspects of arms that can come about so it can be in the
32:11
problem specification. Even before starting should we
32:14
work on the problem to begin with in the specification?
32:18
What are the metrics? We care about what we talked
32:20
about. As you said, question could be
32:23
in the data collection data preparation.
32:26
So one interesting, anecdote, I like to bring up sometimes is
32:31
one of my colleagues maninder. He ran an analysis on this kind
32:36
of well-known problem. Now, where if you use health
32:41
care costs as a proxy for Health Care, need in health, insurance
32:46
decisions and stuff. So preventive care for nursing
32:49
and things like that, then, There is some sort of racial
32:53
bias because in the u.s. is at least African Americans are less
32:57
likely to utilize the Health Care system for the same level
33:01
of how sick they are. So this has become kind of known
33:05
among a lot of people but when in their did was he studied this
33:11
when he kept different cart kinds of health care costs
33:14
separate two separate variables. So inpatient separate from
33:18
outpatient separate from emergency room and so forth.
33:22
If you kept all of these different cause separately,
33:25
there was much, less racial bias that was induced, but when you
33:28
add it, all of them together into a single variable, the
33:31
single cost variable there was much more racial bias in the
33:36
models that that followed. And this is something that a
33:39
practice and data scientists would just naturally do because
33:43
most of us don't actually utilize the Health Care system.
33:46
So we have a lot of zeros in that variable so when you have a
33:50
variable with a lot of zeros, it's E natural to combine it
33:53
with some other variables so that you have fewer zeros and
33:56
you're able to do a better job in modeling, but this
33:59
unconscious Choice actually exacerbates biased quite a bit.
34:05
So yeah. Hey, I'm Vishnu.
34:08
I'm a data scientist at first hand, and I definitely think
34:12
that you should subscribe to the ml Ops coffee sessions podcast.
34:16
It's the best podcast out there to stay on top of what ml Ops
34:19
actually is to talk to the truth, thought leaders in the
34:22
space. And oh, by the way, dimitrios is
34:24
absolutely hilarious. What a weird guy, you should
34:27
definitely subscribe to the podcast.
34:32
You basically talked about that first, those first steps where
34:37
problems can happen, right? And especially when it comes to
34:40
trustworthiness and fairness, you're talking about the problem
34:43
specification, then you go into important metrics and what kind
34:46
of metrics are you looking at? And the data collection, what
34:50
else is there? Because we are talking about
34:53
envelopes and when you deploy something what like when you're
34:57
monitoring something, where else can these problems arise?
35:03
Yeah no that's exactly the point.
35:07
So everywhere so in the modeling itself so once you're past the
35:11
data prep so what we recommend is actually add in some
35:16
mitigation algorithms so our group created an open source
35:21
toolkit named AI fairness 360. Which provides a lot of tooling
35:27
that can just be plugged in. There's a bunch of ways
35:30
medication algorithms. So, Can be part of creating
35:34
modeling itself. Some of them are different types
35:36
of models, so that's something called in processing.
35:40
So there's an extra constraint in the machine, learning
35:43
algorithm itself, to him for fairness, but there's also
35:47
pre-processing and post-processing things that you
35:50
can do. But then going beyond the
35:52
modelling. So once you have created a
35:55
model, there's an important step of evaluation.
35:58
So in the evaluation phase description, Iran was saying I
36:02
mean it's important to bring in all of the different
36:04
stakeholders in doing the evaluation.
36:08
So especially people who have had lived experience of
36:11
marginalization themselves because they're actually better
36:15
able to recognize different types of Harm's even if it's not
36:19
for their own group but for any group.
36:22
So making sure that you do a really good job on the
36:25
evaluation and then on the monitoring side once you have a
36:28
deployed stomach that so I guess I don't have to tell this.
36:32
Crowd button, actually monitoring is important.
36:35
So don't just let the system go and live live on its own.
36:40
So keep monitoring, but then monitor for all of these metrics
36:43
that you cared about things that you specified, tweaking with.
36:46
So compute fairness, metres compute drift metrics bunch of
36:50
things as the mall. So, true.
36:53
And so, Go ahead, especially that's a great point.
36:58
I'll just give you like to give and share an anecdote recently
37:03
in the AC mkdd. Consonants, we presented
37:07
tutorial on missionary model monitoring in practice.
37:12
So what I found really intriguing was that although kdd
37:16
is research conference with a good representation of
37:19
academicians and people from industry, I notice that dearly.
37:25
All the 100-ish attendees were all from industry.
37:28
There is hardly anyone from like you know, Academy or students
37:34
and so so I felt like there's a little bit of Disconnect so for
37:38
people in Industry this challenges associated with
37:41
monitoring models seems like real this is a pain point that
37:44
all of us are facing but when you look at the research a lot
37:48
of research seems to focus on the same data preparation stage
37:53
or model validation like Like is this just my like you'd unique
38:01
perception or is this? Do you do see a similar
38:03
perception of this is like maybe I'm just living from are biased.
38:07
Let's not. You know, I think you're
38:10
absolutely right. So on the academic research
38:14
side, I mean everyone the fun part is the modeling part.
38:20
I mean, that's where I mean, people really enjoy doing it as
38:24
a grad student, I mean, that's how you write papers and I mean,
38:27
all of that stuff. So I think there was a paper, a
38:30
couple of years ago, David donoho.
38:33
So he's a statistics professor at Stanford, he wrote this thing
38:37
and we need talked about what data Sciences or like I think he
38:41
said 50 years of data science or something like that and he laid
38:45
out all the steps and then I mean he spend no time and really
38:49
discussing the modeling part because he said everyone does
38:52
that that's like the thing everyone is drawn to but let's
38:55
talk about everything else and that everything else is really
38:58
what makes or breaks things and without it there is no real
39:05
world deployment. So yeah I mean like my own PhD
39:08
dissertation. And I mean, it was on that
39:11
modeling part kind of came up with this fancy new
39:14
regularization term and I used all sorts of calculus of
39:19
variations and I mean all sorts of stuff.
39:22
But at the end of the day, I mean in real-world practice,
39:27
it's the beginning and the end other the important part and one
39:31
thing I'll say, I mean just connecting back to our airplane
39:35
discussion from the beginning, right?
39:36
So when you think about flying in Clean.
39:39
Right, there's take off. There's cruising and then
39:42
there's Landing. So the way things are headed
39:46
with Automation and everything else that cruising part is like
39:50
the modeling part, right? So that's going to actually get
39:54
automated away pretty soon, but it's the take off, so that
39:59
problem specification figuring out what you're doing and then
40:01
the Landing part. So, the evaluation and, and so
40:04
forth. Those are where the human touch
40:06
is really needed. And those are the toughest Parts
40:09
because you can't really automate those.
40:11
And so I think as things move forward, I think that's where a
40:17
lot of the the human centeredness but also the
40:20
challenges will continue to write.
40:22
That's a fantastic and I legere like like golf or Landing or
40:28
dealing with unexpected situations that we did.
40:32
Like if say there is a hurricane or whatsoever that like
40:35
something I think that's why I like even monitoring models is
40:39
Very important. And also like I think if there
40:42
is anyone from the research Community listening to this, I
40:45
would like to just add that there are a number of exciting
40:49
challenging open research problems even your own
40:52
monitoring modern monitoring itself just like all the
40:56
interesting things happening during validation or model
40:59
modeling itself. Yeah, no, I would agree.
41:02
I mean, one basic thing is, I mean, when you're monitoring you
41:06
often don't have the ground truth labels, right?
41:08
So How do you estimate model drift?
41:12
When you don't have labels that very basic problem is.
41:17
I mean, very open and in there's a lot of room for research
41:19
contributions so I can see why this is so important if we are
41:27
talking about something like you were talking about and how the
41:30
EU classifies like high-stakes, right?
41:33
Like if it's an autonomous vehicle or we're talking about a
41:36
judge that is potentially sending someone to jail for But
41:40
what about that person who is working on a recommender system?
41:45
For we just actually talk to someone last week.
41:48
They're working on a recommender system for GIF website, do they
41:53
still need to be thinking about this or can they give it a slide
41:56
and it's a pass? It's a good question.
42:00
I think they can give it a slide a little bit so I can like the
42:04
way the European Union has some laid out, certain things are
42:08
high risk. Certain things are not.
42:10
And I think the distinction is like what's the consequence of
42:16
the harm? If something goes wrong, so if
42:19
you got the wrong GIF, is it really going to be harmful for
42:23
you or not? And this is exactly the way it
42:27
kind of safety engineers and other engineering disciplines.
42:30
Also think so things are not safety concerns unless the harm
42:34
is high enough. So so yeah, I mean there's no
42:38
difference. I don't think we have a nation
42:42
please. I can see that.
42:43
So let's change gears for a moment because I do want to get
42:46
into your book and a little bit of the nuances around your book
42:52
which is I automatically notice that you had self-published a
42:57
book, you didn't go with man and you didn't go with O'Reilly what
43:00
made you choose that? Yeah, so I did start actually
43:06
working with a publisher and then we parted ways.
43:09
So the reasons there were a lot of them, I think so.
43:14
A few of them one is that? I don't think the trustworthy AI
43:20
is something where, like, a very prescriptive approach to a book
43:25
and work. So a lot of Publishers do want
43:29
something where you're giving the step by step recipe.
43:32
This is exactly what you need to do.
43:35
And what I would rather impart is the fact that you're just
43:39
need to slow down and think for yourself and kind of equipped
43:45
with you, equip you with some of the tools to do so but not give
43:49
you the exact recipe. So that's one sort of reason the
43:53
other is I wanted to make the content accessible as I was self
43:59
reflecting as much as possible to anyone in the world.
44:02
Who wants it because I think that's one of the selflessness
44:07
goals of trustworthy and machine learning is too kind of Empower
44:10
all people, no matter what station in life that they are
44:13
in. So by self-publishing, I can
44:17
make the PDF for the HTML versions free and then the
44:22
paperback as cheap as possible. And it is really, I mean, fairly
44:28
accessible that way. So, for that, I think it's
44:32
Combination of those things. And if you look at the world
44:36
straight now, I mean, where is publishing now?
44:40
I mean there's a few large Publishers, they do have maybe a
44:46
little bit more reach through their marketing.
44:48
But these days, like anyone can really do anything on the
44:53
internet. So it's not very much of a
44:56
limiting or gating Factor. So just to be able to do the
45:01
right things that as I Alton to be self-publishing males.
45:05
And since there is a bit of a question that I think about when
45:10
people do write books, which is how do you ever finish it, and
45:15
don't, you want to just continue to update it especially since
45:18
you do have it on hand? And you can just go back and
45:22
say, all right, well we've had this update in the world, I need
45:26
to address that and then at the end of the day you never get any
45:30
other work done because you're constantly Your book that you
45:33
wrote for years ago? Yeah, no, it's a great point.
45:36
And yeah, through the process that I followed, I could update
45:39
it every day if I wanted to write.
45:42
So but the fact of the matter is for me at least I have so many
45:47
other things to keep me busy that.
45:52
I mean, I mean I sat down, I do have some edits that I'm going
45:56
to do at some point. I haven't updated it at all yet,
45:59
but I think I mean just the fact that there's so much more to
46:04
bring Machine learning into operation.
46:07
So, working in our tool kits, working with clients and other
46:11
nonprofit partners and stuff that keeps me.
46:14
So busy that, I think I would rather spend the effort really
46:19
operationalizing things with. I mean, in a lot of important
46:23
use case is rather than fine-tuned, every single thing
46:28
talking of updates and how fast our field is evolving.
46:32
I'd like one of the recent developments.
46:35
It is around what are often called like large train models
46:40
and large language models or language image models, like
46:46
physically like say let's little bit.
46:48
This lot of talk about large language models like G PT 3 or
46:54
calm or models like Dolly and a variance.
46:59
So what is trustworthy machine learning mean in these settings?
47:04
Can you maybe share? Thoughts.
47:08
Yeah. No.
47:09
I mean this is all this has really exploded in the last
47:11
couple of years. So yeah, when I started writing
47:14
the book, this was back in January of 2020 right before
47:18
covid. And um yeah the world has really
47:20
changed with that but also with these large language models or
47:25
Foundation models or whatever you want to call them.
47:27
So yeah, I mean on the language side, I think it's much harder
47:33
to define the sort of things that we Are talking about, it
47:38
think it's much easier with tabular data, so the reasons for
47:43
that. So there's so many different
47:45
types of Harm's that can come about with language.
47:49
So it's not just about bias, although that is one sort of
47:54
important aspect. So that mean, in terms of bias
47:57
there can be kind of like occupational, gender biases.
48:00
That men are doctors and women are nurses, or things of that
48:04
sort. But in there can also be other
48:07
things like In sentiment classifiers, certain groups, get
48:10
more positive sentiment versus others but the school, I mean,
48:14
keeps going. So there's a lot of toxicity
48:18
that can be generated out of large language models, which is
48:23
a different sort of harm. There's the lack of Common Sense
48:26
reasoning, which is, can lead to a lot of harm as well.
48:32
So there's a whole variety of things that can go wrong and I
48:37
think, You haven't even gotten to a point where we can even
48:41
categorize or collect all of those.
48:44
Those harmful behaviors from large language models.
48:46
So until we even have some understanding of the scope, it's
48:51
even harder to figure out what to do about make the challenges.
48:56
Even like further. Like I saw, I was reading an
49:01
article in the communications of the ACM just over the weekend.
49:05
This is an article written by Professor super Rao come bumpity
49:10
on. The changing nature of Tau
49:14
research that he was comparing EA research, with natural
49:18
sciences and kind of drawing an analogy to engineering versus
49:23
science. Like, if you look at
49:25
traditionally, how lot of computer science has evolved, it
49:29
is still a little bit like engineering.
49:31
We, we decide what we want to achieve, whether through
49:33
algorithms, or software engineering or systems, and
49:36
then, the systems Achieve those. Whereas now with this kind of
49:41
law, a large language models, we are going to this realm of.
49:46
We have trained this models, what all can they achieve that?
49:49
Now this model was trained with some intent in mind.
49:53
Now, it's being able to explain chokes.
49:57
Who knows what are the other functionalities maybe other
49:59
limitations of this models? I think that's that's kind of a
50:02
very interesting Safeway or evolution of how even machine
50:07
learning. Eels are becoming these days.
50:11
Yeah, no I think that's an interesting perspective on it,
50:14
and relating to point D made right at the beginning.
50:18
So we don't understand how medicines work often times these
50:21
drugs range. And so I mean, there's so many
50:26
examples where we didn't know what these drugs were going to
50:29
do and then someone figured out how you can actually repurpose
50:32
this drug for a completely different thing, right?
50:34
So our favorite example of chloroquine on early in the
50:39
covid, Sort of pandemic. People thought, oh, this old
50:43
malaria drug might be useful for for covid and it wasn't bad
50:49
hypothesis. Actually, because chloroquine
50:51
has been repurposed from for other diseases or you take I
50:56
mean, I mean, all sorts of examples that are out there on a
51:01
non drug repurposing. So the same idea here with AI
51:03
models and systems. You kind of Maybe started off
51:07
wanting them to achieve one thing.
51:10
But then you find do, you can repurpose them, fine-tune them
51:14
for something else. And why wouldn't you?
51:16
I mean, this is somehow speaking to this foundational aspect of
51:22
what language is or what perception is what
51:26
decision-making is because if these things are applicable more
51:30
widely, than, that means that we've kind of hit on something.
51:33
More fundamental, I love that, and it is so true.
51:39
It's almost like we have This thing that were poking to see,
51:44
like, what can we do with this. We've got this insane surface
51:48
area, that it covers and let's just see like can we use it for
51:52
something else that we didn't originally plan to use it for?
51:56
So all right transition and again because I really want to
51:59
get to this before we finish I do want to talk about all of
52:03
these open source packages you've been working on Coach can
52:06
you break down what you've been doing?
52:10
All right, so it's not just me. Obviously we have a large team
52:14
that I've been research in several labs around the world
52:18
and India in Dublin, Ireland. Kenya, Israel.
52:23
I mean in the u.s. all, I mean, all sorts of places and together
52:27
we've created several open source tool kits in the space of
52:32
trustworthy. I so we started with two when
52:36
was the AI fairness 360? And the other was Ariel
52:40
robustness 360 and both of those, we kind of pull together.
52:47
Showcasing that, what do practitioners really need?
52:50
What do purpose and data, scientists need to incorporate
52:55
these sort of things that were just in the research.
52:58
Either the bring it down to practice and so that kind of
53:01
motivated us. And then the success of those to
53:06
let us to say, let's keep doing this.
53:08
It's a great way in Not only for the entire Community to come
53:12
together but also to really make a difference on the world.
53:17
And so, since then, we've had explained ability 360.
53:22
So that includes slime and Shop. But also several other IBM
53:27
research algorithms like the contrast of explanations method
53:30
and so forth and then the 360 kind of implies some sort of
53:36
comprehensiveness. So that's where the naming comes
53:39
from. And unexplained ability.
53:41
For example, the comprehensiveness is around
53:45
these different consumers of the explanations.
53:48
So the engineers, the decision-makers, the affected
53:53
users. I mean, there's different
53:55
methods that apply for different constituencies and so the little
53:59
kids help us and then since then we've continued so we have an
54:05
uncertainty quantification 360 toolbox.
54:09
We're working on pulling some things together to create an AI
54:13
privacy 360 as well and we have our fact sheets effort which is
54:18
on transparency. So so that actually is not a
54:22
code base but it is a website with a lot of information on
54:27
what you need to do to enable transparency in there.
54:29
So how do people interact with this?
54:31
And by the way, for anybody listening, we're going to throw
54:33
all these links into the description.
54:35
If you want to go around and play and have a nice little The
54:40
tires a bit and hopefully start incorporating this into your
54:43
workflow. But what is it?
54:44
Is it just like an SDK the PIP install.
54:50
Yeah, so, yeah. So we have GitHub repositories
54:54
for all of these other than the fact sheets.
54:58
So yeah, I mean that's the way you can think about it.
55:01
In fairness, we have both an R version and a python version for
55:05
everything else. It's python only and so yeah,
55:08
you can do it. Install, you can download the
55:12
packages from GitHub. However, you want to do it.
55:15
And then one thing that we spent a lot of time on is, we have
55:19
accompanying websites, which have a lot of reference
55:21
materials. So, for kicking things around.
55:25
If you're don't want to get into the code level, to begin with,
55:28
we have a bunch of interactive web demos for each of them.
55:32
That'll give you a good sense of what's going on.
55:34
There's glossaries kind of materials that help you think
55:39
about Should I use one all of those sort of things so that's
55:44
all there. And first three have been
55:47
actually donated to the Linux foundation.
55:49
So they're not governed by IBM anymore.
55:52
So adversarial robustness fairness and explain ability or
55:55
now Linux Foundation projects and the others.
56:01
Hopefully, we'll get there incredible.
56:03
Well we gotta go now but I also want to ask maybe just a closing
56:08
question. Non, why is IBM a place that
56:14
will let this type of thing happen?
56:18
Like why do they support this type of research?
56:21
And what is it in the ethos of IBM?
56:23
Because I know there is a lot of just responsible, AI fairness,
56:28
ethical AI, all of these Buzz words, as you can say, coming
56:34
out of IBM, what is it that IBM? Has that wants to do this?
56:40
Yeah, and yeah, I mean I don't like to think of them as
56:43
buzzwords for say even though they are right now because it's
56:46
something about the DNA or the fabric of what we're working
56:50
towards because as a company. So IBM stands for International
56:55
Business Machines and the whole idea is to create machines,
57:00
whatever they are, they used to be clocks and cheese slicers
57:05
back in the day, now their computers and AI systems.
57:08
Yeah. Yeah, but it's always been, I
57:10
mean, what can we do? To help other companies meet
57:13
their goals and missions and thinking that where?
57:17
I mean, you first need to have those clients need to have trust
57:21
in idea. But also those companies need to
57:24
have trust from their own cloud and users and their own
57:28
customers, right? So basically I mean trust for me
57:32
it's everything and as where, I mean developing these
57:36
Technologies, we need to help. Everyone.
57:40
I mean really work towards ways to make them responsible and so
57:46
I think that's the way the company has been and it supports
57:50
This research for that reason. Yeah that's maybe how I'll leave
57:55
it. I mean I could offer comparisons
57:58
but I think the best way to just put it as.
58:00
I mean, I DM really is a company where the sort of work is valued
58:06
for for the reason of impacting Society.
58:09
Positively awesome, man. Well, it's been great talking to
58:11
you, and thank you for coming on here.
58:14
Thank you for sharing your wisdom.
58:15
Kristen, ROM. Thank you, man, for coming and
58:17
being an incredible co-host with me.
58:21
This is awesome fellas. It was really fun.
58:26
Thanks, Christian, Iran fence Demetrios.
58:28
And yeah, I think you guys lived up to your promise, this was
58:32
really fun. Fun podcast, thanks H, thanks to
58:36
the truth. This is a great conversation.
58:38
I learned so much from this conversation.
58:41
Likewise cool.