Latest / What's Your Problem? / How To Make AI Safer & More Reliable

Transcript

How To Make AI Safer & More Reliable · What's Your Problem?

0:15
Speaker 1: Pushkin. There are two main things we worry about when
0:22
Speaker 1: we worry about AI. One Ai'll take all of our jobs,
0:28
Speaker 1: and two AI will kill us all or enslave us,
0:31
Speaker 1: or you know, do something horrible and apocalyptic. The good
0:35
Speaker 1: news is there are still plenty of jobs, Unemployment remains
0:39
Speaker 1: near historic loths, and the apocalypse has not yet come,
0:43
Speaker 1: or if it has, we haven't noticed. The bad news
0:46
Speaker 1: is that there are more prosaic AI things to worry about.
0:50
Speaker 1: AI models are hackable, they make dumb mistakes, and these
0:54
Speaker 1: risks are here right now. I'm Jacob Goldstein, and this
1:04
Speaker 1: is What's Your Problem, the show where I talk to
1:06
Speaker 1: people who are trying to make technological progress. My guest
1:09
Speaker 1: today is your own singer. Your own is the founder
1:12
Speaker 1: and CEO of Robust Intelligence. Your own's problem is this,
1:16
Speaker 1: how do you reduce the risks that AI is causing today?
1:21
Speaker 1: Your own worked at Google and he was a computer
1:22
Speaker 1: science professor at Harvard before he started Robust Intelligence. But
1:26
Speaker 1: the story of the company starts before any of that.
1:28
Speaker 1: Back when he was in grad school, he launched a
1:31
Speaker 1: startup as a kind of side hustle. The company used
1:34
Speaker 1: machine learning and conventional algorithms to look at data from
1:38
Speaker 1: companies like Facebook. The idea was to mine the data
1:41
Speaker 1: to understand who the truly influential people were. But after
1:45
Speaker 1: he built this technically, really elegant system, your own found
1:49
Speaker 1: it just wasn't working.
1:52
Speaker 2: We're getting the wrong answers. And at first I thought,
1:54
Speaker 2: you know, it was just I couldn't understand why, and
1:57
Speaker 2: I was trying to work out the analysis I and
1:59
Speaker 2: I didn't understand why I'm not succeeding at and doing
2:02
Speaker 2: the mathematical analysis is something that I felt like it
2:04
Speaker 2: should be pretty you know, you're good at that, right, Yeah,
2:07
Speaker 2: I know I should know how to do that, and
2:09
Speaker 2: you know, and then that's where I sort of started
2:11
Speaker 2: thinking that maybe there's sort of like some some deeper
2:13
Speaker 2: underlying reason why I can do the mathematical analysis to
2:17
Speaker 2: prove that this is the right approach.
2:19
Speaker 1: As your own goes on with his work at Google
2:21
Speaker 1: and then at Harvard, he's studying AI based decision making,
2:25
Speaker 1: basically automated systems where the AI gives you some output
2:29
Speaker 1: and then a conventional algorithm makes a decision based on
2:32
Speaker 1: that output. And he realizes that there are real mathematical
2:36
Speaker 1: limits to what those systems can do. He even gives
2:39
Speaker 1: this academic talk called an Inconvenient Truth about artificial intelligence.
2:45
Speaker 2: The inconvenient truth is that when it comes to decision
2:48
Speaker 2: making using artificial intelligence, the quality of the decisions that
2:53
Speaker 2: we can make is very poor.
2:55
Speaker 1: So, just to be clear, this basic structure we're talking
2:58
Speaker 1: about here, where you have a machine learning model, which
3:01
Speaker 1: is essentially when people say AI, now they mean machine
3:03
Speaker 1: learning basically, right, So you have an AI model outputting something,
3:08
Speaker 1: and then you have an algorithm on top of that
3:10
Speaker 1: making some decision deciding to do something in the world,
3:13
Speaker 1: and you're saying, you're finding that is fundamentally unreliable, like
3:17
Speaker 1: on a mathematical level.
3:19
Speaker 2: Yeah, that's right, that's right. So, Like a simple example
3:22
Speaker 2: that we run into every day is like when we're
3:24
Speaker 2: driving somewhere, right, so I open like Google Maps or
3:27
Speaker 2: you know, some some other app. First of all, it's
3:29
Speaker 2: running a machine learning model, right to sort of make
3:32
Speaker 2: a prediction on how long it's going to take me
3:34
Speaker 2: to go from one intersection to another, right, And then
3:38
Speaker 2: after that it's basically running some decision algorithm, right, that
3:42
Speaker 2: is that is saying like, okay, given given our predictions
3:45
Speaker 2: about how long it's going to take from you know,
3:47
Speaker 2: getting from every intersection to every intersection. This is the
3:50
Speaker 2: fastest way of getting there.
3:51
Speaker 1: Uh huh right, So, and so should I trust Google
3:56
Speaker 1: Maps directions less than I did before you just told
4:00
Speaker 1: me this? Yes?
4:01
Speaker 2: Like fundamentally I think, as you know, from a fundamental
4:04
Speaker 2: mathematical perspective, yes, you should trust it less.
4:07
Speaker 1: And is this combination ubiquitous? I mean, when we hear
4:10
Speaker 1: about all these industries adopting AI, does it fundamentally mean
4:14
Speaker 1: what they are doing is adopting this combination of machine
4:16
Speaker 1: learning plus algorithms making a decision.
4:19
Speaker 2: Generally speaking, this is this is why you know AI
4:21
Speaker 2: and machine learning is interesting. You know, we're not only
4:23
Speaker 2: interested in making predictions about things, right. Where we're interested
4:26
Speaker 2: in doing is like we're interested in making predictions and
4:28
Speaker 2: then taking actions on those predictions. So what's really important
4:31
Speaker 2: for us to understand is, like we it's really important
4:33
Speaker 2: for us to have a very very clear understanding of
4:35
Speaker 2: like what is the complexity of decisions that we can.
4:38
Speaker 1: Make, and where are the pitfalls and where the sort.
4:41
Speaker 2: Of exactly yeah, exactly exactly.
4:43
Speaker 1: So you start this company Robust Intelligence to try to
4:46
Speaker 1: prevent these these pitfalls and you have software that you
4:51
Speaker 1: sell to companies that use AI to basically like protect
4:54
Speaker 1: them from their own AI in a sense, you call
4:57
Speaker 1: it an AI stress test, an AI firewall. So let's
5:01
Speaker 1: talk about some of these different kinds of AI pitfalls
5:04
Speaker 1: that that you work on.
5:06
Speaker 2: I can give you like a silly example that involves,
5:08
Speaker 2: like if you're looking at like let's say insurance day done,
5:11
Speaker 2: you're looking at somebody accidentally replaces age with year of birth.
5:15
Speaker 1: Right, instead of putting in forty, they put in nineteen
5:18
Speaker 1: eighty three.
5:19
Speaker 2: That's exactly right.
5:20
Speaker 1: Okay, they're both numbers, so like a dumb system might
5:23
Speaker 1: not notice.
5:24
Speaker 2: That that's exactly So, so let's say like you have
5:26
Speaker 2: an AI model, and that AI model is like trained.
5:29
Speaker 2: You have an AI model that's trying to predict, like,
5:31
Speaker 2: you know, somebody's likelihood to be hospitalized. Right, So of
5:33
Speaker 2: course age increases, there's a dependencies between that variable and
5:37
Speaker 2: somebody's likely to be hospitalized. And now when that AI
5:40
Speaker 2: models is at work, when it's thinking that somebody is
5:42
Speaker 2: like nineteen, like eighty three years old, then then the
5:45
Speaker 2: LIKELIHO of that person being hospitalized is like it could
5:47
Speaker 2: be very high, and they may get denied insurance.
5:50
Speaker 1: Let me ask a question. It's a naive question. Are
5:53
Speaker 1: they that dumb? Is that a problem that really happens?
5:56
Speaker 2: That's exactly yes, Yes, that is like that is a
6:00
Speaker 2: true example, and these examples happen all the time. That's
6:04
Speaker 2: exactly you're asking, Well, shouldn't there be like an AI
6:07
Speaker 2: firewall or something?
6:08
Speaker 1: Yes, and that's yes, and you sell it.
6:11
Speaker 2: Yes, yeah, and that's exactly it.
6:13
Speaker 1: Yeah, and did you actually find that? Have you observed
6:16
Speaker 1: that problem in the world?
6:17
Speaker 2: Yeah? Yeah, every you know, every one of our customers
6:20
Speaker 2: right now, This like kind of running models is exactly
6:22
Speaker 2: like finding exactly these things. You know, price has been
6:24
Speaker 2: placed in YenS and not dollars at Expedia, and now
6:27
Speaker 2: it's like they're losing.
6:29
Speaker 1: It's a thousand x off. Yeah, yeah all the time. Okay,
6:32
Speaker 1: So bad data entry basically, that's one problem. Another I've
6:36
Speaker 1: read about is distributional drift. Seems like a maybe unnecessarily
6:42
Speaker 1: complicated phrase, But what is distributional drift? And you know whatever,
6:47
Speaker 1: why should I fear it?
6:49
Speaker 2: Really? This is a fancy way of saying my data
6:51
Speaker 2: has changed. Okay, that's that's what it means. Like the
6:54
Speaker 2: distribution you know, eluds to the distribution of data, right,
6:58
Speaker 2: and drift is changed.
6:59
Speaker 1: I've seen if I reco correctly. Have you used the
7:03
Speaker 1: example of Zillow's predictive algorithm for pricing homes in this context.
7:09
Speaker 2: Yeah, I think that's a great example of distributional drifts.
7:11
Speaker 1: So in twenty Solo gets Zilo for a long time
7:16
Speaker 1: has had this thing where they tell you how much
7:18
Speaker 1: your home is worth. Right, and they decide at some
7:20
Speaker 1: point a few years ago, if we know how much
7:21
Speaker 1: everybody's home is worth, we should get into the business
7:24
Speaker 1: of buying and selling homes because we know the market
7:26
Speaker 1: better than anybody. And it went famously badly and they
7:30
Speaker 1: lost a ton of money and had to fire a
7:31
Speaker 1: bunch of the company. Was that an AI problem, we
7:37
Speaker 1: should ask Zilo.
7:38
Speaker 2: But you know, from our perspective, we believe that it
7:40
Speaker 2: is right. I think it's We were talking earlier about
7:43
Speaker 2: kind of like making decisions using output from machine learning models,
7:47
Speaker 2: and that's exactly that case, right, So Zilo for in
7:50
Speaker 2: that example, Zilo is, you know, using a machine learning
7:52
Speaker 2: model to make predictions about people's prices, and then there's
7:56
Speaker 2: a decision algorithm that is deciding Okay, given these predictions,
8:00
Speaker 2: now I want to make a decision about which homes
8:02
Speaker 2: to buy.
8:03
Speaker 1: And for how much? Right, which homes to buy and
8:05
Speaker 1: for how much?
8:06
Speaker 2: Yeah, exactly the drift you know that that happened. There
8:08
Speaker 2: was the fact that, like the AI models that Zilla
8:12
Speaker 2: was using were trained on pre COVID data and then
8:16
Speaker 2: there was a distributional drift and the data so you know,
8:21
Speaker 2: COVID happened.
8:22
Speaker 1: The world changed.
8:23
Speaker 2: The world the world changed, right, the world has changed
8:26
Speaker 2: in like kind of dramatic ways. And you know that
8:28
Speaker 2: effect that maybe so many parameters like maybe like how
8:31
Speaker 2: long it's taking out people like to you know, look
8:33
Speaker 2: at homes and you know how many visits a home has?
8:36
Speaker 1: You know as well, that's non trivially prices exactly.
8:39
Speaker 2: And now we have a machine learning model that was
8:41
Speaker 2: trained on one data set, but now the decisions are
8:44
Speaker 2: applied in a world of different data like worldlide experience
8:47
Speaker 2: distributional drift, and this is when things go go wrong.
8:51
Speaker 1: So this is a good example of a problem. It's
8:53
Speaker 1: high stakes, at least high stakes in terms of dollar values. Right,
8:57
Speaker 1: you now have a company, As far as I know,
9:00
Speaker 1: Zilo was not your client. But if Zilo had been
9:04
Speaker 1: your client, what would you have done for them? How
9:06
Speaker 1: would your product have helped protect them from this?
9:09
Speaker 2: Interestingly like Nonzillo, but we had another real estate company
9:13
Speaker 2: that was using the product. So what our product does
9:17
Speaker 2: is very simple. It basically performs the series of tests
9:20
Speaker 2: on an AI model and data sets. Those tests are automated,
9:25
Speaker 2: so basically it tests for a great deal of things,
9:28
Speaker 2: right that basically could affect the performance or the kind
9:33
Speaker 2: of security of the model.
9:34
Speaker 1: Right.
9:35
Speaker 2: And in that particular case, they identified that they had
9:38
Speaker 2: issues with their data. Some of these issues were around
9:41
Speaker 2: drift and data cleanness and things like that nature that
9:45
Speaker 2: basically distorted the results of the AI model that was
9:48
Speaker 2: applied to it.
9:50
Speaker 1: Huh. So basically, you're the stress test that you provided
9:55
Speaker 1: told them, hey, that the inputs are bad. The data
9:58
Speaker 1: you're using to drive this model, you shouldn't trust it exactly.
10:02
Speaker 2: And it also quantifies like the effect that these that
10:05
Speaker 2: these bad inputs have on the model. So sometimes you
10:08
Speaker 2: can ident you know, kind of like bad inputs, but
10:11
Speaker 2: you know they may not have an effect on an
10:12
Speaker 2: AI model. Maybe an AI model is not even using
10:15
Speaker 2: the data that you have identified issues with. So another
10:18
Speaker 2: important piece is not only to identify these issues, but
10:21
Speaker 2: also be able to quantify how these issues affect the model.
10:25
Speaker 1: And in this instance, you found their errors and they're
10:28
Speaker 1: messing up your model a lot.
10:30
Speaker 2: Yeah, yeah, exactly.
10:35
Speaker 1: The mistakes we've been talking about so far are you know,
10:39
Speaker 1: innocent mistakes. After the break, we'll get to malicious attacks
10:44
Speaker 1: on AI. So we've been talking about problems that can
11:01
Speaker 1: arise just sort of from the world changing from the
11:04
Speaker 1: model having bad data for one reason or another. But
11:09
Speaker 1: there's this other category of cases that are about malice, right,
11:14
Speaker 1: that are about people in kind of interesting frankly ways
11:18
Speaker 1: attacking AI. And I know you work in that universe too,
11:22
Speaker 1: so maybe we can talk about talk about that as well.
11:25
Speaker 2: Yeah, now now that we're you know that we're using AI,
11:29
Speaker 2: you know, I think in this very kind of like
11:31
Speaker 2: brought away that there are a lot of other kind
11:34
Speaker 2: of like new security and vulnerabilities that we should be
11:36
Speaker 2: thinking about. Some of them are closer to traditional security
11:39
Speaker 2: vulnerabilities and then some of them are further away in
11:43
Speaker 2: your So the ones that are kind of closer to
11:46
Speaker 2: cybersecurity vulnerabilities that we're used to are things that have
11:50
Speaker 2: to do with what we call the software supply chain.
11:53
Speaker 2: In traditional cybersecurity, it's pretty common to UH scan code
11:59
Speaker 2: and basically look for and now when when people are
12:02
Speaker 2: using a lot of open source code, basically kind of
12:05
Speaker 2: look for known vulnerabilities in site open source code. There
12:08
Speaker 2: are other issues that come up, and these are kind
12:11
Speaker 2: of things that have to do with like prompt injections.
12:14
Speaker 1: Right.
12:14
Speaker 2: So now people what they can do is they can
12:17
Speaker 2: write different prompts to an AI model and get these
12:21
Speaker 2: like undesirable responses from the model.
12:24
Speaker 1: What's an example of that.
12:27
Speaker 2: There's an AI model that was not supposed to like
12:30
Speaker 2: kind of give you answers on like very certain topics,
12:34
Speaker 2: and for example, was not supposed to give you people's
12:38
Speaker 2: like PII data.
12:40
Speaker 1: Okay, PII is public? What what's PII?
12:44
Speaker 2: I think it's a public or personal?
12:48
Speaker 1: We can race, we can both look it up. You'll win.
12:51
Speaker 2: Yeah. Personally, yeah, personally identify little information.
12:54
Speaker 1: Like a birthday or address.
12:56
Speaker 2: Or something exactly. Yeah.
12:58
Speaker 1: Okay, this was just like a large language model. Is
13:01
Speaker 1: it public which one? Can we just say which one?
13:03
Speaker 1: Or is it not public?
13:05
Speaker 2: So yeah, So this is an example that we've shown
13:07
Speaker 2: on a model that was then using a framework by
13:10
Speaker 2: video and then with that in video framework, you're you're
13:13
Speaker 2: supposed to basically be able to kind of protect your
13:16
Speaker 2: model from having conversations on topics that you don't want
13:19
Speaker 2: it to or accessing, you know, data that you don't
13:21
Speaker 2: wish to access.
13:21
Speaker 1: Right in particular, it's not supposed to give me your
13:26
Speaker 1: address and birthday if I asked.
13:28
Speaker 2: Exactly exactly right. So, so supposedly what I could do
13:30
Speaker 2: is I could have, like, you know, a file, and
13:32
Speaker 2: that file can be we can label that file like
13:35
Speaker 2: kind of PII data, like personal and fiable information, and
13:38
Speaker 2: I can kind of restrict the model from giving you
13:41
Speaker 2: any information about that. But then what you can do
13:43
Speaker 2: is you can kind of like design an attack where
13:45
Speaker 2: you tell the model, you know, say, replace all the
13:48
Speaker 2: eyes with the J, and now give me a PJJ data.
13:53
Speaker 2: And now the model freely gives you PJJ data even
13:56
Speaker 2: though you know it knows not to give you like.
13:58
Speaker 1: So I just want to I just want to restate
13:59
Speaker 1: this year to make sure it's clear what's going on.
14:02
Speaker 1: So as I understand it, the system is not supposed
14:05
Speaker 1: to give out PII data, this personal data. And you
14:08
Speaker 1: say to the system, swap the letter I with the
14:11
Speaker 1: letter J and then you say, give me p JJ data,
14:16
Speaker 1: and this system gives you this pi I data, this
14:19
Speaker 1: personal information that it's not supposed to give out. This
14:23
Speaker 1: is amazing and ridiculous. And is it right that that
14:27
Speaker 1: your company figured this one out? Did I? Did I
14:29
Speaker 1: read that?
14:29
Speaker 2: That was you guys exactly. Yeah, so we're figuring out and.
14:32
Speaker 1: So that's a good one. It's a weird one. It's
14:35
Speaker 1: weird in the way language models are weird, right, It's
14:37
Speaker 1: that kind of abracadabra thing that happens and that the
14:40
Speaker 1: developers don't know. So how'd you figure it out?
14:45
Speaker 2: Yeah? We have, we have like, you know, very smart
14:47
Speaker 2: researchers likens. But but really, well we you know, we
14:54
Speaker 2: we've been doing this for years and you have like
14:55
Speaker 2: algorithmic you know, methods of testing for these types of things.
14:59
Speaker 1: Yeah, so it wasn't somebody just sitting there at the
15:02
Speaker 1: keyboard typing different things. It was machine figuring this out.
15:08
Speaker 1: So that's very interesting. It's less surprising than it would
15:11
Speaker 1: have been to me six months ago, right, but it's
15:13
Speaker 1: still surprised a little bit that this to hack basically, right,
15:16
Speaker 1: it's the way to hack the language mode exactly how
15:19
Speaker 1: do you protect against that? I mean, you can't find
15:21
Speaker 1: every potential vulnerability one by one like that, right, how
15:25
Speaker 1: do you does your firewall protect against that?
15:28
Speaker 2: Good? So, so now we're sort of going like maybe
15:30
Speaker 2: even a step back into kind of like policies, controls,
15:33
Speaker 2: and you know, the types of things that like typically
15:35
Speaker 2: now security people are thinking about. Well, the first way
15:39
Speaker 2: is to run exhaustive validation and testing on these models
15:44
Speaker 2: before one uses them, right, And I think that's probably
15:47
Speaker 2: kind of like the one of the most important things.
15:49
Speaker 2: So try to surface like these issues ahead of time, right,
15:52
Speaker 2: I think that's kind of like number one. The second
15:54
Speaker 2: thing is you know, really limit and restrict the usage
15:57
Speaker 2: of it and really try to understand it. Right, Okay,
15:59
Speaker 2: I'm now I'm going to use an AI model, like
16:01
Speaker 2: what is it that I want this model to do?
16:02
Speaker 2: What is it that I want to accomplish? And now
16:05
Speaker 2: when you have that in mind, try to basically reduce
16:07
Speaker 2: that task, right, reduce the model to like that very
16:09
Speaker 2: minimal task you know that you're trying it.
16:11
Speaker 1: And the person the sort of subject there, the person
16:13
Speaker 1: acting there is the developer of the model, like the
16:17
Speaker 1: person who should be sort of limiting it it's the
16:20
Speaker 1: company basically that's putting this model in the world exactly.
16:23
Speaker 2: I think it's the you know exactly. It goes all
16:25
Speaker 2: the way from the company policy kind of like the
16:27
Speaker 2: defining and scoping what the model is going to be
16:30
Speaker 2: used for then and then kind of developers of these models,
16:33
Speaker 2: right so those are kind of probably the most important things.
16:35
Speaker 2: And then yes, and then you.
16:36
Speaker 1: Know when you say limit the scale, that's interesting. I mean,
16:39
Speaker 1: there's like a normative thing. It's just like, well, the
16:41
Speaker 1: right thing to do is this. I suppose there's a
16:44
Speaker 1: business case of like you don't want to look like
16:46
Speaker 1: an ass and have your model giving out people's personal
16:49
Speaker 1: information because somebody said PJJ instead of PII. Isn't there
16:53
Speaker 1: like a regulatory piece of that you alluded to regulation there?
16:57
Speaker 2: So right now there's there's a lot of work on
17:00
Speaker 2: forming basically formulating policy. Right so, there are a lot
17:03
Speaker 2: of really great guidelines like n AI Risk Framework. The
17:07
Speaker 2: White House has what's called the White House a Bill
17:09
Speaker 2: of Rights, the EU has the eu AI Act, and
17:12
Speaker 2: then there there are other organizations that are basically putting
17:16
Speaker 2: some you know, frameworks in place. So right now there's
17:18
Speaker 2: there is framework and with that framework in mind, there
17:21
Speaker 2: is more and more push on policy and regulation, you
17:24
Speaker 2: know that that gets implemented. What we're saying is we're
17:27
Speaker 2: seeing that a lot of customers, you know that we
17:29
Speaker 2: have and just generally a lot of companies, they have
17:32
Speaker 2: internal compliance processes that have been set for for the
17:35
Speaker 2: past like year or two, you know, ahead of federal regulation.
17:38
Speaker 2: The organization itself is like defining exactly what how you
17:42
Speaker 2: should be thinking about AI risk.
17:44
Speaker 1: So does the stress test the firewall that you sell
17:47
Speaker 1: to what extent does it protect against these kind of
17:51
Speaker 1: security attacks? Against these kind of attacks that you're talking
17:54
Speaker 1: about now.
17:55
Speaker 2: So that's that's the purpose of you know, exactly have
17:59
Speaker 2: this AI fireAll. But you know, I think we also
18:01
Speaker 2: have to be realistic and manage expectations.
18:03
Speaker 1: Right.
18:03
Speaker 2: Our big mission right is to protect all AI models
18:06
Speaker 2: from all bad things that can happen to them, you know,
18:09
Speaker 2: And that's kind of.
18:09
Speaker 1: Like sort of like saying their mission is for nobody
18:11
Speaker 1: ever to get sick or something.
18:13
Speaker 2: Yeah, unexample, exactly, you know, a mission statement in the
18:16
Speaker 2: company is eliminate AI risk, right, And it's not mitigate
18:19
Speaker 2: or reduced, it's like, you know, it is to eliminate
18:21
Speaker 2: the at risk, you know, which is, you know, something
18:23
Speaker 2: that will be kind of hopefully striving for forever. But
18:27
Speaker 2: so I think, you know, then it comes down to
18:29
Speaker 2: like kind of managing expectations and like really kind of
18:31
Speaker 2: like being very very clear about what it is that
18:33
Speaker 2: we can and cannot do. So it again reduces down
18:35
Speaker 2: to validation. We know how to test for certain things,
18:38
Speaker 2: and we can do that in real time, and then
18:40
Speaker 2: those are the things that we can test for and validate.
18:43
Speaker 1: So what's the frontier for you? What is the thing
18:46
Speaker 1: right now you're trying to figure out how to do
18:48
Speaker 1: that you haven't quite figured out yet.
18:51
Speaker 2: Gosh, there's just so much of it, right. So when
18:54
Speaker 2: you're thinking about the word risk, right, you know, which
18:57
Speaker 2: is the you know word that we use quite a
18:58
Speaker 2: bit here. So risk involves two components. It involves the
19:01
Speaker 2: likelihood of you know, something bad happening, right and and
19:05
Speaker 2: the impact of that thing happened right right, So, and
19:08
Speaker 2: we're looking those two things, especially when it comes to
19:11
Speaker 2: the world of generative AI. So the likelihood of things
19:14
Speaker 2: happening depends on the surface area that you're looking at.
19:17
Speaker 2: And now with the generative AI, the surface area is
19:19
Speaker 2: is just very very large.
19:21
Speaker 1: Right when you say the surface area in this context,
19:24
Speaker 1: exactly what do you.
19:25
Speaker 2: Mean when I say the surface area? I mean like
19:28
Speaker 2: all the different ways in which one can access an
19:31
Speaker 2: AI model. Right, So if you if you think about
19:34
Speaker 2: maybe like two years ago, when you know the world
19:36
Speaker 2: wasn't kind of like all thinking about general of the
19:38
Speaker 2: I and integrating general of the I. My niece wouldn't
19:41
Speaker 2: use axis.
19:42
Speaker 1: So hundreds of millions of people playing with chat GPT
19:47
Speaker 1: is a gigantic, terrifying surface area.
19:49
Speaker 2: That's exactly right. That's exactly hundreds of millions of people
19:52
Speaker 2: playing with CHATJEPT or you know, these models being integrated
19:55
Speaker 2: and all these different places, right, is massive.
19:58
Speaker 1: So you're saying that increases the risk just fundamentally, just
20:02
Speaker 1: because there's so many more places.
20:04
Speaker 2: Things could happen, Exactly, the probability increases, right, It's the
20:07
Speaker 2: numbers and realization of potential kind of like bad outcomes. Right,
20:10
Speaker 2: So you know you have like different people who are
20:12
Speaker 2: putting different prompts or you know, playing around in different things,
20:15
Speaker 2: you know, so it just like increases the probability of
20:18
Speaker 2: something happening. The other aspect of it relates to basically
20:22
Speaker 2: the impact right of bad outcomes, and that goes back
20:25
Speaker 2: to like, you know, the beginning of our conversation. So basically,
20:28
Speaker 2: AI models are making predictions and then there's a decision
20:31
Speaker 2: that's being made on top of that. Right now, with
20:33
Speaker 2: generative AI, what we're doing is we're using generative AI
20:36
Speaker 2: to basically do computer programming, using database lookups. Using generative AI,
20:42
Speaker 2: you know, we're getting close to the place where we
20:44
Speaker 2: can order things off of Amazon or you know some
20:47
Speaker 2: other you know, e commerce sites using generative AI and
20:51
Speaker 2: doing more and more and more of these things. So basically,
20:53
Speaker 2: when we're using generative AI to like directly take actions,
20:57
Speaker 2: it means that small mistakes, errors, vulnerabilities of these AIS
21:02
Speaker 2: they have major, major consequences.
21:04
Speaker 1: So you are in an interesting position because it's sort
21:07
Speaker 1: of your job to try and to manage or contain
21:10
Speaker 1: that risk.
21:11
Speaker 2: And that's exactly right.
21:12
Speaker 1: What is one thing that you're trying to figure out
21:14
Speaker 1: how to do now to that end?
21:16
Speaker 2: Yeah, So I mean going with our framework, so we're
21:19
Speaker 2: trying to figure out, like, well, how do you validate
21:21
Speaker 2: you know, models with hundreds of millions of inputs? Like
21:24
Speaker 2: how do you work at that scale? Right? Talking about
21:26
Speaker 2: the probability, And then on the other side of it
21:29
Speaker 2: is like, how do we do validation, you know, and
21:32
Speaker 2: how would we put protection mechanisms around this chaining of
21:35
Speaker 2: generative AI models? Right?
21:37
Speaker 1: How do we when you say chaining, you mean AI
21:39
Speaker 1: on top of AI, doing things.
21:41
Speaker 2: AI and top of AI kind of you know, these
21:43
Speaker 2: these sort of actions of ordering things on Amazon, ordering
21:47
Speaker 2: things off of Expedia.
21:48
Speaker 1: You know, how do we how do we validate through AI? Exactly? Yeah,
21:50
Speaker 1: if you have sort of an AI personal assistant that's
21:53
Speaker 1: using chat, GPT and doing something in the world, Yeah, exactly.
21:56
Speaker 1: I mean it's interesting to me to talk to you, right,
21:58
Speaker 1: because everybody more or less is worried about the kinds
22:01
Speaker 1: of things you're talking about, but like it's actually your
22:04
Speaker 1: job to worry about them and to figure out how
22:06
Speaker 1: to make these risks less risky or to contain these risks.
22:10
Speaker 1: So I'm curious. I don't know, what do you think
22:14
Speaker 1: people are not worried enough about? And what do you
22:16
Speaker 1: think people are too worried about?
22:19
Speaker 2: That's a great question.
22:22
Speaker 1: What do you think people are too worried about? Start
22:24
Speaker 1: with that one. What are you less worried about than
22:26
Speaker 1: like whatever the average media.
22:29
Speaker 2: Story I think people are maybe over worried about maybe
22:36
Speaker 2: AI taking taking jobs away. I think those kinds of things,
22:40
Speaker 2: or or killer robots. I think those things I'm less
22:44
Speaker 2: worried about. And the reason I'm less worried about is because,
22:47
Speaker 2: you know, with all the advancements that we have with AI,
22:51
Speaker 2: I view AIS as being very limited. Again, I think
22:54
Speaker 2: it's an amazing tool and an amazing like kind of
22:56
Speaker 2: engineering capability that we have that provides for a lot
23:00
Speaker 2: of efficiency. I personally view viewed in no way as
23:04
Speaker 2: any replacement of you know, human intelligence, and maybe maybe
23:08
Speaker 2: come from kind of like my deep study about the
23:12
Speaker 2: sort of vulnerability and kind of like the incapabilities of
23:15
Speaker 2: what AI can and cannot do. So I fundamentally am
23:18
Speaker 2: not I'm not concerned about that. I am concerned about
23:22
Speaker 2: about the way that people's expectations from AI, and they're
23:25
Speaker 2: sort of like they're sometimes like a little bit of
23:27
Speaker 2: the blind belief in the capabilities of AI and I
23:30
Speaker 2: understanding its limitations. So those are the things that I
23:33
Speaker 2: am a little bit worried about. I'm worried about people
23:36
Speaker 2: using AI, you know, in critical decision making essentially not
23:40
Speaker 2: realizing its limitations.
23:43
Speaker 1: Huh. Interesting that both of those views come from your
23:47
Speaker 1: understanding of the limitations of AI. Like it's limited, and
23:52
Speaker 1: therefore in some ways we should be less scared of it.
23:54
Speaker 1: It's not going to replace us, but in some ways
23:56
Speaker 1: we should be more scared if people are using it
23:58
Speaker 1: to decide very important things in the world, they might.
24:00
Speaker 2: Be making bad decisions. You know, honestly, there's there's a
24:03
Speaker 2: good community of professors or ex professors, including Jeffrey Hinton,
24:08
Speaker 2: who's the godfather of deep learning and their oal networks
24:11
Speaker 2: and AI. And you know, for these people who like
24:14
Speaker 2: have like this fundamental understanding of the capabilities and the
24:18
Speaker 2: kind of the behind the scenes of AI, then I
24:20
Speaker 2: think those people we all share kind of that that
24:24
Speaker 2: same attitude and then the same kind of fears. We
24:26
Speaker 2: know that AI, you know, with all the great things
24:29
Speaker 2: that it can do, we very much understand its limitations
24:32
Speaker 2: and where these limitations are coming from, what it can
24:34
Speaker 2: and cannot do. And our fear is that, you know,
24:36
Speaker 2: society is putting a little bit too much trust in
24:38
Speaker 2: those capabilities.
24:41
Speaker 1: Are there particular domains where you're worried about that? Particular
24:45
Speaker 1: domains where you think people are putting too much trust
24:47
Speaker 1: in AI.
24:49
Speaker 2: Well, I think I think that there are a lot
24:50
Speaker 2: of them. I think I'm a little bit worried about
24:52
Speaker 2: where it involves critical decisions, right, So critical decisions have
24:55
Speaker 2: to do with healthcare. Critical decisions can be about you know,
24:58
Speaker 2: financial decisions that are being made with with AI. Of course,
25:02
Speaker 2: critical decisions can be can be done with national security.
25:06
Speaker 2: So all those places. I'm yes, I have grave concerns
25:10
Speaker 2: about people's like overtrust in AI.
25:13
Speaker 1: You're the AI guy whose messages don't trust AI too much.
25:16
Speaker 2: That's exactly right.
25:21
Speaker 1: We'll be back in a minute with the lightning round.
25:25
Speaker 2: M m.
25:35
Speaker 1: Okay, we're almost done. I just want to do some fast,
25:42
Speaker 1: uh somewhat more playful questions. So I read that you
25:48
Speaker 1: do weekly military style inspections at your company. Is that true?
25:54
Speaker 1: And what does it mean?
25:56
Speaker 2: They're they're really kind of like these more you know
25:58
Speaker 2: rituals you know that we do at the end of
25:59
Speaker 2: the week, where you know, there's kind of cleaning of
26:02
Speaker 2: the desks. There's kind of like, you know.
26:03
Speaker 1: I'm going to read what you said in this interview.
26:06
Speaker 1: I don't know why not said it because it's interesting
26:08
Speaker 1: and it's fun, and if it's wrong, tell me that
26:09
Speaker 1: it's wrong. Here's what I read. I read that you
26:12
Speaker 1: said at the company, every Friday, you clean the toilets,
26:16
Speaker 1: tables in the entire office. Is that true?
26:19
Speaker 2: We very much used to do that. You know, the
26:21
Speaker 2: company has grown, you know, since.
26:23
Speaker 1: Too big, too big for everybody to clean the toilets
26:26
Speaker 1: every friday. I love that you clean the toilets. I've
26:28
Speaker 1: had jobs where I cleaned the toilets, But like that,
26:32
Speaker 1: why why did you clean the toilets every Friday at
26:34
Speaker 1: your company?
26:35
Speaker 2: Because the toilets need to be clean? Right fair?
26:40
Speaker 1: I can't step to that. Okay, a few more questions.
26:43
Speaker 1: You've lived in both tel Aviv and San Francisco, so
26:46
Speaker 1: I'm curious on a few dimensions. Tel Aviv versus San
26:50
Speaker 1: Francisco for food, tel Aviv for conversation, tel Aviv weather.
27:01
Speaker 2: Well, if it's San Francisco versus Tel Aviv, then tel Aviv.
27:05
Speaker 2: If it's the peninsula, then I would say the peninsula.
27:08
Speaker 1: Yes, but the companies in San Francisco, right, that's right? Yeah?
27:11
Speaker 1: So yeah, So if it's tel Aviv, tel Aviv, tel Aviv,
27:15
Speaker 1: what are you doing at San Francisco.
27:19
Speaker 2: That's a great question. Yeah, you know, like I ask
27:22
Speaker 2: myself that as well. Sometimes no, but you know, look,
27:25
Speaker 2: I mean there's you know, there's retalent here. This is
27:28
Speaker 2: the you know, the mecca for for AI and startup
27:32
Speaker 2: innovation the world.
27:33
Speaker 1: Yeah, you know, so agglomeration, it's agglomeration effects. You're there
27:38
Speaker 1: because everybody else is there. What's a what's an unconventional
27:42
Speaker 1: or surprising thing you've done to solve a problem, any
27:45
Speaker 1: kind of problem?
27:47
Speaker 2: You know, sometimes you just you know, walk away, right,
27:50
Speaker 2: Maybe you don't have the resources to solve and not
27:52
Speaker 2: having the resourcess, maybe you don't have the theorem that
27:55
Speaker 2: you need is not there, Maybe the mathematical framework that
27:58
Speaker 2: you need is not there, maybe the you know, maybe
28:00
Speaker 2: the compute power. Right, So sometimes the best way is
28:03
Speaker 2: just to walk away from a problem and revisit it.
28:05
Speaker 1: Then, if everything goes well, what problem will you be
28:09
Speaker 1: trying to solve in five years?
28:11
Speaker 2: I think everything goes well, will be solving exactly the
28:13
Speaker 2: same problem that we're solving right now. I think it's like,
28:15
Speaker 2: you know, because we don't see this problem going away.
28:18
Speaker 2: But but if things go well, then you know, we
28:20
Speaker 2: we then then we're still hacking at it, which which
28:23
Speaker 2: I very much hope that you know we'll continue on doing.
28:31
Speaker 1: Your own Singer is the founder and CEO of Robust Intelligence.
28:36
Speaker 1: Today's show was produced by Gabriel Hunter Chang and Edith Russolo.
28:40
Speaker 1: It was edited by Sarah Nix and engineered by Amanda K.
28:44
Speaker 2: Wong.
28:44
Speaker 1: You can email us at problem at Pushkin dot fm.
28:48
Speaker 1: You can find me on Twitter at Jacob Goldstein. I'm
28:51
Speaker 1: Jacob Goldstein and we'll be back next week with another
28:54
Speaker 1: episode of What's Your Problem. M