Latest / What's Your Problem? / The Mystery at the Heart of ChatGPT

Transcript

The Mystery at the Heart of ChatGPT · What's Your Problem?

0:15
Speaker 1: Pushkin. Today's show is about no knowns and unknown unknowns,
0:26
Speaker 1: which is to say, we're talking about AI, specifically a
0:30
Speaker 1: type of AI called a large language model, or an LM.
0:35
Speaker 1: The most famous LLM is CHAT GPT, but there are
0:38
Speaker 1: lots of others, and at their core, they all do
0:42
Speaker 1: the same thing. They read a piece of text and
0:44
Speaker 1: they predict what the next series of words should be.
0:48
Speaker 1: Lms are, obviously and quite suddenly, a huge deal in
0:52
Speaker 1: a lot of ways. One thing about them that is
0:55
Speaker 1: particularly wild to me. Lms behave in ways that are
0:59
Speaker 1: surprising even to the people who built them. In other words,
1:04
Speaker 1: large language models are this profoundly powerful, disruptive new thing,
1:09
Speaker 1: and right now we urgently need to figure out what
1:12
Speaker 1: they mean and how they work. I'm Jacob Goldstein and
1:20
Speaker 1: this is What's Your Problem, the show where I talk
1:22
Speaker 1: to people who are trying to make technological progress. My
1:26
Speaker 1: guest today is Sam Bowman. He's an expert in large
1:29
Speaker 1: language models in lms. He's on the faculty at NYU,
1:33
Speaker 1: and he runs a research group at an AI company
1:36
Speaker 1: called Anthropic. All the reason talk about lms inspired Sam
1:40
Speaker 1: to write a paper to clear up what he thought
1:42
Speaker 1: were some misconceptions. The paper is called eight Things to
1:46
Speaker 1: Know about Large Language Models. I am a fan of
1:49
Speaker 1: lists in general, and I loved this list in particular.
1:53
Speaker 1: Among other things, it gave me a deeper sense of
1:55
Speaker 1: the ways in which large language models are still a mystery,
1:58
Speaker 1: even to experts like Sam. That mystery, those unknowns, have
2:04
Speaker 1: important implications for the way we think about, and regulate
2:08
Speaker 1: and develop AI. We're going to start by discussing a
2:11
Speaker 1: pretty simple item on Sam's list. The item is this,
2:16
Speaker 1: brief interactions with llms are often misleading. You write this,
2:21
Speaker 1: You write, brief interactions with lms are often misleading. What's
2:26
Speaker 1: that mean?
2:27
Speaker 2: So when, especially when GPD four came out, and I
2:31
Speaker 2: guess also went when chat GPT first came out, there
2:35
Speaker 2: was very predictably this wave of people on Twitter saying, hey,
2:40
Speaker 2: this system is sentient and it knows where I live
2:44
Speaker 2: and it's ready to take over the world tomorrow because
2:47
Speaker 2: they had one chat with it and it said that
2:50
Speaker 2: it was sentient and it made a few educated guesses
2:52
Speaker 2: that happened to be bright, and you'll get other people
2:55
Speaker 2: on Twitter saying, hey, this system is dumb as bricks.
2:58
Speaker 2: I told it a really simple story and ask it
3:02
Speaker 2: what happened in the story and it got it wrong.
3:04
Speaker 2: There's a couple of things going on here. There's this
3:06
Speaker 2: great analogy that came up in a recent I think
3:08
Speaker 2: Time article by hell in Time owner saying they're basically
3:10
Speaker 2: improv players, where if you put them in some situation,
3:14
Speaker 2: if you put them in this situation of, oh, this
3:18
Speaker 2: is a conversation between a human who thinks the AI
3:20
Speaker 2: is sentient and the AI, then maybe the AA is
3:23
Speaker 2: going to say it's sentient.
3:24
Speaker 1: So specifically, they're improv players in the sense that famously
3:27
Speaker 1: an improv you're supposed to say yes to everything that
3:30
Speaker 1: your improv partner suggests, and so CHATCHYPT and the other
3:34
Speaker 1: llms are there to say yes, yes, and and that's
3:37
Speaker 1: what's going on.
3:38
Speaker 2: That's a decent part of it. Yeah, they're going to
3:41
Speaker 2: say yes. They're going to go along with what you're
3:43
Speaker 2: doing if you make it clear what you expect, if
3:46
Speaker 2: you make it clear, like what kind of narrative you're
3:48
Speaker 2: putting them in, what kind of environment you're putting them in,
3:51
Speaker 2: they'll go along with that.
3:52
Speaker 1: Uh, there are a couple of items on your list
3:56
Speaker 1: that seems directly contrary to assertions I've heard from other
4:00
Speaker 1: people about LMS, so that's fun and exciting. One is
4:06
Speaker 1: human performance on a task is not an upper bound
4:10
Speaker 1: on LM performance.
4:12
Speaker 2: So one of the reasons I think these systems can
4:16
Speaker 2: be better at a lot of tasks than humans is
4:18
Speaker 2: just that they've learned more stuff that they've read and
4:21
Speaker 2: mostly memorized, not just sort of all of the important
4:24
Speaker 2: papers in one little branch of chemistry or all of
4:27
Speaker 2: the important papers in all of chemistry. They've just read
4:29
Speaker 2: and mostly memorized, sort of all of the research papers.
4:33
Speaker 1: In everything, all of the papers in everything.
4:35
Speaker 2: Yeah, and many of the novels and many of the
4:38
Speaker 2: many of the news stories. And even if these systems
4:42
Speaker 2: aren't really great at drawing connections between these and sort
4:45
Speaker 2: of synthesizing a new knowledge out of them, they can
4:47
Speaker 2: do that a little bit. So you can sort of
4:49
Speaker 2: imagine what happens if you get someone who's not especially bright,
4:53
Speaker 2: but basically reasonably intelligent, reasonably competent person who's just gotten
4:59
Speaker 2: a PhD in every single thing you can get a
5:01
Speaker 2: PhD in, I'd expect them to figure some things out
5:03
Speaker 2: and to be able to do some things that no
5:05
Speaker 2: one person can do, and probably they'll notice some things
5:09
Speaker 2: that that'll be really hard for even a team or
5:11
Speaker 2: an organization to do, just because it really it's important
5:13
Speaker 2: that it kind of is, in some sense living in
5:16
Speaker 2: this one person's head.
5:18
Speaker 1: Let me just like lean into that one for a sec.
5:22
Speaker 1: So do you think that in some amount of time
5:27
Speaker 1: in the next few years, say, an LM will make
5:32
Speaker 1: some kind of you know, breakthrough in knowledge, will figure
5:37
Speaker 1: something out that no human has ever figured out that
5:39
Speaker 1: will be a meaningful breakthrough.
5:42
Speaker 2: Yeah, I think so almost. By definition, I don't have
5:45
Speaker 2: a good guess of what that's going to look like
5:46
Speaker 2: or that's going to be.
5:47
Speaker 3: Otherwise you'd be figuring it out right now, right, yeah, yeah, yeah,
5:50
Speaker 3: But no, I can imagine some story like, hey, kind
5:53
Speaker 3: of a bunch of chemists in this field of chemists
5:55
Speaker 3: have noticed this thing, and some biologists in this other
5:59
Speaker 3: subfield have noticed this other thing, and some doctors have
6:01
Speaker 3: noticed this third thing, and together they mean that some
6:05
Speaker 3: very unexpected kind of drug design might treat some new disease.
6:08
Speaker 2: And maybe if you had enough medical researchers trying enough
6:12
Speaker 2: different things, eventually they'd stumble into that. But it seems
6:15
Speaker 2: possible at some point that something like a large language
6:17
Speaker 2: model is just going to notice that, and if you
6:20
Speaker 2: ask it the right way, it's going to tell you,
6:22
Speaker 2: and you might have to second guess it a lot.
6:25
Speaker 2: These systems also make stuff up, But I think it's
6:28
Speaker 2: quite possible that you start seeing these things pretty often
6:31
Speaker 2: tell you surprising new things that happen to be true.
6:34
Speaker 1: There's another item on your list that seems to me
6:38
Speaker 1: to be like a provocation. It seems to me in
6:41
Speaker 1: a good way. It seems like directly contradictory to what
6:47
Speaker 1: I have read, specifically to this idea that all large
6:51
Speaker 1: language models are doing is guessing what the next word
6:55
Speaker 1: in a series is likely to be, and that list
6:58
Speaker 1: item is this. Llms often appear to learn and use
7:04
Speaker 1: representations of the outside world. Llms often appear to learn
7:09
Speaker 1: and use representations of the outside world. So that sounds
7:13
Speaker 1: quite different from just guessing the next word, is it
7:16
Speaker 1: or is it not? Different in a way that I
7:19
Speaker 1: just don't understand.
7:20
Speaker 2: It turns out it's not that different. Okay, this is
7:24
Speaker 2: I want to say it's the big discovery, But it's
7:26
Speaker 2: this big discovery that's spread out over dozens of experiments
7:30
Speaker 2: over the last few years.
7:31
Speaker 1: Can you give me a specific example. It's such an
7:34
Speaker 1: abstract assertion that I think it would be helpful to
7:38
Speaker 1: have a specific example.
7:40
Speaker 2: That we can think about. One great example of this
7:44
Speaker 2: is if you tell a model a story, a simple
7:48
Speaker 2: story that takes place in some sort of physical space
7:51
Speaker 2: where it's it's some characters walking around a house and
7:54
Speaker 2: they're having a conversation while they're walking, and they're picking
7:56
Speaker 2: style up and they're putting it down. You can see
8:00
Speaker 2: inside the activations of the neurons when the model is
8:03
Speaker 2: reading that story. You can pull out a map of
8:06
Speaker 2: the house. You can see that there's a there's a
8:09
Speaker 2: piece the network that says, oh, okay, now they're in
8:11
Speaker 2: the living room, and another piece that says, oh, living
8:13
Speaker 2: room is connected to the bedroom. And you can mess
8:17
Speaker 2: with this in ways that show that it's really sort
8:19
Speaker 2: of it is really representing the house. That if you
8:23
Speaker 2: find the piece of the network that says, oh, Susan
8:26
Speaker 2: is in the living room, and you flip that, flip
8:29
Speaker 2: that from a positive number to a negative number, then
8:32
Speaker 2: the story will continue as though Susan is not in
8:35
Speaker 2: a lot in the living room, or couldn't possibly have
8:37
Speaker 2: been in living.
8:37
Speaker 1: So that does seem like it's representing the physical world
8:41
Speaker 1: in a way that is not just guessing the next word.
8:45
Speaker 2: Yeah. Yeah, so we're finding out these systems are actually
8:50
Speaker 2: representing the objects they're talking about, at least some of
8:52
Speaker 2: the time.
8:53
Speaker 1: They're creating a representation of physical space.
8:56
Speaker 2: Yeah. I should be clear that this is this doesn't
8:58
Speaker 2: always work when when you're giving these systems something really
9:03
Speaker 2: hard and subtle, they're just going to totally botch this stuff.
9:06
Speaker 2: Their internal representations are a mess. But more and more
9:09
Speaker 2: of the time they're really doing it. And as these
9:12
Speaker 2: things get bigger and bigger, they're doing it more and more.
9:15
Speaker 2: And so this feels like this important turning point where
9:17
Speaker 2: it's like, oh, okay, there is some understanding going on
9:20
Speaker 2: here and it's getting better, and that really radically opens
9:24
Speaker 2: up the possibilities for where this technology might go.
9:27
Speaker 1: This what you're saying seems very much at odds with
9:34
Speaker 1: what people generally say about llms, Right, Like the standard
9:39
Speaker 1: line is they're just predicting what the next word is
9:43
Speaker 1: going to be. And they're very good at predicting what
9:44
Speaker 1: the next word is going to be, and there's a
9:45
Speaker 1: lot of powerful things you can do, but what you're
9:48
Speaker 1: saying sounds fundamentally different from that. And so I mean,
9:52
Speaker 1: are the people saying they're just predicting the next word?
9:54
Speaker 1: Are they wrong? Is what you're saying a point of
9:58
Speaker 1: debate among experts or what? Why is this so different
10:01
Speaker 1: than what I've heard before.
10:02
Speaker 2: There's a few things going on. So first, saying that
10:05
Speaker 2: they're just predicting the next word is mostly right. But
10:08
Speaker 2: it turns out that's saying that they just predict the
10:09
Speaker 2: next word is a lot like saying humans are just
10:12
Speaker 2: chemical reactions. It turns out that if you're trying to
10:16
Speaker 2: predict the next word, and if you've got a smaller
10:20
Speaker 2: work that's trying to predict the next word, it's going
10:22
Speaker 2: to learn that sort of the word, the and of
10:26
Speaker 2: an a and those show up often, and that's about
10:28
Speaker 2: all it's going to learn. If you take a medium
10:31
Speaker 2: sized neural network, it's going to learn how to write
10:33
Speaker 2: fluent sentences. This is going to write, oh, okay, sort
10:35
Speaker 2: of adjectives come before nouns, these kinds of nouns come
10:39
Speaker 2: before these kinds of nouns. It might even learn some facts.
10:41
Speaker 2: It might learn that if you talk about the president
10:44
Speaker 2: of the United States, you'll get names like Obama and
10:46
Speaker 2: Bush and Biden and Trump, and it'll start to kind
10:50
Speaker 2: of make sense, but it's still just kind of learning statistics.
10:53
Speaker 2: And if you make the neural work even bigger, it
10:56
Speaker 2: will abstract further away. It will start to reason about
11:00
Speaker 2: the people and the objects and the spaces themselves and
11:04
Speaker 2: use that abstraction to predict the next word. So kind
11:07
Speaker 2: of the more these systems learn about the world, the
11:11
Speaker 2: farther and farther their Internet representations get from just sort
11:13
Speaker 2: of literally what word comes after what other word.
11:17
Speaker 1: So there's another item on your list that seems like
11:20
Speaker 1: it should have interesting implications for the AI industry, right
11:24
Speaker 1: for the business of building lms, I'll just read that one.
11:28
Speaker 1: It goes lms predictably get more capable with increasing investment,
11:35
Speaker 1: even without targeted innovation. So we'll get into it. But
11:40
Speaker 1: just top line, what does that mean?
11:44
Speaker 2: We had language models in almost their modern form back
11:49
Speaker 2: in twenty ten, eleven, twelve. Most of the building blocks
11:53
Speaker 2: for them go back even farther to the eighties or
11:55
Speaker 2: even the sixties. You might have noticed that we weren't
12:00
Speaker 2: We didn't have chat GBT ten or twenty or fifty
12:04
Speaker 2: years ago. What people have been gradually discovering and dually
12:10
Speaker 2: sort of discovering to a greater and greater degree is
12:13
Speaker 2: that if you just take this reldly simple technology and
12:18
Speaker 2: throw more data at it and run it in its
12:22
Speaker 2: sort of training phase for longer and longer by fancier
12:25
Speaker 2: or and France your computers to run it on, it
12:28
Speaker 2: just keeps getting better.
12:29
Speaker 1: But if the technology is not special, I mean, everybody
12:32
Speaker 1: knows the basic sauce, it suggests that GPT might not
12:37
Speaker 1: have an open AI. The company that makes chat GPT
12:41
Speaker 1: might not have like that much of a moat, right.
12:45
Speaker 1: I mean, Google is clearly in this business, as is Anthropic,
12:49
Speaker 1: the company where you're working. Is there any reason to
12:53
Speaker 1: think open AI GPT is going to stay ahead.
12:56
Speaker 2: I think there's not a lot of secret sauce. There
12:59
Speaker 2: are some details of how to build these things that
13:01
Speaker 2: don't get published, but the basic idea is very much
13:04
Speaker 2: out there. And yeah, I think the the closest thing
13:10
Speaker 2: you can really have to emote is just enormous amounts
13:12
Speaker 2: of money. I think at some point you're going to
13:15
Speaker 2: have a relatively small number of labs building the really
13:18
Speaker 2: impressive frontier systems just because at some point these are
13:21
Speaker 2: going to be ten billion dollar projects, and it just
13:25
Speaker 2: seems unlikely that you're going to get that many ten
13:26
Speaker 2: billion dollar projects.
13:28
Speaker 1: If it's the case, as you say that, essentially what
13:31
Speaker 1: you need to build a frontier level LM is a
13:34
Speaker 1: lot of money. I would guess that governments around the world,
13:41
Speaker 1: certainly say China to pick a salient government, are probably
13:45
Speaker 1: building giant lms right now. Does that seem like a
13:48
Speaker 1: reasonable guess?
13:51
Speaker 2: Yeah, that seems right. I know there are a lot
13:55
Speaker 2: of private and private, public and public groups in China
14:00
Speaker 2: working in this stuff, and when I sort of hear
14:02
Speaker 2: people in the field who are following the geopolitical side
14:05
Speaker 2: of this more closely, they're paying a lot of attention
14:08
Speaker 2: to things like the Chips Act and Global Trade in
14:14
Speaker 2: chips in that you really do need. When you're spending
14:17
Speaker 2: these millions or billions of dollars, you're basically spending them
14:19
Speaker 2: to buy or rent very fancy, state of the art
14:23
Speaker 2: computer chips. And it has become a priority for the
14:27
Speaker 2: US to try to make it hard for China to
14:29
Speaker 2: do that, and.
14:33
Speaker 1: To try and make it hard for China to get
14:35
Speaker 1: at the processor level, which in a sense is like
14:38
Speaker 1: the cement that lllms are built from. There is a
14:41
Speaker 1: physical thing. We forget that, but it's fancy chips basically.
14:46
Speaker 2: That's right.
14:46
Speaker 1: Yeah, we've been talking so far about what we know
14:52
Speaker 1: about how large language models work. After the break, we'll
14:55
Speaker 1: get into what I think is the most interesting thing
14:58
Speaker 1: about lms, what we don't know about how they work.
15:09
Speaker 1: That's the end of the ads.
15:11
Speaker 2: Now we're going back to the show.
15:12
Speaker 1: So far, we've basically been talking about how do lllms work.
15:16
Speaker 1: What's going on? There is another bucket in your list,
15:22
Speaker 1: several items, three items that are it seems to me,
15:26
Speaker 1: in quite a different category, and they get at this
15:29
Speaker 1: very very interesting idea about lms, and that is, to
15:35
Speaker 1: some significant degree, nobody knows how they work. The people
15:40
Speaker 1: who build lms, people like you, people who build them
15:43
Speaker 1: and study them, don't understand a lot of what is
15:46
Speaker 1: going on, which is amazing to me and super interesting.
15:49
Speaker 1: So let's start with this list item. It says specific
15:56
Speaker 1: important behaviors in lms tend to emerge unpredictably as a byproduct.
16:02
Speaker 2: Of increasing investment.
16:03
Speaker 1: And you give a couple of examples of this happening
16:07
Speaker 1: for real in the world. I think the best way
16:09
Speaker 1: to understand what's going on here is to talk about
16:12
Speaker 1: one of those examples. Can you just like talk me
16:15
Speaker 1: through one of those examples of this unpredictable new behavior emerging. Yeah.
16:20
Speaker 2: So a specific large language model that people working in
16:23
Speaker 2: the stuff talk about a lot is GPD three. This
16:25
Speaker 2: came out a little less than three years ago and
16:28
Speaker 2: I think sort of kicked off the modern wave of
16:30
Speaker 2: research on this stuff. And one thing researchers would do,
16:34
Speaker 2: as these systems would would come out is give them
16:37
Speaker 2: math puzzles and logic puzzles and see how they did.
16:40
Speaker 2: And this could be as simple as just sort of
16:42
Speaker 2: giving the model reasonably hard arithmetic, sort of asking the model,
16:45
Speaker 2: what is one hundred and twenty five plus four hundred
16:49
Speaker 2: and sixty seven. And what they found is sort of
16:52
Speaker 2: GPD one was bad at this, and GPD two was
16:55
Speaker 2: bad at this, and at least for some of these tasks,
16:57
Speaker 2: GPD three was also bad at this. And they released it.
17:02
Speaker 2: They put it out in the world, they wrote a
17:03
Speaker 2: paper about it, they did some demos to researchers, and
17:06
Speaker 2: then eventually just let anyone sign up and use it.
17:09
Speaker 2: And after a few months people started noticing. Oh, there
17:14
Speaker 2: are some tricks you can use to actually make it
17:15
Speaker 2: quite a bit better at this. If you ask the
17:21
Speaker 2: model the right way, sometimes it'll just kind of reason
17:25
Speaker 2: out loud. Sometimes it will say, well, it'll actually do
17:28
Speaker 2: long edition, we'll actually write out its steps.
17:30
Speaker 1: So give me a specific example. How do you ask
17:33
Speaker 1: it the right way?
17:35
Speaker 2: So it took even a few more months for people
17:37
Speaker 2: to figure out how to do this systematically, but it
17:40
Speaker 2: turned out the trick was you literally say, let's think
17:43
Speaker 2: step by step.
17:44
Speaker 1: You actually type that in, you say that to the machine,
17:48
Speaker 1: to the model.
17:49
Speaker 2: Yes, And if you say what is this number of
17:51
Speaker 2: plus this number question mark, it'll give a wrong answer.
17:55
Speaker 2: If you say, what is this number of plus this number,
17:57
Speaker 2: let's think step by step dot dot, it's going to
18:00
Speaker 2: list out. Okay, let's start with the ones digit, and
18:03
Speaker 2: then the tenth digit, and then the one hundredth digit,
18:05
Speaker 2: and then give you the answer, and it'll very often
18:08
Speaker 2: be right huh. And it turns out this works really
18:10
Speaker 2: generally that for many kinds of sort of math and
18:14
Speaker 2: reasoning problems, even some even sort of ethics problems. There's
18:19
Speaker 2: a huge range of things you might ask one of
18:21
Speaker 2: these ural networks to do where if you just tell it,
18:24
Speaker 2: let's think step by step, it will bring out this
18:28
Speaker 2: whole reasoning ability that is actually really useful, that allows
18:31
Speaker 2: it to do much better at a lot of things,
18:32
Speaker 2: and that it didn't have before. And when this technology
18:37
Speaker 2: was first released, the people who built it, they did
18:39
Speaker 2: not know this was a possibility.
18:42
Speaker 1: That's wild, right, Like it means this thing is incredibly
18:45
Speaker 1: powerful in a way that the people who built it
18:49
Speaker 1: didn't know. And let's think step by step is just
18:52
Speaker 1: like this incantation. It's just like saying abracadabra or something,
18:56
Speaker 1: and the builders didn't know it was there.
18:59
Speaker 2: Yeah, it's it's a bizarre time to be working on
19:02
Speaker 2: this stuff.
19:02
Speaker 1: It Like, here's where it's getting a little sketchy to
19:06
Speaker 1: me at a certain level, right, I mean you've also
19:08
Speaker 1: done a lot of work in AI safety and this
19:10
Speaker 1: kind of section of the interview, I feel like we're
19:12
Speaker 1: getting more toward that, the section of like, the people
19:15
Speaker 1: building this stuff don't understand what it can do. And
19:17
Speaker 1: here should we add another list item here? Like this
19:20
Speaker 1: might be the place Cherkiff, So there's this other item
19:23
Speaker 1: on your eight things to know list that seems germane.
19:25
Speaker 1: Here experts are not yet able to interpret the inner
19:30
Speaker 1: workings of lms, which also wild also kind of goes
19:35
Speaker 1: with this idea of not knowing what the thing can do,
19:39
Speaker 1: right and very not intuitive for a piece of technology.
19:44
Speaker 1: Right If you go back to say the Internet, Sure
19:47
Speaker 1: we didn't know all the social implications of the Internet,
19:51
Speaker 1: but we knew how the technology worked. We knew what
19:54
Speaker 1: was going on with the chips and the wires and
19:56
Speaker 1: the electrons and whatever. Right, Like the amazing thing here
20:00
Speaker 1: is clearly we don't know the social implications of AI.
20:02
Speaker 1: But you're saying, we don't even know what it's doing
20:05
Speaker 1: inside the box.
20:08
Speaker 2: Yeah, that's right. We've got these very crude tools for
20:11
Speaker 2: sort of opening the box and looking inside. I mean,
20:13
Speaker 2: in a literal sense, we know it's going on. We
20:15
Speaker 2: can say, oh, when you put in this word, then
20:18
Speaker 2: it makes this number bigger, which makes that number smaller,
20:20
Speaker 2: which makes this number bigger. And you could keep saying
20:22
Speaker 2: that for twenty years and then you'd have explained what happened.
20:26
Speaker 2: But we haven't figured out any other way of talking
20:29
Speaker 2: about these systems that actually gives us any clarity about
20:33
Speaker 2: what's possible why these systems are doing what they're doing
20:35
Speaker 2: where they're reliable and not it's just this huge mess
20:39
Speaker 2: of connections that we don't really know what to do with.
20:43
Speaker 1: I mean, what should we make of this set of
20:48
Speaker 1: facts that these are incredibly powerful tools that nobody understands
20:54
Speaker 1: at a pretty deep level, that can do unpredictable things,
20:59
Speaker 1: that are able to do things that even their makers
21:03
Speaker 1: don't know they can do.
21:05
Speaker 2: I think it's pretty exciting and also pretty sobering. I
21:09
Speaker 2: think we don't have a good way of predicting how
21:11
Speaker 2: fast this is moving or what we're going to get when.
21:14
Speaker 2: But in the big picture, it seems like there's a
21:18
Speaker 2: lot of momentum toward building these really powerful eye systems
21:21
Speaker 2: over the next few years. We don't understand how they work.
21:25
Speaker 2: Another one of these list items is we also aren't
21:27
Speaker 2: very good at controlling, and we aren't very good at
21:29
Speaker 2: making them do what we want.
21:30
Speaker 1: Yes, let me just pause there, because it's the last
21:32
Speaker 1: list item and you have just walked up to it.
21:34
Speaker 1: So the last item, the item that we haven't mentioned
21:37
Speaker 1: on your list. There are no reliable techniques for steering
21:40
Speaker 1: the behavior of lms, so they're powerful. We don't really
21:44
Speaker 1: understand how they work. They can do things we don't
21:46
Speaker 1: know they're going to do, and we can't really control them.
21:49
Speaker 1: Now we're through the list. Now let's just talk it out.
21:51
Speaker 2: Yeah, and so we're yeah, we're building, we're building these systems.
21:55
Speaker 2: They're getting better, the developing new capabilities. We don't really
21:59
Speaker 2: know how they work. We can't predict which capabilities are
22:03
Speaker 2: showing up when and if they're doing something we don't want.
22:06
Speaker 2: We don't really know how to notice that and mitigate
22:09
Speaker 2: it and prevent it. And that just feels like it's
22:12
Speaker 2: playing with fire at a scale what I'm not sure
22:15
Speaker 2: we've seen before, at least outside of things like nuclear weapons.
22:18
Speaker 2: It's this very sort of sobering situation to be in.
22:20
Speaker 1: What do we do about it?
22:24
Speaker 2: I'm not sure. I wish I had a better answer.
22:28
Speaker 2: There are a few things that will definitely help. Maybe
22:31
Speaker 2: one obvious thing here is just there's probably a lot
22:34
Speaker 2: of regulation that would be good to have here. You
22:36
Speaker 2: really don't want the move fast and break things ethos
22:40
Speaker 2: to be behind a technology that is close to human
22:44
Speaker 2: level ability at a lot of cognitive task That seems
22:48
Speaker 2: like the setup for a bad sci fi movie.
22:49
Speaker 1: Specifically, what regulation do you think is a good idea?
22:54
Speaker 2: One outline of an idea that I'm excited about, and
22:57
Speaker 2: I think this is definitely not the best idea or
23:00
Speaker 2: the only good idea is mandating or standardizing some tests
23:07
Speaker 2: for particularly scary capabilities for things that would be particularly important.
23:11
Speaker 2: And this includes things like an opening. Eyes actually started
23:14
Speaker 2: doing this and inthropics also doing something like this is
23:17
Speaker 2: testing sort of if you ask the system to walk you,
23:21
Speaker 2: a layperson, through building a new biologic weapon, through sort
23:26
Speaker 2: of seeding the start of a new pandemic in your garage,
23:30
Speaker 2: will it Will it help you or will it help you?
23:33
Speaker 2: Sort of much much better than just googling around or
23:35
Speaker 2: talking to your friend of the PhD.
23:37
Speaker 1: And so then you have to think of all of
23:39
Speaker 1: the versions of that. You can think of whatever shutting
23:42
Speaker 1: down the electric grid, poisoning the water supply, building a
23:46
Speaker 1: nuclear bomb, right, I mean, are there people who are
23:47
Speaker 1: just making that list and making sure that chat GPT
23:50
Speaker 1: can't do it?
23:51
Speaker 2: There are people who are making this list, and I'm
23:53
Speaker 2: not sure there are enough of them, and I'm not
23:54
Speaker 2: sure they are involved in testing every system that's being built. Yeah,
23:58
Speaker 2: but it's kind of yeah, running through this checklist of
24:00
Speaker 2: what are the capabilities these systems could have that would
24:02
Speaker 2: be just really disruptive, that we don't want to move
24:07
Speaker 2: fast and break things with that we want to see
24:09
Speaker 2: coming and we want these to sort of influence our
24:12
Speaker 2: decisions about what actually gets deployed when and where.
24:15
Speaker 1: And that there's not some unpredictable abercadabra that nobody can see,
24:19
Speaker 1: but that three months later somebody will figure out.
24:21
Speaker 2: Right, Yeah. This is the big gap of this is
24:25
Speaker 2: I think we can say, Okay, once your system is
24:27
Speaker 2: this dangerous, only deploy it if it's really under control.
24:30
Speaker 2: We don't even know how to define that. We don't
24:32
Speaker 2: even know how you would be sure that a steamer.
24:38
Speaker 1: We'll be back in a minute with the lightning round.
24:50
Speaker 1: Now let's get back to the show. Okay, sign for
24:53
Speaker 1: the lightning round.
24:55
Speaker 2: Are you ready? All right, let's go.
24:56
Speaker 1: Let's go. What's your favorite fictional representation of AI.
25:00
Speaker 2: Off the top of my head, X, Mac and I
25:01
Speaker 2: was pretty good. The premises around I think what people
25:03
Speaker 2: in AI tend to worry about actually look not that
25:06
Speaker 2: far off of it, except that right now we're dealing
25:08
Speaker 2: with bots instead of instead of seductive.
25:11
Speaker 1: Robots, I liked the vibe of X mocking a lot.
25:15
Speaker 1: I like the aesthetic. I like how spare and empty
25:18
Speaker 1: it is. What's your favorite theory for how lms could
25:22
Speaker 1: destroy humanity?
25:24
Speaker 2: Oh, there's so many options and it's so hard to
25:27
Speaker 2: know where this goes.
25:30
Speaker 1: The what's one that's worth mentioning because it's surprising or
25:34
Speaker 1: because it's particularly worrisome, or for any reason.
25:38
Speaker 2: One kind of thing I'm particularly worried about is this
25:41
Speaker 2: sort of slow moving train wreck by way of politics
25:45
Speaker 2: that you get sort of totalitarian states get better and
25:48
Speaker 2: better at surveillance, political persuasion gets better and better, and
25:52
Speaker 2: so democratic political campaigns go more and more off the rails.
25:57
Speaker 2: You wind up with more and more to Helderan states.
26:00
Speaker 2: They're more and more effective, and they themselves are leaning
26:03
Speaker 2: more and more on AI to do important work. And
26:08
Speaker 2: at that point, sort of something like an AIK doesn't
26:11
Speaker 2: seem that crazy.
26:11
Speaker 1: And in particular, what is the large language model doing
26:14
Speaker 1: there in that story.
26:17
Speaker 2: Persuading people one on one, surveiling people one on one,
26:21
Speaker 2: also making political decisions, sort of deciding how resource should
26:25
Speaker 2: be allocated and who should be empowered with any government,
26:28
Speaker 2: and eventually making military decisions and eventually making big economic decisions.
26:32
Speaker 2: I just sort of worry about this world where people
26:34
Speaker 2: put more and more trust in systems because they work,
26:37
Speaker 2: and that helps centralize things more and more into fewer
26:40
Speaker 2: and fewer institutions, and that makes those institutions really really delicate.
26:45
Speaker 2: And if an aisystem goes up the rails and start
26:48
Speaker 2: doing something that even their creators don't want, that gets
26:51
Speaker 2: pretty arbitrarily bad.
26:53
Speaker 1: What's your favorite theory for how llms can help humanity?
26:57
Speaker 2: I think the big ones are education and science. I
27:01
Speaker 2: think it would be pretty cool if you could hire
27:03
Speaker 2: a really world class like sort of Oxford Oxford tutorial
27:08
Speaker 2: quality tutor for just everyone with access to a computer
27:12
Speaker 2: of any kind, and that feels like.
27:13
Speaker 1: On your phone. Close You could do it on your phone.
27:15
Speaker 2: Yeah, yeah, And I don't think we've really figured out
27:18
Speaker 2: how to make that work, but I think if that
27:20
Speaker 2: really works, that could be really transformative for the better.
27:25
Speaker 2: On science, I think there's a lot of just really
27:27
Speaker 2: thorny problems around things like drug development, things like sort
27:31
Speaker 2: of fusion power and clean energy, where it could be
27:34
Speaker 2: that just having these systems that can kind of digest
27:38
Speaker 2: more information understand more at once could unlock a bunch
27:41
Speaker 2: of important stuff that would otherwise take us many more
27:44
Speaker 2: generations to get to.
27:46
Speaker 1: On balance, you think the potential upside of AI outweighs
27:53
Speaker 1: the potential downside.
27:55
Speaker 2: Probably, But I think that really depends on us being
27:58
Speaker 2: careful right now. I think this makes me optimistic in
28:01
Speaker 2: the long run, but I think there's a there's a
28:05
Speaker 2: real chance that things sort of go off the rails
28:07
Speaker 2: if this keeps being kind of a free for all
28:10
Speaker 2: commercial product for more than a few more years.
28:14
Speaker 1: You went viral on Twitter a while ago when you wrote, quote,
28:20
Speaker 1: doing a PhD is in most cases a terrible idea
28:24
Speaker 1: you should put out have a PhD. Also, it's worth
28:28
Speaker 1: pointing out that PhDs have been saying this for I
28:31
Speaker 1: guess as long as there have been PhDs. So there's
28:35
Speaker 1: a lot of questions you could ask here. Well, there's
28:38
Speaker 1: two really, like why do people with PhDs keep saying
28:41
Speaker 1: don't get a PhD? And also why do people keep
28:43
Speaker 1: ignoring them? Why do people keep going to get PhDs?
28:48
Speaker 2: This was in a moment of being being particularly horrified
28:52
Speaker 2: at some of the sort of common outcomes and PhD programs,
28:56
Speaker 2: and I think the average case is really bad the
29:00
Speaker 2: average case, literally, I think the median PhD gets an
29:02
Speaker 2: actual diagnosis of depression or anxiety and often doesn't get
29:07
Speaker 2: that much out of the program, like kind of really
29:08
Speaker 2: struggle in it, and because they're really struggling in it,
29:12
Speaker 2: don't accomplish that much and don't have great job prospects
29:15
Speaker 2: near the end. The best case, if you get a
29:18
Speaker 2: sort of top five percent PhD, it's really great. You
29:20
Speaker 2: get to play around with great resources and do whatever
29:23
Speaker 2: you want and explore new ideas for a few years
29:25
Speaker 2: and it opens up really tremendous opportunities. But yeah, I
29:30
Speaker 2: think it's the kind of thing that people should really
29:34
Speaker 2: really check their motivations and check their resilience before going
29:37
Speaker 2: into it and kind of brace themselves just because it
29:41
Speaker 2: is so often such a such a difficult experience.
29:45
Speaker 1: Why do you think people keep going to get PhDs?
29:48
Speaker 2: I mean, there is some real upside. There are some
29:50
Speaker 2: really cool jobs that you can only get if you
29:52
Speaker 2: have one. But I think there's also this piece, and
29:54
Speaker 2: this is maybe why I had my snippy tweet about this,
29:58
Speaker 2: that if you're a sort of smart, nerdy college student
30:02
Speaker 2: at a research university where you've got lots of opportunities
30:05
Speaker 2: to kind of work in research labs. Then you can
30:09
Speaker 2: get this really strong social signal that just like you're
30:11
Speaker 2: good at school, you should keep doing school, Like doing
30:14
Speaker 2: APHD is what it looks like, keep doing school. This
30:16
Speaker 2: is just the obvious way to use your talents and
30:20
Speaker 2: just kind of jump into that out of momentum, and
30:22
Speaker 2: that's that can be I think a pretty a riskier
30:24
Speaker 2: decision than it looks like.
30:26
Speaker 1: If everything goes well, what problem will you be trying
30:29
Speaker 1: to solve in say, five years?
30:32
Speaker 2: But I don't know. I got into this stuff sort
30:36
Speaker 2: of through the cognitive science, sort of through the idea
30:38
Speaker 2: that you don't really understand something until you can build it,
30:42
Speaker 2: and I really want to understand how minds work, why
30:45
Speaker 2: it is that sort of hooking neurons together in your
30:48
Speaker 2: head this way makes something that can think and that
30:50
Speaker 2: can experience and sort of mixed in with all of
30:55
Speaker 2: this very consequential real world stuff that's going on with AI,
30:58
Speaker 2: as we're building all these tools, we're also building really
31:01
Speaker 2: great tools for just doing cognitive science and sort of
31:03
Speaker 2: figuring out the answers to a lot of really old
31:07
Speaker 2: questions about how the human mind works and if all
31:11
Speaker 2: the practical problems are solved and everything's under control and
31:14
Speaker 2: going great, then I'd be happy to get back into
31:16
Speaker 2: that stuff.
31:17
Speaker 1: So you would have to be less worried about the
31:19
Speaker 1: world than you are now.
31:21
Speaker 2: I think that's right.
31:23
Speaker 1: Well, I hope it goes. I hope you become less
31:25
Speaker 1: worried about the world. I guess I'm not super optimistic
31:28
Speaker 1: about that. I feel like I'm generally a reasonably optimistic person,
31:31
Speaker 1: but this one seems seems like there's a lot to
31:33
Speaker 1: worry about on this one.
31:34
Speaker 2: Yeah, yeah, thanks, thanks for the well wishes. And yet
31:39
Speaker 2: it feels like it feels like sort of decent chance.
31:42
Speaker 2: Things go badly, decent chant things go very well. But
31:45
Speaker 2: I'm it seems pretty sure that stuff is just getting weird,
31:48
Speaker 2: that research five years from now is not going to
31:50
Speaker 2: look like research now, and probably saying with many, many,
31:53
Speaker 2: many other things we do.
32:01
Speaker 1: Sam Bowman is an associate professor at NYU, and he
32:04
Speaker 1: runs a research group at the AI Company and THROP.
32:08
Speaker 1: Today's show was edited by Lydia Jean Kott. It was
32:11
Speaker 1: produced by David Jah and Edith Russelo and engineered by
32:15
Speaker 1: Amanda k Wong. I'm Jacob Goldstein and We'll be back
32:19
Speaker 1: next week with another episode of What's Your Problem.