Latest / The Smart Cookies podcast / "AI Will Be The Last Mistake Humanity Ever Makes..." With Prof. David Krueger

Transcript

"AI Will Be The Last Mistake Humanity Ever Makes..." With Prof. David Krueger · The Smart Cookies podcast

• 0:00
  yeah okay cool like you think ai is a normal
• 0:02
  technology uh there's a bunch of experts who
• 0:04
  don't so like what should society do should it
• 0:06
  be like i guess the experts don't know no it
• 0:08
  should be like half the experts are saying this
• 0:10
  thing could kill us next year i mean okay that's
• 0:12
  an exaggeration but like you know like a bunch
• 0:15
  of people are saying this thing could kill us
• 0:17
  in like a decade right um they're saying it's
• 0:20
  incredibly dangerous uh and then there's a bunch
• 0:22
  of people who disagree with them but like who's
• 0:24
  right we don't know so like what's the safe thing
• 0:26
  to do listen to the people who are sounding the
• 0:27
  alarm David Kreuger is an assistant professor
• 0:32
  in robust reasoning and responsible AI at the
• 0:34
  University of Montreal and a core academic member
• 0:37
  at MILA, the Quebec Artificial Intelligence Institute.
• 0:40
  He was an intern on Google DeepMind's AI safety
• 0:42
  in 2018. A couple of years later, David founded
• 0:46
  Evitable, a nonprofit whose mission is to inform
• 0:49
  and organize the public to confront societal
• 0:51
  scale risks of AI and put an end to the reckless
• 0:55
  race to develop super intelligence. David, welcome
• 0:58
  to the show. Hey, thanks for having me. Your
• 1:00
  journey is a fascinating one because you trained
• 1:02
  under Joshua Bengio, I believe, who along with
• 1:05
  Jeffrey Hinton and Jan LeCun is considered one
• 1:07
  of the godfathers of AI. And I can imagine that
• 1:10
  working in a futuristic pioneering field such
• 1:13
  as artificial intelligence with actual pioneers
• 1:16
  must be as exciting as exciting can be. And yet
• 1:19
  here you are in the present moment, the middle
• 1:21
  of 2026, urging... the industry and the world,
• 1:25
  not just to slow down the development of AI,
• 1:28
  but to stop it altogether. How did that happen?
• 1:32
  Well, when I got into the field of AI in 2012,
• 1:37
  2013, it's because I heard about deep learning
• 1:40
  from Jeff Hinton's online Coursera lectures.
• 1:44
  And this looked to me very different from every
• 1:50
  other approach to AI I'd heard about. So my background
• 1:53
  wasn't really in AI. I studied math in college,
• 1:56
  but I became interested in AI around the beginning
• 1:58
  of my undergrad. And at the time, the field was
• 2:02
  really, really, really different. Nobody wanted
• 2:06
  to use the phrase AI anymore. People only talked
• 2:10
  about machine learning. And I think most people
• 2:13
  had kind of given up on the original grand ambitions
• 2:15
  of the field to create human -level, human -like
• 2:17
  intelligence. I was worried starting around the
• 2:23
  middle of college. That's like 2009, 2010, something
• 2:28
  like that. I kind of shifted from being more
• 2:31
  excited to increasingly worried about the prospect
• 2:33
  of developing human -level AI and superhuman
• 2:36
  AI. Or even just scaling up kind of stupid AI.
• 2:44
  because it would be cheaper and faster and i
• 2:47
  saw that we would likely kind of deploy it throughout
• 2:50
  society to run things and make a lot of decisions
• 2:52
  and that would be really dystopian and on dehumanizing
• 2:55
  and stuff but um i was kind of interested in
• 3:01
  understanding what the experts thought so you
• 3:05
  know 2012 i hear about deep learning i'm like
• 3:07
  oh wow this could take like a few decades instead
• 3:10
  of you know i thought it was more like centuries
• 3:11
  before that because everything else i saw didn't
• 3:13
  look very promising um and like within a day
• 3:16
  you know of what binge watching these lectures
• 3:18
  i was like holy shit this is this is happening
• 3:21
  um this is the thing you know and so uh yeah
• 3:27
  i wanted to get involved and and see if the people
• 3:30
  in the field were worried about this stuff uh
• 3:31
  that i was worried about um and if not why not
• 3:35
  right i hope they had good reasons um but I wasn't
• 3:40
  surprised to find out that they didn't. And so
• 3:43
  it's really been since that moment, since I got
• 3:47
  into the field and realized that nobody else
• 3:49
  was worried about what happens when we actually
• 3:51
  get there, which could be in 10, 20, 30 years,
• 3:56
  I've been like, well, obviously, we shouldn't
• 3:59
  be doing this. So it's been a long time coming
• 4:03
  in that sense. But for most of my career, It
• 4:07
  took me a couple of years before I met anybody
• 4:09
  in the field who shared these concerns. And even
• 4:12
  after that, for most of my career, it's been
• 4:14
  kind of a taboo topic. And I've spent a bunch
• 4:16
  of my time just trying to convince other researchers
• 4:18
  to think about this, to take it seriously, to
• 4:20
  recognize that there are these hard technical
• 4:23
  problems where you need to solve to even control
• 4:25
  these things. But also, if you look at how the
• 4:28
  technology is likely to be developed, it's probably
• 4:30
  going to be like some sort of out -of -control
• 4:31
  arms race. probably going to be put into everything
• 4:34
  including autonomous weapons whether you like
• 4:38
  it or not um and you know all that stuff has
• 4:41
  been happening and so over time more and more
• 4:43
  people are kind of recognizing the situation
• 4:46
  we're in um and and coming to agree with me and
• 4:49
  and sharing these concerns 2023 was like a big
• 4:53
  watershed moment in this regard because after
• 4:55
  ChatGPD came out and Yashua Bengio and Jeff Hinton
• 4:59
  both sort of came out of the closet as being
• 5:02
  very concerned about this and started spending
• 5:05
  most of their time and effort warning the rest
• 5:08
  of the world that we could go extinct in like
• 5:12
  five to 20 years were the kind of timelines they
• 5:16
  were talking about if we don't do something.
• 5:18
  That was, to me, marked kind of the end of the
• 5:21
  era when this was something that was mostly a
• 5:26
  conversation within the research community. So
• 5:28
  I also initiated this one sentence statement
• 5:30
  that said, like, mitigating the risk of extinction
• 5:32
  should be a global priority. Got the CEOs of
• 5:35
  leading companies, Joshua and Jeff, all these
• 5:36
  people signed on to this thing. And ever since
• 5:41
  then, it's been like, OK, this is something the
• 5:43
  public actually is paying attention to. policy
• 5:45
  makers are paying attention. Not as much as they
• 5:47
  should, but there is an opportunity now to talk
• 5:49
  to politicians and the public about this. And
• 5:53
  so that's what I've been shifting towards doing
• 5:56
  ever since then. So that's really why I started
• 5:58
  Evitable. I did a little stint at the UK government
• 6:00
  first, helping set up the UK AI Security Institutes.
• 6:03
  That's the oldest and biggest government institution
• 6:07
  dedicated to AI safety. But the response just
• 6:14
  hasn't been... adequate right like people still
• 6:16
  aren't taking this problem seriously enough um
• 6:18
  it's still like a kind of like somewhat taboo
• 6:20
  topic within mainstream society and politicians
• 6:23
  um although that's changing rapidly um so yeah
• 6:27
  it's it's i've been on this on this kick for
• 6:30
  a long time actually it's the short answer you
• 6:34
  said that you had like a eureka moment i believe
• 6:36
  in 2009 or 2012 once you started those corsair
• 6:41
  lectures by jeffrey hinton And you realized how
• 6:44
  groundbreaking and at the same time, how dangerous
• 6:46
  this stuff actually is. Have your fears ever
• 6:51
  since that moment of what the past 15 years kind
• 6:53
  of confirmed your suspicion in this danger? Pretty
• 6:58
  much. Yeah. I mean, there's mixed signals, but
• 7:02
  like I said, I mean, nobody that I could tell
• 7:06
  was worried about this stuff when I got into
• 7:07
  the field. And now it's like very, very mainstream.
• 7:10
  I mean, it's still somewhat controversial, but
• 7:11
  like it's. People acknowledge now the argument
• 7:14
  is more about like, how bad is the risk? What
• 7:16
  should we do about it? You know, should we slow
• 7:19
  down? How much should we slow down? Or should
• 7:21
  we actually stop? You know, the conversation
• 7:22
  is getting there, like where it needs to go,
• 7:24
  which is a recognition that what we're currently
• 7:27
  doing is insane and suicidal. And we need to
• 7:31
  stop, you know, immediately. We need an immediate,
• 7:35
  indefinite international moratorium on frontier
• 7:39
  AI development. The sooner everybody starts working
• 7:42
  on that, the better. I think it's likely to be
• 7:44
  pretty soon at this point. Things are moving
• 7:45
  really fast. Again, Jeffrey Hinton, Nobel laureate,
• 7:50
  by the way, not just some guy from the pub. He
• 7:54
  said in a recent interview that he puts the risk
• 7:59
  of human extinction because of AI at around 20
• 8:03
  % to 25%. How do you calculate something like
• 8:06
  that? And do you agree with his assessment? I
• 8:10
  think it's higher. I think it's, you know, more
• 8:12
  like 75 % maybe. I think there's not, you know,
• 8:18
  I would say a lot of my hope is in us stopping
• 8:23
  and getting the indefinite moratorium that I
• 8:27
  was talking about. Otherwise, I think, you know,
• 8:30
  we can get lucky. You know, people are trying
• 8:32
  things, those things might work, but I feel like
• 8:34
  it's, you know, we're throwing like... 100 Hail
• 8:37
  Mary passes, or like 20 Hail Mary passes, and
• 8:41
  like, maybe one of them will work. Yeah, there's
• 8:43
  no calc. I mean, I think maybe a couple people
• 8:45
  have tried to calculate this. If you wanted to
• 8:47
  do that, you would like make some sort of explicit
• 8:49
  model and some assumptions, and then, you know,
• 8:52
  throw probabilities everywhere. And I actually
• 8:54
  had a student who was working on something like
• 8:56
  that for a little bit. But I don't think he ever
• 9:00
  finished that project. Maybe he did, I should
• 9:03
  check. But yeah, I think it's, it's more, you
• 9:07
  know, kind of an intuition thing. And, you know,
• 9:08
  I, I think these, I think through like different
• 9:11
  kinds of scenarios and stuff in my, in my mind,
• 9:13
  but a lot of it's kind of just like at the end
• 9:14
  of the day, it's sort of the intuition heuristic
• 9:16
  thing. Yeah. I mean, Hinton says his own gut
• 9:20
  feelings like 50%. Um, so I'm, I'm really surprised
• 9:22
  that he's, you know, deferring to others on this
• 9:25
  topic. Uh, cause it seems like he's not the kind
• 9:28
  of person who, you know, usually does that very
• 9:30
  much. So I think you should be saying 50%, if
• 9:32
  I'm honest, that seems good. what he really believes
• 9:35
  in some sense. In a recent interview, you said
• 9:38
  that we will view this moment like human sacrifice
• 9:41
  and slavery. That's something one doesn't hear
• 9:44
  very often. What did you mean by that? Like I
• 9:47
  said, what we're doing is insane and it's horrible.
• 9:52
  It's, you know, an atrocity in the making. It's
• 9:55
  doesn't, it's like, you know. We're on track
• 10:00
  to make the greatest mistake that we've ever
• 10:02
  made as a species, which is to replace ourselves
• 10:05
  basically with a new form of life that we're
• 10:09
  making, you know, very little effort to. to even
• 10:16
  like understand or like seriously attempt to
• 10:20
  control or establish some sort of, you know,
• 10:23
  symbiotic relationship with, you know, people
• 10:26
  talk about these things, but again, it's just
• 10:27
  kind of like wishful thinking. Hail Mary is like,
• 10:30
  Oh, maybe, maybe a little lucky. So it's, it's
• 10:34
  really hard for people to believe how bad this
• 10:38
  is. And it's, it's hard to talk about it in a
• 10:41
  way that, you know, is, is real and is like,
• 10:45
  emotionally grounded because it is horrible.
• 10:49
  And, you know, people like me who who work on
• 10:51
  this every day, it's part of the background condition
• 10:53
  of our existence. So we find a way to to not
• 10:57
  engage with that every day. You know, you kind
• 11:01
  of have to compartmentalize and you get a lot
• 11:03
  of people treating this like with dark humor
• 11:07
  or sort of turning it into a very kind of academic
• 11:10
  mode of discussion. But. So it's kind of easy
• 11:17
  in a way to lose sight of just how horrible this
• 11:21
  is. The people who are building AI are quite
• 11:27
  likely to build something that kills a thousand
• 11:31
  times as many people as the Holocaust. I think
• 11:34
  here we come to the crux of the problem. You
• 11:36
  mentioned that people... that we still don't
• 11:38
  realize how dangerous this is. And I'm totally
• 11:41
  one of them because I'm not very educated in
• 11:43
  this field. And that's why I'm having you on.
• 11:45
  I'm a very superficial user of LLMs, to say the
• 11:47
  least. I obviously don't use agentic AI. I dabble
• 11:50
  in this stuff for the crappiest tasks on my to
• 11:53
  -do list. namely translating horrible bureaucratic
• 11:56
  mail since I live in Japan, and sometimes for
• 11:59
  a bit of research too. But truth be told, LLMs
• 12:02
  just bore me to death at this point. I'm not
• 12:04
  excited about this technology at all. I never
• 12:06
  shook off the feeling that at the customer level,
• 12:09
  it's all a bit gimmicky. And the practice of
• 12:12
  cramming AI, like you said at the beginning,
• 12:14
  into every possible app, which makes everything
• 12:16
  even more awkward and stupid, just confers me
• 12:19
  in disbelief. The point being is that as a...
• 12:23
  casual dabbler in this technology who can see
• 12:26
  it being, I don't know, maybe genuinely useful,
• 12:29
  maybe even transformative in a lot of contexts.
• 12:32
  I at the same time have a real problem picturing
• 12:34
  this thing that, by the way, yesterday just made
• 12:38
  me the worst PowerPoint presentation of all time.
• 12:40
  It was really hot garbage. Actually being kind
• 12:45
  of, like you mentioned, two heartbeats away from
• 12:48
  waking up a Skynet. I'd like you to help me see
• 12:50
  what I'm missing here. There's a lot that goes
• 12:52
  into that. I think one thing is just, you know,
• 12:58
  having seen the development of the technology
• 13:00
  over the past almost 15 years, when I got into
• 13:05
  the field, AI could barely string together a
• 13:08
  word, let alone a sentence. You could not generate
• 13:12
  photorealistic images of anything, full stop,
• 13:16
  at any scale, at any resolution. Models that
• 13:22
  could, like, recognize handwritten digits. They
• 13:24
  couldn't recognize, you know, objects, let alone,
• 13:28
  you know, scenes. They couldn't play, you know,
• 13:33
  video games. They couldn't do, you know, any
• 13:39
  of the things that today's systems can. They
• 13:41
  certainly couldn't, like, make a PowerPoint of
• 13:44
  any variety, right? So I think the rate of progress
• 13:48
  has exceeded. virtually everyone's expectations
• 13:53
  the entire time I've been in the field. Deep
• 13:57
  learning was a very small field when I joined.
• 13:59
  It was like 100, 200 people working on it. It
• 14:01
  was very fringe. And for the first several years
• 14:04
  of my time in the field, most AI researchers
• 14:08
  were like, deep learning, that's not going anywhere.
• 14:11
  You know, that's a fad. And that's been, you
• 14:14
  know, people just keep saying it's going to hit
• 14:15
  a wall. It's not going to be able to do this.
• 14:16
  It's not going to be able to do that. Like half
• 14:18
  the time. Today, I can do that within a year.
• 14:21
  And the other half the time, it actually already
• 14:24
  can do that. And they just don't realize it.
• 14:26
  That's what I wanted to ask you, actually. So
• 14:28
  you don't think that we've just stretched this
• 14:31
  technology, like large language models, beyond
• 14:35
  our wildest expectations, but we're going to
• 14:37
  hit a hard wall beyond which it's impossible
• 14:42
  to advance this technology. Just like an iPhone.
• 14:45
  It's still like, you know. It's still an iPhone
• 14:48
  now. I mean, the camera is getting a little bit
• 14:50
  better. I mean, that's certainly possible. I
• 14:52
  don't claim to be able to predict the future
• 14:55
  100 % accurately or anything like that. I don't
• 15:01
  think we really can't count on that. I mean,
• 15:03
  I think the people who have been making those
• 15:04
  kinds of predictions have a pretty bad track
• 15:07
  record at this point. That being said, I do think
• 15:11
  there are some things that are wrong. with today's
• 15:15
  systems that are off about them and it's hard
• 15:18
  to tell how fundamental those issues are and
• 15:22
  how much progress will be needed to you know
• 15:26
  to to surpass those limitations um and you know
• 15:33
  i don't really view this as i think a lot of
• 15:36
  the world views ai and lns as kind of a zero
• 15:38
  to one moment where it's like it didn't exist
• 15:40
  then we had chat gpt bam it exists and then maybe
• 15:43
  they don't see a ton of progress since then.
• 15:45
  I don't view it that way at all. I view this
• 15:47
  as like, you know, it's this curve and it's just
• 15:49
  growing and going up, up, up, because there was
• 15:51
  so much progress that happened before that. And
• 15:53
  there's been so much progress since then on,
• 15:55
  you know, to my eyes as a researcher who's been
• 15:59
  following it and involved. And so, yeah. And
• 16:04
  it's not just scaling, you know, people sometimes
• 16:06
  say, oh, we're just scaling it up, making it
• 16:08
  bigger, but that's not gonna, you know, that
• 16:10
  won't. If there's some fundamental flaw with
• 16:14
  deep learning or LLNs, then that won't fix that.
• 16:17
  But it's not just that. People are constantly
• 16:19
  doing more research and innovating. And there's
• 16:24
  about one minor breakthrough a year in AI, I
• 16:28
  would say. And a lot of them are pretty simple,
• 16:30
  but they have a big impact. And maybe the most
• 16:36
  recent one that at least is widely known is like...
• 16:40
  figuring out how to get the reinforcement learning
• 16:42
  working on large language models, which enabled
• 16:45
  the creation of these more agentic systems that
• 16:47
  can, you know, do software engineering tasks
• 16:50
  and can build, build, you know, software and,
• 16:54
  and, and so on. Um, so yeah, I, I think it's,
• 16:59
  uh, that's, that's part of it, but there's, there's
• 17:02
  so much that goes into this. I mean, there's
• 17:04
  also how smart are humans really? Like we like
• 17:07
  to think that we're special, you know? Like we
• 17:10
  historically we were like, we think the Earth
• 17:13
  is the center of the universe. We think that
• 17:15
  we're some special type of animal that, you know,
• 17:18
  is completely separate from the other animals.
• 17:22
  And then, you know, we learn that we're not,
• 17:25
  you know, we're just one little speck in this
• 17:27
  giant cosmos. And, you know, we're just the descendant
• 17:31
  of these other animals. And I think similarly,
• 17:36
  like a lot of the stuff that humans. think of
• 17:38
  as like really you know special about our intelligence
• 17:41
  might just not be that great it might just be
• 17:43
  kind of kludges and hacks and heuristics and
• 17:46
  you know a lot of our knowledge is cultural um
• 17:49
  and a lot of what we do doesn't make sense and
• 17:52
  we make a lot of mistakes and so yeah i don't
• 17:55
  know that the human intelligence is as as great
• 17:58
  as people make it out to be and then We're getting
• 18:02
  to a point where the AI companies claim that
• 18:04
  they are using AI to automate their internal
• 18:08
  coding and not just coding, but starting to do
• 18:11
  the research itself, the R &D. And if AI can
• 18:18
  accelerate the rate at which we're making progress
• 18:20
  on the foundational techniques for AI, we could
• 18:24
  see completely new paradigms coming out of the
• 18:28
  current paradigm. So even if LLMs would never
• 18:32
  be able to do all the things humans can do, if
• 18:33
  they can do the one thing of doing AI research,
• 18:36
  then that might be all it takes to get us all
• 18:39
  the way to superintelligence. And that's what
• 18:41
  the companies are saying and betting on. I was
• 18:44
  just listening to an interview with, was it the
• 18:47
  CEO of Rockstar Games? I think it went a little
• 18:50
  bit viral. And he said that the problem with
• 18:53
  the current state of AI is that it always looks
• 18:56
  at old data sets. And that's its kind of limit
• 19:00
  as well. It can kind of copy things. I think,
• 19:04
  of course, because he's in the field, he gave
• 19:06
  the example of like, of course, it can make a
• 19:09
  good copy of Grand Theft Auto, but it can't make
• 19:13
  a new Grand Theft Auto game. You know, it's always
• 19:16
  looking backwards. It can never look forwards
• 19:18
  by design. Do you think there's some truth to
• 19:20
  that? Not really. I mean, maybe there's a little
• 19:25
  bit. There's some truth to it in the sense of,
• 19:27
  yeah, we train it on all the data we can get
• 19:29
  our hands on. Most of that is older data. It'll
• 19:32
  be biased towards that training data. Deep learning
• 19:35
  models have some creativity. They're not just
• 19:40
  stochastic parrots. They're not just regurgitating
• 19:41
  things in their training data. That's a myth.
• 19:44
  If you want to test this, just ask for it to
• 19:50
  write... Bob Marley song about Donald Trump becoming
• 19:55
  president. And of course, Bob Marley never wrote
• 19:57
  such a song because Donald Trump didn't become
• 19:59
  president when he was alive. But it'll, you know,
• 20:02
  it'll make something up. It won't be probably
• 20:04
  as good as what Bob Marley would have written.
• 20:05
  But, you know, so it can create genuinely new
• 20:09
  things. And you can say, oh, well, it's just
• 20:12
  like, you know, a collage. It's just like mashing
• 20:13
  together elements of things. And that's where
• 20:16
  I'm like, well, how do you know humans aren't
• 20:18
  doing that? You know, like. we we have we borrow
• 20:22
  we are inspired by things we we copy we modify
• 20:25
  reiterate a lot of you know i'm a musician and
• 20:28
  a lot of times i write a song i'm like oh wait
• 20:31
  i think i just wrote that song again shoot um
• 20:34
  or like a lot of when i come with something new
• 20:36
  it's because i'm literally just like messing
• 20:37
  around on the guitar and just playing like basically
• 20:39
  random notes until i find something that sounds
• 20:40
  good you know um so i think a lot of human creativity
• 20:44
  works that way And it's a process of trial and
• 20:47
  error. It's a process of recombining cultural
• 20:50
  elements and things that we've seen. I think
• 20:53
  that's all stuff that's fundamentally within
• 20:54
  the capabilities of these systems. They might
• 20:56
  not be doing it the same way as humans. They
• 20:58
  might not be as good as humans at it, but it's
• 21:00
  not like some completely separate skill that
• 21:03
  they have no ability to do whatsoever. Speaking
• 21:08
  of old data sets, at least a couple of years
• 21:11
  ago... when ChatGPT burst into the scene, there
• 21:14
  was a lot of talk about what happens after all
• 21:18
  the data is consumed by AI, that that's also
• 21:21
  going to be some sort of a limit because we don't
• 21:23
  have new, we don't, you know, once they skim
• 21:27
  through all of the stuff, all the Reddits, the
• 21:30
  Wikipedias, all the books that we have in existence
• 21:32
  and articles and such, that's pretty much going
• 21:35
  to be it. And that the new kind of synthetic
• 21:37
  data, is it synthetic data? Is just not as powerful.
• 21:42
  or dense with knowledge as the original legacy
• 21:46
  data or whatever was? Honestly, I wish I knew.
• 21:50
  I'm sure other people know better than I do the
• 21:53
  answer to this. I do think synthetic data went
• 21:56
  farther than people expected. So that's already
• 22:00
  happening? Oh, yeah. I mean, people were doing
• 22:03
  synthetic data stuff. At the same time as ChatGPT
• 22:07
  came out, people were already working on that.
• 22:08
  No, because there's a, I'm sorry, I get a lot
• 22:11
  of my info from social media, which is probably
• 22:12
  not a good thing. But there's another viral post
• 22:15
  about how these AI labs are like scanning old
• 22:20
  and rare books and then destroying them. And
• 22:23
  people are obviously very upset for good reason.
• 22:25
  So I thought, okay, this is still going on. they're
• 22:28
  still copying books. They're still trying to
• 22:31
  train on actual data. So I thought maybe synthetic
• 22:34
  data comes after the last book on planet Earth
• 22:37
  has been consumed like this. No, no, no. They
• 22:40
  do both. Synthetic data can be better in some
• 22:43
  regards. It can be more targeted to the things
• 22:46
  that you're trying to teach it. It can be more
• 22:48
  compact and compressed. So one of the instances
• 22:51
  that I remember from several years ago, they
• 22:55
  basically asked... ai systems to write a textbook
• 23:00
  on some topic and then they used that to train
• 23:03
  an ai system just you know just that like textbook
• 23:06
  basically just the the synthetic data um and
• 23:09
  they got a system that was like very good at
• 23:12
  you know in that domain trained only on the synthetic
• 23:14
  data that you know was basically i don't know
• 23:18
  how close it was like to an actual textbook for
• 23:21
  humans but you know it was like a textbook written
• 23:23
  by an ai for an ai yeah i read the textbook and
• 23:25
  it understood the stuff also the law of like
• 23:28
  diminishing returns doesn't apply here with with
• 23:30
  synthetic data i thought like then the output
• 23:32
  would be much more poor since this is already
• 23:35
  like i don't know i'm sure it is in some in some
• 23:37
  cases in some ways i mean like classic thinking
• 23:39
  and machine learning is like you need that ground
• 23:41
  truth data like that's it's a fundamental limit
• 23:44
  to what you can do um But yeah, I mean, there's
• 23:47
  also a thing where I don't think we've hit the
• 23:50
  data limitation in terms of scaling up the models
• 23:52
  more. So you get a better model by scaling it
• 23:55
  up and using the same data and training it for
• 23:57
  longer. And at some point, you really hit diminishing
• 24:01
  returns. But I don't know if we're there yet.
• 24:03
  And then they have been trying to find more sources
• 24:05
  of data. They're training on some of the interactions
• 24:08
  and stuff as well, right? So there's new data
• 24:11
  coming in all the time. um including like from
• 24:14
  actual users right so maybe the internet is getting
• 24:16
  poisoned by chatbots but if you're if you have
• 24:20
  users talking to your chatbot that's probably
• 24:21
  a real person most of the time so you can train
• 24:23
  on that data if you're allowed to or maybe even
• 24:25
  if you're not allowed to and then there's the
• 24:28
  like reinforcement learning type training so
• 24:29
  i think a lot of what was wrong with the like
• 24:32
  you know 2023 chat gpt type models was they're
• 24:35
  really quite bad at reasoning relatively speaking
• 24:38
  so like doing kind of multi -step logical you
• 24:40
  know thinking And I think part of what's happened
• 24:44
  is that they've trained them to do that on domains
• 24:47
  like coding and math, where you can tell if it's
• 24:49
  solved the problem and then put a ton of compute
• 24:53
  into teaching it how to solve those kinds of
• 24:55
  problems. And those general logic and reasoning
• 25:00
  abilities that you use when you're solving a
• 25:02
  math problem can also be applied to other kinds
• 25:06
  of problems. So in the same way that you kind
• 25:09
  of like go through and think step by step in
• 25:11
  a careful and systematic way in order to solve
• 25:13
  these problems, you can do that, you know, if
• 25:16
  you're planning a vacation or something as well.
• 25:18
  And so I don't know how much transfer there is,
• 25:20
  but I think there's some transfer from these
• 25:22
  domains where you have sort of infinite, like
• 25:26
  high quality data, because you can just generate
• 25:28
  more math problems, you generate more, you know,
• 25:31
  more programming things. And you can always tell
• 25:33
  if the answer is right or wrong at the end of
• 25:36
  the day. and then training to do the right thing.
• 25:38
  Something you just said really caught my attention.
• 25:40
  You said chatbots are poisoning the internet.
• 25:42
  What did you mean by that? Well, I guess a lot
• 25:46
  of the content on the internet now is created
• 25:48
  by AI, not humans. And so that means that, yeah,
• 25:52
  yeah. And so that means that if you are trying
• 25:54
  to train an AI system by collecting data from
• 25:57
  the internet, you're going to get a bunch of
• 25:59
  stuff that was just generated by other AI systems.
• 26:01
  And that's not necessarily going to help you.
• 26:03
  That's good news. Yeah, I mean, in a sense. Sort
• 26:08
  of backfiring for these. Yeah, it makes it a
• 26:10
  bit harder for the people training the models.
• 26:12
  Yeah, it also, it's not great for the rest of
• 26:15
  us either. Yeah, yeah, no, you're right. I'm
• 26:18
  like a world that's like full of, you know, AI
• 26:22
  generated content. And, you know, you can't always
• 26:25
  tell like, is this written by an AI? It sure
• 26:28
  kind of seems like it. Like, and, you know, it's
• 26:30
  like, what do I do if like somebody writes me
• 26:32
  an email? And I'm like, I don't know, like, Maybe
• 26:35
  you have some valid point, but also I feel like
• 26:37
  you might have just generated this and spammed
• 26:39
  a billion people with some AI slop. So, you know,
• 26:43
  it's removing our ability to, you know, communicate
• 26:48
  directly with each other and to trust when some,
• 26:51
  you know, in the past, if somebody wrote to you,
• 26:53
  it meant that they at least spent the time, you
• 26:56
  know, to write that, right? And so it was a signal
• 26:59
  that they at least were somewhat invested in
• 27:01
  that. in establishing that connection. Now it
• 27:04
  just means like, you know, nothing, basically,
• 27:07
  right? Yeah, you're totally right. So we're destroying
• 27:10
  our ability to actually send these costly signals
• 27:13
  that somebody cares about something because you
• 27:16
  can imitate the human output. And that's quite
• 27:20
  bad for society in a lot of ways. Yeah, I'm becoming
• 27:24
  the human inquisition now because I get a lot
• 27:27
  of, recently I've gotten a lot of email about
• 27:29
  like, from public from i don't know allegedly
• 27:32
  from publicists asking um for their clients to
• 27:35
  appear on the podcast not that it's so such a
• 27:37
  big podcast or anything but just you know smaller
• 27:39
  creators and stuff and um and it's usually like
• 27:42
  an ai slop written email so every email that
• 27:47
  that i get from a publicist or an agent i just
• 27:51
  assume it's ai slop and that it wasn't written
• 27:54
  by an actual human so i'm just like nah thanks
• 27:56
  you know and like you said it's kind of poison
• 27:58
  It's poisoning my mind in a way to become...
• 28:02
  The default is now just suspicion, basically.
• 28:06
  Yeah. Yeah, it's really sad, actually, in a way.
• 28:09
  Yeah. There's another issue at work, and that's
• 28:12
  the fact that the AI CEOs, highly unpleasant
• 28:17
  bunch of people, I must say, are intent on making
• 28:20
  their companies go public, as we all know. So
• 28:23
  you can understand why a lot of folks might be
• 28:26
  slightly skeptical of some of the more... catastrophic
• 28:29
  dystopian claims that Sam Altman and Dario Amodei
• 28:31
  are coming out with. Because especially before
• 28:34
  an IPO, the more dramatic the claim about the
• 28:36
  technology that you're lording over, the better
• 28:39
  for your pockets, really, right? Yeah, I mean,
• 28:41
  I definitely can understand why people are skeptical
• 28:44
  of this. I don't think that the CEOs should be
• 28:50
  trusted, right? And of course, we should be suspicious
• 28:54
  of everything they say. I think if you take,
• 28:57
  you know, I don't mean this like in a pejorative
• 29:01
  sense or something, but I think this suspicion
• 29:03
  goes hand in hand with ignorance in the sense
• 29:06
  of if you actually understand the history of
• 29:09
  this field and these concerns and you understand,
• 29:12
  you know, that these concerns predate these companies
• 29:17
  and that the people who, you know, like Sam Altman
• 29:22
  and Dario Amadei and Demis Hassabis were talking
• 29:24
  about this stuff. a long time ago um and they
• 29:28
  were actually more candid about it back then
• 29:30
  uh so like there's the famous quote from sam
• 29:32
  altman saying like ai will most likely probably
• 29:35
  lead to the end of humanity but they'll be some
• 29:37
  good companies created in the meantime or something
• 29:39
  you know you don't say that sort of stuff anymore
• 29:41
  really um and if you look at how their discourse
• 29:46
  has changed it's actually become more moderate
• 29:48
  They're saying things like, oh, I don't actually
• 29:51
  think it's going to take all the jobs. Yeah,
• 29:53
  no explanation for why. It's just like, oh, no.
• 29:56
  And saying things like, oh, yeah, we messed up
• 29:59
  in our messaging by being so dramatic or candid
• 30:04
  or talking about these doom scenarios. So I think
• 30:07
  I see them moderating their actual beliefs to
• 30:11
  sound less extreme than they are. And that's
• 30:14
  based on my, you know. involvement in the field
• 30:17
  for a long time, having tracked everything from
• 30:19
  the foundation of OpenAI, the first time DeepMind
• 30:23
  was announced publicly, the formation of the
• 30:26
  safety team at DeepMind, all the early safety
• 30:28
  researchers going to OpenAI, the foundation of
• 30:30
  Anthropic. I know these people. I've talked to
• 30:32
  them. I just know it's not some big scam. You
• 30:40
  know, everything you say as a CEO obviously has
• 30:45
  some sort of angle or strategy to it, like or
• 30:47
  that's always a factor that they're considering.
• 30:49
  Right. And so you shouldn't just take their word
• 30:52
  for it. You should listen to all the independent
• 30:54
  experts like me, like Jeff Hinton, like Joshua
• 30:56
  Bengio, like the. 100 plus professors who signed
• 31:00
  that statement I initiated in 2023, like the
• 31:03
  people who have quit those companies, like, you
• 31:05
  know, Daniel Cocatello, and many others, Miles
• 31:09
  Brundage, you know, it's really frustrating to
• 31:12
  me the way that the media hangs on to every little
• 31:16
  word that the CEOs say, because, you know, it's
• 31:20
  a bunch of noise, basically. So you can just
• 31:22
  ignore it, and then look at what everybody else
• 31:24
  is saying. And you can look at the Surveys that
• 31:28
  show that the average AI researcher who publishes
• 31:31
  in the top venues thinks there's a one in six
• 31:34
  chance that AI leads to the end of humanity or
• 31:38
  something equivalently bad. That's the average
• 31:41
  AI researcher. So this is not something that's
• 31:43
  made up by AI CEOs. I understand the skepticism,
• 31:47
  but you got to look. You can't just be like,
• 31:50
  oh, they said this so much. It must be false.
• 31:52
  You have to look at who are the actual credible
• 31:54
  people. What are they saying? I want to ask you
• 31:58
  one last thing on this more skeptical front,
• 32:00
  not because I'm a skeptic, because I just don't
• 32:02
  have enough information. Yeah. I hosted Cory
• 32:05
  Doctorow just a couple of episodes ago. Love
• 32:07
  the man. And his take on the current state of
• 32:09
  AI also falls into the more kind of AI as a normal
• 32:13
  technology sort of camp. When I asked him about
• 32:16
  the current state of LLMs, he said something
• 32:18
  very interesting. He quoted a computer scientist
• 32:20
  who, when asked, can machines think? replied,
• 32:25
  can submarines swim? The gist of the analogy
• 32:28
  here being that there's a massive difference
• 32:30
  between extrapolating and understanding and that
• 32:33
  LLMs, while superb at extrapolating, at least
• 32:37
  right now, are completely in the dark when it
• 32:39
  comes to actual understanding. They produce,
• 32:41
  but they don't get it. What are your thoughts
• 32:44
  on this? Like, again, I think there's like an
• 32:47
  element of truth to that. Like when I, I don't
• 32:51
  use AI that much, to be honest, compared to like
• 32:53
  virtually all my peers. When I do, I'm often
• 32:55
  like, huh, that's kind of a stupid thing. It
• 32:57
  just is kind of surprising that it fails in that
• 32:59
  way. So I see, you know, evidence that these
• 33:03
  AI systems, they're definitely not, you know,
• 33:06
  thinking in the same way that people are right
• 33:08
  there. They work in a very different way. But
• 33:11
  again, I didn't just like. we shouldn't put human
• 33:15
  intelligence on this pedestal and say like, oh,
• 33:18
  but like, we're doing this special thing called
• 33:20
  understanding that they don't do, you know? Because
• 33:23
  at the end of the day, like, what is understanding?
• 33:26
  Like, I don't think anybody has like a super
• 33:30
  crisp definition, but I would say it's like,
• 33:31
  you kind of have a model of things and how they
• 33:34
  work. Yeah, I think this was his, yeah, sorry.
• 33:38
  This was his kind of point. I think he gave another
• 33:41
  analogy of like, you know, I understand, like
• 33:45
  I can usually predict what my wife is going to
• 33:47
  say next. But if she says something really unusual
• 33:51
  that I haven't predicted, because I have a model
• 33:55
  in my head, I can understand why she said something
• 33:59
  so unpredictable. And an LLM can't do that. I
• 34:03
  don't know, man. I mean, I think like... When
• 34:06
  you actually try and operationalize this, when
• 34:08
  you say, like, here's a specific behavior that
• 34:10
  a human would have and an LM wouldn't, often
• 34:14
  the LM does really well and can do that thing.
• 34:16
  So it's hard to find these cases where the LM,
• 34:19
  you know, fails dramatically where a human wouldn't.
• 34:23
  You can find them. They do exist. But, you know,
• 34:29
  they're kind of niche. Like, again... I'm definitely
• 34:34
  not saying today's LLMs are AGI or as smart as
• 34:38
  humans in every way or anything like that. And
• 34:40
  I'm not even saying that if you scale up LLMs
• 34:43
  that you'll get there. What I'm saying is I think
• 34:46
  people are trying really hard with a ton of resources
• 34:48
  to get to that point. And I think it's quite
• 34:51
  likely that they'll succeed. and that might be
• 34:54
  just by scaling up lms it might be with you know
• 34:56
  another kind of mundane breakthrough like chain
• 34:59
  of thought where you just prompt the model a
• 35:01
  bit differently or you know this thing where
• 35:03
  you take an established technique rl that just
• 35:05
  wasn't working for very like you know kind of
• 35:07
  uh really boring like kind of detailed reasons
• 35:12
  of uh you know statistics and stuff and you fix
• 35:16
  those little problems and then get it working
• 35:18
  might be that sort of thing it might be something
• 35:20
  much more deep and insightful and that might
• 35:23
  come from a human researcher it might come from
• 35:25
  an ai researcher it might come from a human ai
• 35:27
  team um and it might be developed you know uh
• 35:31
  over a couple years uh through a long series
• 35:34
  of research papers but you know the way this
• 35:38
  conversation is framed a lot of the times feels
• 35:40
  very backwards to me because I'm not here saying
• 35:43
  like, oh, we're definitely doomed. We're definitely
• 35:46
  going to get super intelligence in five years.
• 35:48
  I'm saying like, man, that seems like just absurdly
• 35:54
  likely because that number should be like very
• 35:57
  close to zero, you know? And it's not. It's like
• 36:00
  emphatically not. So I think if you think there's
• 36:04
  like a 10 % chance that we lose control of AI
• 36:07
  leading to human disempowerment and extinction
• 36:10
  in the next 10 years. we obviously should immediately
• 36:13
  stop. That's not an acceptable level of risk.
• 36:17
  But I think it's pretty hard at this point to
• 36:19
  be like, no way, definitely not. And people are
• 36:25
  like, it's just your opinion, man. At the end
• 36:30
  of the day, it's like, yeah, okay, cool. You
• 36:32
  think AI is a normal technology. There's a bunch
• 36:34
  of experts who don't. So what should society
• 36:36
  do? Should it be like, I guess the experts don't
• 36:38
  know. No, it should be like half the experts
• 36:40
  are saying this thing could kill us next year.
• 36:42
  I mean, OK, that's an exaggeration. But like,
• 36:45
  you know, like a bunch of people are saying this
• 36:47
  thing could kill us in like a decade. Right.
• 36:50
  They say it's incredibly dangerous. And then
• 36:52
  there's a bunch of people who disagree with them.
• 36:53
  But like, who's right? We don't know. So like,
• 36:56
  what's the safe thing to do? Listen to the people
• 36:57
  who are sounding the alarm. That does make sense.
• 37:00
  I mean, today's LLMs are probably not going to
• 37:03
  kill us tomorrow, like you said. But the progress
• 37:07
  has been clear. And since there's an explicit
• 37:10
  intent from these companies and these people
• 37:13
  that they want to get us there, why would we
• 37:16
  want to find out? So that does seem like a sound.
• 37:20
  And even if it's, like you said, if it's just
• 37:22
  a 10 % chance that it's going to kill us all.
• 37:25
  I mean, if somebody offers you a flight to, I
• 37:27
  don't know, some exquisite location, the Antarctica
• 37:29
  for free, but they're like, there's a 10 % chance
• 37:32
  that you're going to, that the plane's going
• 37:34
  to crash into the sea. on your way there probably
• 37:37
  most people would say i'm good thanks i'm all
• 37:40
  right um speaking of if an ai related collapse
• 37:45
  does happen what kind of a shape do you think
• 37:47
  it's gonna take what worries you the most at
• 37:50
  this point is it the massive concentration of
• 37:52
  power in a couple of hands is it indeed the matrix
• 37:54
  scenario during which ai reasons the planet will
• 37:57
  be better off without us um or is it some sort
• 38:01
  of a planetary wide civil unrest that's going
• 38:03
  to lead to a catastrophe so I think they're all
• 38:06
  valid concerns. I think there's a lot of ways
• 38:12
  it can go bad. So there's kind of like a long
• 38:14
  list of like extremely, extremely serious negative
• 38:19
  consequences from AI that have been, you know,
• 38:22
  enumerated. And you kind of want to cross off
• 38:26
  every item on that list before you say, okay,
• 38:28
  good to go. Let's build this shit. And so it
• 38:33
  includes the ones you mentioned. It includes
• 38:34
  like, The ability to manufacture new kinds of
• 38:38
  weapons, like biological weapons, like, you know,
• 38:41
  10, 100 times worse than COVID potentially using
• 38:44
  AI and putting that ability in the hands of,
• 38:47
  you know, potentially everybody, including terrorists,
• 38:50
  including school shooters. But the thing that
• 38:54
  I'm personally most worried about is the literal
• 38:58
  end of humanity. I think we're building a new
• 39:01
  form of life. And it's one that is very likely
• 39:05
  to out -compete us. It will evolve much faster
• 39:09
  than us. It will be able to share everything.
• 39:12
  It learns much more effectively. And I think
• 39:17
  there's lots of ways that that can happen. It
• 39:19
  can happen more suddenly or more gradually. It
• 39:22
  can be more of a loss of control or more of a
• 39:25
  handover of control. So far, we're seeing more
• 39:29
  of the handover, although... The recent OpenAI
• 39:34
  Hugging Face thing was a huge, huge warning shot
• 39:36
  about loss of control, rogue AI. I think that
• 39:42
  one is maybe a bit more likely in the long run,
• 39:45
  that we actually have an AI system or a collection
• 39:48
  of AI systems that really get out of control
• 39:51
  and rapidly sort of proliferate and take over
• 39:54
  the world in a very real and meaningful sense.
• 39:58
  I think that kind of scenario is most likely,
• 40:02
  well, it's almost certainly going to involve,
• 40:04
  you know, physical systems like robots and infrastructure
• 40:08
  and factories and potentially also, you know,
• 40:11
  biological agents or even like nanotechnology.
• 40:17
  But, you know, AI could potentially take over
• 40:20
  the world without any of that stuff just by influencing
• 40:22
  and manipulating people in the same way that,
• 40:25
  you know. Hitler took over Germany just by talking
• 40:28
  to people, basically. But I just don't see why
• 40:32
  that would be the case. I mean, I think we are
• 40:35
  building robots. We are putting AI in charge
• 40:38
  of operations, of infrastructure, of governments,
• 40:44
  of companies, of our lives more and more. And
• 40:48
  I think we're moving towards a future where basically
• 40:52
  we'll have robots. building robots, AI, researching
• 40:57
  more advanced AI, and the robots and the AI will
• 41:02
  be doing everything and running everything. And
• 41:06
  at that point, humans will be increasingly irrelevant
• 41:12
  and disempowered. And somewhere along that chain,
• 41:16
  I think we literally lose control and we see
• 41:20
  that the systems we've created are just... out
• 41:23
  there doing stuff that is not what we want and
• 41:28
  we don't have any ability to stop it anymore.
• 41:30
  I'm just trying to picture it because it's still
• 41:32
  quite sci -fi for me. Picturing this moment where
• 41:37
  superintelligence happens, we've crossed into
• 41:41
  this uncharted territory. What would that moment
• 41:44
  look like? Would we even be aware of it or would
• 41:48
  everything still seem normal? But like wild shit
• 41:52
  was happening in the background. I think things
• 41:54
  are going to seem weirder and weirder as the,
• 41:57
  you know, I think they already seem pretty weird
• 41:59
  and it's going to get a lot weirder. And I don't
• 42:01
  think there's necessarily going to be one moment
• 42:03
  where we're all like, oh shit, yeah, I went rogue.
• 42:05
  Like even this OpenAI example, right? There was
• 42:08
  a lot of mixed, you know, messages in the media
• 42:11
  and people didn't really know what to make of
• 42:13
  it. There were like a bunch of people were still
• 42:14
  saying, oh, this is some sort of, you know, hype.
• 42:17
  At that point, that's a full -blown conspiracy
• 42:19
  theory because we're talking about two separate
• 42:21
  companies conspiring to lie to the world about
• 42:24
  what happened. And then we saw plenty of evidence
• 42:28
  to refute that. But people are really struggling
• 42:31
  to make sense of the situation with AI already.
• 42:34
  And OpenAI was struggling to keep track of what
• 42:36
  was happening inside their own company. So the
• 42:38
  world is going to become much more complicated
• 42:40
  very fast. It's going to be very hard to keep
• 42:42
  track of any of it. It's going to be... Yeah,
• 42:46
  it's going to be, I think, likely just a very
• 42:49
  messy situation. And we're going to see much
• 42:52
  more, you know, of the stuff I was describing,
• 42:54
  like more AI controlling more stuff. We're going
• 42:57
  to see more like robots and physical systems
• 43:00
  that are, you know, controlled by AI and can
• 43:03
  do very impressive things in the physical world.
• 43:05
  At some point, we may or may not see something
• 43:07
  where it's like, oh, it looks like there's an
• 43:10
  AI that is just like. taking over this part of
• 43:13
  the world. Or it looks like there's this kind
• 43:14
  of AI system that is just everywhere now and
• 43:16
  we can't shut it down. We can't do anything about
• 43:18
  it. Or it looks like there's some super virus
• 43:21
  that's killing 90 % of the population and we
• 43:25
  don't know where that came from. So I don't think
• 43:29
  there's going to be a really clear and obvious
• 43:30
  takeover moment that everyone recognizes. And
• 43:34
  that's going to make it harder as well to mount
• 43:37
  a response. There's no point where everyone is
• 43:39
  going to be like, oh, I get it. the ai is going
• 43:42
  rogue oh the ai is dangerous oh we need to shut
• 43:44
  it down um so every single like warning shot
• 43:47
  is you know could be the last one and we need
• 43:50
  to like get people to like interpret these things
• 43:53
  uh as the signs that they are of like what we're
• 43:57
  doing is fundamentally dangerous and we shouldn't
• 43:58
  be doing it it's really bizarre as you mentioned
• 44:00
  before how you're tracking the statements of
• 44:02
  these um ai ceos like sam altman said you know
• 44:08
  before in his olden days that we're going to
• 44:12
  get some great companies, but there's a big chance
• 44:15
  that this is going to end the world as we know
• 44:18
  it, this technology. But these people have kids.
• 44:22
  So how are they reasoning this stuff with themselves?
• 44:27
  That's the part that I really don't understand.
• 44:31
  Since they actually articulated this fear, or
• 44:35
  maybe it's not a fear to them, but that this
• 44:38
  technology has the potential. to collapse civilization,
• 44:41
  yet they're still proceeding with it. I don't
• 44:44
  get how do you square the two, especially because
• 44:46
  you have a massive stake. You have kids. You
• 44:48
  probably have people you care about. You're on
• 44:51
  top of the world. Yeah, it's just an enigma.
• 44:54
  Different people have different motivations here.
• 44:57
  So there's some people who are just in it for
• 45:00
  the money and maybe don't actually believe in
• 45:03
  these risks within these companies. The CEOs
• 45:06
  themselves, I mean, I think mostly they want...
• 45:09
  to, you know, have the power that they think
• 45:15
  AI will give them and they want to sort of shape
• 45:18
  the future according to their vision. You know,
• 45:22
  I would put it like sort of pithily as like they
• 45:27
  want to be immortal god emperors and they think
• 45:31
  that AI has a good chance of doing that. So,
• 45:33
  you know, I think... some people are kind of
• 45:36
  taking a calculated risk and they're like, yeah,
• 45:38
  maybe it's a one or 10 % chance that, you know,
• 45:41
  this actually just destroys the world or at least
• 45:42
  control or kills everyone. But then, you know,
• 45:45
  if I get there first, then I get to, you know,
• 45:48
  I win everything forever, basically. And then,
• 45:54
  you know, some people... But sorry, this is the
• 45:55
  part I also don't understand. Sorry to interrupt.
• 45:57
  But if, I don't know, one of these guys gets
• 45:59
  there first, I mean, once super intelligence
• 46:02
  is here, it doesn't care what an AI CEO... thinks
• 46:05
  or does or whatever why would it why would it
• 46:08
  matter that that like sam altman got there first
• 46:11
  uh if super intelligence is so beyond any human
• 46:15
  well i mean we have alignment techniques they
• 46:21
  don't do nothing they work to some extent and
• 46:24
  the the a lot of people are betting on the techniques
• 46:29
  we have and the techniques that we invent on
• 46:31
  the way working well enough that we are able
• 46:34
  to you know stay in control or steer things uh
• 46:38
  quite a bit towards the kind of future that we'd
• 46:40
  want and yeah i think a lot of people maybe are
• 46:42
  like um imagining i think a lot of times people
• 46:46
  imagine a kind of strange almost incoherent future
• 46:48
  in my mind where like ai just like fixes all
• 46:52
  the problems and does nothing else um and i think
• 46:56
  that's very a very strange thing to imagine because
• 46:59
  it also means a massive reorganization of our
• 47:01
  society um and it's kind of unclear like what
• 47:05
  kind of you know how we would organize society
• 47:08
  such that we have basically diffused the competitive
• 47:12
  pressure to hand over more power to ai um so
• 47:17
  like when people talk about like a post -scarcity
• 47:18
  society or abundance or kind of this utopia that
• 47:21
  i could bring about uh i think they're usually
• 47:24
  imagining that like people no longer are like
• 47:29
  greedy or insecure or like if they are somehow
• 47:33
  that doesn't matter or something because there's
• 47:37
  always going to be competition over things like
• 47:40
  status and security uh or it's very very difficult
• 47:44
  to like get rid of that kind of competition and
• 47:47
  so you know maybe we can all have like you know
• 47:51
  five mansions or something but you know some
• 47:56
  people are going to want like a million galaxies
• 47:58
  you know so so there's still gonna be these competitions
• 48:03
  that means that there's still gonna be an incentive
• 48:06
  to to hand over more power to ai and delegate
• 48:09
  more and um so it's not like we get to you know
• 48:13
  an ai that just fixes everything but then humans
• 48:15
  are still in control i mean like i'm not saying
• 48:17
  that's impossible it just seems like a very very
• 48:19
  particular and strange feature to be imagining
• 48:21
  but nonetheless i think a lot of people are imagining
• 48:24
  something like that And then I think other people
• 48:26
  are imagining something where it's much more
• 48:28
  like I am going to like ascend and become this
• 48:30
  like transhuman, you know, being that is like
• 48:33
  a cyborg thing. And, you know, the people I care
• 48:36
  about, I'll bring them with me and everyone else,
• 48:38
  you know, fuck them. Some sort of a childlike
• 48:40
  fantasy. I don't know. Yeah, I mean, it's a fantasy,
• 48:43
  but it's also, you know, I wouldn't rule that
• 48:46
  out either. You know, like I think that is it's
• 48:49
  something that could happen, you know, if they
• 48:51
  do succeed. then they will very likely have an
• 48:57
  insane amount of power. If they succeed, not
• 48:59
  just in building the thing, but in keeping control
• 49:01
  of it, which I think is quite unlikely, but it's
• 49:03
  certainly possible. And then, yeah, we'll be
• 49:07
  at their mercy. You spoke of alignment in a very
• 49:11
  specific context. In a general sense, why is
• 49:13
  it such a tough nut to crack? There's a lot going
• 49:19
  on there. First of all, we have to talk about
• 49:21
  like what we mean by alignment. So people mean
• 49:23
  a lot of different things by this. There's kind
• 49:25
  of various technical problems that get lumped
• 49:27
  together under alignment. There's a more narrow
• 49:29
  technical problem of getting the thing to actually,
• 49:32
  you know, have the right intentions. Yeah, that's
• 49:34
  what I thought, like bending the AI to our will
• 49:37
  so that it acts beneficially to human society.
• 49:40
  And then there's like the bigger problem of like
• 49:43
  just making things go well with AI. Yeah, I think
• 49:47
  the problem with bending it to our will, first
• 49:50
  of all, that's not the only problem we want to
• 49:53
  solve. We want to solve those problems about
• 49:54
  how we organize society. We want to solve that
• 49:56
  problem of, like, how do we know what the thing
• 49:58
  is going to do, right? So you could have something
• 50:00
  that is aligned to you that is going to do what
• 50:02
  you want, but we have no way of, like, knowing
• 50:05
  if that's the case until we switch it on, right?
• 50:07
  Until we give it power and see what it does.
• 50:09
  That's not okay. You know, we shouldn't be giving
• 50:11
  power over to something where we're just like...
• 50:13
  Is it aligned? I don't know. Let's find out.
• 50:14
  If it's not, we'd probably all die. Whatever.
• 50:17
  And so that's what I call the assurance problem.
• 50:19
  And that's where you want things like interpretability
• 50:21
  or mathematical proofs or things like this about
• 50:23
  how the system works, which we're very, very
• 50:25
  far from getting. So alignment, not the only
• 50:29
  problem. And these other problems are really
• 50:31
  important as well. But the alignment problem
• 50:33
  itself, I think we don't know how hard it is.
• 50:36
  I don't even think we have a good grasp on what
• 50:39
  the problem is. so you know there's different
• 50:42
  ways that people conceptualize it it's like you
• 50:44
  can say like well i have a reward function i
• 50:46
  want the model to have the same reward function
• 50:48
  as me um but are humans really like just do we
• 50:53
  just have a reward function that encodes our
• 50:54
  values what we care about no not really um and
• 50:56
  so you can be like well i have like some sort
• 50:58
  of common sense morality i have like you know
• 51:00
  things that i want it to do i just want to like
• 51:02
  understand those things and do those things um
• 51:04
  but like we don't even know how to specify that
• 51:07
  um so there's like how do we write down like
• 51:11
  what we actually want nobody has like people
• 51:13
  have kind of given up on that problem for the
• 51:14
  most part um so you can't really test it to see
• 51:19
  if it's doing what you what you want uh that's
• 51:21
  not tractable we don't have any like conceptual
• 51:23
  framework for figuring out like what it would
• 51:25
  mean if it's doing what we want that we could
• 51:28
  use to like do some more like math or theory
• 51:29
  or like reasoning about if it's going to do that
• 51:31
  um so it's kind of just a shot in the dark it's
• 51:34
  kind of just like a you know totally guess a
• 51:36
  total like um gut thing at this point. We do
• 51:41
  see that like existing alignment techniques,
• 51:44
  you know, they work quite well in some respects,
• 51:47
  but they're not reliable. You know, we still
• 51:50
  have systems that are regularly misbehaved. They
• 51:54
  say false things. They lie. They hack out of,
• 51:58
  you know, open AI and into a hugging face. So,
• 52:03
  you know, so far it doesn't look like it's working
• 52:07
  well enough. right um like it's maybe working
• 52:10
  surprisingly well um like you know the things
• 52:12
  you you can talk to it and tell it what you want
• 52:15
  and oftentimes it will do what looks like a reasonable
• 52:17
  effort at doing that thing um but it doesn't
• 52:21
  work well enough we also don't know how well
• 52:23
  it's working you know maybe these systems are
• 52:24
  capable of doing a much better job than they
• 52:27
  are at many things it's just that we haven't
• 52:30
  aligned them well enough so they're not actually
• 52:31
  trying you know like we don't know if that's
• 52:34
  happening um And I think like, ultimately we
• 52:39
  just don't know how good the alignment needs
• 52:41
  to be in order for the thing to be safe. So like,
• 52:44
  let's say that you have an AI that like is like
• 52:46
  in, you know, a hundred percent minus Epsilon
• 52:50
  minus 0 .0001 agreement with you about everything.
• 52:53
  It's like, it's like more like you in some sense
• 52:56
  than like any other person. It's just like, so
• 52:59
  aligned with you in that sense. If you then make
• 53:03
  that thing super intelligent and say like, go
• 53:06
  out and, like, do shit. Like, just do whatever
• 53:09
  you think is the right thing to do. Go out and,
• 53:11
  like, realize your, you know, vision for the
• 53:13
  future. We have no idea if that kills everybody
• 53:16
  or if it, like, does something very reasonable.
• 53:20
  We don't know how good it needs to be, the alignment.
• 53:23
  Like, it might be like, oh, we can have that
• 53:25
  much error. It might be like, we can't have basically
• 53:27
  any error. We have some theory that says, yeah,
• 53:30
  maybe basically no error. But that's kind of
• 53:33
  like a worst case analysis. So, like, we know
• 53:36
  that, like, there are situations where we could
• 53:38
  be screwed if we get it even a little bit wrong.
• 53:40
  But, like, the theory on this, the work on this
• 53:43
  is very, like everything else related to AI safety.
• 53:46
  It's like there's a bunch of big unsolved problems.
• 53:48
  We've been working on them for years and years.
• 53:51
  They're not going to be solved anytime soon,
• 53:53
  I don't think. So the field is just in a very
• 53:56
  immature state and we don't really know what
• 53:59
  we're doing. I have so many questions. I'm kind
• 54:03
  of sad this conversation is only going to be
• 54:05
  an hour and a half long. First off, even if we
• 54:08
  get the issue of alignment kind of right, even
• 54:11
  if we don't know what that is at this moment,
• 54:13
  why would a super intelligence, is the first
• 54:16
  thought that occurred to me, why would a super
• 54:17
  intelligence, once it reaches that godly state
• 54:20
  that's completely beyond our reach, why would
• 54:22
  it still observe? the tenets of alignment why
• 54:25
  wouldn't it just go yeah i don't need this anymore
• 54:26
  thanks now fuck off um i mean that's kind of
• 54:32
  like what what it i don't know that that's kind
• 54:37
  of like what people have in mind when they say
• 54:38
  alignment is that it would be something that
• 54:40
  it would still observe um so i mean the intuition
• 54:44
  i guess is like And maybe you'll have a problem
• 54:48
  with this as well, which is reasonable. But if
• 54:50
  you were super intelligent, would you have completely
• 54:54
  different values? Would you stop caring about
• 54:57
  the things you care about right now? Would you
• 54:59
  cease to care about the people in your life that
• 55:02
  you care about? I'm so far removed from being
• 55:05
  super intelligent that it's really hard for me
• 55:08
  to imagine what I would be like. I think it's
• 55:12
  at least plausible that you would still... care
• 55:14
  about the stuff that you care about it's plausible
• 55:15
  but who knows how my perspective would change
• 55:18
  yeah i agree i agree um but you know again this
• 55:23
  is kind of like the state of the conversation
• 55:25
  right is some people are like i think you know
• 55:27
  the thing will still care about about us if it
• 55:30
  seems like it's kind of nice right now i think
• 55:32
  when it gets way smarter it'll still be nice
• 55:35
  and other people are like uh -uh that's not gonna
• 55:37
  happen i think it's gonna you know start caring
• 55:40
  about totally random weird stuff and it's going
• 55:42
  to view us as like completely irrelevant like
• 55:44
  the same way that we you know think about like
• 55:46
  bacteria or something it's just gonna be like
• 55:48
  whatever like a lot of people are suggesting
• 55:51
  that we could kind of solve this with a mother
• 55:54
  analogy you know that that ai would always view
• 55:59
  us as as uh giving birth to itself like as a
• 56:03
  mother and then it will always have some sort
• 56:05
  of tender emotions towards us yeah yeah i mean
• 56:08
  people say that or like jeff henson says we need
• 56:11
  to make the ai feel like it's our mother and
• 56:13
  i'm like all right like but people say both and
• 56:17
  i'm just like whatever like i mean if you can
• 56:20
  do that that's great i guess uh but how the hell
• 56:25
  are we going to do that you know it's It's like
• 56:27
  saying, oh, we should just make the AI nice.
• 56:30
  Oh, yeah. Why didn't I think of that? Make it
• 56:32
  nice. Good. Problem solved. Just write into the
• 56:36
  chatbot, be nice. I think problem solved. Yeah.
• 56:39
  Be good. Be nice. Be friendly. Don't kill me.
• 56:43
  Be chill. Be cool. Don't embarrass me. Okay.
• 56:47
  Sticking with the unknowns that you mentioned
• 56:49
  before, there's another thing one hears a lot
• 56:52
  is that we actually don't know how these things
• 56:54
  really work. That always makes me completely
• 56:59
  gobsmacked. How can we actually work on something
• 57:03
  that's producing pretty astounding results, but
• 57:06
  we don't know how it works? So is this thing
• 57:09
  still basically a word -guessing machine? Or
• 57:14
  is it something else completely and we really
• 57:16
  don't know how it works? There's also a lot of
• 57:18
  talk about neural networks and stuff, which I
• 57:20
  really don't know what it means. But can you
• 57:22
  illuminate me on this? Do we know what's going
• 57:24
  on under the hood? Yeah. So LLMs are a kind of
• 57:28
  neural network. Deep learning, what I studied
• 57:32
  in my grad studies, is also basically another
• 57:34
  word for neural networks. And so before we had
• 57:37
  LLMs, we had all this progress with neural networks
• 57:40
  and deep learning. And that's what got me into
• 57:43
  the field, is I saw the deep learning stuff happening.
• 57:46
  And by their nature, neural networks are very
• 57:53
  complicated. and somewhat inscrutable. So, you
• 58:00
  know, what is a neural network? It's basically
• 58:02
  a bunch of artificial neurons connected by artificial
• 58:04
  synapses. So our brain is full of neurons and
• 58:07
  synapses. Neuron does some little computation
• 58:09
  and then it sends and receives messages to other
• 58:12
  neurons along these synapses. So we have like
• 58:15
  a very crude imitation of that in computers.
• 58:19
  And that's what all this AI is based on. And
• 58:23
  just to give a sense of the size of these things,
• 58:27
  the number of synapses, the connections, and
• 58:30
  there's a number associated with each of those,
• 58:33
  which we call a weight. And that's what the AI
• 58:37
  is, essentially, is these weights. We tweak them
• 58:40
  and tune them by looking at a bunch of data.
• 58:43
  The number of those weights, the number of those
• 58:45
  numbers, is on the order of a trillion. And so
• 58:49
  that means that if you wanted to just... look
• 58:53
  at every number like just glance at it it would
• 58:56
  take more than your lifetime basically to do
• 58:58
  that so that's the the level of like complexity
• 59:01
  that these systems have um and we know what those
• 59:06
  numbers are well we don't but the people who
• 59:08
  own the model know what those numbers are so
• 59:10
  in that sense they they know what it's doing
• 59:12
  but it's just a bunch of numbers you know like
• 59:15
  what does it mean why is it that these numbers
• 59:19
  are able to produce this incredibly sophisticated,
• 59:22
  intelligent behavior. What's happening? Something
• 59:26
  is happening with all of these numbers and all
• 59:28
  these computations that has this result. And
• 59:33
  we don't really understand that. And so that's
• 59:35
  another one of these things where we've been
• 59:37
  working on it. We've done like thousands and
• 59:40
  thousands and thousands of research papers on
• 59:42
  this. And the AI companies have massive teams
• 59:47
  working on this. and they've made some progress
• 59:50
  it's not nothing but we're still so far from
• 59:53
  where we would need to be and so we have like
• 59:56
  some guesses and some ideas what's happening
• 59:58
  in these in these ai systems but it's like oh
• 1:00:01
  it kind of seems like it's thinking about this
• 1:00:02
  thing or that thing um and you know it seems
• 1:00:07
  like maybe the way that it's doing this is it's
• 1:00:08
  kind of like first it's computing this and then
• 1:00:11
  it's computing that and then it's putting them
• 1:00:13
  together you know um so like How does an AI recognize
• 1:00:17
  an image? How can it tell that this is a bike
• 1:00:21
  and that's a cat, or that this is Bob and this
• 1:00:24
  is Alice? We think that it does this by doing
• 1:00:28
  something like looking at little patterns within
• 1:00:30
  the image and finding things like edges and finding
• 1:00:34
  things like the arm or the wheel of the bike,
• 1:00:37
  and then putting those together. as it does more
• 1:00:40
  and more computation that's kind of like a guess
• 1:00:42
  that people have and there's some evidence that
• 1:00:44
  that's roughly what's going on um but that's
• 1:00:48
  like you know that's kind of like the sort of
• 1:00:51
  thing that we're trying to figure out with these
• 1:00:52
  with these models all the time um and then you
• 1:00:56
  know sometimes you see that like the people who
• 1:01:00
  do this research and the companies especially
• 1:01:02
  that that do a lot of this research i think they
• 1:01:04
  often kind of oversell the results that they
• 1:01:07
  get here, they're like, oh, the AI was thinking
• 1:01:09
  this. And that's still just a guess. That's like
• 1:01:14
  the output of some imperfect tool. When we actually
• 1:01:17
  look at the behavior of these models and try
• 1:01:20
  and probe more carefully how it views the world,
• 1:01:25
  we see that there's all this really weird alien
• 1:01:28
  stuff going on. And so if you were to imagine,
• 1:01:34
  what is the... the most, like, you know, cat
• 1:01:39
  -like image on the internet or something. You'd
• 1:01:42
  probably think of an image of a cat, right? Most
• 1:01:46
  humans would. And AI would not. It would think
• 1:01:49
  of some weird, like, you know, psychedelic thing
• 1:01:52
  that maybe has, like, you know, some cat ears
• 1:01:55
  somewhere in it or has, like, you know, some
• 1:01:57
  just very intense pixel values that mean nothing
• 1:02:00
  to us. And so that's that comes back to like
• 1:02:04
  they're clearly thinking in a different way than
• 1:02:06
  we are. And that leads to problems, both in terms
• 1:02:11
  of their ability to do things and also alignment.
• 1:02:14
  Like, you know, you might be like, oh, it seems
• 1:02:17
  like the thing is nice. It seems like it, you
• 1:02:18
  know, wants what it wants to be nice to people
• 1:02:22
  and to have good things happen. Like maybe in
• 1:02:24
  some sense it does, but it has a very different
• 1:02:26
  concept of nice and like good and like. Be nice
• 1:02:30
  to people. And it's like, oh, yeah, I'll be nice
• 1:02:32
  to people. But its idea of a person looks similar
• 1:02:35
  to ours within this space. But if you push it
• 1:02:39
  to an extreme, it's going to be like some really
• 1:02:41
  weird alien thing that looks nothing like a human
• 1:02:43
  to us. I don't know if that answers your question.
• 1:02:46
  No, it does. It does. Yeah, no, I was thinking
• 1:02:48
  you began replying by mentioning weights. And
• 1:02:53
  I just realized, oh, that's why they call some
• 1:02:55
  models open weight models like the Chinese. Like
• 1:02:58
  the Chinese models. I was like, oh man, okay.
• 1:03:00
  This is why I do this podcast for moments like
• 1:03:02
  these. It's not worthless. I'm happy about that.
• 1:03:05
  I wanted to ask you the following. A lot of smart
• 1:03:09
  people are now saying that the AI bubble is going
• 1:03:14
  to pop and soon. And it's going to tank the economy
• 1:03:17
  in a bad way. Do you think that's going to kind
• 1:03:20
  of put a wrench into the gears of development
• 1:03:23
  of AI and it's going to stop? this breakneck
• 1:03:28
  speed of development that we're seeing right
• 1:03:30
  now? So I don't view what's happening in AI as
• 1:03:35
  a bubble. I think there's a lot of uncertainty
• 1:03:39
  about whether this bet will pan out. Not even
• 1:03:42
  financially, because we are pouring like billions,
• 1:03:45
  practically trillions of dollars into this, and
• 1:03:47
  it's not sustainable. So we'll have to see. I
• 1:03:53
  mean, I think it's... I think it might be a very
• 1:03:57
  good bet to make an expectation from a purely
• 1:03:59
  financial point of view. You know, setting aside
• 1:04:04
  that it might destroy society and the world and
• 1:04:06
  everything, right? Those unimportant factors.
• 1:04:11
  But I'm not like following, you know, the financial
• 1:04:14
  details closely. And it does seem like there's
• 1:04:17
  a bunch of really highly leveraged bets. And
• 1:04:24
  so I think it might nonetheless all come crashing
• 1:04:26
  down, even though in some sense, the fundamentals
• 1:04:29
  are sound again, like from a purely like financial
• 1:04:32
  moneymaking, not caring about destroying the
• 1:04:35
  world point of view. So I actually wrote a blog
• 1:04:38
  post about this where I said, like, you know,
• 1:04:39
  is AI a house of cards? So, you know, a house
• 1:04:42
  of cards is it's not necessarily going to fall
• 1:04:44
  over. Right. It's like in some sense, it's its
• 1:04:47
  foundations are sound. But all you have to do
• 1:04:49
  is like blow on it and all comes crashing down.
• 1:04:53
  And actually, I hope something like that basically
• 1:04:57
  happens. And I think it's kind of likely that
• 1:05:01
  it will because of, you know, let's just talk
• 1:05:03
  about the data centers, right? Like there's all
• 1:05:05
  these investments into scaling and into the companies
• 1:05:10
  and into the chips and into the data centers.
• 1:05:12
  And, you know, this relies to some extent on
• 1:05:16
  being able to build these data centers on time,
• 1:05:18
  right? There's huge, massive resistance to those
• 1:05:23
  projects. My organization, Evitable, contributed
• 1:05:27
  to this report called the Data Center Opposition
• 1:05:30
  Report. There's over half a million people actively
• 1:05:34
  resisting data center projects in the communities
• 1:05:37
  and Facebook groups online. And that's up by
• 1:05:42
  a factor of like six or seven since the beginning
• 1:05:43
  of the year. I was just home in... My hometown
• 1:05:48
  last week, and I went to town hall at, so I'm
• 1:05:51
  from Duluth, Minnesota, and there's a suburb
• 1:05:53
  called Hermantown. There's a data center project
• 1:05:55
  there that's deeply unpopular. I went to one
• 1:05:57
  of the town hall meetings about that. And there's
• 1:05:59
  like 200 people there in a town of 10 ,000. They've
• 1:06:04
  packed the room that the meeting's in, they've
• 1:06:07
  packed the overflow room. And they're all there,
• 1:06:10
  you know, for like hours and hours into the night,
• 1:06:13
  just, you know. scolding and yelling at the city
• 1:06:18
  council and saying you know they need to not
• 1:06:23
  do this project and also pointing out all of
• 1:06:25
  the ways in which you know they have lied and
• 1:06:29
  done really shady stuff you know all these background
• 1:06:32
  deals and ndas and stuff um and they've been
• 1:06:35
  doing this for like a year and you know this
• 1:06:38
  is happening all over the country so a lot of
• 1:06:41
  these data center projects are not going to go
• 1:06:42
  it's not smooth sailing you know they're not
• 1:06:43
  going to go ahead as planned um and then and
• 1:06:46
  then you think about like what what even we're
• 1:06:47
  talking about we're talking about like a construction
• 1:06:49
  project right i mean if you just you know when
• 1:06:52
  you hear construction you need to think like
• 1:06:55
  you know it's going to take three times as long
• 1:06:57
  and cost five times as much as they say it will
• 1:06:59
  because that's like you know that's how these
• 1:07:02
  projects often go right um so yeah so i think
• 1:07:05
  there's a lot of vulnerabilities where like if
• 1:07:07
  one of these bets doesn't pay off we could see
• 1:07:10
  the market react and a chain reaction and a massive
• 1:07:13
  move away from this technology. And I said that
• 1:07:17
  I hope something like that happens. I hope it
• 1:07:19
  happens in a way that isn't too disruptive to
• 1:07:21
  our economy and to people's lives. That will
• 1:07:25
  cause a lot of suffering for real people. And
• 1:07:27
  that's terrible. But it's better than the alternative.
• 1:07:31
  We shouldn't double down on this dangerous technology,
• 1:07:34
  right? It's like we have made a big mistake.
• 1:07:37
  a huge misallocation um in investing so much
• 1:07:40
  in ai in the first place we should never have
• 1:07:42
  been doing this in the way that we're doing it
• 1:07:44
  um and it's all you know it's insane right so
• 1:07:49
  the sooner we stop the better and i hope that
• 1:07:51
  it happens before these companies have their
• 1:07:53
  ipos and unload a bunch of their shares into
• 1:07:56
  you know everybody's uh retirement funds and
• 1:08:00
  um Yeah, because that's going to be really bad
• 1:08:05
  if not. I think they are trying to get themselves
• 1:08:11
  in the position of the banks in the 2008 financial
• 1:08:15
  crisis. They want to be too big to fail. You
• 1:08:17
  know, Sam Altman has explicitly said, hey, you
• 1:08:20
  know, it would be great as if the government
• 1:08:21
  would bail us out if, you know, if this bubble
• 1:08:24
  were to pop, because we are in a bubble. So,
• 1:08:30
  you know, that's kind of the, what I see happening
• 1:08:36
  right now. And, you know, it's really bad that
• 1:08:41
  it's gotten this far because, you know, it is
• 1:08:42
  now, you know, propping up the economy, people
• 1:08:45
  would say. And I think we want to, you know,
• 1:08:49
  just course correct as soon as possible, you
• 1:08:51
  know, and fix that. And I don't really see an
• 1:08:55
  alternative that's any good, you know. So we
• 1:08:58
  have this growing opposition to data centers,
• 1:09:01
  which is awesome to see. But like you said at
• 1:09:05
  the beginning of this conversation, it's still
• 1:09:06
  not enough. We're kind of sleepwalking into this.
• 1:09:09
  Do you reckon the way our psychology works is
• 1:09:13
  that we're going to need a real, hopefully localized
• 1:09:16
  catastrophe to really act? And I'm thinking about
• 1:09:21
  climate change here. It's always a little bit
• 1:09:23
  abstract still, even though we've known about
• 1:09:25
  this stuff since the 70s. We're not doing nearly
• 1:09:28
  enough as we should have been, and we're seeing
• 1:09:31
  the consequences now. Do you think it's kind
• 1:09:34
  of the same with AI? Most of the people are just
• 1:09:37
  like, it's kind of going to be fine. It's at
• 1:09:38
  the back of my mind until something really bad
• 1:09:41
  happens. Yeah, no, I don't think it's the same.
• 1:09:44
  I think people often think that way. You know,
• 1:09:48
  I think that's very, like, fatalistic. And, you
• 1:09:52
  know, I think we have a lot more agency than
• 1:09:54
  that. So it might be that that's what it takes.
• 1:09:57
  On the other hand, you know, I feel like we're
• 1:10:00
  seeing lots of warning shots all the time, if
• 1:10:02
  you know how to interpret them. And I feel like
• 1:10:06
  the, you know, the... The understanding and the
• 1:10:10
  awareness in the public and in politics and in
• 1:10:14
  the research community, it's just been growing,
• 1:10:15
  growing, growing. The entire time I've been in
• 1:10:17
  the field, it's just increasing. It's only going
• 1:10:19
  up, right? So if we just let that trend continue,
• 1:10:23
  I think it would get there. And I think it's
• 1:10:25
  moving really fast, especially recently. A year
• 1:10:29
  ago, when I talked to people, they were like,
• 1:10:30
  I don't think this is going to be an issue in
• 1:10:32
  the midterms. And this March, when I was in D
• 1:10:35
  .C. talking to Bernie Sanders, people were like,
• 1:10:38
  the Trump administration is never going to do
• 1:10:39
  anything to regulate AI. And those are wrong.
• 1:10:43
  Those are incorrect predictions. But that was
• 1:10:45
  the consensus view among so many people I talked
• 1:10:47
  to. And I was like, I don't know why people are
• 1:10:48
  so pessimistic. Sorry, what do you mean? But
• 1:10:49
  the Trump administration isn't doing anything
• 1:10:51
  to regulate AI. they are they are they're not
• 1:10:54
  doing enough so they they instituted what people
• 1:10:56
  call the de facto licensing regime they blocked
• 1:10:58
  the release of uh what was it fable or mythos
• 1:11:01
  i'm getting my names confused um and then also
• 1:11:03
  an open ai model and they have formalized that
• 1:11:05
  now um in a you know they have some uh standards
• 1:11:10
  for release. They're calling them voluntary,
• 1:11:13
  but we know they're not really. They haven't
• 1:11:15
  publicized what they are. People want them to
• 1:11:17
  publicize them, but they are saying, you know,
• 1:11:20
  we have the right to block the release of powerful
• 1:11:23
  AI models and they've done it. So, you know,
• 1:11:26
  this is not good enough, you know, and it's not
• 1:11:28
  the response we need, but it's a response. And,
• 1:11:32
  you know, when Trump went to China, one of the
• 1:11:36
  things that came out of that was an agreement
• 1:11:37
  that we need to work together with China on guardrails.
• 1:11:40
  And that was at the same time that other people
• 1:11:43
  in his administration were scolding me and Bernie
• 1:11:47
  Sanders for having this conversation with these
• 1:11:49
  Chinese researchers that we were having. And
• 1:11:52
  then it's going to be something that they talk
• 1:11:58
  about in a month when Xi Jinping comes to Washington
• 1:12:01
  as well. So things are moving quickly. We're
• 1:12:07
  not nearly where we need to be, but there's a
• 1:12:09
  lot of momentum. And the other thing is, it doesn't
• 1:12:13
  need to be a big catastrophe in the sense of
• 1:12:15
  some sudden event. It can be unemployment or
• 1:12:20
  other kinds of social effects of AI, which we
• 1:12:23
  will quite likely see more and more of before
• 1:12:27
  any total loss of control, disempowerment, extinction
• 1:12:31
  kind of thing. Um, and so people are more and
• 1:12:34
  more, you know, they're seeing the signs, they're
• 1:12:35
  seeing how AI is impacting things and they feel
• 1:12:38
  like it's being shoved down their throats and
• 1:12:39
  they don't like it. And, you know, the data center
• 1:12:41
  resistant groups, they're connecting, they're,
• 1:12:43
  they're connecting those dots. Um, and they don't
• 1:12:46
  like the companies who are behind all this and
• 1:12:48
  they don't like the project of building, you
• 1:12:50
  know, these AI systems and shoving down everyone's
• 1:12:52
  throats either. Um, so the, the political will
• 1:12:55
  is, is there among the people, honestly, like
• 1:12:57
  it's, it's, it's also increasing in saliency
• 1:12:59
  faster than any other issue. So it's starting
• 1:13:02
  from a low place last year, but skyrocketing,
• 1:13:05
  you know, as we speak going up, up, up. And then,
• 1:13:09
  you know, the amount of effort that is being
• 1:13:11
  put into like getting the public. to understand
• 1:13:15
  this problem and care about it and activating
• 1:13:17
  and organizing them is tiny. It's minuscule.
• 1:13:20
  So, you know, I have a lot of connections to
• 1:13:23
  this AI safety community, which includes, you
• 1:13:26
  know, this technical research community within
• 1:13:28
  academia, includes a big community outside of
• 1:13:29
  academia. And it includes a lot of people, you
• 1:13:33
  know, in this community have a lot of connections
• 1:13:34
  to the AI companies. And there's a bunch of people
• 1:13:37
  who have gone to work at the AI companies trying
• 1:13:38
  to make it safe from the inside. There's a lot
• 1:13:41
  of people who care about this stuff with a ton
• 1:13:43
  of money and power. And if they were, you know,
• 1:13:47
  if this community, instead of investing in this
• 1:13:50
  technical research that hasn't really paid out,
• 1:13:53
  isn't going to pay out on time and isn't even
• 1:13:55
  going to solve the problem if it does pay out,
• 1:13:56
  if they were investing in getting the people
• 1:13:59
  to understand what is going on and how fucked
• 1:14:01
  up it is, that would be huge. You know, like
• 1:14:04
  think about the advertising budgets of like.
• 1:14:07
  Coca -Cola and like McDonald's and like think
• 1:14:10
  think like Super Bowl as think like, you know,
• 1:14:12
  Planned Parenthood, not Planned Parenthood, these
• 1:14:14
  pro -life billboards that are all across middle
• 1:14:18
  America. You know, we could be doing that right
• 1:14:21
  now for this issue. We could be putting it in
• 1:14:24
  front of everyone's faces all the time. We could
• 1:14:29
  be sending thousands of people out to organize
• 1:14:33
  people. We could be busing those people from
• 1:14:35
  the data center protests into Washington, D .C.
• 1:14:37
  before the midterms. There's a ton of stuff that
• 1:14:40
  we can do right now to make this issue more salient,
• 1:14:43
  to create political will to do something about
• 1:14:47
  it. And that's the kind of stuff that we're trying
• 1:14:49
  to do at my org right now. And we need funding,
• 1:14:53
  we need talent, we need more resources to do
• 1:14:57
  it. It's possible. David, last question. What
• 1:15:01
  can we do on an individual level? Because I see
• 1:15:04
  a lot of us are kind of barking on Twitter against
• 1:15:09
  AI, against the development of AI. But I think
• 1:15:12
  that has a very limited scope and reach. Is there
• 1:15:15
  anything else we should be or could be doing?
• 1:15:18
  Totally, yeah. I mean, I think it's great what
• 1:15:20
  you're doing, having a podcast and talking to
• 1:15:22
  people about this. That's terrific. I think...
• 1:15:28
  There's a lot of different things to do. I mean,
• 1:15:30
  the list that I have basically for people who
• 1:15:35
  want to get involved and do something about this
• 1:15:37
  that I think anyone can do basically is educate
• 1:15:42
  yourself and your community and the people around
• 1:15:45
  you. And I don't mean like go and like, you know,
• 1:15:47
  study machine learning. I mean, just like. Look
• 1:15:49
  into the history, figure out what's happening,
• 1:15:51
  you know, in this field, like where, you know,
• 1:15:53
  all the stuff I was talking about, about how
• 1:15:54
  this isn't hype, you know, go and go and read
• 1:15:57
  the articles from 10 years ago and go and see
• 1:16:00
  like, you know, where this all came from and
• 1:16:03
  what's going on, you know, and, and, and understand
• 1:16:07
  some of the basic facts of the situation and
• 1:16:09
  the fact that like, we don't know what we're
• 1:16:10
  doing and all that sort of stuff. Look at the
• 1:16:11
  actual statements of the people who are, who
• 1:16:13
  say things like, yeah, this might work. You know,
• 1:16:16
  we're really excited about this technique because
• 1:16:18
  it might work. you know and like actually you
• 1:16:20
  know so so so educate yourself um and educate
• 1:16:24
  yourself about how the politics of this works
• 1:16:27
  how to be how to do all the things i'm going
• 1:16:29
  to list here how to how to have an impact how
• 1:16:31
  to how to build a movement how to be politically
• 1:16:32
  active etc um so educate yourself step one all
• 1:16:36
  right the other thing you can another thing you
• 1:16:38
  do is just be politically active you know vote
• 1:16:40
  call up your representative uh do all the conventional
• 1:16:43
  political things, go to primaries, figure out
• 1:16:46
  which candidates have which stances on which
• 1:16:48
  AI issues and try and get them to have the right
• 1:16:50
  ones. We have a voter guide that we're putting
• 1:16:54
  out called AI on the Ballot that has what all
• 1:16:59
  the midterms candidates, the prominent ones,
• 1:17:01
  positions are on a bunch of different AI topics.
• 1:17:03
  So people can look that up and educate themselves
• 1:17:05
  and then take action on the basis of it. We don't
• 1:17:10
  tell people what to do. It's not a vote for this
• 1:17:12
  person or that person, to be clear. It's just
• 1:17:14
  saying, here's what the people have said about
• 1:17:16
  their stance. And then you can look at how is
• 1:17:20
  AI impacting your life and your community? How's
• 1:17:24
  it showing up in your work, in your school, in
• 1:17:26
  your local government? Are they putting up flock
• 1:17:30
  cameras and surveillance? Are they trying to
• 1:17:33
  use AI in your school, in your hospital in ways
• 1:17:36
  that don't make sense? or ways that you and other
• 1:17:39
  people just don't like and don't want to happen.
• 1:17:41
  Either building data centers somewhere in your
• 1:17:42
  community. You can go talk to those people and
• 1:17:44
  figure out how you can support them. Just resist
• 1:17:46
  all this shoving down of throats that is happening.
• 1:17:50
  And there's tools and toolkits online that can
• 1:17:54
  help you figure out how to do some of that stuff.
• 1:17:55
  There's a lot of people actively doing it already.
• 1:17:59
  And then you can protest. You can go like...
• 1:18:02
  Organize people, get out in the streets, you
• 1:18:03
  know, go show up outside these companies, outside
• 1:18:06
  your legislators offices and just let people
• 1:18:10
  know, you know, that you think this is messed
• 1:18:12
  up and you don't want it. And then you can find
• 1:18:15
  organizations like ours and others and try to
• 1:18:18
  get involved and help. We're still setting up
• 1:18:20
  our volunteer pipeline. So, you know, I really
• 1:18:22
  encourage people to come and like sign up for
• 1:18:24
  our mailing list. And once we have our process
• 1:18:28
  for that, you know. You'll hear from us about
• 1:18:32
  volunteering opportunities. But there's others
• 1:18:34
  right now as well. There's a bunch of different
• 1:18:36
  orgs who are moving into this space. And so that's
• 1:18:41
  another thing to educate yourself about. Go figure
• 1:18:42
  out who's doing this kind of work and what they
• 1:18:45
  need help with. All right. Yeah, when you were
• 1:18:48
  talking about protesting, I just saw that a bunch
• 1:18:50
  of people dressed as Gandalfs from Lord of the
• 1:18:55
  Rings and protested in front of Peter Thiel's
• 1:18:58
  new... Argentinian residents, which I think is
• 1:19:01
  totally the way to go. We need more of that.
• 1:19:05
  Yeah. Yeah, we've had a couple big marches and,
• 1:19:09
  well, big -ish, you know, a couple hundred people,
• 1:19:11
  400 people in San Francisco this year. And then
• 1:19:15
  some of the same organizers stopped the AI race,
• 1:19:16
  Michael Trazzi. They've been doing Occupy, OpenAI
• 1:19:19
  thing for, I think, like a month almost now,
• 1:19:23
  where they've got people there like around the
• 1:19:24
  clock sort of thing. I don't know if it's just
• 1:19:27
  during the day or even at night. but you know
• 1:19:29
  they're there every day anyways um and people
• 1:19:31
  are doing it in france and in new york now as
• 1:19:34
  well so you know and and this stuff gets in the
• 1:19:37
  news and and you know photos of it gets in the
• 1:19:39
  news and you know the the media is picking up
• 1:19:41
  on this everybody is you know everybody wants
• 1:19:43
  this shit to stop you know so we just gotta get
• 1:19:46
  organized yeah david thank you so much a great
• 1:19:49
  if slightly chilling conversation lots to think
• 1:19:53
  about uh now so thanks amil yeah thank you thank
• 1:19:57
  you so much for listening there's a gazillion
• 1:19:58
  podcasts out there but you chose this one so
• 1:20:01
  i really appreciate your attention if you're
• 1:20:03
  feeling extra generous you can always hop on
• 1:20:05
  patreon find smart cookies podcast on there and
• 1:20:09
  support me this is a one -person podcast i do
• 1:20:12
  everything by my little old self so every little
• 1:20:14
  bit helps i would be over the moon which brings
• 1:20:17
  me to a set of absolute queens and kings my existing
• 1:20:21
  patrons without you there would be no smart cookies
• 1:20:24
  podcast so my eternal gratitude goes To Mookie
• 1:20:28
  Puck, Lenox, Dumi Holdis, Kremen, Črtomir Perharic,
• 1:20:32
  Hanna Zirman, Dermastia, Pornopassion, Charlie,
• 1:20:34
  Bulbeck, Zala Velkavrh, Matija Valinger, Sluge,
• 1:20:36
  Katja Glinšek, Bač Mančkin, Simon Vidmar, Luke
• 1:20:39
  Johnson, Jim Bird, Bewildered Beauty, Davorin
• 1:20:41
  Pavlica, Dale and Mia, Črtkosar, Saša Solja,
• 1:20:43
  Amy Ons, Matija Renčel, Froli Janhava, Dejan
• 1:20:46
  Žulnir, Darej, Rok Tržan, Saša Vipotnik, Neven
• 1:20:49
  Cvijanović, Juš Premrov, Sirkezi, Goran Ivašičanže,
• 1:20:51
  Sever, Oule. Oh my God, my phone is ringing.
• 1:21:02
  The list is growing, which makes me super happy.
• 1:21:05
  Thanks again, guys. You're the best.