Latest / Razib Khan's Unsupervised Learning / Theo Jaffee: monitoring the “singularity”
Transcript
- 0:00SPEAKER_00: This podcast is brought to you by the Albany Public Library Main Branch and
- 0:03SPEAKER_00: the generosity of listeners like you.
- 0:09SPEAKER_00: What is a podcast?
- 0:16SPEAKER_00: God, Daddy, these people talk as much as you do.
- 0:26SPEAKER_00: Razib Khan's Unsupervised Learning.
- 0:29SPEAKER_02: Hey, everybody.
- 0:29SPEAKER_02: This is Razib with the Unsupervised Learning Podcast,
- 0:31SPEAKER_02: and I'm here with my old friend, Theo Jaffe.
- 0:34SPEAKER_02: Theo and I have an interesting history.
- 0:36SPEAKER_02: I was actually an early guest on one of his early podcasts back when he was
- 0:38SPEAKER_02: an undergrad back in Florida.
- 0:41SPEAKER_02: Now he has relocated to San Francisco,
- 0:43SPEAKER_02: like many young people interested in technology,
- 0:46SPEAKER_02: the frontiers of technology, artificial intelligence.
- 0:48SPEAKER_02: He is now co-hosting a video podcast show.
- 0:54SPEAKER_02: They've got streams.
- 0:55SPEAKER_02: They're doing all sorts of things, but it's kind of like TBPM a little bit, that sort of thing.
- 1:01SPEAKER_02: MTS Live, which is monitoring the situation or monitoring the singularity.
- 1:05SPEAKER_02: I've seen both of those.
- 1:05SPEAKER_02: I actually saw the second,
- 1:07SPEAKER_02: and then I've seen you saying monitoring the situation,
- 1:09SPEAKER_02: which is a catchphrase, so it's kind of hard to figure out.
- 1:11SPEAKER_02: You do have some co-hosts,
- 1:13SPEAKER_02: so it's not just you, but you're holding up one-third of the sky or something?
- 1:17SPEAKER_01: Yep.
- 1:18SPEAKER_02: All right.
- 1:19SPEAKER_02: And so I kind of wanted to have you on because,
- 1:22SPEAKER_02: you know, you are monitoring the situation and we are living in interesting times.
- 1:27SPEAKER_02: Things are changing really fast.
- 1:28SPEAKER_02: Like as we are recording,
- 1:31SPEAKER_02: one of the Millennium problems,
- 1:32SPEAKER_02: the Navier-Stokes problem, I guess, was solved by OpenAI, supposedly.
- 1:38SPEAKER_02: So, you know,
- 1:39SPEAKER_02: a lot of mathematicians are stressed out right now
- 1:42SPEAKER_02: because they're wondering what they're going to do for the future.
- 1:45SPEAKER_02: And, you know,
- 1:46SPEAKER_02: They've been predicting,
- 1:47SPEAKER_02: you know, AI,
- 1:49SPEAKER_02: execs like Dario and I have been predicting, the waves of unemployment and all that.
- 1:53SPEAKER_02: That hasn't quite happened yet.
- 1:54SPEAKER_02: So maybe it'll happen.
- 1:55SPEAKER_02: Things change slowly or not at all until they change really fast sometimes.
- 1:58SPEAKER_02: So, you know, I know Dario a little bit, not well anymore.
- 2:02SPEAKER_02: But he is a very smart fellow, very, very smart.
- 2:05SPEAKER_02: So I take anything he says seriously.
- 2:07SPEAKER_02: And obviously, I know a lot of other people in that space who are like that.
- 2:11SPEAKER_02: To start out with, let me just ask you right now, we are recording September 9th, 2026.
- 2:18SPEAKER_02: Things could change by the time this posts next week.
- 2:20SPEAKER_02: So I'm going to just ask you real quick, what's your P-doom over the next 10 years?
- 2:26SPEAKER_02: Let's say doom as in like the collapse of technological civilization.
- 2:30SPEAKER_02: We don't need to deal with extinction here.
- 2:32SPEAKER_01: Well, I think the collapse of technological civilization,
- 2:35SPEAKER_01: my odds of that are quite low because I think even in a very bad scenario,
- 2:40SPEAKER_01: even in a scenario that does result in human disenvironment or extinction,
- 2:44SPEAKER_01: technological civilization will very much continue.
- 2:47SPEAKER_01: It will just be the AIs running the civilization rather than the humans.
- 2:51SPEAKER_02: Yeah, I should have said human technological civilization because we are
- 2:53SPEAKER_02: a human-centric podcast here.
- 2:58SPEAKER_02: Basically, what are the prospects for our species in your opinion?
- 3:01SPEAKER_02: Are you an optimist?
- 3:02SPEAKER_02: Are you a pessimist?
- 3:03SPEAKER_02: Depending on how you define that, I guess.
- 3:05SPEAKER_01: I think I am relatively optimistic compared to many of the AI Doomers out there.
- 3:10SPEAKER_01: I think if I had to put a number on it, what is my PDoom over the next 10 years?
- 3:15SPEAKER_01: I would say like 10%.
- 3:18SPEAKER_02: Okay, that's not a trivial
- 3:20SPEAKER_01: PDoom though.
- 3:21SPEAKER_01: No, it's not.
- 3:22SPEAKER_01: It's not trivial.
- 3:23SPEAKER_01: Like there are very real, serious and significant risks.
- 3:28SPEAKER_01: However, I think it's mitigated by a few different things.
- 3:31SPEAKER_01: First, I think that international coordination, domestic coordination, pacing frontier.
- 3:37SPEAKER_01: We'll play into it more.
- 3:39SPEAKER_01: I think my P2 would be higher if I basically expected labs
- 3:43SPEAKER_01: to just race their superintelligence as quickly as possible without much alignment.
- 3:49SPEAKER_01: But I don't think that's going to happen.
- 3:51SPEAKER_01: The preference cascade that we've seen so far in just the past day In just
- 3:56SPEAKER_01: the past 24 hours since that anthropic researcher Jacob Coxon resigned
- 3:59SPEAKER_01: and posted his resignation note,
- 4:02SPEAKER_01: you're seeing politicians from across the aisle,
- 4:04SPEAKER_01: Bernie Sanders,
- 4:06SPEAKER_01: Ed Markey,
- 4:07SPEAKER_01: Ted Cruz,
- 4:08SPEAKER_01: Ana Polina Luna, Ron DeSantis, Chris Blumenthal, all posting about their concerns.
- 4:14SPEAKER_01: So like I do expect political considerations
- 4:16SPEAKER_01: to slow progress down pretty substantially in the near future.
- 4:19SPEAKER_01: And I expect the same will happen for China as well.
- 4:22SPEAKER_01: I also think that it is just difficult to destroy the world.
- 4:27SPEAKER_01: In order to actually destroy human civilization,
- 4:30SPEAKER_01: which is really robust,
- 4:33SPEAKER_01: consider COVID barely touched our long-term growth trajectory.
- 4:38SPEAKER_01: Even World War Two barely touched our long term growth trajectory.
- 4:41SPEAKER_01: If you look at like the GDP of the world over time,
- 4:44SPEAKER_01: World War Two only set us off by like a few years in the grand scheme of things.
- 4:48SPEAKER_01: Technological civilization is quite robust and quite hard to destroy.
- 4:52SPEAKER_01: Not that A super intelligent AI couldn't do it,
- 4:55SPEAKER_01: but it would have to be both wildly super intelligent and able to fully automate
- 5:00SPEAKER_01: the entire economy.
- 5:02SPEAKER_01: Because remember,
- 5:03SPEAKER_01: if there are still like humans running the power plants in the data centers,
- 5:08SPEAKER_01: then the AI would not be able to sustain itself without us.
- 5:11SPEAKER_01: So humans would have to be completely superfluous and the AI would have
- 5:14SPEAKER_01: to be much smarter than all of humanity combined plus our defense mechanisms
- 5:19SPEAKER_01: in order to disempower and defeat us.
- 5:21SPEAKER_01: And I'm not saying that couldn't happen.
- 5:22SPEAKER_01: But I'm saying it's not overwhelmingly likely.
- 5:25SPEAKER_02: Well, I mean, you know, this is showing, I didn't do like my research.
- 5:29SPEAKER_02: Like I've only known approximately because I follow you on socials and stuff like that.
- 5:33SPEAKER_02: You know, we are friends.
- 5:34SPEAKER_02: How long have you been doing MTS now?
- 5:36SPEAKER_01: For about four and a half months.
- 5:38SPEAKER_01: So since our launch date was April 20th.
- 5:41SPEAKER_01: We were working on it internally for like a month before then.
- 5:45SPEAKER_01: Like internally, like what do you mean by that?
- 5:47SPEAKER_01: At A16Z.
- 5:48SPEAKER_01: Okay, yeah.
- 5:50SPEAKER_02: This is sponsored by A16Z.
- 5:52SPEAKER_01: Not sponsored by A16Z.
- 5:54SPEAKER_01: A16Z was an early backer and early investor in MTS.
- 5:58SPEAKER_02: Okay.
- 5:59SPEAKER_02: Okay.
- 5:59SPEAKER_02: So it's still invested.
- 6:01SPEAKER_02: Okay.
- 6:02SPEAKER_01: It is an investment.
- 6:03SPEAKER_01: We do have some sponsors who are great companies.
- 6:07SPEAKER_01: We love our sponsors, but A16Z is not a sponsor.
- 6:09SPEAKER_01: It's an investage.
- 6:10SPEAKER_02: So are you guys, just to hear, are you an LLC then?
- 6:14SPEAKER_02: Yeah.
- 6:15SPEAKER_02: I mean, how are you guys organized?
- 6:18SPEAKER_02: Like, now I don't know.
- 6:18SPEAKER_02: I don't know what's going on here.
- 6:20SPEAKER_02: So are you three the only employees?
- 6:23SPEAKER_02: Like, you know, are you guys incorporated?
- 6:24SPEAKER_02: Like, what's going on with that?
- 6:25SPEAKER_01: We're a separate entity.
- 6:26SPEAKER_01: We're editorial independence.
- 6:29SPEAKER_01: And we currently have a team of around on the order of 10 people-ish.
- 6:33SPEAKER_02: Oh, okay.
- 6:35SPEAKER_02: Stuff is happening.
- 6:36SPEAKER_02: All right.
- 6:37SPEAKER_02: Back to the real stuff in terms of technology.
- 6:41SPEAKER_02: So it's been four months.
- 6:44SPEAKER_02: And let's go back to the beginning.
- 6:47SPEAKER_02: You're not that old.
- 6:48SPEAKER_02: As in, you are quite young.
- 6:50SPEAKER_02: In fact,
- 6:50SPEAKER_02: your whole adult life has spanned the new era of AI with LLMs and all this stuff,
- 6:55SPEAKER_02: right?
- 6:56SPEAKER_02: Yeah.
- 6:57SPEAKER_02: Yeah.
- 6:59SPEAKER_02: Tell me your history.
- 7:01SPEAKER_02: Actually, I'll tell you mine just really quickly about AI.
- 7:05SPEAKER_02: I moved to the Bay Area in 2007,
- 7:08SPEAKER_02: and I was adjacent to the Eliezer Yudkowsky group of rationalists,
- 7:15SPEAKER_02: and they were very worried about artificial intelligence.
- 7:18SPEAKER_02: And at the time, they were trying to develop friendly AI.
- 7:21SPEAKER_02: And I think Eliezer gave that up around 2010 or so,
- 7:26SPEAKER_02: 2011, I don't remember, but that was the initial period.
- 7:29SPEAKER_02: And, you know, they were worried that AI was going to, you know, exterminate us.
- 7:34SPEAKER_02: But also there was a Kurzweil faction and the Kurzweil tendency,
- 7:37SPEAKER_02: which is like, oh,
- 7:38SPEAKER_02: the singularity will,
- 7:39SPEAKER_02: you know, it'll be like the rapture or something, exaltation, right?
- 7:43SPEAKER_02: And so, you know, I went to grad school, you know, did other things, moved to Austin.
- 7:47SPEAKER_02: But, you know, I've always been like around these people.
- 7:49SPEAKER_02: And, you know,
- 7:50SPEAKER_02: when GPT came out,
- 7:52SPEAKER_02: you know,
- 7:53SPEAKER_02: all my friends that had been talking about how big this was going to be in 2010,
- 7:59SPEAKER_02: Well, I mean,
- 7:59SPEAKER_02: they were kind of vindicated in a way that they weren't expecting because,
- 8:03SPEAKER_02: you know,
- 8:04SPEAKER_02: obviously LLMs and Transformers and stuff like that were not where they are.
- 8:08SPEAKER_02: I mean, I think Transformers are relatively new.
- 8:10SPEAKER_02: Anyway, my point is— 2017, I believe.
- 8:13SPEAKER_02: Yeah, 2017.
- 8:13SPEAKER_02: There you go.
- 8:14SPEAKER_02: So totally new technology.
- 8:15SPEAKER_02: But, you know,
- 8:17SPEAKER_02: the general argument they made that artificial intelligence was going to
- 8:20SPEAKER_02: a real thing Well, I mean, it's a real thing right now.
- 8:23SPEAKER_02: Here we are, right?
- 8:25SPEAKER_02: And so talk about when you encountered artificial intelligence,
- 8:29SPEAKER_02: when you started thinking about it,
- 8:30SPEAKER_02: and why you're in this space now, what are you excited about?
- 8:34SPEAKER_01: Well, I just turned 22 a couple weeks ago.
- 8:37SPEAKER_01: So yeah, I am pretty young, I would say.
- 8:40SPEAKER_01: I think I first encountered artificial intelligence for the first time,
- 8:43SPEAKER_01: the concepts of artificial intelligence, like as a kid.
- 8:46SPEAKER_01: I was maybe 11, 12 years old,
- 8:51SPEAKER_01: And around that time was Bostrom Superintelligence,
- 8:54SPEAKER_01: was Elon Musk and Sam Wallman co-founding OpenAI,
- 8:58SPEAKER_01: Elon Musk and Stephen Hawking warning about superintelligence
- 9:01SPEAKER_01: and human disempowerment to human extinction.
- 9:04SPEAKER_01: CGP Grey, one of my favorite YouTubers at the time, did a video on it.
- 9:09SPEAKER_01: And...
- 9:11SPEAKER_01: I remember thinking at the time,
- 9:12SPEAKER_01: like, wow, okay, this artificial intelligence is going to be a risk for the future.
- 9:17SPEAKER_01: But I thought it was some long way out.
- 9:20SPEAKER_01: I thought it was going to be a thing in my lifetime.
- 9:22SPEAKER_01: I assumed that it was a few decades away.
- 9:24SPEAKER_01: So maybe by the time of my 50s or 60s, AI would become a thing.
- 9:29SPEAKER_01: And it turns out, no, AI will not become a thing in my 50s and 60s.
- 9:33SPEAKER_01: It will become a thing in my 20s.
- 9:34SPEAKER_01: So In 2022, when I was 18 years old, I just started college.
- 9:39SPEAKER_01: It was my freshman year, my fall semester of my freshman year.
- 9:43SPEAKER_01: I discovered...
- 9:46SPEAKER_01: Actually, first,
- 9:46SPEAKER_01: I had been keeping tabs on AI a little bit because of Stable Diffusion that year
- 9:50SPEAKER_01: and Codex.
- 9:52SPEAKER_01: Actually,
- 9:52SPEAKER_01: back in 2021,
- 9:54SPEAKER_01: I had used GPT-3,
- 9:56SPEAKER_01: and I thought of it as interesting, but still basically a toy, not very useful.
- 10:02SPEAKER_01: I had played with Stable Diffusion.
- 10:04SPEAKER_01: I had played with Meta's Galactica,
- 10:06SPEAKER_01: which was their large language model,
- 10:07SPEAKER_01: which they released in November 2022 before TragedyVT.
- 10:10SPEAKER_01: But when TragedyVT came out, like immediately it blew me away.
- 10:17SPEAKER_01: Like I had no idea.
- 10:18SPEAKER_01: I had just started by coincidence.
- 10:20SPEAKER_01: I just started using Twitter a couple months earlier.
- 10:22SPEAKER_01: And so I saw people posting about ChatGPT as soon as it came out.
- 10:27SPEAKER_01: And I was completely blown away.
- 10:29SPEAKER_01: I was like, Okay, this is it.
- 10:30SPEAKER_01: Like AI is coming soon.
- 10:32SPEAKER_01: My timelines need to be much shorter.
- 10:36SPEAKER_01: Also around the same time, maybe a few months earlier, I had discovered LessWrong.
- 10:41SPEAKER_01: And I had read it mostly for the rationality stuff.
- 10:44SPEAKER_01: But In late 2022, early 2023, I started reading it for the AI stuff as well.
- 10:50SPEAKER_01: And what I saw really shocked me.
- 10:52SPEAKER_01: It was like people who were very smart,
- 10:55SPEAKER_01: serious,
- 10:56SPEAKER_01: rigorous people who were like,
- 10:58SPEAKER_01: AI might actually just destroy the world and kill everyone in the near future.
- 11:02SPEAKER_01: And so I took it very seriously.
- 11:04SPEAKER_01: I was quite Doomer for quite some time through much of 2023, I think.
- 11:12SPEAKER_01: Not that I think this stuff is implausible now,
- 11:15SPEAKER_01: but I definitely think it is less preordained than I believed at the time.
- 11:22SPEAKER_01: GBT4 obviously was another big update.
- 11:24SPEAKER_01: I was like, okay, this is really happening now.
- 11:28SPEAKER_01: And so in summer 2023, I started my podcast.
- 11:33SPEAKER_01: The TLBR of this story is I got some people on.
- 11:38SPEAKER_01: In August 2023, I visited SF for the first time and I knew I wanted to move out.
- 11:44SPEAKER_01: The nature is amazing.
- 11:46SPEAKER_01: The people are amazing.
- 11:48SPEAKER_01: In summer 2024,
- 11:50SPEAKER_01: I moved here temporarily for the summer with a grant from Emergent Ventures,
- 11:54SPEAKER_01: Tyler Callen's program to do my podcast.
- 11:58SPEAKER_02: I remember.
- 11:59SPEAKER_01: I think that's where we met roughly about that time.
- 12:02SPEAKER_02: We met in the spring.
- 12:03SPEAKER_02: We met, yes.
- 12:05SPEAKER_02: In April, 2024.
- 12:08SPEAKER_02: You were coming through to Austin with your fam.
- 12:10SPEAKER_02: The Eclipse Party.
- 12:12SPEAKER_02: Yeah, it was Eclipse.
- 12:13SPEAKER_02: I should say, I checked the Theo Jaffe podcast on YouTube.
- 12:18SPEAKER_02: I am the second highest number of views of all of your interviews next to Scott Aronson.
- 12:24SPEAKER_01: That's awesome.
- 12:25SPEAKER_02: That was
- 12:26SPEAKER_01: Austin, Texas.
- 12:27SPEAKER_02: Yeah, and that was two years ago.
- 12:30SPEAKER_02: What was the date?
- 12:31SPEAKER_02: It was like, yeah, it was like two years ago.
- 12:33SPEAKER_02: Yeah, was it the summer?
- 12:35SPEAKER_02: Or was it before?
- 12:35SPEAKER_02: Was it before we met IRL?
- 12:37SPEAKER_02: It might be before we met IRL.
- 12:38SPEAKER_02: It was before we met IRL.
- 12:40SPEAKER_02: Okay, okay.
- 12:40SPEAKER_02: Anyway, so sorry, I'm sure people are super interested in how we met.
- 12:45SPEAKER_02: But anyway, so yeah, I will say that GPT, fall of 2022, that was an inflection point.
- 12:52SPEAKER_02: It was interesting,
- 12:53SPEAKER_02: and people have heard me say this on the podcast before,
- 12:55SPEAKER_02: that the basics were already there,
- 12:58SPEAKER_02: but until OpenAI decided to go Go consumer hardcore.
- 13:03SPEAKER_02: It didn't really impact the culture, even though stuff was quote unquote happening.
- 13:09SPEAKER_02: People at DeepMind have told me that, look, they could have done that years earlier.
- 13:14SPEAKER_02: It's just that Google put the brakes on it.
- 13:17SPEAKER_02: The levels of safety, security, and paranoia about how it would impact Alphabet's brand.
- 13:23SPEAKER_02: And so OpenAI obviously didn't have that baggage.
- 13:25SPEAKER_02: That went with it and it transformed our culture in many ways.
- 13:29SPEAKER_02: I will say,
- 13:30SPEAKER_02: obviously like all the software people like at my company,
- 13:33SPEAKER_02: you know, they're using AI and cursor and all these things.
- 13:35SPEAKER_02: AI has influenced our lives.
- 13:37SPEAKER_02: But I mean, it's been four years, almost four years.
- 13:44SPEAKER_02: It seems like it's not...
- 13:46SPEAKER_02: I mean, okay, the Navier-Stokes, the millennial problem.
- 13:48SPEAKER_02: There's stuff that's happening, but it hasn't transformed our lives.
- 13:53SPEAKER_02: This is like a Peter Thiel complaint, like the car.
- 13:57SPEAKER_02: But now some people are saying this is the biggest change since agriculture.
- 14:01SPEAKER_02: So can you talk about that, your take on that?
- 14:04SPEAKER_02: Is it just we haven't,
- 14:06SPEAKER_02: like we're very early on on the curve,
- 14:08SPEAKER_02: or we just don't perceive a lot of back office things changing?
- 14:11SPEAKER_02: What do you think?
- 14:12SPEAKER_01: Right.
- 14:12SPEAKER_01: I mean, there's two views here.
- 14:14SPEAKER_01: Obviously,
- 14:14SPEAKER_01: there are bottlenecks to AI simply You know,
- 14:18SPEAKER_01: teleporting to immediate superintelligence that transforms absolutely everything.
- 14:24SPEAKER_01: Some people will say,
- 14:25SPEAKER_01: oh, these bottlenecks will just fall, like all past bottlenecks have, very quickly.
- 14:30SPEAKER_01: Like, for example, people assumed that hallucinations would be a major bottleneck to an option.
- 14:34SPEAKER_01: Turns out, no, actually, hallucinations are pretty easy to overcome.
- 14:38SPEAKER_01: You just like RL the models for truthfulness and also give them access to the internet.
- 14:42SPEAKER_01: People assumed, for example, that Long context would be a bottleneck.
- 14:48SPEAKER_01: The original ChadGBT had a 4,000 token context window, if you remember that.
- 14:52SPEAKER_01: And it was a huge deal at the time when GBT4 in late 2023...
- 14:57SPEAKER_02: Talk about what a context window is, just real quick.
- 14:59SPEAKER_01: Yeah, a context window is just the maximum amount of tokens,
- 15:01SPEAKER_01: basically the maximum input size that you can give to the model that it can think
- 15:05SPEAKER_01: about in a single forward pass.
- 15:09SPEAKER_01: It was a huge deal in late 23 when GBT4 had a 32,000 token context window.
- 15:14SPEAKER_01: Now all the models have like a million tokens of context.
- 15:17SPEAKER_01: So that ended up not being a bottleneck.
- 15:19SPEAKER_01: People thought its inability to do long horizon tasks would be a bottleneck to coding.
- 15:25SPEAKER_01: That turned out not to be the case.
- 15:27SPEAKER_01: So under these assumptions, it's like, oh, right now, research taste is a bottleneck.
- 15:31SPEAKER_01: The AI is good at writing code and running experiments.
- 15:34SPEAKER_01: It's not so good at deciding which experiments to run.
- 15:36SPEAKER_01: Some people say, oh, the AI will just get way better at that.
- 15:40SPEAKER_01: And then it will just take over every niche of human intelligence very quickly.
- 15:45SPEAKER_01: And then another view is that...
- 15:48SPEAKER_01: The AI has gotten really good at what we have devoted billions of dollars
- 15:52SPEAKER_01: to training it to do,
- 15:53SPEAKER_01: which is namely coding,
- 15:55SPEAKER_01: agentic white-collar work,
- 15:57SPEAKER_01: and mathematics,
- 15:59SPEAKER_01: which are areas that the AI labs have devoted huge amounts of money towards.
- 16:04SPEAKER_01: They've not gotten as good at, say,
- 16:05SPEAKER_01: creative writing or full automation of labor,
- 16:10SPEAKER_01: which the labs have devoted relatively less to.
- 16:13SPEAKER_01: So one bottleneck could be just that it is fundamentally easier for AIs in the way
- 16:20SPEAKER_01: that they're currently being developed to do Millennium Prize problems than for them
- 16:23SPEAKER_01: to complete someone's job.
- 16:26SPEAKER_01: They might just be very deeply jagged in this way.
- 16:28SPEAKER_01: They might get super, superhuman in some areas while remaining below human level in others.
- 16:34SPEAKER_01: Yeah.
- 16:36SPEAKER_01: And then there's another idea of computational irreducibility,
- 16:39SPEAKER_01: which is that even if you are very intelligent,
- 16:45SPEAKER_01: some things in reality might just be fundamentally hard,
- 16:48SPEAKER_01: even for extreme superintelligence to be able to do.
- 16:52SPEAKER_01: So maybe, for example, restoring brain function to someone with Alzheimer's disease.
- 16:59SPEAKER_01: A lot of that information is basically gone.
- 17:01SPEAKER_01: If someone has Alzheimer's disease,
- 17:02SPEAKER_01: it's a neurodegenerative disease,
- 17:05SPEAKER_01: a lot of the information that makes up the patterns
- 17:08SPEAKER_01: that make up people's brain ends up just being deleted.
- 17:13SPEAKER_01: And so even if you're super intelligence, it seems difficult to get that back.
- 17:20SPEAKER_02: So actually,
- 17:21SPEAKER_02: we're throwing out a lot of terms,
- 17:22SPEAKER_02: and I can't make you define everything,
- 17:24SPEAKER_02: people need to do their homework, but talk about what the singularity is.
- 17:28SPEAKER_01: The singularity means many things to many people.
- 17:33SPEAKER_02: The singularity in the AI context,
- 17:35SPEAKER_02: obviously, not the singularity of a black hole and all that stuff.
- 17:37SPEAKER_01: Right, right.
- 17:38SPEAKER_01: I mean, like Patrick Collison,
- 17:40SPEAKER_01: Stripe earlier,
- 17:40SPEAKER_01: a couple months ago,
- 17:41SPEAKER_01: put out a letter where they were like,
- 17:43SPEAKER_01: we believe the singularity started in the beginning of 2026 because the rate
- 17:47SPEAKER_01: of firm foundation increased.
- 17:49SPEAKER_01: And it's like, that's not what that means.
- 17:51SPEAKER_01: The singularity, to me, is...
- 17:54SPEAKER_01: The point at which artificial superintelligence becomes so powerful
- 17:58SPEAKER_01: that we have no idea how to predict anything that happens after that point.
- 18:04SPEAKER_01: I don't necessarily believe it will actually shake out like that.
- 18:09SPEAKER_01: I think that there will be ways in which reality remains very understandable,
- 18:13SPEAKER_01: very easily understandable to humans for quite some time.
- 18:16SPEAKER_01: Well, other aspects of it might be inscrutable to us already.
- 18:21SPEAKER_01: You know,
- 18:21SPEAKER_01: many have made the point that we are currently in the long singularity that began,
- 18:25SPEAKER_01: I don't know,
- 18:26SPEAKER_01: when the first humans developed the tools that led us to the Industrial Revolution
- 18:31SPEAKER_01: or fire before that or Homo sapiens before that or the formation of life before
- 18:36SPEAKER_01: that.
- 18:37SPEAKER_01: So it seems like,
- 18:38SPEAKER_01: you know, according to some views, we've always been riding up this exponential.
- 18:42SPEAKER_02: Yeah.
- 18:43SPEAKER_02: And obviously,
- 18:44SPEAKER_02: the term was popularized,
- 18:46SPEAKER_02: information explosion,
- 18:47SPEAKER_02: technological singularity,
- 18:49SPEAKER_02: by Ray Kurzweil,
- 18:50SPEAKER_02: who is still,
- 18:51SPEAKER_02: you know,
- 18:52SPEAKER_02: for those of you who are younger,
- 18:54SPEAKER_02: so like basically when you were born,
- 18:56SPEAKER_02: you know,
- 18:57SPEAKER_02: these sorts of ideas that you're talking about are basically, were,
- 19:02SPEAKER_02: not basically,
- 19:02SPEAKER_02: they were the purview of Ray Kurzweil,
- 19:05SPEAKER_02: he's the one that really, author,
- 19:08SPEAKER_02: The Singularity Is Near, When Humans Transcend Biology was published in 2005.
- 19:14SPEAKER_02: It sold a little over 20 years ago.
- 19:17SPEAKER_02: It was published in 2005.
- 19:18SPEAKER_02: Kurzweil was the one that really popularized these ideas.
- 19:22SPEAKER_02: He's 78 now.
- 19:25SPEAKER_02: He seems to be relatively quiet at this point.
- 19:27SPEAKER_02: I wonder why that is.
- 19:29SPEAKER_02: Do you have any hunch?
- 19:31SPEAKER_02: I think he predicted the singularity.
- 19:33SPEAKER_01: He's just old.
- 19:35SPEAKER_01: Yeah, not much more to it.
- 19:36SPEAKER_01: He's just, he's just kind of boomered out.
- 19:38SPEAKER_01: Similar with Nick Land,
- 19:39SPEAKER_01: actually,
- 19:40SPEAKER_01: you know, I have a tremendous amount of respect for Nick Land as a thinker.
- 19:45SPEAKER_01: He predicted AI much better than just about anyone else in the 90s.
- 19:49SPEAKER_01: And he spends most of his time on Twitter posting about like immigration.
- 19:52SPEAKER_01: And it's like, bro, you know, we're on the eve of like human level machine intelligence.
- 19:56SPEAKER_01: You know, this is what you've talked about for forever.
- 19:58SPEAKER_01: Like,