Latest / Best of Business / Sam Dickie: Fisher Funds expert on the AI researcher quitting Anthropic and warning of the tech's risks

Transcript

Sam Dickie: Fisher Funds expert on the AI researcher quitting Anthropic and warning of the tech's risks · Best of Business

• 0:09
  Speaker 1: You're listening to a podcast from Newstalk ZB. Follow this
• 0:13
  Speaker 1: and our wide range of podcasts now on iHeartRadio.
• 0:17
  Speaker 2: Now, as you well know, an anthropic researcher has quit,
• 0:19
  Speaker 2: saying the industry is gambling with our lives. This comes
• 0:22
  Speaker 2: just a few weeks after Sam Dickey from Fisher Funds
• 0:25
  Speaker 2: talked about the AI safety and research company and what
• 0:27
  Speaker 2: could be the biggest initial public offering in history. And
• 0:30
  Speaker 2: Sam is back with us.
• 0:31
  Speaker 3: Hi, Sam. Good evening.
• 0:33
  Speaker 2: So what has actually happened here?
• 0:37
  Speaker 3: Well, this one really went viral, but many high-profile AI
• 0:41
  Speaker 3: executives have been sounding the alarm. And we talked about
• 0:45
  Speaker 3: Dario himself, who's the founder of Anthropic, when you and
• 0:48
  Speaker 3: I talked three weeks ago, was talking about the risks
• 0:51
  Speaker 3: of his product. But Jacob Coxon just resigned from Anthropic
• 0:53
  Speaker 3: after three years working at both Anthropic and OpenAI. And
• 0:57
  Speaker 3: he said these leading-edge AI companies are racing to build
• 1:01
  Speaker 3: increasingly powerful systems before they know how to make them
• 1:03
  Speaker 3: completely safe. And then the real twist there was Evan,
• 1:06
  Speaker 3: who is still employed anthropic, publicly backed him, putting the
• 1:11
  Speaker 3: chance of AI killing all humans within the next decade
• 1:14
  Speaker 3: at around 10%, which sounds really scary. But there's a
• 1:17
  Speaker 3: couple of important qualifications there. The first one is he
• 1:20
  Speaker 3: also says the risk from today's AI is low. Everyone
• 1:24
  Speaker 3: can tap the brakes if they want, but they're worried
• 1:27
  Speaker 3: the other one won't. So it's that prisoner's dilemma. So
• 1:29
  Speaker 3: everyone has their foot to the floor, flat out race.
• 1:33
  Speaker 2: Okay, so what is it that they're actually worried is
• 1:35
  Speaker 2: going to happen?
• 1:37
  Speaker 3: Well, it's not a robot suddenly deciding to take over Auckland.
• 1:41
  Speaker 3: The fear is AI is improving AI, so recursive self-improvement.
• 1:46
  Speaker 3: So one generation of AI helping build the next and
• 1:49
  Speaker 3: the next until the machines are improving faster than humans
• 1:52
  Speaker 3: can keep up. We are not there today. But there
• 1:55
  Speaker 3: has been some recent incidents that have created a bit
• 1:58
  Speaker 3: of hysteria, the Hugging Face hacking incident, for example. And
• 2:02
  Speaker 3: it's revealing something more mundane, actually. AI models being tested
• 2:06
  Speaker 3: in supposedly sealed environments have accessed real-world systems because of
• 2:11
  Speaker 3: a simple setup error. So that's not a machine plotting
• 2:14
  Speaker 3: its escape. and it's just humans that got the safety
• 2:17
  Speaker 3: barrier wrong. So what may be more useful as a
• 2:20
  Speaker 3: warning to us and to AI model manufacturers is we're
• 2:25
  Speaker 3: building increasingly powerful technology increasingly quickly, and the models become
• 2:30
  Speaker 3: more capable. The blast radius of a simple human mistake
• 2:33
  Speaker 3: is significantly larger. So for us, we just need to
• 2:36
  Speaker 3: be aware of what data you want AI models to access.
• 2:39
  Speaker 1: Yeah.
• 2:40
  Speaker 2: Now, this is obviously a really bold claim. So what's
• 2:42
  Speaker 2: the counterargument?
• 2:44
  Speaker 3: Well, the strong counter-argument is, and we talked about this
• 2:46
  Speaker 3: a couple of weeks ago, is with founder Dario and
• 2:50
  Speaker 3: many people within Anthropic themselves, they spend an awful amount
• 2:53
  Speaker 3: of time warning us about their own product. And if
• 2:56
  Speaker 3: you think about it, it's called regulatory capture, arguably. So
• 3:00
  Speaker 3: if governments believe AI could be extraordinarily dangerous, they regulate it.
• 3:04
  Speaker 3: And regulation, of course, is much easier... for a company
• 3:08
  Speaker 3: with hundreds of billions of dollars to absorb than a
• 3:10
  Speaker 3: new competitor. So it really shuts out smaller competitors. That's
• 3:13
  Speaker 3: why it's called regulatory capture. So the fear itself can
• 3:16
  Speaker 3: become a competitive advantage. But here's the twist. The fear
• 3:20
  Speaker 3: can be sincere and commercially useful. Those aren't contradictory. So remember,
• 3:24
  Speaker 3: Anthropic itself built a model called Mythos, and they said
• 3:28
  Speaker 3: it's too powerful for you guys. But that was a
• 3:29
  Speaker 3: wonderful marketing campaign as well.
• 3:33
  Speaker 2: Now, what does this mean for investors, do.
• 3:35
  Speaker 3: You think, Sam? I don't think we need to be
• 3:38
  Speaker 3: frightened of our phone quite yet. But I think the
• 3:41
  Speaker 3: better question is for all of us is how much
• 3:43
  Speaker 3: freedom should we give increasingly powerful AI and who gets
• 3:47
  Speaker 3: to decide that? So if you listen to Dario from
• 3:50
  Speaker 3: Anthropic and some of the more overzealous Anthropicas, they think
• 3:55
  Speaker 3: they're the only company on earth that has got the
• 3:57
  Speaker 3: God-given talent and right to control this beast that they're creating.
• 4:02
  Speaker 3: So for investors, we should probably watch three things. How
• 4:04
  Speaker 3: fast the technology improves, particularly that recursive self-improvement where AI
• 4:09
  Speaker 3: is building AI at a faster pace, how governments respond,
• 4:12
  Speaker 3: because when this fear-mongering gets viral, you can imagine that
• 4:16
  Speaker 3: governments might do some silly things, and whether these companies
• 4:20
  Speaker 3: themselves can prove they can make it safe enough to trust.
• 4:23
  Speaker 2: Yeah, very good point. Hey, thank you very much, Sam.
• 4:25
  Speaker 2: Really appreciate it, as always. We'll talk to you next week.
• 4:27
  Speaker 2: Sam Dickey, official funds 20 away from seven.
• 4:30
  Speaker 1: For more from Newstalk ZB, listen live, on air or
• 4:33
  Speaker 1: online and keep our shows with you wherever you go
• 4:36
  Speaker 1: with our podcasts on iHeart Radio.