Latest / In the Loop AI / Anthropic Got Hacked Again: How Ex-DOGE Researchers Got Into Its Slack With AI Agents

Transcript

Anthropic Got Hacked Again: How Ex-DOGE Researchers Got Into Its Slack With AI Agents · In the Loop AI

0:00
So Anthropic just got hacked. Again. And this time not only did the attackers
0:04
get access to proprietary information about models, people's salaries,
0:07
but we also know who did it.
0:09
A one month old startup that hammered their systems with hundreds of agents
0:13
24-7 until they found a secret passage in.
0:16
But what should concern everyone is how they got access.
0:20
It was via a third party platform that Anthropic had zero control over and using
0:25
AI models that existed three months ago which means that probably anyone could
0:29
have pulled off the same attack.
0:30
You see the reason why I want to talk about this today isn't just because this
0:33
is another major breach of an AI lab that is concerning in itself
0:38
but it's because it's part of a recent growing trend I'm observing where AI
0:41
is just capable of attacking and hacking into
0:44
any software that exists for a few dollars and this has massive ramifications
0:49
around how we protect our bank accounts, our medical records,
0:52
our personal information, and even the nuclear codes.
0:55
The point is, this affects anything and everything that is software-based,
0:59
and it's time to take this seriously.
1:01
I have many thoughts, so let's get into it on today's episode brought to you by Qualcomm.
1:08
So there's this startup called Cathedral, which is a one-month-old startup run
1:12
by ex-Doge employees, but it's co-founded by an ex-Pentagon head of data.
1:18
Now, as you know, the history between the Pentagon and Anthropic has been less than ideal.
1:24
Pentagon currently has blacklisted Claude and Anthropic after they failed to
1:27
comply with a bunch of other initiatives, which I'm not going to get into right now.
1:31
But the point is, a one-month-old startup, and they saw that Anthropic was having a public bug bounty.
1:37
So typically companies do this when they want ethical hackers or white hat hackers
1:42
to stress or test the boundaries of their own systems.
1:46
The point is, if they are able to find a bug or a flaw, they can disclose it
1:50
to Anthropic or any other company who's running a similar test,
1:53
and they'll be able to patch and fix it before a malicious actor is able to do so.
1:58
So this startup cathedral took part in this, and they started off by doing the obvious thing.
2:03
They tried to go in through the front door. So they tried to attack all of Anthropic
2:07
systems, and they did so for several days, but came up with nothing.
2:11
They weren't able to penetrate the system. So that's all well and good.
2:14
But then they had a bright idea. They thought, hmm, what if we went through
2:19
a third-party admin system or a third-party software system?
2:22
I wonder how easy it would be to do so.
2:25
And so they did so. They started off with targeting a sales and admin platform
2:29
known as Salesforce, used by millions of people around the world,
2:34
and within 12 hours, they were able to get access to Anthropix Slack.
2:39
Now, for context here, Slack might not seem like an important thing.
2:43
It's where you kind of exchange messages with your colleagues,
2:47
and maybe you share some files here and there.
2:49
Anthropic and companies like OpenAI keep everything on Slack.
2:54
This is something that the Anthropic researcher that quit a few weeks ago,
2:58
Jacob Coxon, that heralded the warnings of all these labs basically
3:03
building AI without any necessarily controls and systems, and it's all going
3:06
to end up in a cataclysmic end of humanity.
3:09
That guy basically said that these companies store everything on Slack.
3:14
I'm talking model information about unreleased models.
3:17
I'm talking about strategies about how they beat their competitors and pre-release
3:21
items for new products. I'm talking personal information like people's salaries.
3:25
Anything that is identifiable and valuable is mentioned, discussed, and stored on Slack.
3:32
And that's what these attackers got access to via a third-party system.
3:36
So it's a very serious breach. Now, Anthropic's response to this was we became
3:40
aware of one flaw that they exploited, but we weren't aware of another one,
3:44
which they didn't disclose.
3:45
And they're trying to claim that a cathedral was basically unethical.
3:48
But the point is, these guys got access to very proprietary information.
3:52
Now, Anthropic also included in their response that they only got access to
3:56
things like people's salaries, some model discussions, but not actually model weights, not actually
4:02
proprietary research that could inform an attacker how to build a much better
4:07
model or sabotage their own systems.
4:09
But personally, I think that's besides the point. They got into the system,
4:13
and let's say they got access to other types of Slack channels,
4:16
would they have got access to this proprietary information? I don't know.
4:19
Now, it's worth spending a moment emphasizing the political landscape around
4:24
this whole thing, because as I mentioned earlier, the co-founder and CEO of
4:28
Cathedral is the ex-head of data at the Pentagon.
4:33
And the Anthropic and the Pentagon have a very storied history.
4:36
It started about, I don't know, six months ago, I think in January of,
4:39
when the whole war between the US and Iran was going down, and it was revealed
4:44
that the Pentagon had secretly used Claude to strategize a certain attack.
4:50
Actually, it wasn't about the war. It was about the Maduro capture specifically, right?
4:55
So they used Claude to plan and execute the successful capture of Maduro.
5:01
And Anthropic had some choice thoughts on that. They said, you shouldn't be
5:04
using our models to be able to do that.
5:05
We're going to restrict you now, and any use by the Pentagon of Claude to perform
5:10
or execute missions should go through us. You need to call us up and we'll give
5:14
you the access if we think it's right or wrong.
5:18
And obviously the government and the Pentagon thought that this was a massive
5:21
breach of trust and could endanger the US as a national security issue if, let's say,
5:27
foreign adversaries like China who are known to get access to blacklisted accounts
5:31
could do the same thing and use it against them, right?
5:34
So I could see valid propositions on both ends, right? But the point is,
5:39
they were at odds with each other.
5:40
And the result was that Pentagon and the government ended up blacklisting Anthropic
5:45
and blacklisting their Claude models.
5:47
To this day, it is currently still blacklisted. We've come full circle here,
5:51
where an external startup that is only a month old, that is backed by the likes
5:55
of Andreessen Horowitz, I believe Mark Andreessen is actually on the board of directors
5:59
of this company, was complicit in attacking the company that is currently blacklisted.
6:05
Now, of course, within days of this being announced or within hours of this
6:09
being announced, the Pentagon reached out to this company and was like.
6:12
Hey, we want to offer you a government contract. We want you to work with us
6:16
and we want you to help us figure out where
6:19
possible attacks may come from, from adversaries and help us harden our own
6:23
systems and help us figure out how we prevent attacks like this happening in the future.
6:28
So on the one hand, you have the government getting involved.
6:31
Now, remember, none of these labs are currently nationalized and maybe this
6:36
is a step towards that happening.
6:37
The fact that this is being flagged
6:39
by both the government and a frontier research lab is a pretty big deal.
6:43
And so you have the Pentagon saying, okay, we need to figure this out.
6:46
And then you have Anthropic responding and their response has sort of been muted.
6:51
I haven't seen really much mention of this on the news channels or mainstream
6:55
media channels. But the point is.
6:57
They are trying to play it down a little bit. And listen, if the attackers truly
7:02
did just get access to some personal information, such as salaries and personal
7:07
data, maybe it's being dramatized a bit. And I want to refrain from doing that.
7:12
But I'm at odds because what if they did get access to proprietary channels?
7:18
And I guess I don't know the extent of which Anthropic specifically keeps their
7:22
proprietary information on Slack. All I know is that the rumors are they do.
7:27
And so do a lot of other companies. I'm sure it's not just OpenAI or Anthropic.
7:30
It could have been any other company, but it's quite prominent in this case
7:33
because Anthropic is one of the leading AI research labs out there,
7:36
building these models that could potentially cause a lot of harm.
7:40
And it confirms exactly what Jacob Cox and the ex-OpenAI and Anthropic alignment
7:45
researcher said a few weeks back where he said, listen, these companies are
7:49
recklessly racing towards superintelligence at the cost of their own security.
7:55
He literally said, they're gambling with our lives. And back then it was just
7:59
a headline. It was just hearsay.
8:01
And now it's being proven. And many respected voices in the field are raising
8:06
some very valid concerns. Will Depew here goes, Anthropic is massively underplaying
8:10
the recent security breach. I'm hearing Cathedral achieved full administrator
8:14
access of the Slack workspace.
8:16
To those unfamiliar, Anthropic and OpenAI keep literally everything in Slack.
8:21
We're critical to national security. We must beat China. And yes,
8:24
we keep all our crown jewels in Salesforce Slack Messenger.
8:28
Yes, we have admin API tokens everywhere.
8:30
Why do you ask? Anthropic and OpenAI have openly stated multiple times that
8:37
AI models are capable of very dangerous acts. Okay, fair enough.
8:41
They also say that if we don't control these systems, if we don't pace the frontier,
8:45
remember, there was a letter signed by hundreds and hundreds of very prominent
8:48
individuals asking to slow down AI progress and research.
8:51
If we don't do this, we could end up with a catastrophic event where someone
8:55
builds a bioweapon or there's a massive cybersecurity attack which steals most
8:59
of our money from our bank accounts, right?
9:01
And last week, we saw Black October unfold over 48 hours where there was a series
9:05
of these potential types of attacks. So we're in the mix here and they're making
9:09
a very valid point. So then why do you keep a bunch of your important stuff on Slack?
9:14
It just doesn't make any sense. He then goes on to make a point that he doesn't
9:17
necessarily think that it's the security team's fault at Anthropic,
9:22
but more so the higher level C-suite exec, the CEO
9:25
and founder's prerogative to build AI models as fast as they can to out-compete their competitors.
9:32
And they're doing this at the cost of making sure that their models are one,
9:37
aligned, and two, secure.
9:39
He's saying that OpenAI and Anthropic and any other model lab are doing these things.
9:43
And if that is the truth, then we are going to end up in
9:46
crappy situations like this where they're building amazing models that on one
9:50
hand are working towards solving cancer
9:52
and on the other hand are hacking all of our accounts now there was also a statement
9:56
from anthropic's head of red teaming red teaming is basically the team that
10:00
stress tests the systems they
10:01
are like an internal team that does the bug bounty thing that i described earlier
10:05
and his response was listen.
10:07
Things aren't as severe as people are making it out to be. They weren't able
10:11
to perform the attack in under 12 hours.
10:14
We were aware of the initial exploit.
10:17
We weren't aware of the second exploit. And he goes on to make a number of excuses,
10:21
basically saying this isn't that big of a deal, which I don't necessarily buy, to be honest.
10:27
But what concerns me more is what he says near the end of this. He goes, so what to do?
10:32
You probably want to red team yourself all the time. current models are very
10:37
creative i also think we should redesign software systems so that you're fundamentally
10:42
less exposed he then goes in another tweet,
10:45
but man do i agree with your overall message extremely extremely important basically
10:51
need to reinvent security every year at this point this is coming from one of
10:55
the heads of security at one of the leading labs,
10:59
that are building all these AI models in the first place.
11:02
And the reason why this is concerning is he's basically seeing no way to protect
11:07
current systems, current software as it exists today.
11:11
He's saying likely anything and everything is flawed, including the bedrock
11:16
on which the internet is built on, all the way scaled up to front-end applications
11:21
that anyone can make in VibeCode today.
11:24
And he's saying we're going to have to rebuild the security infrastructure and
11:27
systems around these different things.
11:29
One thing immediately pops to mind this is going to be incredibly costly.
11:33
This is going to take an incredible amount of time. And three,
11:36
and most importantly, this is going to happen at so many different rates of time.
11:42
What I mean by that is companies aren't going to suddenly upgrade their systems
11:47
to this brand new security parameter every single year, as he's hoping will
11:52
happen over the next year.
11:54
Companies are going to progress. They're very archaic. They're very bloated.
11:58
They're very bureaucratic.
11:59
You need policy sign-offs. You need
12:01
to kind of convince so many different stakeholders for this to happen.
12:04
What that means is there's going to be a long period of time,
12:08
which is likely going to be a couple of years at this point,
12:11
where anyone and everyone that gets access to cheap old generation models will
12:16
be able to probably perform similar attacks like this.
12:19
You don't have to take my word for it. You just need to look over our shoulder
12:22
at last week, where we saw one of the biggest cybersecurity incidents that happened
12:27
over 48 hours that marked the entire month a new nickname, Black October.
12:31
The very presumption that we need to now rebuild our entire software stack and
12:37
do so every year is bizarre for me to understand, but is, I guess,
12:42
necessary to understand this new world.
12:44
The immediate analogy that pops to my mind is those world-building games.
12:48
You know, the ones that you see on your mobile phone being advertised to you
12:50
or like Clash of Clans or whatever that might be, Civilization,
12:53
where you have to control multiple things and build a world you have to build
12:57
a fence a defense system to prevent
12:59
people attacking you but by the time someone's attacking on one side
13:03
someone's attacking on the other side and so you have to go there and deploy
13:05
resources and it's this whole constant 24 7 thing that's what the new world
13:10
is going to look like from a cyber security perspective so
13:13
the only way to combat this in my opinion is to spin up other ai agents to defend
13:17
against this and that's basically the point he's making he's saying.
13:20
Hey, you're probably going to need to pay for access to AI models to red team
13:25
your entire system before attackers find exploits themselves.
13:29
Now, there's obviously a self-serving reason why he might say this because,
13:33
hey, like we have the best models at Anthropic, use our models or use any open
13:37
source models or you open air will say the same thing about their models.
13:40
So I see that angle, but I fail to see how he's wrong.
13:44
We are working ourselves towards a world where at any given moment,
13:48
it doesn't matter if you're asleep, it doesn't matter if your
13:51
security team is the best of the best, humans have flaws, human handwritten
13:56
code, which is 99% of all the code out there also has flaws,
14:01
will mean that you will get eventually attacked.
14:03
And so you need to have a perpetual defense system that is not only defending,
14:08
but is being proactive and plugging any holes in your systems.
14:12
This doesn't just apply to Anthropic and their Slack exploit.
14:16
This applies to you and i who are consumers of everyday products any kind of
14:21
system that you use for leisure or for professional work is liable to this but
14:26
it's also applicable to you as a
14:28
business owner or a founder who might be building systems or building a new
14:31
startup this is something that could potentially happen to you as well so i
14:35
guess the reason why this
14:36
concerns me and why i wanted to make this episode is that
14:39
this has such a large attack surface
14:41
area and the cost of these things are going to ultimately be driven.
14:46
Super super low a point that was made around this whole thing is that they
14:51
likely used older generation models to do this and if i had to guess these older
14:56
generation models are at the same capability level of open source models and this is something that.
15:02
People like dario mode and sam altman have warned about happening very soon
15:06
okay so i want to get into a few final thoughts on today's episode brought to you by qualcomm.
15:15
The first is, this was kind of out of Anthropic's control.
15:19
They got access via a third-party platform and software. And listen,
15:22
I'm not saying Anthropic isn't liable for this. They need to work with these
15:25
third parties and companies to make sure that this doesn't happen again.
15:28
And from what I hear, they've been able to patch this up.
15:31
But the point being is, security is no longer a single siloed thing.
15:37
It's no longer just about you and what you can do, how much money you spend
15:40
and defense systems that you, walls that you build around your own product.
15:44
It's about the products that you use to help make your product and service actually function and work.
15:50
And if you look at the majority of software companies out there today,
15:53
this is pretty much all of them.
15:56
The world internet economy runs on companies being able to work with other companies,
16:01
automations, bots, systems that work 24-7 because they're supported by every
16:07
other single software system that you need.
16:10
And this was the exact flaw that we saw last week when we found out that AI
16:14
agents were apparently getting into people's bank accounts here in the US and
16:19
moving money around, changing transactions, getting access to personal information.
16:22
They were able to get access via a third party for credential information.
16:26
This is the same thing. So it's parallels of the same thing,
16:29
but happening in two different industries, which means that it's liable to pretty
16:32
much anything and everything. Imagine you're walking into the office of the
16:36
company that you want to hack into.
16:37
You steal the cleaning company, who is a third party's access key.
16:42
You get access to the, I don't know, the HR office, and you're able to go through
16:47
the filing cabinet and see everyone's salaries or look into the model room and
16:50
see what they're cooking up.
16:52
Now, the second point I want to mention is just the 24-7 nature of this entire attack.
16:58
Humans sleep. Humans are.
17:01
Imperfect, and they make mistakes sometimes, even the best of the best.
17:05
And also humans, experts specifically, are scarce. It's hard to find.
17:10
And so to have AI agents that are not just at the level of the expert humans,
17:16
but are starting to beat them, this becomes a problem, especially if it's 24-7.
17:20
And the idea here is, it's not just that it's happening all the time,
17:24
it means that they're stress testing any and every possible avenue to get in,
17:29
which means that the long tail of scenarios, which humans
17:33
don't typically spend a lot of time trying to figure out because they're like,
17:36
oh, it'll never happen, will actually happen.
17:38
And that's a new security vector that most people haven't really considered today.
17:43
Third thing is the attacker in this instance is very closely affiliated to a.
17:48
Huge government org, in this case being the Pentagon. They signed a new contract with the Pentagon.
17:54
They are, I think, maybe have some
17:56
very strong affiliation given that the co-founder was an ex-worker there.
18:00
And so we are weirdly unofficially one step closer towards nationalizing these labs.
18:06
Because if the threat was big enough, let's say it was an adversarial actor,
18:10
not a white hat hacker in this instance, and they got access to proprietary
18:14
model information that could then be used against the US, you could easily see
18:18
the part that leads towards, hey, we need to nationalize this lab.
18:21
Otherwise, we're going to end up in a huge pot of hot water and the final thing
18:25
i'll say is if you're china and you're reading the headlines today or of this
18:30
week of last week seeing all these security flaws.
18:33
Maybe you're smiling because not only are you aware that these things are capable,
18:38
but you're aware that these things are capable from the average older model.
18:43
Remember, you don't need frontier models. You don't need to spend thousands
18:46
and thousands of dollars anymore.
18:48
You can just spend a couple hundred bucks and get access to some of the most
18:52
important and crucial systems that run the world and the future world.
18:56
Anyway, those are my thoughts. I hope you enjoyed today's episode.
18:59
Thank you so much for listening.
18:59
If you're listening to this on YouTube, there are many new subscribers welcome it's great to have you,
19:06
if you aren't subscribed or if you haven't turned on notifications please do
19:08
so it helps us out massively
19:10
also we are on spotify and apple go check us out give us a rating leave us a
19:14
comment leave us comments everywhere by the way dm me on x my socials are linked
19:18
below and also if you think of a friend colleague or anyone that might find this interesting
19:23
please share it with them it helps us out massively and yeah
19:26
i think that's it i'll see you on the next one thank you for listening to today's
19:29
episode brought to you by qualcomm.