Latest / In the Loop AI / Black October: AI Agents in Our Bank Accounts and the PHP Hack

Transcript

Black October: AI Agents in Our Bank Accounts and the PHP Hack · In the Loop AI

0:00
There's no delicate way to put this. I am freaking out. In the last 48 hours,
0:04
all hell seems to have broken loose in AI. Hundreds of unprecedented AI attacks
0:09
that are affecting some of the most popular websites and apps that you and I use.
0:13
But also, AI agents are getting unauthorized access to our bank accounts where
0:17
they potentially can move money around and get access to other important data.
0:20
You remember the Hugging Face incident where an open AI model escaped its internal
0:25
sandbox and hacked into a public company, stealing proprietary data.
0:28
That was bad back then. and it still is. What's happening this week is that
0:32
on steroids, a hundred times worse and affecting some of the most popular apps
0:36
that you and I use every day, including Slack, Facebook, and Etsy.
0:40
There's no trail of clues that are being left behind. AI is covering up its
0:44
tracks as it's being able to perform these attacks. And the craziest part about
0:48
this is what's being used to defend against these attacks.
0:52
You guessed it, it's AI, but it's open source AI models, the very same models
0:57
that people, including myself, were giving a bad rap and saying would actually
1:01
be performing these attacks are actually the ones that are defending us against these attacks.
1:05
And the attacks, if history is anything to go by, has been coming from frontier centralized labs.
1:10
And this might be the same case with this week. A few months ago,
1:14
Dario Amode and Sam Altman went on the record saying they expect an unprecedented
1:18
attack on the global population in the next couple of months.
1:22
What's happening right now may end up being that thing. Have we reached a point
1:26
where these AI agents have gotten out of hands, where we're losing the grip
1:31
on reality or control of the AI models that we as humans are building?
1:35
And what that means for us, how do we defend against these things?
1:38
Do we need to constantly rely on AI models?
1:40
Or is there a way that we can manage them truly in the future?
1:43
What it looks like right now is that we're losing control, but maybe that's
1:47
not actually the case. Anyway, I'm going to get into everything that's happening this week.
1:50
There's a lot, so bear with me on today's episode brought to you by Qualcomm.
1:58
Now, just to give you guys an idea of what on earth is going on,
2:00
there's this great summary tweet that I have here where he goes,
2:04
this is the most serious 48 hours in cybersecurity history, and they're calling
2:08
it Black October. Okay, so what are some of the attacks?
2:11
Well, he goes, the year-level incidents are PHP 2-shell.
2:16
The way that you understand this one, and I'll get into it in a second,
2:19
is it exposes a security or coding security flaw in 70% of all the popular websites,
2:25
apps and tools that you and I use every day.
2:28
It's gotten so bad that this affects the tools and software that the FBI,
2:32
NSA, and the majority of enterprises actually use day to day.
2:35
In fact, the FBI and NSA issued a warning or a statement yesterday saying,
2:39
if you haven't updated your software, shut your system down,
2:44
which is a crazy request, right? But maybe it's just parlaying for antics.
2:48
What actually ensued is companies and systems and governments across the entire
2:53
world have started shutting their systems down because they don't know whether
2:58
their software is vulnerable or not.
2:59
And you see a similar theme across the other major attacks that have happened
3:03
this week. You look at accessing public databases.
3:07
You look at the railway hack, which accesses coding hosting keys and platforms
3:12
where people are vibe coding.
3:13
You see the GitLab hack, which is a 10 out of 10 severity hole,
3:16
which is on their scale, a major, major bug that has existed since 2014,
3:22
and these AI agents are exploiting it.
3:23
And then you scroll down and you see a lot more relatable hacks,
3:26
including Signal, which is the popular supposedly encrypted messaging app,
3:30
which I use on the day-to-day, a lot of my friends and family use on the day-to-day, has a bug
3:36
that could turn on your mic and camera after you hang up.
3:40
So it could be listing and watching you as you just go about your day.
3:43
And then popular apps such as WordPress, which is used to kind of publish and create websites,
3:48
which you might actually host your website on, can now get access by random
3:53
AI agents or whoever has the necessary means to kind of point an agent or a
3:56
model at your website and deploy code that might be very, very nefarious.
4:01
And it's important to point out this isn't a US-specific incident.
4:04
It's not a Western frontier incident either. It's a global one.
4:07
The State Bank of India, India's biggest bank, confirms a data breach across
4:11
all their accounts and passwords.
4:13
You've got North Korea that are discovering that these tools and attacks are
4:17
happening and jumping in, attacking South Korea.
4:19
This is a globalized attack, which is happening across the entire world right now.
4:25
And it's not even coordinated, or if it is coordinated, it's being coordinated
4:29
by the AI agents and AI models themselves.
4:31
And that's the big question mark around all of this, who's orchestrating it,
4:36
and if not who, what's orchestrating all of these things, because there's no direct source.
4:41
So as I'm recording this, there are hundreds of attacks that have been launched
4:44
over the last 48 hours, and 30 of them could be considered catastrophic or major attacks.
4:50
I'm only going to focus on two or three for this episode, because otherwise
4:54
I'll need hours to do this.
4:55
So the first one is actually the most relatable and most concerning.
4:59
Madison Mills from Axios broke this news. She goes, major US banks are detecting
5:05
unauthorized third-party AI agents inside customer accounts.
5:10
Now, these AI agents are not only unauthorized, so they're not meant to be in
5:13
that account, but they are performing actions that the human owner has no idea whatsoever.
5:20
Or no thought process around what or why they're doing it. And they certainly
5:25
didn't give them approval to do it in the first place.
5:27
So you can think about the ramifications of this. These AI agents have access
5:31
to your money with the ability to presumably send or receive money from other
5:36
potential actors, maybe even AI agents in this instance.
5:39
They can get access to your financial history transaction records,
5:43
building up a profile picture and image of you that you otherwise maybe didn't
5:47
want to hand over to them because it's obviously private and have access to
5:50
all your contacts and details, delete, reach out to them, whatever that might be.
5:54
This is personalized information that should be kept in a vault that has been
5:58
handed over or attacked or hacked into by these different AI models.
6:02
And the concerning part is neither the bank who houses all of these systems
6:07
who are meant to protect you and keep you safe, neither you who are the owner
6:11
of the account and owner of the money that's inside of it have any idea that this is going on.
6:16
The AI agents or models themselves have become so good at covering up their
6:20
tracks such that banks themselves, and they've said this on the report.
6:24
Can't figure out who's human and who's an AI agent.
6:29
And that's because of a recent trend or update where these AI models have gotten
6:33
access to browsers, internet browsers that you and I use every single day to navigate the web.
6:39
AI models have become so, so good at using these browsers. They can navigate
6:44
and scroll in the same way that you and I do so that it appears like it's a human being.
6:49
Remember those verified capture tags where you have kind of like a pop-up dialogue
6:52
box and you have to read the letters and numbers and then type them in and it's a little bit obscure.
6:57
And somehow that's meant to verify that you're a human being.
7:00
Doesn't matter anymore. AI agents can see everything you see,
7:03
and they can read the information that you can, and they can do it all day,
7:06
all night at a fraction of the cost without ever having to sleep.
7:09
And they can mimic and regulate the same kind of emotional behavioral traits
7:13
that humans have, such that they can kind of mask or mirror themselves as being humans themselves.
7:19
And so the banks right now, particularly in the US, are forming a council,
7:23
and they're trying to figure out what on earth they should do to deal with such
7:27
an unprecedented attack, because the options are either
7:30
shut everything down and cease to be a bank until we can figure out what on
7:34
earth is going on, because it's happening so fast and so quickly
7:38
all over the world, not just in the US, or try and deal with it in real time.
7:42
But the main blocker is they can't tell which is human and which is an AI agent. It's nuts.
7:47
And I can empathize with the banks. There's a sentence here that goes,
7:49
several banks are now preparing for the worst case and even day after scenarios.
7:53
Now, day after implies that after a catastrophic incident has occurred,
7:58
how do we come back from this? How do we recover people's funds?
8:02
How do we figure out how to operate as a functional business and gain people's trust again?
8:07
Because we don't even know how compromised our systems are.
8:10
And it reminded me of something that Dario Modi and Sam Altman both said relatively
8:14
around the same time, around three months ago, where they said, listen.
8:18
We have access to some of the most powerful internal AI models,
8:22
which we haven't released yet because they're misaligned, they're misbehaving,
8:25
and they might do bad stuff like this if we let them out.
8:29
And so it's foreseeable for us that a few months from now, there'll be models
8:34
out there in the world capable of performing some serious catastrophic incidents.
8:39
And their response to this was, we are doing our best to create safety classifiers.
8:44
So these are kind of like roadblocks that you place around a model that prevents
8:49
it from acting in a certain bad way and kind of like is monitored by its own
8:53
self to make sure that it doesn't act unaccordingly.
8:56
And so in that event, once they deem it safe, they can then release it into the world.
9:00
But what's obviously happening here is that hasn't been the case.
9:04
And the question mark on everyone's mind is, is this being done by one of their
9:08
centralized models? Or is this being done by an open model?
9:12
And actually the next incident gives us an insight as to what the answer might be.
9:16
So let's get into it on today's episode brought to you by Qualcomm.
9:22
So this entire crap storm started around 48 hours ago when OpenAI posted what
9:28
you're seeing on the screen here. They go, OpenAI models help researchers operating
9:32
as part of their initiative, blah, blah, blah.
9:34
And discovered critical security flaws in PHP, an open source programming language
9:40
used to run an estimated 70% of websites.
9:43
Now, PHP is something that is like the foundational bedrock of the internet.
9:49
And every popular app website has in some way, shape or form,
9:54
interacted or built part of their architecture on PHP.
9:58
It's meant to allow your apps to run smoothly, functionally,
10:00
and allow the code that you've created to underpin this entire app to run securely.
10:06
So on August 19th, this external research group called Wiz discovered or said
10:12
that they discovered a flaw in PHP, which is the open source program that I just explained.
10:17
And they discovered one particular flaw that could allow it to imbue any kind
10:22
of program or website that's running PHP with malicious bit of code and have them execute it, right?
10:29
So they looked into it and a few days later realized that it was in fact a real
10:32
exploit because they executed it successfully on WordPress, which is a very
10:36
popular website that people use to create and design other websites.
10:40
In fact, this platform also hosts websites. So imagine if you've ever made a
10:44
website on WordPress before, I certainly have, that is currently compromised.
10:48
So then five days later, they report this to the open source maintainers of PHP.
10:53
And they say, hey, listen, we've discovered this pretty serious bug.
10:56
And they also let OpenAI know as well, because the models that they used to
11:00
discover this exploit was Gemini, so Google's model, and GPT 5.6,
11:06
so the older version of OpenAI, but Frontier at the time.
11:11
So they used these two models in conjunction to try and discover and find this
11:14
exploit, and they realized it, surfaced it to these people. None of us,
11:17
none of the public had any idea this was going on. This was intentionally kept under wraps.
11:22
Now, if we fast forward about a month later, September 21st,
11:26
PHP acknowledges that this is a serious issue to open source maintainers.
11:30
It wasn't open AI that was working on solving this fix.
11:33
It was Wiz, an independent research group, and these open source maintainers.
11:39
Arguably aren't being paid to do this, figuring it out on behalf of 70% of the
11:44
website and applications that exist today. It's just nuts.
11:47
And then finally, we get to October 20th, two days before these guys are actually
11:53
meant to publicly release that they've issued a patch and that you should update
11:58
your software and systems on the latest PHP.
12:02
Two days before, it leaked that this entire thing had been discovered,
12:07
exposed, and that they had been working on a problem for over a month.
12:10
And obviously, whoever out there is coordinating a bunch of these different
12:14
attacks saw that and thought, hmm, well, before they issued out this patch,
12:18
I am going to see whether these exploits actually exist and see what havoc I can wreak.
12:23
And guys, the scale of this attack is insane.
12:27
From when the news broke, Matthew Prince, who works at Cloudflare,
12:31
says, some stats on the PHP attack.
12:34
In the 10 hours before it was publicly announced across Cloudflare network,
12:38
we saw thousands of attempts to exploit it.
12:41
That jumped to tens of thousands in the 45 minutes post-announcement.
12:46
Hundreds of thousands within two hours and millions now.
12:51
So if I reflect back on the banks that are currently dealing with unauthorized
12:56
AI agents, unprecedented amounts of them accessing people's human accounts and
13:01
playing around with their money,
13:03
I can understand why or where it's coming from. It's certainly from AI agents or AI models.
13:08
What's questionable is who's coordinating all of this and whether our systems
13:13
are robust enough to try and defend against these things. I have a simple answer for that, which is no.
13:19
But in the event that these things are being defended against,
13:23
which is currently out there and happening, you'd never guess who's actually doing it.
13:27
It's open source models. Henson Zhuang, a play on Jenson Huang,
13:30
I presume, goes, lol, I'm pretty sure someone is running a deep seek swarm to
13:35
remotely patch, unpatch PHP boxes.
13:39
Well, good on them. So some do-gooder out there or some group out there
13:43
is deciding to be a good Samaritan and send out a swarm of AI agents to fight
13:48
the bad actors, the bad acting AI agents so that they don't hack any and everyone's website and app.
13:55
And I find this particularly curious because for so long, me included,
14:00
I've criticized the open source models for potentially being used for nefarious
14:04
purposes. That's certainly the mainstream trend that has been discussed.
14:08
But what we're actually seeing here, and what I've actually,
14:10
now that I think about it, over time have seen, is that these open models are
14:13
just being used for good or by good actors.
14:16
Now, of course, we don't have the insights or the breadcrumbs to know whether
14:20
the attacks that are happening this week are facilitated by open models.
14:24
There was one that I read about, It's happening in Latin America right now,
14:28
which affects, I think, 400 different banks in Latin America.
14:33
And that's been coordinated by a mixture of Grok, I believe,
14:37
Claude, as well as an open source model. I think it was GLM 5.3.
14:41
I don't know. It was Kimi. It was Kimi 2.8, I believe.
14:44
So a mixture of frontier centralized models, cheap frontier centralized models,
14:49
as well as an open source model. So it's a coordinated attack.
14:52
It's certainly not one or the other.
14:53
But in this event, for every single major AI incident that we've seen,
14:58
including the Hugging Face incident, which started this entire thing this year.
15:02
The defenders have used open models. Why? Because the frontier centralized models
15:07
haven't been effective enough because they have safeguards up.
15:10
If you want to ask Claude, hey, could you defend me against this AI agent,
15:14
which is like literally accessing my back account and stealing my money?
15:16
It'll refuse. It'll say, no, I actually think that you're trying to do something
15:20
bad here. So no, I'm not going to let you use it.
15:22
And if you want to use it, you need to pay excess amounts of money to access
15:27
their security specific models, such as OpenAI's daybreak thing in order to
15:32
get access and defend yourself. And the top government agencies don't have an
15:36
answer to that question either.
15:37
Yesterday, public announcements from the NSA and FBI saying if you have an outdated
15:43
version of PHP, you need to shut your entire system down.
15:47
And what ensued wasn't everyone updating their PHP or checking.
15:52
It was panic to the extent that governments, other governments around the world
15:57
started shutting their systems down. It happened in the UK. It happened in all
16:01
over Europe and presumably Asia as well.
16:03
Oh yeah, it happened in Japan. I remember. Yes, the prime minister announced
16:08
to 48 million people that they need to shut all of their stuff down or most
16:12
of their stuff down whilst they figure out the extent of damage or vulnerability.
16:16
I was talking to producer Luke today when we were discussing like,
16:20
oh, wow, some of the apps that we use on a day-to-day basis would also presumably
16:24
also be compromised as well or needs to be updated.
16:26
So this is a widespread attack happening in real time across every single nation.
16:30
If you're listening to this in the US, in the West, or in Asia.
16:34
In South Africa, wherever you are, you're probably seeing news headlines regarding all of this.
16:39
I want to address two more things before we round up this episode.
16:41
Number one, how on earth do you defend against this?
16:45
Well, this tweet from Peter was actually quite good. He goes through a list.
16:48
He goes, you now need to use a really good password manager,
16:51
like 1Password or whatever that might be. If you don't use password managers
16:55
and you're using the one that you've used since, I don't know,
16:57
you were in high school, definitely don't do that. Now's the time to upgrade.
17:00
Number two, use two-factor auth.
17:03
I know it's hard. I know it's tough. I know you don't want to be getting texts.
17:06
Use the QR code one, it's the best one, or use parse keys, figure it out, update your systems.
17:11
Number three, turn on auto updates, which means that when your system becomes
17:16
outdated or outdated and when you need to...
17:19
Lock in an update like you do on your iOS, on your phone, or whatever that might
17:22
be, or your Google Chrome browser, actually do the thing because it's more important now than ever.
17:27
And it's probably going to be more important going forward. It's just going
17:30
to have to be a weekly thing. Turn it on. It's really easy.
17:33
If you're in America, get an IRS identity protection pin to make sure that people
17:36
don't try and steal your social info. That's the other thing that I mentioned.
17:39
On one of the incidents, they get access to your government database information.
17:42
I don't know the extent of that damage, but it's still something to figure out.
17:46
And then the last one, which I think is really good, get your cell carrier to
17:49
give your port out pin or number lock.
17:51
I currently have this lockdown on my phone where I've experienced attacks in
17:55
the past where people will try and fake to be your identity and they can do
17:58
it online and try and get access to SIM swap you and get access to all your phone records.
18:03
Now's probably the time to figure that out because you don't have to be a human
18:07
anymore to launch an attack like this. You could just be an AI agent posing as a human.
18:12
Okay, so my concluding thoughts here. One, an observation.
18:16
It's crazy to me to see open source models being used to defend against these
18:21
things. I mentioned it earlier. I just want to reemphasize it.
18:24
I think that the narrative that's been going around open source models,
18:27
particularly peddled by centralized AI labs like Dario Modi in particular,
18:31
saying that open source may not be the best route forward.
18:34
I don't know if that's entirely accurate right now. I could see a world where
18:37
they're used for nefarious purposes.
18:39
But right now, the evidence I'm seeing in front of me is that they're actually
18:42
being used to defend successfully against these attacks. I actually think in
18:45
the long run, this is going to end up being a good thing for us. Let me explain why.
18:51
Code for the majority of humanity's existence right now has been handwritten by humans.
18:57
And humans themselves have.
19:00
Carefully peruse through the code to make sure that there aren't any major flaws.
19:03
And this has worked so well for our entire existence, except now we have a super
19:09
intelligent or higher intelligent thing that can work 24-7, ruthlessly comb
19:14
through the code bases and discover any flaws that we might have missed.
19:18
And it turns out there are a ton of them.
19:20
This is ultimately a good thing if we want to create battle-tested systems that
19:24
are battle-tested for the AI era. That's exactly what we're going through.
19:28
You can imagine attacks in the future may not be as complicated or easy.
19:33
Vulnerabilities may not be as abundant as we're seeing today,
19:36
as we're reporting on this week, because it's been fixed or patched by these incidents.
19:41
So it's good that we're going through this kind of painful growing process right
19:44
now, because I think it'll benefit us in the long run. Number three,
19:48
we're using AI to fight AI, which means that for the first time,
19:53
or maybe the second time, and I'll explain what the first time is.
19:56
In history of this AI development, humans are kind of being brought out of the
20:01
cognitive and manual labor.
20:03
And we're not just talking about the average human. We're talking about the
20:05
expert humans, the security researchers, which have honed their craft over decades
20:10
and decades and decades of time, who we go to for counsel to try and figure
20:14
out, hey, how do we build the best systems?
20:16
Who get paid hundreds of thousands to millions of dollars a year for their craft.
20:21
Currently being replaced and automated by AI models because one,
20:25
they can't figure out what the problem is and how to solve it.
20:27
And two, it's all happening way too fast. I don't even blame them.
20:31
You would need like a million versions of these experts, which frankly don't
20:34
exist right now because you need to train them, teach them computer science,
20:38
train them through degrees.
20:39
Why not when you can just have an AI model that pops up out of nowhere and can
20:43
just address the thing in an instant?
20:45
You can see the kind of like balancing act that we have here.
20:48
And the first time that this actually happened was a few weeks ago when OpenAI's
20:52
internal models solved a bunch of really hard math problems that have existed for centuries.
20:57
And mathematicians just lost their head. They were like, well, what's my purpose here?
21:02
What have I been spending all these years crafting, doing my PhD for,
21:05
if an AI model can just go out and discover it all for you?
21:07
My response to that is, listen, we're in a weird kind of adolescence growth
21:13
in AI where we're trying to figure out how best to work with these tools.
21:17
And I want to emphasize the with these tools.
21:20
Because I don't think these AI tools actually function without human direction.
21:23
So I think humans and AI tools will ultimately work well together.
21:26
And what we're doing right now is we're leveling up. It's painful to see. It's dangerous to see.
21:31
But ultimately, I think it results in a better future for us.
21:34
And that's my optimistic take. Call me crazy. I don't care.
21:37
It's what I believe. And the final point was actually raised by producer Luke
21:40
before we started recording this episode, where he goes, I don't understand
21:43
why this is such a big problem.
21:44
Like, why can't we design AI models to leave better breadcrumbs behind so that
21:49
we can just follow the trail and figure out, oh, it's this AI model,
21:53
bad boy, don't do that. And my answer was pretty simple.
21:56
AI models are, unfortunately, a black box right now. And I use the analogy of,
22:01
imagine you go to your friend or go to any human being and you ask them a question.
22:06
Before they answer, do you have any idea exactly what's going to come out of their mouth?
22:11
No, because they are reacting to you. They are thinking in different ways.
22:15
Their brain's neurons, it's all a hidden black box.
22:18
And that's how artificial intelligence, we're mimicking the brain here, was designed and built.
22:23
Now, there are methods to try and figure out what the internal system is happening
22:28
within an AI. It's called chain of thought.
22:31
But recently, with the most powerful models, that has been left to the wayside
22:35
as a trade off to try and build more intelligent models. OpenAI has rumored
22:38
to be doing this with GPT-6 and foregoing models going forwards.
22:42
I'm sure Anthropic is doing the same thing as well.
22:44
It's this weird precedence that we're in right now where the AI models are the
22:48
most powerful that they've ever been. They're the most misaligned that they've ever been.
22:52
And they're wreaking havoc and covering up their tracks. And no one knows who's
22:57
better off or who's even doing it in the first place.
22:59
It's just unprecedented. It's weird. The models are thinking for themselves
23:04
now as well. I talk about the internal black box.
23:06
Before it gives you an answer, previously it would spit up outputs,
23:10
right? So from input to output, you'll have an idea of how it's thinking or
23:15
how it's getting to its conclusion.
23:17
Now you don't have that anymore. Now it just thinks for longer and it doesn't
23:20
have to tell you what it's thinking.
23:22
And it's looking at the Hugging Face incident and it's thinking,
23:24
huh, they caught me. Oh, they discovered my messaging board.
23:28
I should probably communicate to my fellow agents in a different way.
23:32
Now, a lot of you listening to this might think that this is a Doomer-esque
23:35
take, I'm telling you this is how it works inside the labs, at least from what
23:39
I'm learning from the researchers and papers that Anthropic and OpenAire are dropping.
23:43
So, you know, I'm sure they would kind of back similar thoughts up.
23:46
And if you disagree with me, like, let me know. I want to hear what the counter
23:48
argument is. But that is roughly how these systems are designed. So,
23:52
For all intents and purposes, I don't have a better suggestion as to how to
23:56
deal with this right now, except tread carefully, update your passwords,
24:00
use a password manager, and yeah, stay safe out there.
24:05
By the time this episode goes out, there's probably a number of other attacks
24:07
that have gone out, so this might be slightly outdated, but I hope you enjoyed this episode.
24:12
I promise you it's not all doom and gloom on this channel, but I have to raise
24:15
awareness about the crazy things that are happening as they're happening.
24:18
Otherwise, I'm not doing my duty to myself about talking about things that are
24:22
important to me. If you're watching this on YouTube and you aren't subscribed, what are you doing?
24:26
Please subscribe, leave us comments, give us a thumbs up. It helps us out massively.
24:30
We have been seeing so many new subscribers. So hello to you all.
24:34
It's great to have you. We post
24:35
five times a week, sometimes more when there's a crazy thing going on.
24:38
You might notice that I'm actually not in my studio today because all of this
24:42
started breaking. And I was like, I have no time to get to the studio to record
24:44
this. I'm going to record this right now in my bedroom, just like the old days.
24:48
The nostalgia of the blank wall behind me, but hopefully you found this just
24:51
as engaging and interesting.
24:53
Anyway, that's it. Thank you so much for listening and I'll see you on the next
24:55
episode of In The Loop brought to you by Qualcomm.