Latest / Cybersecurity Under Pressure. Real Attacks, Real Lessons / OT Recovery: Proving a Safe Restart Beyond the Backup
Transcript
- 0:45Welcome to cybersecurity under pressure, real attacks, real
- 0:48lessons. Today is Monday, June 15th,
- 0:512026. This story starts with a routine
- 0:54check and ends with someone losing their job and their
- 0:58sleep. Yeah, a lot of sleep.
- 0:59I mean, probably month of it honestly.
- 1:01Exactly. So I want you to imagine, just
- 1:03for a second, that you are holding a standard old fashioned
- 1:06stopwatch. You click the button to start
- 1:08it. One second passes.
- 1:10You just lost $600.00. You blink.
- 1:12Another second passes. That is $1200 gone.
- 1:15You wait one single minute to go.
- 1:17I don't know, pour yourself a cup of coffee.
- 1:19You were down $36,000. Which is a pretty expensive cup
- 1:22of coffee. Right.
- 1:23And if you let that stopwatch run for exactly 1 hour, you have
- 1:26effectively incinerated 2 point $3,000,000.
- 1:29It's a genuinely staggering burn rate.
- 1:32I mean, it sounds made-up. It does, but the terrifying part
- 1:34is that this isn't some theoretical math exercise.
- 1:37That's a real number affecting real people right now.
- 1:40Yeah, it is that $2.3 million an hour is the exact calculated
- 1:44cost of unplanned downtime for a modern automotive manufacturing
- 1:48plant today. 2.3 million. An hour.
- 1:51Every single hour. But here is the thing that
- 1:53completely rewrites how we you know how we have to look at this
- 1:56stuff? That massive factory doesn't
- 2:00have to be hit by a Category 5 hurricane or suffer some like
- 2:05catastrophic mechanical failure to stop running.
- 2:08All it takes is the wrong piece of invisible code crossing a
- 2:11digital boundary. Which is wild because usually
- 2:15when we talk about this, and welcome to the deep dive, by the
- 2:17way, we're looking at a massive paradigm.
- 2:20Shift. Oh, absolutely.
- 2:21For the last three decades, when the average person or even the
- 2:24average executive talked about cybersecurity, they were almost
- 2:27exclusively talking about information technology IT.
- 2:31Yeah, the digital stuff. We were trained to think about
- 2:34protecting data. You know, your personal emails,
- 2:37your credit card numbers, a corporation's intellectual
- 2:39property. The servers hosting a website,
- 2:42cloud storage environments. Right.
- 2:43The stuff that essentially lives trapped behind glass.
- 2:46It's on a screen. And if it gets stolen, I mean,
- 2:48it's a huge headache, It's a privacy violation.
- 2:50It might cost a lot of money in lawsuits and fines, but it
- 2:54remains a digital problem. It stays behind the glass.
- 2:58But today, in this deep dive, we are stepping out from behind
- 3:01that glass. We are crossing over into
- 3:04operational technology OT. The physical world.
- 3:07Exactly. This is the realm where computer
- 3:10code reaches out and manipulates the physical world.
- 3:12The actual kinetic movement. Yes, we are talking about the
- 3:16microprocessors that physically open and close massive steel
- 3:20valves in municipal water treatment plants.
- 3:23We're talking about the systems that regulate the boiling
- 3:25temperature of like volatile chemical vats, the controllers
- 3:30that route high voltage electricity across the national
- 3:34power grid. And those robotic arms on the
- 3:36assembly lines? Right, the ones burning millions
- 3:39of dollars an hour when they suddenly freeze up because of a
- 3:41rogue command over a remote access portal.
- 3:43The. Stakes are incredible, probably
- 3:45high. I mean the invisible digital
- 3:46borders, keeping our physical world safe or just quietly
- 3:49collapsing. So today, our mission for this
- 3:52deep dive into our source material is to look at exactly
- 3:55how and why that has happened. We have to unpack it.
- 3:57You're going to pull apart the mechanics of how adversaries are
- 4:00evolving from just stealing data to actively targeting physical
- 4:05machinery. We're going to break down the
- 4:07anatomy of a real world hack that cost the global economy
- 4:12nearly £2 billion. The JLR incident, yeah, it's
- 4:16brutal. It's insane.
- 4:17And most importantly, we're going to explore what it
- 4:20actually takes to survive a digital attack that has real
- 4:24world kinetic consequences. And to do that properly, we
- 4:28really have to start by tearing down a lot of preconceived
- 4:30notions. Because if you take the absolute
- 4:33best, most rigorous IT security practices in the world.
- 4:37Like the playbooks used by top banks?
- 4:39Yeah, top banks, big tech companies.
- 4:41If you take those playbooks and you simply drop them onto a
- 4:44factory floor, you don't just fail to secure the factory, you
- 4:47can actually cause a physical disaster yourself.
- 4:49OK, wait, help me understand that.
- 4:51Because on the surface, you know a computer is a computer.
- 4:54Right, that's the trap. Why can't we just take the
- 4:56firewalls, the antivirus software, the routine security
- 5:00scans we use for our corporate laptops and just plug them into
- 5:03the systems running a chemical plant?
- 5:05Because it comes down to a fundamental clash and
- 5:07philosophy. It's completely backward.
- 5:10In the traditional IT security world, the absolute gold
- 5:14standard guiding every single decision is known as the CIA
- 5:18triad. Right confidentiality,
- 5:20integrity, and availability. Exactly, and with
- 5:23confidentiality sitting right at the absolute top of the pyramid.
- 5:26Always. I mean, let's look at a major
- 5:28bank. Their primary non negotiable
- 5:30directive is to keep their customer's financial data
- 5:33secret. Yes, if the bank's online portal
- 5:36goes down for an hour, which is a loss of availability,
- 5:39customers will definitely be annoyed.
- 5:41They'll complain on social media for sure.
- 5:43Right. But if the bank's database is
- 5:45breached and millions of account numbers are published on the
- 5:47dark web, which is a loss of confidentiality, that is an
- 5:50existential crisis for the business.
- 5:52It's game over. So therefore every IT system is
- 5:55engineered to aggressively protect secrets.
- 5:58So if a security program detects something suspicious on a
- 6:01server, the immediate reflex is just to kill it.
- 6:04Kill it instantly. The standard automated IT
- 6:07playbook is rapid isolation. You quarantine the infected
- 6:11machine, You sever its network connection.
- 6:13You lock the remote access users out.
- 6:16You protect the sensitive data at the expense of keeping the
- 6:19machine running. OK, so that is the IT mindset,
- 6:22CIA confidentiality first. What happens to that philosophy
- 6:26when we step onto the factory floor, like into the OT space?
- 6:30In operational technology, you have to take that entire
- 6:32triangle and completely inverted the model becomes AIC.
- 6:36Availability first, then integrity and confidentiality is
- 6:39what last. It's a distant, almost
- 6:41irrelevant third. Availability is the absolute,
- 6:45unquestionable priority. Because the machines have to
- 6:47keep moving. They have to because in OT we
- 6:49are no longer dealing with spreadsheets, you know, we are
- 6:51dealing with physics, chemistry, thermodynamics, momentum.
- 6:55Real world forces, yeah. If you are managing a blast
- 6:58furnace or a high pressure natural gas pipeline, the
- 7:01continuous uninterrupted oeration of that system, its
- 7:04availability, is very often the only thing keeping the physical
- 7:07process from destroying itself. Man, I'm trying to picture the
- 7:11real world consequence of this. Let's say we put that aggressive
- 7:15banking IT security software onto the computer controlling a
- 7:19chemical reactor. The outcome could literally be
- 7:23lethal. Imagine a scenario where the
- 7:25chemical reaction is exothermic. Meaning it generates its own
- 7:28heat. Right, and the computer is
- 7:30constantly adjusting a cooling valve to keep it from boiling
- 7:33over if your aggressive IT security software suddenly
- 7:37detects a strange network ping from, say, a vendor's remote
- 7:42access tunnel, and it decides, hey, that looks like malware and
- 7:45automatically quarantines that control.
- 7:47You don't just get an error message on a screen.
- 7:49No, the controller is locked out, which means it stops
- 7:51adjusting the cooling valve. And the chemical reaction
- 7:54doesn't care that your antivirus software is running a scan.
- 7:57Exactly. It doesn't care.
- 7:58The heat builds, the pressure rises, and because the
- 8:01controller is isolated from the network, the human operators in
- 8:05the control room might not even see the pressure spiking and IT
- 8:09failure results in data loss and OT failure results in an
- 8:13environmental disaster or loss of human life.
- 8:16That makes perfect sense in those extreme scenarios, but I
- 8:19want to push back a little here. Surely there are routine IT
- 8:23maintenance tasks that factories like have to do.
- 8:28You'd think so, yeah. What about just scanning the
- 8:30network for vulnerabilities in a corporate office?
- 8:34IT departments run automated scans constantly.
- 8:36They ping every device, ask it what operating system it's
- 8:39running, check to see if any weird digital doors are left
- 8:42open. What is the harm in just looking
- 8:45at the network? It sounds harmless, I know,
- 8:47until you look at the actual hardware running a factory.
- 8:50OT environments rely heavily on devices called Programmable
- 8:53Logic Controllers or PLCS. These are not modern laptops
- 8:57with multi core processors. They are essentially ruggedized
- 9:00industrial pocket calculators. Pocket calculator?
- 9:03Seriously. Yeah, basically they are
- 9:05designed to do 1 highly specific physical task incredibly well,
- 9:09like reading a temperature sensor thousands of times a
- 9:12second. And they do it using hardware
- 9:14that might be 20 years old. So they aren't exactly packing a
- 9:17lot of processing power to handle extra tasks.
- 9:20They have almost none to spare. Their entire CPU is dedicated to
- 9:24maintaining the physics of their assigned task.
- 9:26Their network communication stacks are notoriously fragile.
- 9:30So if you ping them too much. If you hit a legacy TLC with a
- 9:33standard IT network scan which sends, you know, thousands of
- 9:37unexpected queries in a matter of seconds, you will completely
- 9:40overwhelm its tiny processor. It just panicked.
- 9:43It completely panicked, the processor gets flooded with
- 9:46these, IT requests, it runs out of memory and it drops its
- 9:49primary physical task to try and answer them.
- 9:51The PLC crashes or it forces a fail safe physical shutdown.
- 9:55Wow, so you have essentially D dos your own factory floor just
- 10:00by trying to check its security posture?
- 10:02Yeah, you took yourself offline. Good job it.
- 10:04Okay so active scanning is out, but what about patching?
- 10:07I mean, in the IT world, Patch Tuesday is a religion.
- 10:10Yeah, it's mandatory. Microsoft or Apple finds a
- 10:13security flaw, they push out a fix, and millions of computers
- 10:17automatically update and reboot. I'm struggling to see how
- 10:20running unpatched vulnerable software in a critical
- 10:23infrastructure facility isn't just like, gross negligence.
- 10:27How do compliance auditors even allow them to skip patches?
- 10:30Well, they don't skip them entirely, but they have to
- 10:32approach them with extreme caution because an OT system is
- 10:36a finely tuned orchestra of timing.
- 10:39You cannot auto deploy a Tuesday patch to a robotic arm moving
- 10:43heavy car chassis on an assembly line.
- 10:45Because of the reboot required. Not just the reboot, The patch
- 10:48itself alters the code. Even a routine operating system
- 10:52update can change how a processor handles timing by a
- 10:54fraction of a millisecond. Wait, a millisecond matters.
- 10:57In OT, absolutely. In an IT environment, if Excel
- 11:00opens 2 milliseconds slower after an update, a human being
- 11:03literally cannot perceive it. But if a robotic welding arm
- 11:07receives its command 2 milliseconds late.
- 11:09It misses its mark. It misses its mark.
- 11:11It collides with the chassis and destroys $1,000,000 worth of
- 11:14equipment. Oh man, it sounds like applying
- 11:17IT security to OT is like trying to perform mandatory brain
- 11:21surgery on a marathon runner while they are mid Sprint.
- 11:25You can't just pause their heart to install a new driver.
- 11:28That is a very, very accurate way to visualize it.
- 11:31Patching requires scheduled downtime, immense engineering
- 11:35analysis, and running the patch in a simulation environment for
- 11:38weeks before it ever touches the live factory floor.
- 11:42OK, I hear you. The internal OT systems are
- 11:44incredibly fragile. They can't handle security
- 11:47scans. They can't run modern antivirus
- 11:50and they are incredibly difficult to patch.
- 11:52If I'm listening to this and I'm running a manufacturing plant,
- 11:54I'm screaming at my dashboard right now.
- 11:57Just unplug the factory from the Internet.
- 11:59The classic solution. If the systems are that
- 12:02delicate, why on earth are they connected to the outside world
- 12:04in the 1st place? I always hear about the air gap.
- 12:06Keep the factory isolated and you solve the problem.
- 12:09The air gap is the most dangerous, pervasive myth in
- 12:12modern industrial security. It's a ghost.
- 12:15A myth, but it used to be real, right?
- 12:17It used to be reality. Sure, perhaps 30 years ago a
- 12:21factory floor was a genuinely closed electrical loop.
- 12:24But the global economic drive for extreme efficiency, data
- 12:27analytics and what the industry calls Industry 4 Point O has
- 12:31permanently demolished the air gap.
- 12:33Industry 4 point O, meaning the digitization and networking of
- 12:38manufacturing. Yes, the pressure to connect
- 12:41everything is immense. Aceo, sitting in a boardroom in
- 12:44New York, wants to pull out their iPad and see the real time
- 12:47production metrics of a facility in Germany.
- 12:49Which requires a data bridge. Exactly.
- 12:51And then you have the vendor who sold the factory a multi
- 12:54$1,000,000 turbine. They demand remote Internet
- 12:58access to the machine so they can perform predictive
- 13:00maintenance and monitor wear and tear from their own
- 13:02headquarters. Oh, so this is a third party VPN
- 13:04tunnel going straight into the factory?
- 13:06Yes, that requires AVPN bridge. Supply chain chains are now so
- 13:10deeply integrated that the moment a bin of screws gets low
- 13:13on the factory floor, an automated signal is sent through
- 13:16the cloud to a supplier to ship more.
- 13:18So the plant's internal fragile systems are constantly talking
- 13:22to the corporate IT networks, which are connected to the open
- 13:24Internet. Inevitably, yes.
- 13:26We see engineering workstations directly connected to enterprise
- 13:30IT domains. We see SCATTA telemetry.
- 13:34Scatta, meaning the massive software systems that human
- 13:37operators use to visually monitor the plant.
- 13:40We see those systems pushing raw data directly into third party
- 13:44cloud analytics platforms. The bottom line is that the air
- 13:47gap is gone. It's just Swiss cheese now.
- 13:49It really is. Every single efficiency tool,
- 13:52every remote maintenance port, every cloud dashboard creates a
- 13:56digital bridge between the relatively exposed corporate IT
- 13:59network and the highly sensitive physical OT network.
- 14:03Which means if an attacker compromises a board accountant's
- 14:06laptop in the HR department or cracks A vendor's remote access
- 14:09port portal, there might be an invisible pathway straight down
- 14:13to the chemical reactors. And adversaries have spent the
- 14:15last five years figuring out exactly how to map and exploit
- 14:19those specific pathways. Which is terrifying.
- 14:21Let's talk about those adversaries because Chapter 2 of
- 14:25our deep dive here. The data on the current threat
- 14:28landscape from the Drago's report is deeply unsettling.
- 14:32It's a whole new world out there.
- 14:34We are no longer talking about individuals in basements trying
- 14:36to steal credit card databases to sell on the dark web.
- 14:40The attackers targeting physical infrastructure have evolved into
- 14:43highly coordinated, specialized ecosystems.
- 14:46Yeah, we are tracting a maturation in the adversary
- 14:49landscape that honestly mirrors the complexity of a Fortune 500
- 14:52company. Like they have HR departments.
- 14:54Basically, these are not isolated actors, they are
- 14:57syndicates with distinct departments.
- 15:00Threat intelligence analysts like the team at Drago's, track
- 15:03these groups using specific naming conventions, often based
- 15:06on minerals, to differentiate their unique tactics and goals.
- 15:10OK. Help me understand how these
- 15:11groups operate because you mentioned they are specialized.
- 15:13Do we have different groups handling different parts of the
- 15:16attack like contractors? We do.
- 15:19It's a gig economy for cyber war.
- 15:20Consider a group tracked under the designation Sylvanite.
- 15:24Their specialty is the Vanguard Operation the Breach.
- 15:27Right. They are experts at exploiting
- 15:29standard IT vulnerabilities like an unpatched router or
- 15:33exploiting a weak remote access VPN password to establish a
- 15:37solid persistent foothold inside targets corporate network.
- 15:41They kick the door open and hold it, but they don't necessarily
- 15:44execute the final destructive attack on the physical machinery
- 15:47themselves. Instead, they operate almost
- 15:49like an initial access broker. They sell the access.
- 15:52Yeah, they secure the IT foothold and then hand off that
- 15:56access to a much more specialized group known as Volt
- 15:59site. A literal handoff or relay race
- 16:02for cyber warfare. And Volt Site is deeply
- 16:04concerning because they possess what we call Stage 2
- 16:07capabilities in the industrial control systems Kilchain.
- 16:10Stage 2 meaning what? They are not interested in the
- 16:14IT networks billing data. Once they get the baton from
- 16:17Sylvanite, they push deep into the OT environment.
- 16:20They actively manipulate engineering workstations.
- 16:23Go for the throat. They do.
- 16:25They silently dump configuration files to understand how the
- 16:28factory is wired. They pull alarm data to see what
- 16:30the human operators are currently monitoring.
- 16:33So they are stealing the blueprints to the physical
- 16:35processes. They are doing their homework on
- 16:37the physics of the target. They are investigating what
- 16:40specific digital commands will force a physical valve to stick
- 16:44open, or what sequence of code will override a temperature
- 16:48warning. That is a terrifying level of
- 16:50reconnaissance. I mean, it's one thing to steal
- 16:52a password, it's a completely different level of
- 16:55sophistication to study engineering schematics so you
- 16:58know exactly which valve to manipulate to build up lethal
- 17:01pressure. And they are not alone.
- 17:04Another group, Camasite, has been observed systematically
- 17:07mapping control loops across vast swaths of US critical
- 17:12infrastructure, performing massive, quiet reconnaissance,
- 17:16just waiting, just mapping it out.
- 17:17Meanwhile, groups like Pyroxyny have escalated to actual
- 17:21destruction during region's conflicts, deploying custom
- 17:24wiper malware. Wiper malware, meaning it
- 17:26doesn't hold anything for ransom, it just destroys.
- 17:30Its sole purpose is to permanently erase the logic on
- 17:32the controllers and brick the systems, causing maximum
- 17:35physical disruption and requiring total hardware
- 17:38replacement. You have to buy new machines.
- 17:41Wow. You mentioned ransom.
- 17:44The statistics surrounding industrial ransomware are just
- 17:46exploding. We are seeing a nearly 50%
- 17:49increase in attacks year over year, impacting thousands of
- 17:53industrial organizations globally.
- 17:55It's everywhere. But there's something I'm
- 17:57struggling with here. When I think of ransomware I
- 17:59think of a pop up on a screen saying pay $1000 in Bitcoin or
- 18:03we delete your file. Right, the classic IT
- 18:06ransomware. Why is that such a massive
- 18:08threat to a factory? Isn't ransomware just an IT
- 18:11problem? If my e-mail server gets
- 18:13encrypted, the robotic arm on the floor can still physically
- 18:17Weld the car door. Right, you are echoing a very
- 18:20common and very dangerous misconception held by a lot of
- 18:23corporate executives right now. They assume that ransomware only
- 18:27encrypts Windows files, spreadsheets and databases.
- 18:30Which it does. It does.
- 18:31Therefore, they think the physical plant which runs on
- 18:34those specialized legacy PLCS we discussed should be perfectly
- 18:38fine. Right.
- 18:38The PLC is an industrial calculator.
- 18:41It doesn't run Windows. It shouldn't care about Windows
- 18:43ransomware. The PLC itself might not run
- 18:45Windows, no, but we have to look at how human beings interact
- 18:49with that PLC. You don't have engineers walking
- 18:52around the factory floor plugging keyboards directly into
- 18:55hundreds of tiny PLCS. That would be absurd.
- 18:58It would all of those PLCS feed their data up to centralized
- 19:02human machine interfaces, Hmis, and massive engineering
- 19:06workstations in a comfortable air conditioned control room.
- 19:09The big screens showing all the dials, gauges and pressure
- 19:12levels. Exactly.
- 19:13And those Hmis and engineering workstations almost exclusively
- 19:16run on standard commercial Windows operating systems.
- 19:20They are just as vulnerable to standard ransomware as an
- 19:23accountant's laptop. Oh oh, I see where this is
- 19:26going. If a ransomware group reaches
- 19:28the network and encrypts the Windows based HMI, the
- 19:31underlying PLC down on the floor might still be mechanically
- 19:35running the robotic arm or pumping the chemical.
- 19:37But the human operator sitting in the control room is now
- 19:40completely blind. Yes, their screens are locked.
- 19:42They cannot see the pressure levels.
- 19:43They cannot see the core temperature.
- 19:45They cannot send new commands to slow the process down.
- 19:48The collateral damage immediately bleeds from the IT
- 19:51software layer straight into the physical operations layer.
- 19:55And we are seeing adversaries build custom tools specifically
- 19:58for the scenario. There is a ransomware variant
- 20:01known as Termite built using leaked source code.
- 20:04Termite. Yeah, it has aggressive of
- 20:07custom coded capabilities designed to specifically hunt
- 20:10down and terminate industrial processes running on those
- 20:14Windows machines before it encrypts them.
- 20:16So it knows what an OT environment looks.
- 20:18Like it knows exactly what it's looking at.
- 20:20It ensures that the operator is totally disconnected from the
- 20:22physical process before they even realize what's happening.
- 20:25I want to ground this in reality because talking about
- 20:29theoretical malware variants can feel, you know, abstract.
- 20:33Let's look at how this actually manifests in the wild.
- 20:36I want to talk about the Colonial Pipeline incident from
- 20:382021 A. Classic.
- 20:40This feels like the textbook case study of how an IT problem
- 20:43physically paralyzed half the United States.
- 20:46Colonial Pipeline perfectly illustrates the danger of the
- 20:49ITOT convergence we've been discussing.
- 20:52To understand what happened, we have to look at the access
- 20:54point. How they got in?
- 20:56Right, the attackers didn't use some impossibly complex 0 day
- 21:00exploit. They found a single compromise
- 21:03pass word for a legacy VPN account.
- 21:05A virtual private network, basically a remote access
- 21:09digital tunnel into the companies internal network.
- 21:12A tunnel that was meant for employees to access the
- 21:16corporate IT systems from home. The attackers walked right
- 21:19through that remote access tunnel and deployed ransomware
- 21:22strictly on the IT side of the house.
- 21:24They encrypted the corporate billing systems, the ticketing
- 21:26systems, the business operations data.
- 21:29Yep, but they didn't actually hack the pipeline itself.
- 21:32They didn't touch the OT network or the pumps moving the fuel.
- 21:35Wait. They didn't.
- 21:36No, they didn't need to think about what a pipeline billing
- 21:39system actually does. It isn't just generating
- 21:42invoices at the end of the month.
- 21:43It is a highly complex metering system.
- 21:46It tracks the exact volume of liquid, passes through specific
- 21:49physical checkpoints in real time, assigning financial value
- 21:52based on fluctuating commodity markets and determining exactly
- 21:56whose fuel is going where. So without the IT billing
- 21:58system, Colonial Pipeline was flying blind commercially.
- 22:01Totally blind, they didn't know who they were pumping fuel to,
- 22:04how much was going where, or how to legally bill for the
- 22:07transfer. Furthermore, there was massive
- 22:10internal panic. The executive and the security
- 22:13teams did not have total confidence that the ITNOT
- 22:16networks were perfectly segregated.
- 22:18They. Didn't know if the air gap was
- 22:19real. Exactly.
- 22:21They feel the ransomware might discover one of those invisible
- 22:23bridges and crossover into the physical control systems.
- 22:26So they made a choice. Out of an abundance of caution,
- 22:30Colonial proactively ordered the physical shutdown of the
- 22:33pipeline operations themselves. They stopped the pumps.
- 22:35Wow, a purely IT compromise. A hacked billing network via a
- 22:41remote access VPN caused a massive physical fuel shortage
- 22:45across the entire US East Coast. Leading to panic buying, gas
- 22:49stations running dry and grounded airplanes.
- 22:51That is a staggering, cascading effect from a single stolen
- 22:55password. But as bad as Colonial was, you
- 22:57know the attackers were just after money.
- 22:59They weren't trying to blow the pipeline.
- 23:00Up. No, they were just greedy.
- 23:02Right, I want to look at a darker escalation.
- 23:04Let's talk about the Triton attack in Saudi Arabia in 2017.
- 23:08Triton Triton represents A fundamental crossing of the
- 23:11Rubicon in industrial cybersecurity.
- 23:14This was an attack on a major petrochemical facility and the
- 23:18adversaries were not looking for a ransom.
- 23:20They wanted to break. Things they wanted to kill
- 23:22people. They were not targeting the IT
- 23:24network and they weren't even targeting the primary OT systems
- 23:27running the daily chemical process.
- 23:30They specifically targeted the safety instrumented system
- 23:33thesis. OK, how many visualize the
- 23:35architecture here? What is a safety instrumented
- 23:38system and how does it relate to the rest of the plant?
- 23:41So in any hazardous environment, a petrochemical plant, an oil
- 23:45refinery, a nuclear facility, you have your primary
- 23:48distributed control systems. These are the PLCS and computers
- 23:51actively mixing chemicals, managing heat, and moving
- 23:54fluids. The day-to-day workers.
- 23:56Right, but engineering acknowledges that primary
- 23:58systems can fail, valves can stick, sensors can break.
- 24:02So completely independent of the primary controls, you install an
- 24:05SIS. An independent backup.
- 24:07More than a backup, it is the absolute last line, a physical
- 24:10defense. It has its own dedicated
- 24:12sensors, its own wiring, and its own specialized controllers.
- 24:16Its only job is to sit quietly and monitor for critical safety
- 24:19thresholds. Like a silent guardian.
- 24:21Exactly. If the primary systems fail and
- 24:25a chemical VAT starts building pressure toward an explosive
- 24:28limit, Thesis detects the anomaly and automatically
- 24:32triggers an emergency physical shutdown.
- 24:34It overrides everything to prevent a catastrophe.
- 24:37It is the physical emergency brake, or like the airbags in a
- 24:41car, you never use them to drive, but if you crash, they're
- 24:44the only thing that saves you. A perfect analogy.
- 24:47Now consider the Triton malware. The attackers managed to
- 24:49infiltrate the plants network, likely through a remote access
- 24:53point, and locate the specific engineering workstations that
- 24:56interact with the safety systems.
- 24:58The facility used proprietary safety controllers manufactured
- 25:01by Schneider Electric called Triconex.
- 25:03Proprietary, meaning the code is an open source, you can't just
- 25:07Google how it works. No, you cannot.
- 25:10The attackers had to acquire a Triconics controller, take it
- 25:13into a lab, and painstakingly reverse engineer it's
- 25:17specialized, undocumented communication protocol.
- 25:20That takes serious funding and time.
- 25:21State level funding. They then wrote custom malware
- 25:25Triton specifically designed to inject malicious logic directly
- 25:29into the safety controllers. Wait, hold on.
- 25:31Why attack the safety system? If you want to cause an
- 25:34explosion, wouldn't you attack the primary system that controls
- 25:37the pressure? Think about it.
- 25:39If you attack the primary system and cause a pressure spike, the
- 25:42independent safety system will simply detect it and shut the
- 25:45plant down safely. Your attack fails.
- 25:48Oh wow. The Triton adversaries
- 25:50understood this. Their goal was to secretly alter
- 25:52the logic of the safety system first, essentially disabling the
- 25:55airbags so that when they subsequently triggered a
- 25:58catastrophic failure in the primary process, the fail safes
- 26:01would be blind to the danger and would not trigger.
- 26:04They were systematically dismantling the safety Nets to
- 26:07ensure kinetic destruction. That is absolutely terrifying.
- 26:10It is a direct, premeditated attempt to cause massive kinetic
- 26:13damage, environmental devastation, and massive loss of
- 26:17human life. The only reason it failed was a
- 26:20slight error in the attacker's custom code, which accidentally
- 26:23tripped an alarm and alerted the plant engineers before they
- 26:26could execute the second stage. A typo.
- 26:28Save the plant. Literally a coating bug saved
- 26:31lives. We've talked about a stolen VPN
- 26:34password at Colonial and reverse engineered proprietary hardware
- 26:38with Triton. I want to look at one more
- 26:40historical precedent before we get to the present day, because
- 26:43it highlights just how trivial the access point can be,
- 26:46especially when we talk about remote access.
- 26:48Yeah, let's hear it. Let's look at the water
- 26:50treatment facility in Oldsmar, FL in 2021.
- 26:53Oldsmar is a chilling example of the visibility crisis we face in
- 26:57municipal infrastructure. Water treatment plants are
- 27:00critical, but they are often severely underfunded and
- 27:03understaffed. Which means they take shortcuts.
- 27:05To manage the workload, engineers frequently rely on
- 27:07remote access tools. In Oldsmar, an attacker gained
- 27:10access to the plants internal systems using seemingly dormant
- 27:14commercial remote desktop software.
- 27:17Like Teamviewer, the stuff IT support uses to fix your laptop
- 27:21screen remotely. Exactly Commercial off the shelf
- 27:24remote access software team 4. The attacker breached the IT
- 27:29environment, found the remote access tool, and pivoted
- 27:32directly onto the OT control interface.
- 27:34Just bypassed everything. Yep, and an actual human
- 27:38operator sitting in the water plant suddenly noticed their
- 27:41mouse cursor moving across the screen on its own.
- 27:44It's like a ghost in the machine.
- 27:45That's so creepy. The operator sat there and
- 27:47watched the cursor navigate the chemical control software.
- 27:50The attacker opened the controls for sodium hydroxide.
- 27:53Which is Lye, right? Yes, Lye is used in tiny amounts
- 27:57to control water acidity and prevent lead from leaching out
- 28:00of the pipes. So in large amounts, Lye causes
- 28:02severe chemical burns to human tissue.
- 28:04It's poison. The attacker attempted to change
- 28:07the sodium hydroxide levels in the municipal water supply from
- 28:10a safe 100 parts per million to 11,100 parts per million, a
- 28:14lethal, highly corrosive dose. Just by typing numbers into a
- 28:17text box on a remote screen over Teamviewer.
- 28:20Yes, thankfully the human operator was physically present,
- 28:24saw it happening, and immediately reverse the change.
- 28:27There were also downstream physical chemical alarms that
- 28:29likely would have caught the spike before it reached homes.
- 28:32Thank goodness. But Oldsmar proves A terrifying
- 28:34point. An attacker crossed from a basic
- 28:37commercial IT remote access tool directly into the real time
- 28:41manipulation of a life safety process.
- 28:44Which brings us perfectly to the core of our deep dive today,
- 28:47Chapter 3. We've talked about the theory.
- 28:50We've talked about the historical evolution of these
- 28:52attacks. Now I want to look at what
- 28:54happens when this reality collides with the global economy
- 28:58right now. It gets expensive.
- 28:59It gets devastating. Let's walk through the anatomy
- 29:02of an absolute economic disaster, the Jaguar Land Rover
- 29:06hack of late 2025. The Jaguar Land Rover incident
- 29:10JLR is already being studied as a watershed moment in industrial
- 29:14cybersecurity. It is the ultimate manifestation
- 29:16of the ITOT convergence risk we've been outlining.
- 29:19I mean, we opened this whole discussion with the statistic of
- 29:22a plant costing $2.3 million an hour in downtime.
- 29:26The JLR attack is that statistic brought to life on a massive
- 29:30global scale. What started as an intrusion on
- 29:33the IT network forced a total synchronized shutdown of vehicle
- 29:38production worldwide for over a month.
- 29:40And to truly grasp the scale of the devastation, we have to look
- 29:43beyond the company itself. Economic analysis estimates that
- 29:47this single cyber attack cost the British economy roughly £1.9
- 29:52billion. That is nearly 2.5 billion U.S.
- 29:55dollars. It is officially the most
- 29:57economically damaging cyber event in the history of the
- 30:00United Kingdom. OK, let's break down exactly how
- 30:02this happened. We just talked about Triton
- 30:04reverse engineering safety controllers.
- 30:05Did the attackers write some incredibly sophisticated 0 day
- 30:08exploit to hack the robotic welding arms of the JLR assembly
- 30:12line? They didn't touch the robotic
- 30:14arms. They didn't need to write a
- 30:15single line of custom OT code. Wait, really?
- 30:18Nope. The attackers are linked to a
- 30:20syndicate known as Shiny Hunters, specifically a
- 30:22notorious subgroup tracked as Scattered Spider.
- 30:26I've heard of them, they target IT mostly.
- 30:28Their primary weapon is not complex programming, It is
- 30:31aggressive, highly sophisticated social engineering.
- 30:34Social engineering? Yeah.
- 30:36Hacking the human. Specifically, a technique called
- 30:39fishing Voice fishing. Scattered Spider operators are
- 30:43incredibly fluent in corporate culture.
- 30:45They research employees on LinkedIn.
- 30:47They figure out who works in the IT help desk.
- 30:50Then they pick up the phone and call the help desk, perfectly
- 30:54impersonating an employee who claims to be locked out of their
- 30:57remote access account. Wow.
- 30:58They. Just call them.
- 31:00They understand the internal jargon, they use psychological
- 31:02pressure, and they manipulate the support staff into resetting
- 31:06passwords or bypassing multi factor authentication protocols
- 31:09for their remote VPN. So they just talked their way
- 31:12through the front door. They convinced a human being to
- 31:15hand over the keys to the IT network.
- 31:16Precisely, they established a massive privileged presence
- 31:20inside the corporate IT environment.
- 31:22But the critical question is why an IT breach forced a global
- 31:26manufacturing shutdown. Right, because if they are just
- 31:29in the e-mail servers or the HR database, why stop building
- 31:32cars? This goes back to industry 4
- 31:34point O. Modern automotive production is
- 31:37a marvel of integration. JLR does not just build cars,
- 31:41They operate a tightly wound synchronized logistics network.
- 31:45They utilize complex manufacturing execution systems
- 31:49and Just in Time supplier portals.
- 31:51Just in time, meaning they don't keep thousands of steering
- 31:54wheels in a warehouse somewhere. The parts arrive exactly when
- 31:57they are needed. Exactly.
- 31:58You cannot build a modern, highly customized vehicle if the
- 32:01IT system cannot. It needs to be bolted into which
- 32:09specific chassis at exactly 2.14 PM.
- 32:13It's a continuous stream of data.
- 32:14The entire production sequence relies on a continuous,
- 32:17uninterrupted flow of data from the corporate IT layer down to
- 32:21the PLCS. So it isn't just a bridge
- 32:23between IT and OT, it's a completely interwoven nervous
- 32:26system. If the IT brain stops sending
- 32:28signals, the OT body can't move. That is the vulnerability.
- 32:32When JLR security teams detected the Scattered Spider intrusion
- 32:35on their enterprise remote access networks, they were
- 32:38forced into a brutal calculation.
- 32:40Yes, they knew the threat actor. They knew Scattered Spider
- 32:44routinely deploys highly destructive ransomware.
- 32:47And they knew that if the infection utilized that deeply
- 32:50interwoven nervous system to cross from the IT servers onto
- 32:54the factory floor OT networks, the physical damage to the
- 32:57machinery and the required recovery time would be
- 33:00catastrophic. So GLR had to make a choice.
- 33:03They chose to amputate to contain the blast radius and
- 33:06protect the physical factories from being encrypted.
- 33:09JLR proactively severed the connections and shut down their
- 33:13own IT systems. They pulled the plug on
- 33:15themselves to save the patient. Yes, and that decision meant the
- 33:19just in time data stopped flowing, which meant the parts
- 33:22stopped arriving, which meant sales, registration and global
- 33:26profession lines were brought to an absolute, immediate
- 33:29standstill. And the ripple effect was
- 33:30staggering. It wasn't just JLR taking the
- 33:33hit. Reports indicate that over 5000
- 33:36different supply chain organizations were impacted.
- 33:38It's an ecosystem collapse. Because if JLR isn't assembling
- 33:41cars, they aren't buying alternators or microchips or
- 33:44windshield wiper motors. The suppliers, many of which are
- 33:48smaller companies operating on thin margins, stop making money
- 33:52the very same day. The cascading economic failure
- 33:55was so severe and so rapid that it triggered emergency
- 33:59government intervention. The government had to step in.
- 34:01The UK government had to step in and underwrite loans.
- 34:04JLR themselves had to launch a massive emergency financing
- 34:08scheme just to inject upfront cash into their supply chain so
- 34:12those smaller businesses wouldn't go bankrupt while the
- 34:14main factories sat idle. It's an incredible scenario.
- 34:17The attackers use a phone call to steal a remote access
- 34:20password. They never touched a piece of
- 34:22factory machinery and they effectively crippled an entire
- 34:25national manufacturing sector. This highlights a crucial
- 34:28concept outlined in the Mitre ATTNCK for ICS framework.
- 34:32In traditional IT security, we are obsessed with data theft or
- 34:36data encryption. In OT security, we focus on a
- 34:39tactic known as loss of view or denial of control.
- 34:42Loss of view, meaning the operators literally cannot see
- 34:45what the machinery is doing. Exactly as we discussed earlier
- 34:49with the Termite ransomware, the attacker doesn't necessarily
- 34:52have to break the physical machine or alter the logic
- 34:54inside the PLC. If they simply encrypt the
- 34:57screens of the engineering workstations, or if they sever
- 35:00the network connection between the control room in the factory
- 35:03floor, as JLR was forced to do, the plant manager suffers a loss
- 35:07of you. They are flying completely
- 35:10blind. They have no idea if the boiler
- 35:12is operating at a normal pressure or if it is 5 minutes
- 35:16away from exploding. And in industrial safety,
- 35:19operating a high risk physical process without visibility is
- 35:22unacceptable. If you suffer a loss of you,
- 35:25your safety protocols mandate that you must hit the emergency
- 35:28stop button. The attacker doesn't shut down
- 35:31your plan, they create a condition where you are forced
- 35:33to shut it down yourself. I'm struggling with the detail
- 35:36here though. Let's move into chapter 4, the
- 35:38visibility crisis. When JLR discovered the hackers
- 35:41in the IT network, why couldn't they just hunt them down?
- 35:44What do you mean? Why couldn't they look at the
- 35:46internal network traffic, see exactly which specific machines
- 35:50the hackers were touching via that remote access, and isolate
- 35:54just those machines instead of shutting down the entire global
- 35:57company? You are hitting on the core
- 35:59systemic vulnerability that plagues almost every the
- 36:02industrial sector right now. It is the massive industry wide
- 36:06visibility crisis. If you look at the forensic data
- 36:09from Dragos, the statistics are incredibly jarring.
- 36:13I have right here. According to the reporting, 88%
- 36:16of OT networks struggle with basic detection and response
- 36:20capabilities. Even more critically, 56% of
- 36:23organizations cannot see anything happening below the
- 36:26ITOT boundary. That 56% figure is the root
- 36:30cause of why these attacks succeed.
- 36:32More than half are. Total blind Completely blind.
- 36:34To understand what below the boundary means, we need to look
- 36:37at how modern industrial networks are structurally
- 36:39designed. The global industry standard for
- 36:42architectural security is called the Purdue Enterprise Reference
- 36:45Architecture, commonly known as the Purdue.
- 36:47Model OK, let's unpack the Purdue model.
- 36:49Help me visualize this structure.
- 36:51Think of a manufacturing plant as a massive multi story
- 36:54building. The Purdue model organizes the
- 36:56network into hierarchical floors, or levels.
- 37:00Level 0 is the basement. This is the actual physical
- 37:03process, the steel pipe, the electric motor, the mechanical
- 37:06valve and dumb metal, right. Level 1 is the ground floor
- 37:09occupied by the basic controllers, the PLCS we talked
- 37:12about that directly send electrical voltages to the
- 37:15valves in the basement. OK, that's level 1.
- 37:17Level 2 is the supervisor level, the local control rooms with the
- 37:20HMI screens where human operators watch the PLCS.
- 37:24Level 3 is site wide operations, managing the entire facility's
- 37:27workflow. Still mostly on the factory
- 37:29side. Exactly.
- 37:31Finally, levels four and five are the top floors, the
- 37:33corporate enterprise, IT network, e-mail servers, remote
- 37:37access, VPNs and the Cloud O. It's a structural hierarchy.
- 37:42Physical machines in the basement.
- 37:43Corporate executives in the penthouse.
- 37:45Exactly. And the critical boundary we
- 37:48were talking about sits right between Level 3, the top of the
- 37:51OT factory floor, and level 4, the bottom of the corporate IT
- 37:54network. International Security standards
- 37:56dictate that you must create a strict Demilitarized Zone ADMZ
- 38:01sitting at level 3.5. Level 3.5.
- 38:04It's the security checkpoint between the penthouse and the
- 38:06factory floor. An organization might have
- 38:14state-of-the-art security visibility on their corporate
- 38:17laptops. At level 4, they can see every
- 38:19file being opened, every website visited.
- 38:21But once a piece of data crosses down through the DMZ into the
- 38:25PLC's at level 1, it becomes a complete black box.
- 38:28They are blind. Wait, earlier we established why
- 38:31they are blind. You told me we cannot install
- 38:34standard IT security software like an antivirus agent onto a
- 38:37level 1 PLC because it will crash the fragile processor and
- 38:41shut down the plant. That remains true.
- 38:42You cannot put active IT security agents on OT hardware.
- 38:45It's too brittle. So if they can't patch them and
- 38:47they can't scan them and they can't install antivirus on them,
- 38:51how on earth is a security team supposed to know if an attacker
- 38:54like Volt CIT is inside the factory network?
- 38:58The answer is entirely different from IT.
- 39:00You rely on passive network monitoring and deep packet
- 39:03inspection. Passive monitoring, meaning you
- 39:05don't interact with the devices at all.
- 39:07You don't ask the devices questions.
- 39:09Instead, you sit quietly on the network switches and you listen
- 39:12to every single piece of data flowing between the machines.
- 39:15Like wiretapping your own factory.
- 39:17Yes, but here is the challenge. OT systems do not speak the same
- 39:21language as IT systems. They use highly specialized
- 39:25legacy industrial protocols languages like Modbus, DNP 3,
- 39:29Profinet, or OPC UA. So standard IT security software
- 39:34wouldn't even understand the traffic if it saw.
- 39:36Standard IT tools are completely protocol blind.
- 39:38If an IT firewall sees a packet of Modbus data, it just sees a
- 39:41string of numbers. It doesn't know the difference
- 39:44between a routine automated command checking a boiler's
- 39:47temperature and a malicious command sent by a hacker
- 39:50instructing the boiler to turn off its cooling pump.
- 39:52OK, I think I have an analogy for this.
- 39:54Tell me if this works instead of having an active security guard
- 39:58in every single room checking ID badges, which would be like an
- 40:02IT antivirus agent. You essentially need a highly
- 40:05trained linguist sitting quietly inside the factory's air ducts.
- 40:10Oh, I like that. They just listen to the raw
- 40:12native language of the factory, humming along, waiting to notice
- 40:16if someone suddenly starts speaking with a malicious accent
- 40:18or issuing strange commands. That is a brilliant way to
- 40:21conceptualize it. The linguists in the air ducts
- 40:24needs to be perfectly fluent in Modbus.
- 40:26You need specialized out of band monitoring solutions.
- 40:30There are systems designed specifically for this, utilizing
- 40:33deep packet inspection to decode the industrial languages.
- 40:37So they translated on the fly. Exactly.
- 40:39Some advanced solutions even bypass the network entirely and
- 40:42monitor the raw electrical signals coming directly from the
- 40:46physical sensors. That's smart because by
- 40:48monitoring the physical electrical signals, the attacker
- 40:51can't hide their tracks. Even if they alter the software
- 40:54logs on the network to cover their tracks, they can't fake
- 40:56the raw voltage coming off of physical pressure gauge.
- 40:59Precisely. You spend months passively
- 41:02baselining the normal, healthy communication patterns of the
- 41:05factory. You learn exactly what the
- 41:07facility is supposed to sound like.
- 41:09Then your system instantly flags any behavioral anomaly.
- 41:13APLC talking to a machine it has never spoken to before or a
- 41:17controller receiving a firmware update command at 3:00 AM on a
- 41:20Sunday over a remote vendor portal.
- 41:23But to even set up that kind of passive monitoring, you need an
- 41:26incredible understanding of your own infrastructure.
- 41:29You need to know where all the cable.
- 41:31Which is why Incident Response Frameworks hammer home the
- 41:34absolute necessity of a Collection Management Framework,
- 41:36or CMFACMF, is a rigorous, painstakingly detailed inventory
- 41:42of every single data log and network switch in your OT
- 41:46environment. Every single one.
- 41:47Every single one. You have to know exactly how
- 41:49long logs are kept, who has access to them, and what
- 41:52specific forensic questions they can answer.
- 41:54Because if you suffer an attack and you don't have ACMF, your
- 41:56incident responders will waste the 1st crucial 48 hours just
- 41:59trying to figure out if security data even exists in the factory,
- 42:02let alone what it says. OK, so we have our structural
- 42:05Purdue model, we have our DMZ at level 3.5, we have our passive
- 42:09linguist in the air ducts decoding Modbus, and we have a
- 42:12rigorous CMF that is the ideal defense playbook.
- 42:16But what happens when all of that fails?
- 42:19It often does. Right.
- 42:20What happens when Scattered Spider gets through the DMZ via
- 42:23some compromised remote access, decrypts the engineering
- 42:26workstations, the operator loses view and the plant manager is
- 42:29forced to hit the emergency stop?
- 42:31You transition from defense to incident response, and as the
- 42:34data clearly shows in Chapter 5, recovering a physical OT plant
- 42:38is a completely different universe from rebooting an IT
- 42:41server. And this brings us to what I
- 42:42find the most fascinating psychological component of this
- 42:45entire deep dive. The Macrium report refers to it
- 42:47as the backup paradox. It is a phenomenal phrase that
- 42:51captures the exact dilemma plant managers face.
- 42:54Macrium surveyed the state of backup and recovery in the
- 42:57manufacturing sector. What they found is that
- 42:59manufacturers have actually invested heavily in backup
- 43:02technology. 75% of organizations are utilizing complex hybrid
- 43:08backups combining local on premise hardware with redundant
- 43:11cloud storage. So the technology's there, they
- 43:14have the data saved, it's protected.
- 43:16They have the data, but the paradox lies in the actual
- 43:19performance during a crisis. Despite all of that investment
- 43:23in hybrid technology, only 18% of manufacturers can actually
- 43:27meet their recovery time objectives during a simulated
- 43:29test. 18% I hear that, but I'm struggling with the logic.
- 43:34If you have a clean protected backup of your systems,
- 43:37shouldn't you just wipe the infected machines, click restore
- 43:39and get the assembly line back to work?
- 43:41Why is the failure rate so high? Because you are thinking like an
- 43:45IT professional. In an IT environment, restoring
- 43:48a server is a predictable linear process.
- 43:51If your e-mail server is encrypted with ransomware, you
- 43:55wipe the hard drives, you pull the clean image from the cloud
- 43:58backup, you reinstall it, and the server is back in business.
- 44:02The restoration of the data is the recovery.
- 44:05You got your files back, You're good.
- 44:06Right, but in OT, restoring the data isn't the end of the story.
- 44:11Not even. Close.
- 44:11Not even close. In OT, restoring a PLC's logic
- 44:15file from a cloud backup is not enough to restart a massive
- 44:18physical plant. The limiting factor is a concept
- 44:21called operational confidence. Operational confidence.
- 44:24Let's put you in the shoes of a plant manager.
- 44:26You run a massive facility producing volatile industrial
- 44:30chemicals. Your remote access networks were
- 44:32breached, the screens went dark, and the safety systems
- 44:35automatically triggered a massive screeching halt to the
- 44:38entire facility. OK, I'm stressed already.
- 44:40You should be. The pipes are full of half
- 44:43reacted chemicals cooling down. 24 hours later your IT security
- 44:48team walks into your office. They say great news, we wipe the
- 44:51servers, we restored the control logic from Tuesday's backup and
- 44:55the network is clean. We are ready to restart the
- 44:58reaction. As the plant manager, my first
- 45:00reaction is relief. Let's fire it up.
- 45:04Time is money. But then you stop and think.
- 45:06The question you have to ask isn't just do we have a backup?
- 45:09The critical life or death question is why is it safe to
- 45:12hit the start button right now? How do you definitively know
- 45:17that the attacker wasn't silently inside your network for
- 45:19a month before they triggered the final shutdown?
- 45:22Oh man, and we know from the reports that adversaries love to
- 45:26linger. The average dwell time for a
- 45:28threat actor inside an OT network before they detected is
- 45:3142 days. Exactly.
- 45:33So if you blindly restore a backup from Tuesday or last week
- 45:36or even last month, you have a very high probability of simply
- 45:39restoring the attackers hidden back doors.
- 45:41You're just inviting them back in.
- 45:42But it gets worse. What if three weeks ago, Volt CD
- 45:47subtly manipulated the chemical recipe logic inside the PLC
- 45:50backup itself? What if they changed the cooling
- 45:53threshold from 100° to 200°, but they didn't execute the final
- 45:57ransomware payload until today? So if I restore that backup, the
- 46:01IT security software says the file is clean from malware, but
- 46:05the engineering logic inside the file is physically poisoned.
- 46:08If you do not have an unbroken mathematically proven evidence
- 46:12change showing that the engineering logic in that
- 46:15restored backup is perfectly pristine, restarting that
- 46:18physical process could cause a catastrophic chemical spill.
- 46:21It could trigger an explosion. The quote from the Macrium
- 46:23report summarizes this perfectly.
- 46:26In IT, restoring service is often the visible milestone.
- 46:29In OT, the real milestone is operational confidence.
- 46:33You cannot just cross your fingers, hit, start on a blast
- 46:36furnace and hope for the best. You cannot.
- 46:37And yet the psychological pressure on that plant manager
- 46:40is immense. As we established, downtime in
- 46:43automotive is $2.3 million an hour.
- 46:45The executive board is probably screaming on a conference call
- 46:48for production to resume. The pressure to simply hit the
- 46:50start button can easily become stronger than the engineering
- 46:53confidence required to do it safely, and that is exactly the
- 46:57moment when lethal mistakes happen.
- 46:59This friction between downtime and recovery connects directly
- 47:02to a truly wild paradox highlighted in the Siemens True
- 47:06Cost of Downtime report. Siemens points out that the
- 47:09industries massive push toward predictive maintenance, using
- 47:13thousands of IoT sensors and AI algorithms to predict when a
- 47:17physical machine is going to break before it actually fails
- 47:20has been incredibly successful. It has.
- 47:22It's reduced the sheer volume of downtime incidents
- 47:24significantly. 5 years ago, plants averaged 42 unplanned
- 47:28incidents a month. Today, they average only 25.
- 47:32The predictive technology is undeniably working.
- 47:34It is catching the degrading pump bearings and the
- 47:36overheating motors early. Right.
- 47:38But here is the paradox. Even though plants are breaking
- 47:41down less frequently, when they do go down today, it takes them
- 47:44significantly longer to recover. Right.
- 47:46The average recovery time per incident jumped from 49 minutes
- 47:49five years ago to 81 minutes today.
- 47:52It's a fascinating statistical inversion, and it perfectly
- 47:55illustrates the compounding complexity of the modern OT
- 47:58environment. Because as predictive
- 48:00maintenance has solved all the easy problems, the worn out
- 48:03conveyor belts, the simple mechanical faults, the only
- 48:06problems that remain, the ones that actually managed to cause
- 48:09downtime now are the incredibly complex systemic disasters.
- 48:13The deep software integration failures.
- 48:16Catastrophic cyber attacks spanning IT and OT.
- 48:19Global supply chain shortages for highly specialized silicon
- 48:22parts. The easy fixes are gone.
- 48:24When a plant stops today, it's a structural nightmare.
- 48:27And as the incident response data points out, the human
- 48:30beings who actually know how to untangle these structural
- 48:32nightmares are disappearing. Right.
- 48:34We are facing a massive skilled labor shortage in manufacturing.
- 48:38The demographic shift is severe. The master electricians and
- 48:41veteran Mechanical Engineers who built these plants 30 years ago
- 48:44are retiring. Which brings us to the most
- 48:46critical realization about recovering from an industrial
- 48:49cyberattack. OT incident response cannot
- 48:52under any circumstances rely on an IT playbook.
- 48:55It's not a guy in a hoodie sitting at a laptop looking at
- 48:57code. Not at all.
- 48:59The ISA Global Cybersecurity Alliance, the Isai SEA, has
- 49:03developed a specialized program for this exact scenario called
- 49:07ICS. For ICS, it stands for Incident
- 49:11Command System for Industrial Control Systems.
- 49:13That sounds heavily inspired by government disaster response
- 49:16like FEMA. It is directly based on the
- 49:19National Incident Management System used by firefighters,
- 49:22paramedics, and 1st responders during physical disasters.
- 49:25Because responding to an OT cyberattack looks much more like
- 49:28fighting a complex multi story structural fire than it does
- 49:31traditional digital forensics. Walk me through what that
- 49:34actually looks like on the ground.
- 49:36It involves establishing a rigid physical command structure.
- 49:39You aren't just coordinating software engineers, you are
- 49:41coordinating Mechanical Engineers, chemical engineers,
- 49:44and plant safety officers. The.
- 49:45Whole multidisciplinary team. Before anyone can even look at a
- 49:50computer screen, you have to secure physical safety.
- 49:53You have to ensure the environment is stable.
- 49:55You have to make sure the incident responders are wearing
- 49:57steel toed boots, high visibility vests, hard hats and
- 50:00personal protective equipment before they step onto the
- 50:02factory floor to inspect a compromised PLC.
- 50:05That is so far removed from an IT server room.
- 50:08You might be coordinating with local environmental agencies to
- 50:11manage hazardous material spills caused by the attack.
- 50:15You are managing a violent physical crisis that just
- 50:18happens to have a digital root cause.
- 50:20Wow. OK, We have covered an immense
- 50:22amount of ground today and we've gone incredibly deep into the
- 50:25mechanics of this invisible war. We started with the staggering
- 50:29financial reality of the stopwatch, $2.3 million an hour
- 50:33in automotive downtime. Yeah, that stopwatch is still
- 50:36ticking. It is, we unpacked A fundamental
- 50:39philosophical clash between IT's confidentiality, first CIA triad
- 50:43and OT's availability, first AIC triad.
- 50:46And why running a standard security scan can crash a
- 50:49factory? And why the air gap is a myth,
- 50:51especially with all these remote access tunnels.
- 50:53Exactly. We look at how adversaries like
- 50:56Voltsite are evolving from data thieves into industrial
- 50:59saboteurs, studying blueprints to reverse engineer safety
- 51:03systems like in the Triton attack.
- 51:05It is an incredibly complex, rapidly evolving battlefield.
- 51:08We walked through the economic devastation of the Jaguar Land
- 51:11Rover attack where a single phone call to an IT help desk to
- 51:15steal AVPN password forced a multi billion dollar global
- 51:18shutdown because of the tightly wound nature of just in time
- 51:22manufacturing. The butterfly effect of a single
- 51:24vision call. We explored the Purdue model,
- 51:27The absolute necessity of the passive linguist in the air
- 51:30ducts, coding Modbus, and finally the agonizing backup
- 51:34paradox where restoring the data means absolute nothing without
- 51:37the operational confidence to safely restart the physical
- 51:40machinery. And every single one of these
- 51:42elements is compounding on the others.
- 51:44It's a battlefield that remains completely invisible to the vast
- 51:47majority of us, yet it's silently underpins every single
- 51:52aspect of our modern lives. The clean water coming out of
- 51:55your kitchen tap, the steady electricity keeping your
- 51:58refrigerator cold, the car you drive to work, all of it relies
- 52:02on the uninterrupted, highly delicate balance of these
- 52:06industrial control systems. It really does.
- 52:08Understanding this vulnerability isn't just a niche topic for
- 52:11cybersecurity professionals. It is fundamentally about the
- 52:14resilience of modern human civilization.
- 52:17If we are going to leave the listener with one final
- 52:19provocative thought to chew on, I think it connects directly
- 52:22back to that recovery time statistic and the demographic
- 52:25shift we discussed. As these cyber attacks get more
- 52:28sophisticated and as our factories become more deeply
- 52:31interwoven with the cloud and remote access portals, we are
- 52:34slowly realizing that our ultimate defense isn't a magical
- 52:37piece of AI software or a new firewall.
- 52:40The ultimate defense is the institutional knowledge of the
- 52:43human operators. The people on the floor.
- 52:45The men and women who intuitively know what the
- 52:47factory floor is supposed to sound like.
- 52:49The veteran engineers who can place a hand on a metal casing
- 52:52and know the exact healthy vibration of a turbine, or who
- 52:56can look at a chemical reaction through a reinforced window and
- 52:59know if the pressure gauge is lying to them.
- 53:02But with that skilled manufacturing workforce rapidly
- 53:05aging out. Right, With them aging out and
- 53:07tech focused remote IT security personnel increasingly taking
- 53:12over the operational controls from hundreds of miles away, we
- 53:16have to ask a very difficult question.
- 53:18As an industry, are we losing the physical human intuition
- 53:22required to save our most critical infrastructure when the
- 53:25digital screens inevitably go dark?
- 53:27If the linguist in the air ducts doesn't understand the language
- 53:30anymore and the engineer who knows the sound of the machine
- 53:33is retired, we are in deep trouble.
- 53:35That is a heavy critical thought to leave on.
- 53:38That's all for this episode of Cybersecurity Under Pressure.
- 53:41Real attacks, real lessons. It makes you look twice at every
- 53:44switching valve out there. See you in the next chapter.