Latest / The Operator's Manual with Fexingo: Daily Decisions, Tactical Wins, and Business Execution / How One Supply Chain Hack Cost Maersk 300 Million Dollars
Transcript
- Lucas: So June 27th, 2017. A routine Wednesday morning at Maersk's headquarters in Copenhagen. And then, simultaneously, across every Maersk office worldwide, every screen goes dark. Not a blue screen of death — just black. No cursor. Nothing. Luna: That's the NotPetya attack. I remember reading that it wasn't even financially motivated — it was a nation-state weapon aimed at Ukraine. Lucas: Exactly right. The malware was designed to cripple Ukrainian infrastructure. It spread via a compromised update to a popular tax accounting software called M.E.Doc. But here's the operational detail that matters for this show: Maersk had a subsidiary in Ukraine called Maersk Drilling. Someone there downloaded that software update, and within minutes, the worm had jumped onto the corporate network. Luna: And because Maersk is a global company with a single Active Directory domain — one set of credentials for the whole company — the malware didn't stop at the Ukrainian border. Lucas: That's the critical piece. A single domain. So once the worm had admin credentials in Ukraine, it could authenticate against servers in Rotterdam, in Mumbai, in Los Angeles. Same password, same access. Within an hour, 4,000 servers and 45,000 PCs were encrypted. Every single one. Maersk's entire digital infrastructure was gone. Luna: I want to pause on that number — 45,000 devices. That's not a small office going offline. That's a company that moves about 20 percent of the world's container trade. Ships at sea, containers in ports, trucks on highways. Lucas: And the backup systems — this is where it gets really painful. Maersk had a backup strategy, but they stored their backups on the same network as their production servers. So the worm encrypted the live data and the backups simultaneously. The only thing that saved them was one small data center in a remote location that happened to be offline for maintenance when the attack hit. That data center had a partially intact copy of their Active Directory. Without that, they might never have rebuilt. Luna: So they had to rebuild from scratch essentially — reinstall 4,000 servers, reimage 45,000 laptops and desktops. What did that look like day-to-day? Lucas: They had to run the entire global shipping operation on paper and phone calls for ten days. Think about that. A modern container terminal — every container movement, every truck dispatch, every customs document — it's all digital. Maersk had to print shipping instructions and fax them to ports. Employees were handwriting cargo manifests. One terminal in Rotterdam was using whiteboards to track which containers were on which vessel. Luna: And the cost? I've seen $300 million as the headline number, but I think that's just the direct cost. Lucas: That's the figure Maersk reported — $300 million. But that's only the direct cost of the IT rebuild and the lost revenue from the ten-day shutdown. It doesn't include the reputational damage. Some customers lost perishable goods because containers sat in the wrong ports. A few major clients reportedly shifted volume to competitors like MSC or CMA CGM afterwards. The total cost of the attack in terms of long-term business lost is probably much higher. Luna: It's a reminder that operational resilience isn't just about having backups — it's about having backups that are actually isolated from the attack surface. Lucas: And that's the core lesson from Maersk. After the attack, they did two things. First, they split their IT infrastructure into multiple domains with separate admin credentials for different regions. Second, they air-gapped their critical backup systems — physically disconnected from the corporate network. If a regional domain gets compromised, the other regions stay clean. And the backups can't be encrypted because they're not accessible from the network. Luna: You know, this is a good place to mention something. If you're getting value from these deep dives into operational disasters and recoveries, there's a way to help us keep the show ad-free and focused on these kinds of case studies. Lucas: Yeah, listener support is what lets us spend time drilling into a single company's breakdown instead of rushing through headlines. If these conversations have moved your work forward in some small way, you can support us at buy me a coffee dot com slash fexingo. Luna: And we mean it — no pressure, no perks, just a sincere thanks. It honestly makes a difference. Lucas: Alright, back to Maersk. So the air-gapped backups solved the encryption problem, but they introduced a new operational question: how do you keep those backups current when they're disconnected? Maersk's approach was to do periodic offline syncs — physically copying data to the air-gapped systems on a scheduled basis. It's slower, but it's far more secure. Luna: I want to talk about the human side. During those ten days, how did Maersk keep ships moving? I mean, a container ship at sea still needs to know where to dock and what to unload. Lucas: They had to rely on the ship captains' local copies of voyage plans. Normally, a vessel's route is updated constantly from shore via satellite. Without that, captains reverted to paper charts and pre-printed schedules. Port agents — people on the ground — were running between terminals with handwritten cargo lists. One manager told a reporter that he spent three days in a port office with a fax machine and a stack of papers, manually reconciling containers. Luna: It sounds like something from the 1980s. Lucas: And yet it worked, barely. Ten days is a long time in global logistics. Maersk lost about 20,000 container moves during that period. But they didn't lose a single ship. No collisions, no groundings. That's a testament to the skill of the crews and the port teams, but also to the fact that the safety-critical systems — navigation, engine controls — were on separate networks and not affected. Luna: So the attack was on the administrative and operational IT, not the operational technology. That's a distinction a lot of companies don't make. Lucas: Right. Maersk's shipboard systems ran on isolated industrial control networks. The malware couldn't reach them because they weren't connected to the corporate LAN. That separation probably saved them from a much worse outcome. But most companies — especially smaller ones — run everything on one flat network. If the CEO's laptop gets malware, the factory floor could go down too. Luna: So the broader lesson is: segment your network. Don't let one compromised workstation give an attacker the keys to the whole kingdom, and don't let your IT network connect to your OT network. Lucas: Exactly. Maersk's post-mortem led to a complete network redesign. They now have what they call 'zones of trust' — different levels of access with firewalls between them. A Ukrainian office can't authenticate against a server in Denmark. A booking system in Rotterdam can't talk to a container terminal's crane controls. It's more complex to manage, but it's survivable. Luna: And they also changed their backup strategy. I read that they now have multiple copies of critical data in different geographic locations, and at least one copy is offline. Lucas: They adopted the 3-2-1 rule. Three copies of data, on two different media types, with one copy offsite and offline. It's a standard practice in IT, but Maersk proves why it matters. The attack happened four years after that rule was well-known. It's easy to let operational convenience override security — 'Oh, we'll just keep the backup server on the same network, it's faster to restore.' Until it's gone. Luna: And the insurance piece — did their cyber insurance cover this? I know a lot of policies exclude state-sponsored attacks. Lucas: That's a great question. NotPetya was attributed to the Russian military, and many insurers invoked 'acts of war' clauses to deny coverage. Merck, the pharmaceutical company that was also hit, sued its insurers and eventually won a court ruling that the attack wasn't an act of war because it wasn't part of a traditional armed conflict. But Maersk reportedly didn't disclose the full details of their insurance recovery. The safe assumption is that they absorbed most of the $300 million themselves. Luna: Which means that for many companies, relying on cyber insurance is not a substitute for actual operational resilience. Lucas: It's exactly the wrong substitute. Insurance pays out cash, but it doesn't un-encrypt your servers. It doesn't restore customer trust. The Maersk story shows that the real cost is the downtime and the lost business, not the IT replacement hardware. The $300 million figure is almost entirely lost revenue and recovery labor, not hardware. Luna: So if a listener is running operations for a mid-size company, what's the one thing they should take away from this? Not the whole network redesign — just one actionable step. Lucas: I'd say: go check whether your backup system can be encrypted by the same malware that infects your production servers. If the backup server is on the same network and uses the same admin credentials, it can. And if the answer is yes, you need an offline backup — a tape drive, a disconnected hard drive, something that requires physical access to destroy. That's the Maersk lesson in one sentence. Luna: It's a simple test that would have saved them weeks of agony. Lucas: And $300 million. Not a bad return on a manual backup drive. Luna: Next episode, let's look at the 2020 hack on Garmin — similar story, very different outcome. Lucas: I like that. Ransomware in a consumer-facing company. That's a good contrast. See you next time. Luna: See you.