Latest / Decrypted: The UK Cyber Briefing / A mislabelled maintenance job, and the outage it exported to Britain
Transcript
- 0:03This is Decrypted. Monday, the twenty-seventh of July.
- 0:06A mislabelled maintenance job, and the outage it exported to Britain.
- 0:12Thursday morning, a routine maintenance job on network kit in Microsoft's West US datacentre went wrong in a
- 0:21small, unglamorous way. An automated system miscounted which devices it was allowed to touch.
- 0:27It pulled network routes from more of them than intended.
- 0:31For close to five hours, the facility lost its connection to the rest of the internet.
- 0:37And with it went Teams, Outlook, SharePoint, OneDrive and Copilot, for a chunk of Microsoft's customers world
- 0:44wide. Plenty of them were in the UK.
- 0:48There, those services double as the phone system, the sales inbox and the shared drive for thousands of small
- 0:55and medium businesses that had never heard of West US.
- 0:58No attacker was involved. That's exactly why it belongs in a column about design.
- 1:05Nobody drew a line around how much damage one automated change was allowed to do.
- 1:10Next. What actually broke.
- 1:13Microsoft's preliminary account, published this week, blames fibre maintenance work.
- 1:20That work fed into an automated request-conversion system.
- 1:24And that system mislabelled extra network devices as being in scope for the maintenance window.
- 1:30It stripped their I P routes along with the intended ones.
- 1:34The result? West US was cut off from Microsoft's wider network between 14:44 and 19:41 U T C on 23 July.
- 1:45Engineers spotted the fault within the hour.
- 1:49But the rollback needed to fix it still took most of an hour to complete.
- 1:54Fully restoring stability took longer again.
- 1:57Here's the detail that matters.
- 2:00Nothing running entirely inside West US failed.
- 2:03Only traffic trying to get in or out did.
- 2:06And that's exactly the traffic that counts when your email and your video calls live somewhere else.
- 2:12And finally. The blast radius nobody bought.
- 2:16Think of a fuse box. A competent electrician isolates one circuit before working on it.
- 2:23A mislabelled panel should never take out the whole street.
- 2:27Microsoft's automation mislabelled the panel.
- 2:30And there was no independent check to catch the mismatch before it executed at scale.
- 2:35That's a Secure by Design failure, in the resilience sense.
- 2:40Changes to shared infrastructure need a hard ceiling on how many devices one action can touch.
- 2:46The tooling itself should enforce it.
- 2:48It shouldn't be left to whoever wrote the change ticket.
- 2:51So what should UK organisations take from this?
- 2:55Not that you should abandon Microsoft 365.
- 2:58It's that a single provider's internal change-control bug is now a boardroom-level dependency.
- 3:05Microsoft's own advice is to spread critical workloads across more than one region.
- 3:10That's sound. But it says the quiet part aloud.
- 3:14When the fault sits entirely inside someone else's automation, the customer holds no lever to pull.
- 3:20That gap between dependency and control?
- 3:24It's worth pricing into your continuity plans now.
- 3:27That was Decrypted. The written version and every source are linked in the show notes.
- 3:33Thanks for listening.