Latest / Decrypted: The UK Cyber Briefing / Cloudflare's shared disks kept the last customer's data, and the fix was a default
Transcript
- 0:03This is Decrypted. Friday, the twenty-fifth of September.
- 0:07Cloudflare's shared disks kept the last customer's data, and the fix was a default.
- 0:14On Thursday, Cloudflare and the researcher who found it published details of a bug.
- 0:20It let one customer's container read leftover data from a previous customer's container.
- 0:27Cloudflare says it found no evidence anyone else used the method, and that customers need do nothing.
- 0:34The interesting part is the cause.
- 0:37One storage setting, named skip_block_zeroing, which says exactly what it did.
- 0:43This is a story about a safe default that someone switched off.
- 0:48Next.
- 0:53Cloudflare Containers, and the Sandboxes built on them, give each customer a virtual disk.
- 1:00It's carved from a shared pool, and handed out in 64 KiB blocks by Linux thin provisioning.
- 1:08When a container is deleted, its blocks go back to the pool.
- 1:13Normally, each block is wiped before the next customer gets it.
- 1:17On the affected pools, the wipe was turned off.
- 1:21Picture a hotel that gives you a 64-page notebook but never removes the previous guest's notes.
- 1:28You write on four pages, and the other sixty are still theirs.
- 1:33In practice, a 4 KiB write into unused space triggered allocation of a whole 64 KiB block.
- 1:42The other 60 KiB could hold a previous tenant's bytes.
- 1:47Oren Yomtov of Accomplish reported it through Cloudflare's bug bounty on 4 September.
- 1:54The researchers recovered directory listings, SQLite databases, Chromium profiles,.
- 2:01env files and credential files from other customers' containers.
- 2:05No exploit chain was needed.
- 2:08And also.
- 2:13Almost nothing. The read happens at the block layer of your own disk.
- 2:19No login, no failed access, no alert.
- 2:23Cloudflare looked back by writing signatures for the telltale pattern, a 4 KiB write landing in a previously
- 2:30unmapped region. Then it searched historical disk telemetry.
- 2:35It found only authorised testing by the researchers and its own engineers.
- 2:40That's a good answer, but it's only as good as the telemetry retained.
- 2:44And no customer could have run that check.
- 2:47The fix was quick. It was confirmed in production the same day, and a runtime fix was merged about six hours
- 2:54after the report. Rollout finished on 7 September.
- 2:58The slow part came next. Removing the setting protects only new allocations, so Cloudflare retired every runn
- 3:06ing container disk and cleared cached image snapshots.
- 3:10It finished on 19 September.
- 3:12And also.
- 3:17A shared platform sells you a promise: your neighbour can't see your data.
- 3:22Tenant isolation is the product.
- 3:25So nothing tuned for speed or convenience should be able to weaken it quietly.
- 3:29The decision that would have prevented this is fail-safe defaults.
- 3:33Zero on allocation, always, with any override reviewed like code and covered by a test.
- 3:40Write four kilobytes, read the sixty beside them, and fail the build if they aren't zeros.
- 3:47Cloudflare's fix was simply to restore the default.
- 3:50And finally.
- 3:55This is opinion, not advice.
- 3:58You can't audit a supplier's storage configuration, so ask for evidence.
- 4:03How do you test isolation between tenants?
- 4:05How long do you keep the telemetry that lets you look back?
- 4:09If.env files or credentials lived on Containers or Sandboxes before the cleanup finished, rotating them cost
- 4:17s an afternoon. Cloudflare found no exploitation, so that's insurance, not a response.
- 4:24The longer-term fix is fewer long-lived secrets on disks you don't control.
- 4:29That was Decrypted. The written version and every source are linked in the show notes.
- 4:35Thanks for listening.