Latest / Blast Radius / The Night the Database Died
Transcript
- 0:01The thirty-first of January, twenty seventeen.
- 0:05Late evening.
- 0:07An engineer types a command to wipe a database directory...
- 0:12and presses enter.
- 0:15Then he sees which machine he's on.
- 0:18D B one.
- 0:20Not D B two.
- 0:23This is the production database of GitLab dot com.
- 0:28And it is deleting itself, while he watches.
- 0:33That was the disaster.
- 0:36It was not the catastrophe.
- 0:39The catastrophe came next...
- 0:41when the team reached for the backups.
- 0:45I'm Nora Whitfield.
- 0:46Before we begin...
- 0:48one honest disclosure.
- 0:50Everything in this story is real.
- 0:52The company, the timestamps, the commands.
- 0:55All of it happened, and all of it comes from the public record.
- 1:00The voice about to tell it to you...
- 1:02is not real.
- 1:03Neither is mine.
- 1:04We are synthetic...
- 1:06generated with AI.
- 1:08The facts are human.
- 1:10The reconstruction is ours.
- 1:13I'm Julian Ashcroft, and I'm your narrator.
- 1:16This is Blast Radius...
- 1:19stories from the nights our systems failed...
- 1:21and the people who brought them back.
- 1:26This is a true story.
- 1:28Incident declared.
- 1:30Start the clock.
- 1:33January, twenty seventeen.
- 1:36GitLab is a young company, founded in twenty fourteen...
- 1:40about a hundred people, many of them working remotely.
- 1:44Their product is where developers keep their code...
- 1:48their bug reports, their code reviews, their conversations.
- 1:52Trust is the entire business.
- 1:54You give them your work, and they keep it safe.
- 1:59Behind GitLab dot com there are two big database servers.
- 2:04D B one...
- 2:05the primary.
- 2:07Every issue, every comment, every merge request lands here.
- 2:11And D B two...
- 2:13the secondary.
- 2:15Its only job is to follow along, copying everything the primary does,
- 2:20a heartbeat behind.
- 2:22If the primary ever dies, the secondary steps in.
- 2:25That's the design.
- 2:27That's the safety net.
- 2:30Keep those two names in mind.
- 2:32D B one.
- 2:34D B two.
- 2:36One letter of difference...
- 2:38on machines that look identical when you're logged in.
- 2:42That day, GitLab is under attack.
- 2:46Not a clever one — spammers, hammering the site, creating garbage at industrial speed.
- 2:53The database is drowning in load.
- 2:56And under that load, something snaps: the secondary falls behind.
- 3:01Replication...
- 3:02stops.
- 3:06Now there is no follower.
- 3:09No copy, a heartbeat behind.
- 3:13Whatever happens to D B one...
- 3:16happens to GitLab.
- 3:18An engineer on the infrastructure team steps up to fix it.
- 3:22It's late in Europe.
- 3:24He'd meant to sign off around eleven.
- 3:27But replication is broken...
- 3:29so he stays.
- 3:31One more fix...
- 3:33and then sleep.
- 3:35Oh — and one small detail.
- 3:39Earlier that afternoon, before testing some new software, an engineer took a snapshot of the production database by hand...
- 3:48and loaded it into the staging environment.
- 3:51A copy for testing.
- 3:53Nobody thought about it twice.
- 3:56Remember it anyway.
- 3:58The fix for broken replication is fragile.
- 4:03GitLab's own report would later call it badly documented...
- 4:07a handful of shell scripts.
- 4:10The secondary's data is stale and inconsistent...
- 4:14so you wipe it clean, and let it copy everything fresh from the primary.
- 4:19Erase the follower...
- 4:21so it can follow again.
- 4:23But tonight, the copy tool won't cooperate.
- 4:28The team changes settings.
- 4:30Restarts the database.
- 4:32Tries again...
- 4:33and it just hangs.
- 4:35Silent.
- 4:36No error.
- 4:38No progress.
- 4:41Anyone who has run a production system knows this exact feeling.
- 4:46Eleven at night...
- 4:48a tool sitting there doing nothing...
- 4:50and not one message to tell you why.
- 4:54He decides the secondary's data directory isn't empty enough.
- 5:00Fine.
- 5:01Wipe it properly, then.
- 5:03One command.
- 5:04He's typed it a thousand times.
- 5:06Here is what the record shows.
- 5:09He believed he was on the secondary.
- 5:13He was on the primary.
- 5:17Later, GitLab would change the prompt on every one of its servers...
- 5:21so the hosts would be easier to tell apart.
- 5:25One letter of difference.
- 5:28The command went to D B one.
- 5:32Seconds later, he sees it.
- 5:34Cancels it.
- 5:35Four and a half gigabytes left...
- 5:38out of three hundred.
- 5:41What happens next tells you everything about this company.
- 5:46No cover-up.
- 5:47No quiet scramble.
- 5:49They take the site down, and say so in public.
- 5:53And then, just after midnight, the company's status account publishes a sentence...
- 6:00that no communications department in the world would have approved.
- 6:04Listen.
- 6:06We accidentally deleted production data...
- 6:09and might have to restore from backup.
- 6:13That's real.
- 6:15They said it.
- 6:16In public.
- 6:17That night.
- 6:19And they went further.
- 6:21The internal document tracking the disaster was opened to the whole world...
- 6:27and later, they put the recovery itself on a live stream.
- 6:33Thousands of strangers, watching engineers try to resurrect a database in real time...
- 6:40like a surgery theater with a chat window.
- 6:45But before any of that can work...
- 6:48they need something to restore from.
- 6:50They need the backups.
- 6:54Time to line up the suspects.
- 6:57Suspect number one: the spammers.
- 7:00They lit the fire — the load that broke replication in the first place.
- 7:05Guilty of arson, maybe.
- 7:08But they didn't delete anything.
- 7:11Suspect number two: the tooling.
- 7:14The copy process that hung, silently, whatever they tried...
- 7:19and pushed a tired human toward a manual fix at eleven at night.
- 7:25Closer.
- 7:27Suspect number three...
- 7:28the engineer.
- 7:30The man at the keyboard.
- 7:32The obvious answer.
- 7:34The one everyone reaches for first.
- 7:37Hold that thought.
- 7:39Because this show has a rule, and the rule exists for a reason:
- 7:44if one tired person typing one command in the wrong window can erase your company...
- 7:52the person was never the problem.
- 7:55The system was.
- 7:56The question is not who.
- 7:58The question is...
- 8:00what else had already failed, silently, before he ever touched the keyboard.
- 8:06And no — in case you're wondering.
- 8:09Not this time.
- 8:11This time, it wasn't DNS.
- 8:14It was about to be something much worse.
- 8:18Just after half past eleven.
- 8:21The team calls for help, and starts down the list of backups.
- 8:25GitLab had five separate ways to recover a lost database.
- 8:30Five layers of safety net.
- 8:33Count them with me.
- 8:35Layer one.
- 8:36The regular database dumps.
- 8:38A scheduled job, exporting everything.
- 8:41The team goes looking for the files...
- 8:45and finds nothing.
- 8:49The dump tool had been failing...
- 8:51every single time it ran.
- 8:53A version mismatch: the backup binary was too old for the database it was backing up.
- 9:00It failed silently...
- 9:01and nobody knew.
- 9:04Why did nobody know?
- 9:07Because the failure notifications...
- 9:09the emails that scream when a backup breaks...
- 9:13were never arriving.
- 9:15Rejected by the mail system for a technical formality.
- 9:21The alarm was ringing in a room with nobody in it.
- 9:25Layer two.
- 9:27The uploads to cloud storage.
- 9:30The bucket is empty.
- 9:32The job feeding it had depended on layer one.
- 9:36Layer three.
- 9:38Azure disk snapshots.
- 9:41Never enabled for the database servers.
- 9:44Layer four.
- 9:46The disk-level snapshots on the machine itself.
- 9:49Normally taken just once every twenty-four hours.
- 9:54The last regular one could be up to a day old.
- 9:57Restoring it could mean erasing a whole day of the world's work.
- 10:02And layer five?
- 10:04Replication itself.
- 10:07The very thing that broke tonight.
- 10:11Five safety nets.
- 10:14Not one of them holding.
- 10:16And the whole building is on fire.
- 10:20And then someone remembers one more copy.
- 10:25The one nobody thought of as a backup at all.
- 10:28The snapshot for staging.
- 10:31The copy an engineer took by hand that afternoon, for the testing environment...
- 10:38about six hours before the deletion.
- 10:43It exists.
- 10:45It's real.
- 10:47It is, by pure luck...
- 10:51the only survivor.
- 10:53So here is the reveal.
- 10:57The true root cause of the GitLab data loss...
- 11:01is not a command typed in the wrong window at eleven at night.
- 11:06The database didn't die when he pressed enter.
- 11:10Parts of GitLab's safety had been dead long before that night.
- 11:15The dumps that produced nothing.
- 11:17The alerts that reached no one.
- 11:19The snapshots that were never enabled.
- 11:23The deletion didn't cause the catastrophe.
- 11:26The deletion revealed it.
- 11:31A backup you have never restored...
- 11:35is not a backup.
- 11:37It's a hope.
- 11:39GitLab had five hopes...
- 11:43and one accident.
- 11:46The recovery is brutal and slow.
- 11:50The staging environment runs on cheaper infrastructure...
- 11:53so copying the data back takes hour...
- 11:57after hour...
- 11:58around eighteen of them, streamed live, to an audience of thousands.
- 12:05The next evening, GitLab dot com comes back.
- 12:10The final damage: about six hours of database data, gone.
- 12:16Issues, merge requests, users, comments, snippets.
- 12:21The code itself — the repositories — survived untouched.
- 12:26They were never in that database.
- 12:29Painful.
- 12:30Survivable.
- 12:32Lucky.
- 12:34Then comes the part that turned a disaster into a legend.
- 12:39The postmortem.
- 12:41Public.
- 12:42Complete.
- 12:44Unflinching.
- 12:45Every failed backup layer, named and explained.
- 12:50And the engineer?
- 12:52In GitLab's reports, he is never named.
- 12:55The postmortem doesn't stop at the man at the keyboard.
- 12:59It goes after the systems that let one command do this much damage.
- 13:05The sentence, then.
- 13:07Not for a person — for a system.
- 13:11Backups, actually monitored...
- 13:13alerts that actually arrive...
- 13:16restores, actually tested, on a schedule, forever.
- 13:22And on every database prompt...
- 13:24the hostname, impossible to miss...
- 13:27so no one ever again has to guess which window is loaded.
- 13:33They fell in public...
- 13:34so they rebuilt in public.
- 13:37Rigor.
- 13:38Respect.
- 13:40And the discipline to say: the system failed the human...
- 13:44not the other way around.
- 13:47The dashboards are green again.
- 13:50The runbook is longer than it was yesterday.
- 13:54Somewhere in the Netherlands...
- 13:55an engineer finally gets some sleep.
- 13:59So.
- 14:01The verdict.
- 14:02It wasn't the spammers.
- 14:04It wasn't the replication lag.
- 14:07It wasn't the man at the keyboard — it was never really him.
- 14:13It was the backups that never ran...
- 14:16the alerts that never arrived...
- 14:18the restores that were never tested.
- 14:23And honestly?
- 14:24This time...
- 14:25it wasn't even DNS.
- 14:28One question before you go, for everyone with a pager on their nightstand:
- 14:35have you tested your restore this quarter?
- 14:38Not the backup.
- 14:40The restore.
- 14:43Blast Radius is a synthetic-voice production, made with AI and based entirely on the public record — GitLab's own postmortem and their live-streamed recovery.
- 14:55Links, timestamps, and sources are in the show notes.
- 14:59Thanks for keeping me company tonight.
- 15:01Incident resolved.
- 15:04Stop the clock.
- 15:07...
- 15:08Sleep with your pager on...