skinny.

Latest / Blast Radius

The Night the Database Died

31 January 2017: a tired engineer wipes a database directory on the wrong server, and GitLab.com's production database starts deleting itself. Then the team reaches for five layers of backups.

Sources

GitLab: Postmortem of database outage of January 31

GitLab: GitLab.com database incident

GitLab.com database incident: live incident notes (public Google Doc)

The Register: GitLab.com melts down after wrong directory deleted, backups fail

TechCrunch: GitLab suffers major backup failure after data deletion incident

GitLab infrastructure issue #1684: [meta] Listing all issues related to Jan 31st…

Log in to save or queue this episode.

The skinny

The skinny isn't ready yet — notes appear once the transcript is processed.