
The Night the Database Died
31 January 2017: a tired engineer wipes a database directory on the wrong server, and GitLab.com's production database starts deleting itself. Then the team reaches for five layers of backups.
Sources
GitLab: Postmortem of database outage of January 31
GitLab: GitLab.com database incident
GitLab.com database incident: live incident notes (public Google Doc)
The Register: GitLab.com melts down after wrong directory deleted, backups fail
TechCrunch: GitLab suffers major backup failure after data deletion incident
GitLab infrastructure issue #1684: [meta] Listing all issues related to Jan 31st…
The skinny
The skinny isn't ready yet — notes appear once the transcript is processed.