A Replica Fell Behind—and Production Took the Hit
January 2017 saw GitLab.com relying on a PostgreSQL primary, db1.cluster.gitlab.com, and a live replica, db2.cluster.gitlab.com, for resilience. This architecture, common for high-availability setups, aimed to keep db2 in sync for failover, a crucial but often brittle defense.
That evening, a load spike caused db2 to fall so far behind that replication ceased. The standard fix: reinitialize db2 by wiping its data folder and copying everything from db1. A simple procedure, in theory.
However, during this reinitialization, an engineer mistakenly executed the rm -rf command not on db2, but on db1, the production primary. The command ran for a mere second or two before being stopped, but the damage was done: approximately 300 GB of data was deleted. This wasn't a slow corruption; it was a sudden, catastrophic erasure of live production data. The consequences of this one misdirected command would soon unfold, revealing a far deeper problem.
Every Backup Looked Safer Than It Was
GitLab’s disaster unfolded not just from a single misstep but from a house of cards built on untested assumptions. The organization believed it had a robust backup strategy, yet every layer proved fatally flawed, demonstrating that an untested backup is merely a hypothesis, not protection.
Confidence in a daily pg_dump export to Amazon S3 was misplaced. The job, intended to safeguard the PostgreSQL 9.6 database, bizarrely used a pg_dump tool from a much older PostgreSQL 9.2 version. This mismatch caused the export to error out silently, leaving the S3 bucket empty.
Crucially, nobody caught the systemic failure. Notification emails, meant to flag the failing backup job, were summarily rejected by the mail server due to authentication issues. Further compounding the oversight, production database snapshots, a fundamental layer of resilience, were never enabled. LVM snapshots existed only for staging, and Azure Cloud snapshots covered OS disks but not critical database volumes.
With the primary database compromised and the replica already wiped during the recovery attempt, GitLab faced a terrifying reality: no viable recent backup existed. The only path back was a roughly six-hour-old manual snapshot, taken by an engineer for an unrelated load test. This stroke of luck, rather than any engineered solution, underscored the precariousness of their entire backup philosophy. Copying this snapshot from slow disks took about 18 hours, mirroring the total outage duration and highlighting the true cost of neglected data recovery preparedness.
The Fix Played Out in Front of 5,000 Viewers
GitLab’s crisis response was a masterclass in radical transparency. Engineers shared real-time updates and troubleshooting publicly through a Google Doc, social media, and a YouTube livestream. This decision turned a catastrophic technical failure into a highly visible test of trust, with over 5,000 people watching at the peak.
The fix, painfully slow, relied on a manually created snapshot from six hours prior, residing on slow-storage disks. Restoring this snapshot consumed approximately 18 hours, the duration of the entire outage. This slow recovery meant GitLab.com was down for a significant period.
Ultimately, GitLab lost about six hours of database records. This included approximately:
- 5,000 projects
- 5,000 comments
- 700 user accounts
Repository data, thankfully, remained intact on separate storage. While the outcome was painful, the public display of resilience and honesty garnered significant respect. For a deeper dive into the specifics, consult the Postmortem of database outage of January 31 - GitLab. This incident fundamentally shifted how many in the industry view incident response and radical transparency.
Enjoying this? Get one like it in your inbox each morning.
one email a day · unsubscribe in two clicks · no third-party tracking
The Lesson Was Ownership, Not Blame
GitLab’s postmortem didn’t seek a scapegoat; it diagnosed a systemic failure of ownership. The central issue wasn't an engineer’s mistake, but the absence of a dedicated role for backup reliability and routine restore verification. Nobody was accountable, so critical safeguards atrophied.
The incident catalyzed significant internal reforms. GitLab implemented automated restoration tests, ensuring backups were not just present but functional. They also strengthened production access guardrails, making destructive actions harder to perform accidentally, and assigned explicit responsibility for database reliability to a dedicated team.
For any engineering organization, the lesson is clear: assign a specific owner for backup integrity. Establish verified alert channels for backup failures—don't rely on unnoticed emails. Crucially, regularly test restores to validate your recovery strategy, and implement robust safeguards to prevent accidental data loss. Your backups are only as good as your last verified restore.
Frequently Asked Questions
What caused GitLab’s 2017 database outage?
While fixing a replica that had fallen behind, an engineer accidentally deleted data from the production database instead of the replica.
Why couldn’t GitLab restore from its backups?
The daily export failed because it used an incompatible Postgres tool version, and failure emails were rejected. Production snapshots were also unavailable.
How long was GitLab down?
The recovery took about 18 hours, largely because restoring a manually created snapshot from slow storage took time.
How much data did GitLab lose?
GitLab lost about six hours of database data, including roughly 5,000 projects, 5,000 comments, and 700 user accounts.
What is the main lesson from the incident?
A backup is only trustworthy when teams regularly test that they can restore it—and someone is accountable for that process.