Software Quality · Recovery and Reliability · Software Failures and Quality Decisions
GitLab’s Database Incident: A Backup Is a Claim Until You Restore It
A practical recovery-testing approach inspired by GitLab’s 2017 database incident: restoration, data integrity, business workflows, and observable recovery targets.
The reassuring dashboard
A scheduled job finishes. A file exists in object storage. A dashboard is green. Someone concludes that recovery is covered.
Each of those observations can be useful. None demonstrates that the product can be restored to a usable state.
GitLab’s public postmortem describes its 31 January 2017 database incident, which began with accidental deletion of data from the primary database. The outage involved unrecoverable changes to some database records. Git repositories and wikis were not affected by that data loss, although access was disrupted. The report also describes problems encountered with backup and restoration arrangements. GitLab’s incident postmortem
The distinction matters. I would not summarise that event as “GitLab lost everyone’s code.” An inaccurate incident summary is a poor foundation for a quality argument.
Begin with a recovery promise users can recognise
“Restore the database” is an infrastructure task. A product recovery promise needs to describe the user-visible state that should exist afterwards.
For an internal approvals platform, that might mean people can sign in, see the correct pending requests, identify completed approvals, and continue work without granting permissions they did not previously have.
Before a recovery exercise, the team should agree the acceptable loss window and recovery duration. These are often discussed as recovery point and recovery time objectives. Their values should follow business needs and technical capability; I would not invent a universal target for every product.
The test then measures a real attempt against those expectations. A target documented in a plan is not an achieved result.
Restore somewhere isolated
The exercise should use an environment where restoring data cannot overwrite production or contact real customers. That requires attention to credentials, outbound integrations, scheduled jobs, notifications, and third-party callbacks.
A restored environment can be dangerous if it wakes up and begins executing old work. An email queue is a good example: recovering the data does not automatically mean it is safe to resume delivery.
I would ask the platform owners to establish the isolation controls first, then test the recovery workflow through its documented interface. QA can verify the result without claiming ownership of database administration.
Seed evidence before simulating loss
An exercise is easier to interpret when it starts with a small set of known records across important states.
For a hypothetical approvals product, the dataset might include an active user, a disabled user, an unapproved request, a completed request, an attachment, and a permission change. Record when each item was created relative to the backup.
After restoration, inspect what exists, what is missing by design, and what is unexpectedly inconsistent. This creates a basis for judging the result rather than relying on an impression that the application looks normal.
The important records should come from the product’s risks. A randomly generated thousand rows may be less useful than six carefully selected states.
Check relationships as well as counts
A row count can match while the business state is wrong. A request may exist without its approval history. An attachment reference may point to an object not included in the recovery. A restored account might have stale permissions.
A useful verification matrix could look like this:
| Area | Observable evidence |
|---|---|
| Identity | Intended users can authenticate; disabled users remain blocked |
| Relationships | Requests, owners, history, and attachments remain connected |
| Permissions | Restored state respects the agreed access rules |
| Time | Known records align with the measured recovery point |
| Integrations | Queued effects do not replay without deliberate control |
These are proposed checks for a sample product. They are not claims about GitLab’s test plan.
Run a business journey after the restore
Database health and application usability are related but different. Once infrastructure owners confirm the restored system is ready, run a representative workflow through the UI and API.
Can a user finish an existing request? Does the API return the expected state? Does a write persist after a subsequent read? Are audit records consistent? Can the team safely identify work that needs reconciliation?
Automate the repeatable checks where practical. Keep the restore timing, backup identifier, environment configuration, and test results together so that another person can understand what was demonstrated.
Include the awkward dependencies
A backup can be intact while the recovery procedure depends on an unavailable secret, an outdated runbook, or a person who is on leave.
I would include a rehearsal where the usual author is not the operator. This is a test of the procedure’s usability, not a judgement of the substitute’s skill. Steps requiring undocumented knowledge become findings to resolve.
The exercise should also observe failure: what happens when a backup cannot be read, a required artifact is missing, or a prerequisite check fails? A procedure should stop clearly rather than progress into an ambiguous state.
Make recovery evidence expire
A successful rehearsal remains evidence for the system and conditions exercised. A new storage layout, authentication change, database upgrade, or integration can make parts of it obsolete.
Recovery checks belong in change planning. Teams can define which changes trigger a focused re-test and which require a fuller exercise. The frequency should follow risk and the rate of meaningful change.
For software quality engineering, recovery is a concrete way to connect non-functional requirements with the work people actually need to resume. The question is not simply whether a copy exists. It is whether the team can demonstrate a trustworthy path back to useful software.