GitHub’s 2018 Incident: A Healthy Failover Can Still Break the Product

Availability has more than one meaning

A component can be reachable while the product is inconsistent. A database can accept a connection while a workflow is no longer safe to continue.

GitHub’s analysis of its October 2018 incident describes a network partition followed by database topology changes and a prolonged service disruption. The response prioritised protecting data integrity while restoring service. This was a distributed-system recovery problem with user-visible consequences, not merely a web page that needed restarting. GitHub’s post-incident analysis

I take from it a testing question that applies well beyond large infrastructure: when the system is partially available, which product promises still hold?

Start with the workflow, then map dependencies

An infrastructure diagram tells us where components connect. A quality model also needs to tell us what a user expects to remain true across those connections.

Imagine a SaaS approvals tool. A user submits a request, an API stores it, a worker sends a notification, and another user approves it. The interface shows status based on data read from a replica.

That simple journey can become inconsistent in several ways. A write succeeds but the next read looks old. A notification arrives before the corresponding request is visible. A timed-out operation is retried and produces two jobs. None of these requires the entire system to be down.

These are hypothetical examples. Their purpose is to make the test conditions concrete rather than attribute unreported behaviour to GitHub.

Write product invariants that survive partial failure

For the approvals example, I would ask the team to agree conditions such as:

• A request cannot be approved by an identity that lacks permission.
• A completed operation remains identifiable after a retry.
• A user is not told that an operation definitely failed when its result is actually unknown.
• A queued notification refers to a state that can be reconciled.
• Recovery does not silently turn an approved request back into an actionable pending request.

Some products allow stale reads for a period. Others cannot tolerate them for particular decisions. The acceptable behaviour needs to be explicit before an automated test can judge it.

A test suite that assumes every read must be immediately current can be wrong for the architecture. A suite that accepts every stale result can be wrong for the business.

Exercise failure transitions, not only steady states

A useful resilience test observes the system before, during, and after a controlled fault. Testing a fully healthy environment and a completely unavailable environment misses the transitions where ambiguity appears.

With the relevant platform owners, I would choose an isolated environment and a bounded disturbance: delaying one dependency, interrupting a connection, or holding a queue consumer. The exact method depends on the system and available controls.

Before the disturbance, create identifiable work. During it, attempt the important operations and record the returned state. After recovery, reconcile what actually happened. The result should connect client observations with durable product state.

Fault injection without agreed expected behaviour is a demonstration. It becomes a test when the team can explain what should happen and assess the evidence against it.

Treat retries as new opportunities for inconsistency

Clients often retry because they cannot tell whether an operation completed. The server may also retry downstream work. Together, those mechanisms can produce more effects than either team expected.

For an API with an agreed idempotency contract, useful cases include repeated submission of the same intent, a connection loss after acceptance, and recovery followed by another client attempt. The assertion should inspect business effects, not only HTTP status codes.

Where no such contract exists, the finding may be a requirement question before it is a defect: what should the product do when the caller has an uncertain result?

That distinction is central to good QA. Tests should expose an unresolved decision instead of inventing a convenient oracle.

Make degraded behaviour understandable

The UI is part of resilience. A disabled action with an explanation can protect a workflow better than a spinner that permits repeated clicking. A status message can distinguish delayed information from confirmed failure.

I would inspect whether the interface tells the truth under the selected fault conditions. Does it preserve the user’s input? Can the person identify the operation later? Does it explain when retrying is safe? Are accessibility announcements consistent with the visible state?

These are small product decisions with important consequences. They connect product engineering to the underlying reliability model.

Verify the return to normal

Recovery can leave a backlog, delayed notifications, or records needing reconciliation. A test should not stop when the first health check turns green.

For the sample workflow, I would compare submitted request identifiers with final stored results, inspect the queue’s eventual state, and check that the UI converges to the agreed view. Operations whose outcome remains unknown should be recorded as such.

A concise report can separate what stayed correct, what was delayed, what needed intervention, and what was not tested. That gives decision-makers useful boundaries instead of an absolute claim that failover works.

What QA contributes

Quality engineering does not replace distributed-systems design or operations expertise. It connects their mechanisms to observable promises and tests the assumptions between teams.

The most useful deliverable may begin as a short state model and a handful of API checks. Its value comes from making inconsistency visible and defining what safe behaviour means when the system cannot provide its normal service. Explore my quality engineering approach.