Software Quality · Performance and Risk · Software Failures and Quality Decisions
Cloudflare’s Regex Outage: Performance Belongs in the Definition of Correct
How a rule can return the right answer and still fail the product: performance test design, adversarial inputs, budgets, and safe rollout.
A rule that works can still be unacceptable
Ask whether a validation rule matches the intended examples and the answer may be yes. Ask what it costs to evaluate under an awkward input and the answer may change the release decision.
Cloudflare’s account of its 2 July 2019 outage describes a Web Application Firewall rule whose regular expression exhausted CPU resources. The company reported a 27-minute outage and explained that the failure involved the wider testing and rollout system, not only the expression itself. Cloudflare’s detailed postmortem
For a product team, the useful lesson is that performance requirements belong beside functional expectations. Producing the right result too late can be a product failure.
Define the cost the workflow can tolerate
“Must be fast” cannot tell a tester whether a result is acceptable. A response budget should identify the operation, expected conditions, measurement method, and threshold that matters to the user or downstream system.
Consider a hypothetical API that evaluates a customer-defined rule before accepting a request. Its budget should account for the rule evaluation, other application work, and the time callers can wait. If the rule can consume the entire request budget on its own, its functional correctness provides little protection.
I would agree that expectation with engineering and product owners before writing a benchmark. The acceptable limit depends on the product; an arbitrary number copied from an unrelated system creates false precision.
Select input families deliberately
A performance suite should contain more than average-sized happy-path examples. Different families of input reveal different risks.
For a parser or pattern matcher, I would consider normal matching inputs, normal non-matches, long prefixes that almost match, repeated characters, empty values, and combinations at the supported size limit. The intent is to find classes of expensive behaviour, not to reproduce production traffic byte for byte.
This matters because a random sample can contain plenty of ordinary data and almost none of the input that triggers a costly path. A test plan needs an argument for why its examples exercise the relevant algorithmic risks.
The matrix should also respect the product contract. Unsupported input may be rejected, but the rejection itself must be bounded and safe.
A small experiment can reveal a large problem
For an expensive operation, measure how runtime changes as input grows. Double the input length through several controlled sizes and observe the trend, while keeping the environment and execution limits clear.
This is a proposed investigation technique, not a claim about which experiment Cloudflare did or did not perform.
A simple record might contain:
| Input family | Size | Evaluation time | Outcome |
|---|---|---|---|
| Valid example | Small | Measured | Accepted |
| Near-match | Small | Measured | Rejected |
| Near-match | Twice the size | Measured | Rejected or timed out |
| Near-match | Supported maximum | Measured | Compared with the budget |
A timeout is a finding, not a missing measurement to discard. The test runner should isolate the work and stop safely so that the investigation does not exhaust a shared environment.
Test concurrency separately
A single request under budget does not establish that a shared service remains useful when many requests arrive together. Contention, queues, and resource limits can change the picture.
A follow-up test should represent a justified workload: ordinary traffic mixed with expensive-but-valid inputs, for example. Observe latency distribution, error rate, resource use, and behaviour of unrelated requests.
The assertion should include what happens to other users. A customer-configurable rule that slows only its own evaluation has a different consequence from one that consumes a shared worker and blocks everyone else.
This is where resource isolation becomes part of the product’s quality model. QA can identify the expected boundary; engineering determines and implements the appropriate mechanism.
Build performance checks that engineers can trust
A noisy benchmark can become an ignored gate. Useful automation needs repeatable fixtures, controlled execution, and enough context to explain a regression.
I would distinguish a fast budget check from a broader investigation suite. The first should detect an obvious unacceptable change quickly. The second can explore input growth and concurrency with richer measurements. Neither should report a generic “performance passed” without identifying the conditions exercised.
When a threshold fails, preserve the input, software revision, environment, and result. Without those, the team spends its time debating the test rather than understanding the product behaviour.
Delivery controls contain what tests miss
Performance testing samples a space; it does not prove every future input is harmless. A release plan should still account for the consequences of an expensive rule reaching users.
A limited rollout can provide useful evidence when it monitors the receiving service’s health and has an actionable stop condition. A kill switch needs to remain reachable under the failure being considered. Those are properties to exercise, not reassuring labels to add to a document.
The CrowdStrike analysis explores a related question: changes delivered as content still deserve release controls when they alter critical runtime behaviour.
Bring performance into review conversations
During requirements review, I would ask which operations can consume unbounded work, which input sizes are permitted, and what the user should see if a limit is reached. During test design, I would turn those decisions into input families and measurable expectations.
That is quality engineering work before a load-test dashboard exists. It makes performance part of the product contract, where it can influence design, automation, and release decisions rather than appear as an unpleasant discovery after deployment.