The sidecar GitHub's auto-scaler didn't know existed

On August 17, GitHub went down. Not a flaky endpoint, not one feature degraded while everything else limped along. Authentication, Issues, Pull Requests, Actions, Copilot, gone, globally, for 7 hours and 47 minutes. No bad deploy. No fat-fingered config push. Reading the postmortem, the thing that stands out isn't any single failure. It's how ordinary each individual failure was, and how far a chain of ordinary failures can travel when nothing stands between the links.

The traffic doubled and the infrastructure didn't notice

GitHub's monthly commit volume went from 1.4 billion to 2.9 billion since April. That's not a slow drift, that's a new peak arriving faster than the capacity plan accounted for. Nothing about that growth is a bug. It's success, the kind that quietly rewrites every assumption baked into a scaling policy written for the old peak.

Where it actually started: a pod nobody was watching

The failure's first domino was a service mesh sidecar, an Istio proxy sitting next to the main application container, routing its internal traffic. When the new traffic level arrived, that sidecar hit its own concurrency ceiling. It was supposed to scale out automatically. It didn't, because the auto-scaler was configured to watch the main host service's limits and never checked the sidecar's. The sidecar shares a pod with the thing everyone was measuring, but it isn't the thing everyone was measuring.

Roughly, the shape of the miss looks like this, not GitHub's actual policy, just the pattern:

# what the auto-scaler watched
metrics:
  - containerName: app          # the main service
    target: cpu.utilization

# what it never watched
#   - containerName: istio-proxy  # the sidecar in the same pod
#     target: concurrent_connections

A pod's resource limits are the sum of every container inside it, not just the one you remembered to name in the scaling policy. The main service could be sitting well under its own ceiling while its sidecar quietly maxes out a few centimeters away, and nothing about the host's own metrics would ever show it.

One gateway, every product

A sidecar that can't scale means traffic backs up behind it. That backup cascaded until four internal HAProxy load balancers ran out of flow capacity entirely. Those load balancers sat in front of the primary gateway authentication path, and authentication is a shared dependency every product on the platform calls through. A localized saturation problem in one Central US datacenter stopped being localized the moment it touched the one component nothing else on GitHub.com can route around.

The retry storm, or how a slowdown becomes a DDoS you did to yourself

Degraded authentication meant slower responses everywhere, including to Copilot token requests from VS Code. That's where a second, unrelated bug turned a bad afternoon into an outage. Every time a Copilot token failed to process in time, the VS Code client didn't back off, it retried aggressively and optimistically, on the assumption that the next attempt would probably just work. Multiply that assumption across every editor window on every laptop hitting the same degraded endpoint, and you get amplification, not recovery.

The Copilot Token Service normally runs 7,000 to 9,000 requests per second. Under the retry storm it hit 70,000 to 100,000, roughly 10x, self-inflicted, no external attacker required. And this is the detail that makes the recovery so painful to read about: systems couldn't be safely brought back online, because the instant they came up, the backlog of retries that had been queuing the whole time crushed them again. The outage kept re-triggering itself on the way out the door.

Roughly what was missing on the client side, again just the pattern, not GitHub's code:

// a retry budget: spend tokens to retry, refill slowly, stop when empty
class RetryBudget {
  private tokens = 100;
  private readonly costPerRetry = 10;

  canRetry(): boolean {
    return this.tokens >= this.costPerRetry;
  }

  recordRetry() {
    this.tokens -= this.costPerRetry;
  }

  refill(amount: number) {
    this.tokens = Math.min(100, this.tokens + amount);
  }
}

A retry without a budget isn't resilience, it's an unbounded loop that hasn't found its trigger condition yet. "Try again" is a reasonable instinct for one client talking to one flaky endpoint. It stops being reasonable the moment every client makes the same decision at the same time against the same already-struggling service.

What GitHub is actually changing

• Scaling policies now have to watch the whole pod, sidecars included, not just the one container someone remembered to name.
• Retry budgets and circuit breakers become mandatory across service-to-service calls, and the VS Code client gets patched so a slow response stops turning into a hundred slow responses.
• Authentication gets pried apart from everything that isn't authentication, so one region's bad day stays that region's bad day.
• A read architecture that scales with the number of readers instead of against them, landing first on the repos big enough to have broken the old one.
• More load moving off on-prem and into Azure, now 58% of the platform, up from 12% a few months back. Read into that whatever you want about how the on-prem side held up.

What I take from this

I've written on this blog about a retry that shipped as a silent success, about a rate limiter that overshot because check-and-increment wasn't atomic, and about chasing a hidden bottleneck through a request waterfall. Different scale, same root habits: a system deciding on its own how many times it's allowed to try again with nobody enforcing a boundary, and a shared dependency nobody was watching until it was the only thing left standing. GitHub's outage is what those habits look like once they're running at billions of requests instead of one confused user session or one slow embedding call. Individual components, service meshes and third-party clients alike, need strict limits, load-shedding, and circuit-breaking built in on purpose. Otherwise a single local bottleneck eventually finds the one shared dependency that turns it into everyone's problem at once.