Anatomy of a Cluster-Wide Outage: When One Component's Bad Luck Breaks Every Node

The alert was blunt: a production Kubernetes cluster’s health checks were failing across the board. Roughly two-thirds of the worker nodes went NotReady within a few minutes of each other. The instinct in that moment is to suspect the underlying cloud provider — a bad availability zone, a hardware fault, a networking change. None of that was it.

The chain

Most CNI plugins split responsibilities: a fan-out/sync component talks to the API server on behalf of the whole cluster, and a lightweight per-node agent applies routing rules locally based on what the sync component tells it. That split exists so every node doesn’t hammer the API server directly — but it also means the sync component is a single point of failure for something that looks, from the outside, like “the network.”

Here’s what happened, in order:

1. The sync component’s horizontal autoscaler reacted to a transient dip in perceived node count and scaled it from 2 replicas down to 1.
2. Both remaining/rescheduled replicas kept landing on nodes that were themselves becoming unhealthy — so the sync component never had a stable place to run.
3. Without it, every node’s local routing agent failed its readiness probe and stopped programming Service routes — including the in-cluster route to the API server itself.
4. That’s the detail that made this a cluster-wide incident rather than a handful of bad nodes: a node whose kubelet was still reporting healthy failed the exact same connectivity checks as a node that had gone fully NotReady, because the fault was in Service routing, not in any individual node’s health.
5. A safety controller watching for degraded node/API signals reacted by scaling down the cluster’s own management controllers — which stalled infrastructure reconciliation entirely, on top of the original networking failure.

Two-plus feedback loops, triggered by one component losing its footing.

Ruling things out with evidence, not assumption

The instinct to blame the cloud provider is natural and needs to be checked, not trusted. In this case:

• Every affected instance reported healthy status from the provider’s own side throughout — nothing was replaced or flagged.
• Infrastructure audit logs showed zero manual network-affecting changes (security groups, routing, load balancers) in the relevant window.
• A concurrent capacity scale-up succeeded cleanly with no quota or availability errors, ruling out “the cloud ran out of room” as a contributing factor.

Each of those took minutes to check and eliminated a plausible, distracting hypothesis. Incident investigations move faster when you spend those minutes early rather than half-building a narrative around an unverified assumption.

The fix that wasn’t really a fix

The cluster recovered when an unrelated, already-planned capacity increase gave the scheduler enough healthy nodes to finally land the sync component somewhere stable. Once it came back up, the local agents re-synced, routing recovered, and the reconcile backlog cleared within minutes.

That’s a lucky resolution, not an engineered one. The actual fix is making sure a component this critical can’t lose both replicas to a run of bad scheduling luck in the first place — pod anti-affinity so replicas don’t share fate, a PriorityClass so it’s never the thing evicted to make room, and a second look at whether an autoscaler tuned for a much larger pool should be scaling a 2-replica singleton-adjacent component down to 1 at all.

The takeaway

A component doesn’t need to be labeled “critical” in anyone’s architecture diagram to be a single point of failure for the whole cluster. If losing it breaks something as basic as Service routing, it needs the scheduling guarantees of a critical component — regardless of what it’s called or how small its replica count looks on paper.