The NetworkPolicy Drop That Looked Like Three Different Bugs

A service in one of our clusters started failing its Kafka consumer health checks. The error looked like a TLS problem:

Connection to node -N (/10.x.x.x:9094) terminated during authentication. This may happen due to:
(1) Firewall blocking TLS traffic, (2) Transient network issue.

Nothing about that message points at the real cause, and that ambiguity cost the investigation three false starts before landing on it.

False lead #1: the service mesh sidecar

The sidecar proxy seemed like an obvious suspect — it intercepts all pod traffic, and there was no explicit route configured for this particular external endpoint. A debug pod on the same node, deployed without a sidecar, completed the TLS handshake cleanly where the real app pod hung. That looked like confirmation.

It wasn’t. The debug pod also didn’t carry the labels the NetworkPolicy selector was matching on — so it bypassed the policy entirely, not just the sidecar. Two variables had changed between the “control” and the “real” pod, not one. A test that changes two things at once and reports one conclusion is worth re-running with only one thing changed.

False lead #2: a real bug, but not this bug

Along the way, a genuine routing gap turned up: a VPC peering route that existed for a set of legacy subnets but had never been extended to the newer Kubernetes worker subnets added during a later migration. Worth fixing on its own merits — but fixing it changed nothing. Packets flowed correctly in both directions afterward, and the app still couldn’t complete a handshake. It’s easy to declare victory on the first real bug you find in a multi-cause investigation; confirm it actually explains the symptom before moving on.

The actual cause

A NetworkPolicy egress rule allowlisted a handful of ports to the right CIDR block — except the Kafka port. One port, missing from one list:

egress:
  - to:
      - ipBlock: { cidr: 10.0.0.0/8 }
    ports:
      - { port: 8080, protocol: TCP }
      - { port: 8081, protocol: TCP }
      # 9094 (Kafka) was never added here

A NetworkPolicy DROP is silent — no RST packet, nothing in the connecting pod’s logs to distinguish it from a hung TLS handshake on the wire. That silence is exactly what sent the sidecar theory and the routing gap down the priority list ahead of the one-line fix.

The fix reverted itself — twice

Patching the policy live resolved it immediately. It also came back a few days later, identical symptom, same missing port. The live kubectl patch had never been committed to the Helm chart backing the deployment, so the next IaC-driven redeploy quietly reset it to the broken version. It happened once, got caught and re-patched, and happened again during an unrelated investigation days later.

The lesson generalizes past Kafka and past NetworkPolicy: a live patch to a resource your IaC also owns is not a fix, it’s a loan — due back, with no warning, on the next reconcile. If the fix matters, it belongs in the source the pipeline deploys from, not just in the live cluster state. Anything else will look fixed, report clean, and quietly regress on a schedule you don’t control.