When the Backend Recovers but the App Still Can't Reach It

A set of Cloud Foundry applications sharing a message broker instance started failing with connection timeouts and JMS exceptions. The natural assumption — the broker is down — turned out to be wrong within minutes of looking at the broker itself. It stayed wrong for a while longer before the real explanation surfaced.

The broker was fine

The broker ran as an active/standby/monitor trio. Its own logs told a clean story: the standby detected a volume-attach failure on the active node, promoted itself, and completed the failover in about three minutes with zero dropped connections on its side and no crash loop. By every measure the broker itself exposed, it was healthy again well before the client applications recovered.

And yet, fifteen-plus minutes after that clean failover, client apps were still logging fresh connection timeouts. A healthy backend and failing callers, at the same time, for a sustained period — that combination means the problem isn’t in the backend. It’s in the path to it, or in the client’s own state.

Two culprits, both worth checking whenever you see this pattern

1. Load balancer health-check convergence. The Service in front of the broker used externalTrafficPolicy: Local — meaning only the node hosting the currently active pod passes the load balancer’s own health check. A failover doesn’t just change which pod is active; it changes which node the load balancer needs to be routing to. That re-registration isn’t instant, and for however long it takes to converge, new connection attempts can land on a node the balancer hasn’t caught up to yet — even though the broker itself has been ready and accepting connections the whole time.

2. Client-side DNS caching. The affected clients were long-lived JVM processes with retrying connections. The JVM caches successful DNS lookups indefinitely by default (networkaddress.cache.ttl), so a client that had already resolved and connected before the failover can keep retrying against a resolution it made minutes or hours earlier, rather than re-resolving to whatever the failover changed. New processes, or apps configured to respect a sane DNS TTL, don’t have this problem — long-lived ones do, silently.

Neither of these shows up in the backend’s own health signals. Both show up as “the thing everyone’s staring at looks fine, and the callers are still broken.”

Diagnosing it without guessing

The useful next step wasn’t restarting the broker again — it was clean already. It was restaging one affected app instance to force a fresh DNS resolution and a genuinely new connection attempt, which confirms or rules out the client-side caching theory in one step, and checking the load balancer’s target-group registration state directly rather than inferring it from symptoms.

The generalizable version

Whenever “the backend’s own logs say healthy” and “clients still can’t connect” are both true at the same time, resist the urge to keep re-diagnosing the backend. Look at what sits between the two: load balancer health-check semantics (especially anything like externalTrafficPolicy: Local that ties routing to a specific node), and whether your clients are long-lived enough to be holding onto a DNS answer that’s no longer correct. The backend being fine is data, not a dead end.