A single URL can represent multiple application instances. One successful response establishes only that the selected path worked at that moment. Broader verification needs repeated probes, backend identification and an understanding of selection rules.
Collect a comparable request series
Repeat the same URL with the same Host and method. Record status and timing. A 200/502 pattern suggests heterogeneous paths, but does not by itself identify the balancing algorithm.
Six successful probes are not mathematical proof of full health. They are a short hypothesis check to combine with replica probes and backend-selection logs.
for n in 1 2 3 4 5 6; do
curl -sS --max-time 3 -o /dev/null -w "%{http_code} %{time_total}\n" http://localhost/
done200 0.004
502 0.003
200 0.005
502 0.003Probe every upstream directly
Read the address list from nginx -T. Probe application health and the relevant operation for each instance while preserving the required Host. An open TCP port does not guarantee a useful application response.
Compare the failure layer as well as the status: connection refused, timeout or an application error. They need different repairs even when the entry proxy presents the same 502 page.
sudo nginx -T
curl -sS -i --max-time 3 http://127.0.0.1:8081/health
curl -sS -i --max-time 3 http://127.0.0.1:8082/healthUnderstand how failures influence selection
nginx passive checking uses real traffic. max_fails and fail_timeout control behavior for counted failures; which responses count also depends on upstream error-handling configuration.
Process checks and user-path checks can disagree: an application may listen while using the wrong configuration or failing to reach a dependency. Readiness should test the capability that makes the instance eligible for traffic.
Compare replicas and verify recovery
Find the difference first: port, version, environment, data access or configuration. If an instance is temporarily excluded, record the capacity impact and the condition for returning it to service.
After the change, verify each restored instance and repeat the same requests through the entry URL. Valid nginx syntax, a successful reload and a healthy pool are three different observations.
From symptom to check
| Observation | Next check | How to interpret it |
|---|---|---|
| Entry URL is intermittent | Repeat the probe and identify the backend | Look for a relationship between outcome and selected instance. |
| One instance does not respond | Address, port, process, log | Locate the difference before changing the whole pool. |
| Open port, failing health | Application response and dependencies | A socket is not application readiness. |
Common mistakes
- Attributing every other failure to strict round-robin without logs.
- Permanently leaving one replica out without accounting for reduced capacity.
- Checking only an entry URL that may have selected the healthy backend.
How to prove recovery
- All upstream addresses map to the expected processes.
- Every replica passes an application-level probe.
- The entry route passes a repeat series after configuration is applied.
MINI INCIDENT
A half-healthy pool
nginx -t succeeds. Of two direct probes, one returns 200 and the other connection refused. What conclusion is supported, and what remains unknown?
8081/health → 200
8082/health → connection refused
nginx -t → successfulExplore the diagnosis
The configuration is syntactically valid, but one address is not serving the expected port from this context. Check the listener, upstream target and intended process. The evidence does not establish that the balancer alone is broken or that reload will start the missing backend.
WORKSHOP
Workshop: explain the evidence and choose an action
Both replicas pass an overly narrow liveness probe.
replica A: /health=200 /api/orders=200
replica B: /health=200 /api/orders=500
entry /api/orders: intermittent errorsWhy does green health not establish B’s readiness?
Compare your reasoning
Its contract does not test the required capability failing on B. Identify the dependency required by that route.
How can the difference be localized without changing the whole pool?
Compare your reasoning
Compare configuration, version and /api/orders logs on A and B using identical requests. Successful /health no longer separates the hypotheses.
What should count as verification after repair?
Compare your reasoning
Test the real operation on each replica and through the entry address, and define a useful readiness criterion. Health needs a link to the actual contract.
CHECK YOUR UNDERSTANDING
Why does /health need a clear contract?
Show explanation
A balancer can use it to decide traffic eligibility. If it checks only process liveness, an instance with a broken required dependency can incorrectly appear ready.
No comments yet. Share your first impression or ask a question about this topic.