Lesson 5 of 10Diagnostics

14 min · theory and practice · Basics

Load balancing: replicas and health checks

Investigate intermittent failures without assuming a random network problem.

Connect a failure to a replica and verify the entire pool after a change.

A single URL can represent multiple application instances. One successful response establishes only that the selected path worked at that moment. Broader verification needs repeated probes, backend identification and an understanding of selection rules.

Collect a comparable request series

Repeat the same URL with the same Host and method. Record status and timing. A 200/502 pattern suggests heterogeneous paths, but does not by itself identify the balancing algorithm.

Six successful probes are not mathematical proof of full health. They are a short hypothesis check to combine with replica probes and backend-selection logs.

Commands
for n in 1 2 3 4 5 6; do
  curl -sS --max-time 3 -o /dev/null -w "%{http_code} %{time_total}\n" http://localhost/
done
Example output
200 0.004
502 0.003
200 0.005
502 0.003

Probe every upstream directly

Read the address list from nginx -T. Probe application health and the relevant operation for each instance while preserving the required Host. An open TCP port does not guarantee a useful application response.

Compare the failure layer as well as the status: connection refused, timeout or an application error. They need different repairs even when the entry proxy presents the same 502 page.

Commands
sudo nginx -T
curl -sS -i --max-time 3 http://127.0.0.1:8081/health
curl -sS -i --max-time 3 http://127.0.0.1:8082/health

Understand how failures influence selection

nginx passive checking uses real traffic. max_fails and fail_timeout control behavior for counted failures; which responses count also depends on upstream error-handling configuration.

Process checks and user-path checks can disagree: an application may listen while using the wrong configuration or failing to reach a dependency. Readiness should test the capability that makes the instance eligible for traffic.

Compare replicas and verify recovery

Find the difference first: port, version, environment, data access or configuration. If an instance is temporarily excluded, record the capacity impact and the condition for returning it to service.

After the change, verify each restored instance and repeat the same requests through the entry URL. Valid nginx syntax, a successful reload and a healthy pool are three different observations.

From symptom to check

ObservationNext checkHow to interpret it
Entry URL is intermittentRepeat the probe and identify the backendLook for a relationship between outcome and selected instance.
One instance does not respondAddress, port, process, logLocate the difference before changing the whole pool.
Open port, failing healthApplication response and dependenciesA socket is not application readiness.

Common mistakes

  • Attributing every other failure to strict round-robin without logs.
  • Permanently leaving one replica out without accounting for reduced capacity.
  • Checking only an entry URL that may have selected the healthy backend.

How to prove recovery

  1. All upstream addresses map to the expected processes.
  2. Every replica passes an application-level probe.
  3. The entry route passes a repeat series after configuration is applied.

MINI INCIDENT

A half-healthy pool

nginx -t succeeds. Of two direct probes, one returns 200 and the other connection refused. What conclusion is supported, and what remains unknown?

Evidence to investigate
8081/health → 200
8082/health → connection refused
nginx -t → successful
Explore the diagnosis

The configuration is syntactically valid, but one address is not serving the expected port from this context. Check the listener, upstream target and intended process. The evidence does not establish that the balancer alone is broken or that reload will start the missing backend.

WORKSHOP

Workshop: explain the evidence and choose an action

Both replicas pass an overly narrow liveness probe.

Sample evidence
replica A: /health=200 /api/orders=200
replica B: /health=200 /api/orders=500
entry /api/orders: intermittent errors
  1. Why does green health not establish B’s readiness?

    Compare your reasoning

    Its contract does not test the required capability failing on B. Identify the dependency required by that route.

  2. How can the difference be localized without changing the whole pool?

    Compare your reasoning

    Compare configuration, version and /api/orders logs on A and B using identical requests. Successful /health no longer separates the hypotheses.

  3. What should count as verification after repair?

    Compare your reasoning

    Test the real operation on each replica and through the entry address, and define a useful readiness criterion. Health needs a link to the actual contract.

CHECK YOUR UNDERSTANDING

Why does /health need a clear contract?

Show explanation

A balancer can use it to decide traffic eligibility. If it checks only process liveness, an instance with a broken required dependency can incorrectly appear ready.

Documentation

Guest progress is stored separately in this browser.

TAKE IT TO THE TERMINAL

Practice on an incident

Choose a linked case and write three diagnostic checks before changing configuration. After the repair, repeat the original user operation.

DID IT MAKE SENSE?

Was this article helpful?

Rate the article or tell us what needed more explanation. Your feedback helps improve the lessons.

Helpful 0 Not helpful 0

Sign in to rate this article and leave a comment.

Discussion

0

Join the discussion →

No comments yet. Share your first impression or ask a question about this topic.