Lesson 8 of 10Diagnostics

12 min · theory and practice · Advanced

Service-to-service 401s: tokens and proxies

Locate the failing trust boundary in Gateway → Nginx → Internal API.

Distinguish a stale secret, a stripped header and a time-validity failure.

Health returns 200 while a business request returns 401. These observations cover different routes and checks. Map who creates the header, who forwards it and who validates its trusted value. TCP reachability does not establish application trust.

Locate the component returning 401

Repeat /api/orders, then correlate its time with Gateway and Internal API logs. If the internal service receives it and rejects a token, the failure is beyond a successful connection. If it never arrives, investigate the intermediate route.

Commands
curl -sS -i --max-time 5 http://localhost:8080/health
curl -sS -i --max-time 5 http://localhost:8080/api/orders
systemctl list-units --type=service --no-pager
Example output
GET /health       200
GET /api/orders   401

These URLs are for this case’s Gateway. Inspect journals for the units you actually identify.

Trace the secret source

Compare TOKEN_FILE paths and effective configuration on both sides. A new file does not mean a long-running process reread it. Coordinated rotation checks both producer and consumer rather than only a file timestamp.

Commands
systemctl cat gateway
systemctl cat internal-orders
ls -l /etc/gateway /etc/orders-auth

Find exact units and filenames on the server. Compare secrets locally using fingerprints; do not paste values into shared logs, screenshots or chat.

Inspect header forwarding through nginx

This exercise uses X-Internal-Token, a service-specific convention rather than a standard OAuth header. proxy_set_header can replace or remove a forwarded value. Match the relevant location and proxy_pass destination to the actual request.

Commands
sudo nginx -T

Check both the file and its applied reload. A direct Internal API success does not prove the nginx path works.

Distinguish secret format from validity checks

An opaque token may be compared as a string, whereas a JWT may require signature and time checks. token used before issued points to clocks or validity windows; token mismatch suggests another hypothesis. Do not apply a JWT setting to a service that does not use JWTs.

Establish the contract first. The rotation case needs the current secret and header; the clock case needs correct time.

Verify both acceptance and rejection

After the fix, the normal Gateway request must succeed through the whole chain. Separately check rejection with no secret and with a deliberately wrong training value. Disabling authentication to obtain 200 breaks the other half of the contract.

Use isolated training secrets. In a live system, coordinate such checks with the service owner without exposing current keys.

From symptom to check

ObservationNext checkHow to interpret it
Health 200, business request 401Route and rejecting componentHealth does not exercise the same authentication contract.
Token sources have different fingerprintsTOKEN_FILE and loaded versionEstablish the current rotation version and process context.
Direct call works; proxied call failsX-Internal-Token forwardingCheck the relevant location and whether it replaces the header.

Common mistakes

  • Disabling authentication to obtain 200.
  • Pasting a live secret into chat for comparison.
  • Restarting only one side of a coordinated rotation.

How to prove recovery

  1. Verify the normal request through Gateway, nginx and Internal API.
  2. Confirm both sides loaded the intended secret version.
  3. Verify rejection without a token and with a deliberately wrong training value.
  4. Verify consistent behavior after a normal restart.

MINI INCIDENT

Where should you look after rotation?

Gateway and Internal API use different TOKEN_FILE paths. A direct request with the current training token works, but Gateway receives 401. How can you narrow the cause without disabling validation?

Evidence to investigate
Gateway source: previous.token
Internal API source: current.token
GET /health: 200
GET /api/orders: 401
Explore the diagnosis

Compare the rotation contract, file fingerprints and what Gateway actually loaded. Then inspect header forwarding through nginx. Once aligned, normal requests must work while invalid-token requests remain rejected.

WORKSHOP

Workshop: explain the evidence and choose an action

401s began after a rotation. Compare component contracts without disclosing secrets.

Sample evidence
gateway: credentials version=v2
internal-orders: credentials version=v1
nginx forwards header X-Internal-Token
  1. Which hypothesis is stronger than a network timeout here?

    Compare your reasoning

    The components use different credential versions. A 401 confirms an application refusal rather than a missing connection. Inspect the loaded configuration sources on both sides.

  2. How can a wrong credential be separated from a lost header?

    Compare your reasoning

    Compare the expected header name and format with effective proxy configuration and receiver logs. Compare values locally or by version rather than posting the secret in a comment.

  3. Which test pair is required after repair?

    Compare your reasoning

    A correctly authenticated request must succeed while a missing or invalid credential remains denied. Otherwise availability may have returned by disabling the control.

CHECK YOUR UNDERSTANDING

Why is /health returning 200 insufficient after token rotation?

Show explanation

Health may exercise a different route without the same authentication. Test a business request through every component and verify unauthorized rejection.

Documentation

Guest progress is stored separately in this browser.

TAKE IT TO THE TERMINAL

Practice on an incident

Restore service trust while preserving rejection of missing or invalid credentials.

DID IT MAKE SENSE?

Was this article helpful?

Rate the article or tell us what needed more explanation. Your feedback helps improve the lessons.

Helpful 0 Not helpful 0

Sign in to rate this article and leave a comment.

Discussion

0

Join the discussion →

No comments yet. Share your first impression or ask a question about this topic.