Health returns 200 while a business request returns 401. These observations cover different routes and checks. Map who creates the header, who forwards it and who validates its trusted value. TCP reachability does not establish application trust.
Locate the component returning 401
Repeat /api/orders, then correlate its time with Gateway and Internal API logs. If the internal service receives it and rejects a token, the failure is beyond a successful connection. If it never arrives, investigate the intermediate route.
curl -sS -i --max-time 5 http://localhost:8080/health
curl -sS -i --max-time 5 http://localhost:8080/api/orders
systemctl list-units --type=service --no-pagerGET /health 200
GET /api/orders 401These URLs are for this case’s Gateway. Inspect journals for the units you actually identify.
Trace the secret source
Compare TOKEN_FILE paths and effective configuration on both sides. A new file does not mean a long-running process reread it. Coordinated rotation checks both producer and consumer rather than only a file timestamp.
systemctl cat gateway
systemctl cat internal-orders
ls -l /etc/gateway /etc/orders-authFind exact units and filenames on the server. Compare secrets locally using fingerprints; do not paste values into shared logs, screenshots or chat.
Inspect header forwarding through nginx
This exercise uses X-Internal-Token, a service-specific convention rather than a standard OAuth header. proxy_set_header can replace or remove a forwarded value. Match the relevant location and proxy_pass destination to the actual request.
sudo nginx -TCheck both the file and its applied reload. A direct Internal API success does not prove the nginx path works.
Distinguish secret format from validity checks
An opaque token may be compared as a string, whereas a JWT may require signature and time checks. token used before issued points to clocks or validity windows; token mismatch suggests another hypothesis. Do not apply a JWT setting to a service that does not use JWTs.
Establish the contract first. The rotation case needs the current secret and header; the clock case needs correct time.
Verify both acceptance and rejection
After the fix, the normal Gateway request must succeed through the whole chain. Separately check rejection with no secret and with a deliberately wrong training value. Disabling authentication to obtain 200 breaks the other half of the contract.
Use isolated training secrets. In a live system, coordinate such checks with the service owner without exposing current keys.
From symptom to check
| Observation | Next check | How to interpret it |
|---|---|---|
| Health 200, business request 401 | Route and rejecting component | Health does not exercise the same authentication contract. |
| Token sources have different fingerprints | TOKEN_FILE and loaded version | Establish the current rotation version and process context. |
| Direct call works; proxied call fails | X-Internal-Token forwarding | Check the relevant location and whether it replaces the header. |
Common mistakes
- Disabling authentication to obtain 200.
- Pasting a live secret into chat for comparison.
- Restarting only one side of a coordinated rotation.
How to prove recovery
- Verify the normal request through Gateway, nginx and Internal API.
- Confirm both sides loaded the intended secret version.
- Verify rejection without a token and with a deliberately wrong training value.
- Verify consistent behavior after a normal restart.
MINI INCIDENT
Where should you look after rotation?
Gateway and Internal API use different TOKEN_FILE paths. A direct request with the current training token works, but Gateway receives 401. How can you narrow the cause without disabling validation?
Gateway source: previous.token
Internal API source: current.token
GET /health: 200
GET /api/orders: 401Explore the diagnosis
Compare the rotation contract, file fingerprints and what Gateway actually loaded. Then inspect header forwarding through nginx. Once aligned, normal requests must work while invalid-token requests remain rejected.
WORKSHOP
Workshop: explain the evidence and choose an action
401s began after a rotation. Compare component contracts without disclosing secrets.
gateway: credentials version=v2
internal-orders: credentials version=v1
nginx forwards header X-Internal-TokenWhich hypothesis is stronger than a network timeout here?
Compare your reasoning
The components use different credential versions. A 401 confirms an application refusal rather than a missing connection. Inspect the loaded configuration sources on both sides.
How can a wrong credential be separated from a lost header?
Compare your reasoning
Compare the expected header name and format with effective proxy configuration and receiver logs. Compare values locally or by version rather than posting the secret in a comment.
Which test pair is required after repair?
Compare your reasoning
A correctly authenticated request must succeed while a missing or invalid credential remains denied. Otherwise availability may have returned by disabling the control.
CHECK YOUR UNDERSTANDING
Why is /health returning 200 insufficient after token rotation?
Show explanation
Health may exercise a different route without the same authentication. Test a business request through every component and verify unauthorized rejection.
No comments yet. Share your first impression or ask a question about this topic.