A server has hundreds of files. Start by asking which component failed and where it leaves evidence. Configuration often lives in /etc, application data in /var/lib or /srv, and file logs in /var/log. These are starting points; get the actual paths from the service configuration.
Find your starting point
pwd shows the current directory. ls -lah includes hidden files, permissions, owners and sizes. A path starting with / is absolute; a relative path starts from your current directory.
Look at ownership and modification times. A new configuration beside an old one can help investigate a recent deployment, but a timestamp alone does not prove a cause.
pwd
ls -lah /etc/nginx
ls -lah /var/log/nginxUse less or cat to inspect a file rather than executing it.
Read a bounded slice
tail -n 50 reads the last 50 lines. In less, search with / and quit with q. Context matters: what happened before the error, and which request triggered it?
tail -n 50 /var/log/nginx/error.log
less /var/log/nginx/error.logFile names in this example may differ on your server.
Separate evidence from a hypothesis
Find error lines, then return to adjacent lines and the incident time. An old ERROR may be unrelated. Match the path, upstream address and timestamp against configuration and a user request.
grep -n -i 'error' /var/log/nginx/error.log | tail -n 20connect() failed (111: Connection refused) while connecting to upstreamThis line points to the upstream address and availability. It does not explain why the backend is refusing connections.
Build a short evidence chain
For the store, separate the public 502 from shop-api’s startup failure. Record the request time, nginx message, service state and first application error. If the journal reports No space left on device, investigate the relevant filesystem rather than rewriting nginx.
After the fix, collect a new pair of observations: a successful request and no recurrence in fresh logs. Historical ERROR lines remain valid history; clearing them to make the output look clean is unnecessary.
sudo journalctl -u shop-api -n 30 --no-pager
curl -sS -i --max-time 5 http://localhost:8080/healthFrom symptom to check
| Observation | Next check | How to interpret it |
|---|---|---|
| 502 Bad Gateway | error.log and upstream | Identify the nonresponding backend; the status does not explain why it failed. |
| shop-api: failed | journalctl -u shop-api | Find the first application error before systemd’s exit message. |
| No space left on device | df and du on the reported path | Move from the log to filesystem checks while preserving user data. |
Common mistakes
- Treating an old error as current without matching timestamps.
- Dumping the whole journal and losing the relevant context.
- Deleting logs instead of fixing the application failure.
How to prove recovery
- Repeat the original user request and record its time.
- Verify the intended service is running, not only nginx.
- Inspect fresh logs after the successful request.
- Keep a concise symptom → check → confirmed cause chain.
MINI INCIDENT
What matters more than an old ERROR?
The store started returning 502 at 12:10. A file contains an error from 08:00 and a shop-api startup error from 12:09. Which evidence should you correlate, and which hypothesis comes first?
08:00 unrelated maintenance warning
12:09 shop-api: failed to write heartbeat
12:10 nginx: connect() failed to upstreamExplore the diagnosis
Correlate the last two lines, upstream address and user-request time. Read the complete write-failure reason in shop-api and inspect that resource. The 08:00 error is not yet linked to this outage.
WORKSHOP
Workshop: explain the evidence and choose an action
These three lines describe different moments. The user noticed the failure at 14:22. Choose the related evidence rather than the most alarming word.
13:10 ERROR old worker stopped
14:22 nginx: connect() failed upstream=127.0.0.1:8081
14:22 request GET /api/orders status=502Which two lines form a testable connection?
Compare your reasoning
The 502 request and connection failure share a time and path. The older ERROR remains separate context until a connection to the current failure is established.
Which next experiment could disprove the upstream hypothesis?
Compare your reasoning
Probe the named address and port from the same network context, then inspect the listener and process. Success on another port does not test this hypothesis.
What should be preserved for the post-repair comparison?
Compare your reasoning
Preserve method, URL, Host, time and initial status. Repeat that operation after the change and check the fresh log for the previous connection failure.
CHECK YOUR UNDERSTANDING
You found an ERROR from an hour ago. Is that enough for a diagnosis?
Show explanation
No. Compare its timestamp with the failure, reproduce the request and inspect new lines. Then test a specific hypothesis: a file path, port or service state.
No comments yet. Share your first impression or ask a question about this topic.