Lesson 2 of 10Foundations

12 min · theory and practice · Basics

Files and logs: the first clues

Find the right file and read an error without drowning in terminal output.

You will be able to find configuration and match an error to the time of a failure.

A server has hundreds of files. Start by asking which component failed and where it leaves evidence. Configuration often lives in /etc, application data in /var/lib or /srv, and file logs in /var/log. These are starting points; get the actual paths from the service configuration.

Find your starting point

pwd shows the current directory. ls -lah includes hidden files, permissions, owners and sizes. A path starting with / is absolute; a relative path starts from your current directory.

Look at ownership and modification times. A new configuration beside an old one can help investigate a recent deployment, but a timestamp alone does not prove a cause.

Commands
pwd
ls -lah /etc/nginx
ls -lah /var/log/nginx

Use less or cat to inspect a file rather than executing it.

Read a bounded slice

tail -n 50 reads the last 50 lines. In less, search with / and quit with q. Context matters: what happened before the error, and which request triggered it?

Commands
tail -n 50 /var/log/nginx/error.log
less /var/log/nginx/error.log

File names in this example may differ on your server.

Separate evidence from a hypothesis

Find error lines, then return to adjacent lines and the incident time. An old ERROR may be unrelated. Match the path, upstream address and timestamp against configuration and a user request.

Commands
grep -n -i 'error' /var/log/nginx/error.log | tail -n 20
Example output
connect() failed (111: Connection refused) while connecting to upstream

This line points to the upstream address and availability. It does not explain why the backend is refusing connections.

Build a short evidence chain

For the store, separate the public 502 from shop-api’s startup failure. Record the request time, nginx message, service state and first application error. If the journal reports No space left on device, investigate the relevant filesystem rather than rewriting nginx.

After the fix, collect a new pair of observations: a successful request and no recurrence in fresh logs. Historical ERROR lines remain valid history; clearing them to make the output look clean is unnecessary.

Commands
sudo journalctl -u shop-api -n 30 --no-pager
curl -sS -i --max-time 5 http://localhost:8080/health

From symptom to check

ObservationNext checkHow to interpret it
502 Bad Gatewayerror.log and upstreamIdentify the nonresponding backend; the status does not explain why it failed.
shop-api: failedjournalctl -u shop-apiFind the first application error before systemd’s exit message.
No space left on devicedf and du on the reported pathMove from the log to filesystem checks while preserving user data.

Common mistakes

  • Treating an old error as current without matching timestamps.
  • Dumping the whole journal and losing the relevant context.
  • Deleting logs instead of fixing the application failure.

How to prove recovery

  1. Repeat the original user request and record its time.
  2. Verify the intended service is running, not only nginx.
  3. Inspect fresh logs after the successful request.
  4. Keep a concise symptom → check → confirmed cause chain.

MINI INCIDENT

What matters more than an old ERROR?

The store started returning 502 at 12:10. A file contains an error from 08:00 and a shop-api startup error from 12:09. Which evidence should you correlate, and which hypothesis comes first?

Evidence to investigate
08:00 unrelated maintenance warning
12:09 shop-api: failed to write heartbeat
12:10 nginx: connect() failed to upstream
Explore the diagnosis

Correlate the last two lines, upstream address and user-request time. Read the complete write-failure reason in shop-api and inspect that resource. The 08:00 error is not yet linked to this outage.

WORKSHOP

Workshop: explain the evidence and choose an action

These three lines describe different moments. The user noticed the failure at 14:22. Choose the related evidence rather than the most alarming word.

Sample evidence
13:10 ERROR old worker stopped
14:22 nginx: connect() failed upstream=127.0.0.1:8081
14:22 request GET /api/orders status=502
  1. Which two lines form a testable connection?

    Compare your reasoning

    The 502 request and connection failure share a time and path. The older ERROR remains separate context until a connection to the current failure is established.

  2. Which next experiment could disprove the upstream hypothesis?

    Compare your reasoning

    Probe the named address and port from the same network context, then inspect the listener and process. Success on another port does not test this hypothesis.

  3. What should be preserved for the post-repair comparison?

    Compare your reasoning

    Preserve method, URL, Host, time and initial status. Repeat that operation after the change and check the fresh log for the previous connection failure.

CHECK YOUR UNDERSTANDING

You found an ERROR from an hour ago. Is that enough for a diagnosis?

Show explanation

No. Compare its timestamp with the failure, reproduce the request and inspect new lines. Then test a specific hypothesis: a file path, port or service state.

Documentation

Guest progress is stored separately in this browser.

TAKE IT TO THE TERMINAL

Practice on an incident

Collect evidence in the introductory incident, then apply the same approach in the next case.

DID IT MAKE SENSE?

Was this article helpful?

Rate the article or tell us what needed more explanation. Your feedback helps improve the lessons.

Helpful 1 Not helpful 0

Sign in to rate this article and leave a comment.

Discussion

0

Join the discussion →

No comments yet. Share your first impression or ask a question about this topic.