Lesson 5 of 10Diagnostics

12 min · theory and practice · Basics

Processes, CPU and memory

Find a resource-hungry process and distinguish a symptom from its cause.

You will be able to connect a PID and command to CPU or memory usage.

High CPU and low memory need different checks. A snapshot shows the present; repeated observations reveal a trend. Before restarting, record the PID, command, resource and time.

Look at the whole system

top shows processes and changing system metrics. In free -m, inspect available as well as free: some memory is used for reclaimable cache. Low free memory alone is not evidence of a failure.

Commands
uptime
free -m
top

Linux load average is not a CPU percentage: it also includes some waiting tasks. Compare it with the CPU count and other metrics.

Identify a specific process

ps takes a snapshot. Sorting by %cpu or %mem brings candidates to the top. RSS is resident memory in KiB; ps %CPU is an average over process lifetime and can differ from the current value in top.

Commands
ps -eo pid,ppid,user,%cpu,%mem,rss,args --sort=-%cpu | head
ps -eo pid,ppid,user,%cpu,%mem,rss,args --sort=-rss | head
Example output
PID  PPID USER  %CPU %MEM    RSS COMMAND
842     1 app   96.0  8.4 172000 python worker.py

The command and parent PID help identify who launched it. The values shown here are illustrative.

Check the consequences

If an application disappeared, inspect the kernel journal for OOM messages. If it is alive but saturating CPU, investigate its task and recent changes. The busiest process may be doing useful work; choose a remedy based on the user-facing symptom.

Commands
sudo journalctl -k -n 100 --no-pager | grep -i -E 'oom|killed process'

After a fix, repeat the user request and observe resources again. A falling graph alone is not verification.

Trace the setting generating the load

In the CPU case, worker count, work intensity and loop delay affect load differently. For OOM, inspect whether queued work is bounded, which process was killed and what restarts it. Killing a process lowers the graph without changing the setting that caused the load.

Compare repeated snapshots before and after one change. Preserve useful request handling: an application that is disabled and consumes no CPU is not recovered.

Commands
ps -eo pid,ppid,%cpu,rss,args --sort=-rss | head -n 12

From symptom to check

ObservationNext checkHow to interpret it
High CPU, process stays alivetop and CPU-sorted psRelate the busy process to worker count and behavior.
Killed process / OOMjournalctl -k and pre-failure RSSConfirm memory pressure and its victim rather than merely a new PID.
RSS grows under unchanged loadRepeated snapshots and queue boundsInspect retained work or a leak; one snapshot cannot establish a trend.

Common mistakes

  • Assuming low free memory proves OOM.
  • Restarting a continuously growing process and treating lower RSS as a fix.
  • Stopping the useful API merely to obtain zero CPU.

How to prove recovery

  1. Compare repeated CPU/RSS snapshots under comparable load.
  2. Verify a business request as well as process state.
  3. Ensure no new OOM events occur.
  4. Confirm the corrected setting persists across a normal restart.

MINI INCIDENT

The service recovered after kill. Is it resolved?

Stopping a heavy process restored the API. A minute later a new PID grows RSS at the same request rate. What must you investigate before another restart?

Evidence to investigate
12:00 PID 410 RSS 180 MiB
12:01 PID 410 RSS 320 MiB
12:02 PID 410 RSS 470 MiB
Explore the diagnosis

Inspect the launcher and the setting that bounds retained work. Growing RSS at a steady request rate calls for memory-retention analysis; another restart only resets the observation. Verify both stable memory and a working API after the change.

WORKSHOP

Workshop: explain the evidence and choose an action

Compare three snapshots of one service. A large RSS alone does not establish a leak.

Sample evidence
t=0s   PID=711 RSS=120MiB available=600MiB
t=30s  PID=711 RSS=240MiB available=470MiB
t=60s  PID=711 RSS=360MiB available=350MiB
  1. What changed, and what remains unknown?

    Compare your reasoning

    One PID’s RSS grows while available memory falls. Workload, expected caching and any eventual plateau remain unknown. A comparable series is needed before calling it a leak.

  2. What can the kernel log add?

    Compare your reasoning

    It can identify an OOM event and the selected victim. No OOM message does not negate memory growth; it represents a different threshold and event.

  3. What should be checked after reducing concurrency?

    Compare your reasoning

    Repeat a similar workload, compare process count and RSS over time, then test the user response. A post-restart memory drop shows a reset, not lasting correctness.

CHECK YOUR UNDERSTANDING

free shows little free memory but plenty available. Is the server definitely short of RAM?

Show explanation

No. Memory may be used for cache. Compare available memory, OOM messages, per-process usage and application behavior.

Documentation

Guest progress is stored separately in this browser.

TAKE IT TO THE TERMINAL

Practice on an incident

These cases cover CPU pressure and memory shortage. Pick one and test your hypothesis with commands.

DID IT MAKE SENSE?

Was this article helpful?

Rate the article or tell us what needed more explanation. Your feedback helps improve the lessons.

Helpful 0 Not helpful 0

Sign in to rate this article and leave a comment.

Discussion

0

Join the discussion →

No comments yet. Share your first impression or ask a question about this topic.