Lesson 8 of 10Diagnostics

14 min · theory and practice · Advanced

File descriptors: a limit or a leak

Investigate Too many open files in the running process, not your shell.

Compare descriptor usage with its limit and distinguish legitimate load from a leak.

Sockets, regular files and pipes consume file descriptors. Notifications may stop even with normal CPU and disk readings: the process cannot accept a connection or open a log. Ask how many descriptors are used, which limit applies and why usage grows.

Find the current PID and limit

ulimit -n describes the current shell and its descendants. A service launched earlier by systemd has its own soft and hard limits. A restart changes the PID; retrieve it again before comparing values.

Commands
systemctl show notify-gateway -p MainPID -p LimitNOFILE
PID=$(systemctl show -p MainPID --value notify-gateway)
grep "open files" /proc/$PID/limits
Example output
Max open files            1024                 4096                 files

The example has a working soft limit of 1024. A hard limit of 4096 does not by itself give the process 4096 usable descriptors.

Count and classify usage

/proc/PID/fd contains links for open descriptors. Repeat the count under roughly constant load. Socket usage should relate to clients; thousands of references to one log with an unchanged client count call for a different explanation.

Commands
sudo ls /proc/$PID/fd | wc -l
sudo ls -l /proc/$PID/fd | head -n 25
sudo lsof -p "$PID" | head -n 25

lsof may be unavailable; /proc is enough for an initial comparison. Small differences between snapshots are normal for a live process.

Separate two causes

A low limit with a predictable number of client connections is a capacity-setting problem. Continuous growth without increasing load suggests a leak. Raising the limit then only postpones failure and increases potential resource use.

In this incident, inspect both resource policy and audit mode: EMFILE alone cannot choose between them.

Make the change persist

For systemd, inspect the unit and drop-in that define LimitNOFILE. Changing limits.conf for PAM sessions does not change the existing service. Unit edits require reloading definitions and restarting, followed by checking the new process.

Commands
systemctl cat notify-gateway
systemctl show notify-gateway -p LimitNOFILE

Choose a limit from the load profile and headroom, not the largest possible number. If usage leaks, fix its source.

Demonstrate stable client handling

After the fix, wait for the expected clients to connect and check health. Repeat the fd count: it should stabilize rather than merely drop immediately after restart. Another restart tests persistence, but does not replace observation under load.

From symptom to check

ObservationNext checkHow to interpret it
fd usage nears soft limit as clients increaseProcess limit and load profileThe limit may be too low for expected work.
fd usage grows with steady clientsDescriptor types and repeated pathsInvestigate retained resources or a leak.
Limit remains unchanged after restartUnit, drop-in and new /proc/PID/limitsCheck that the source changed and the new process inherited it.

Common mistakes

  • Raising ulimit only in your SSH shell.
  • Increasing the limit without explaining repeated references to one file.
  • Treating an immediate post-restart fd drop as proof a leak was fixed.

How to prove recovery

  1. Retrieve the current MainPID and its effective soft limit.
  2. Wait for the expected clients to connect.
  3. Check health and repeated fd snapshots under the same load.
  4. Verify configuration persists across a normal restart.

MINI INCIDENT

A larger limit, or another cause?

Client count stays at 300 with a soft limit of 4096. fd usage grows from 500 to 3000 in two minutes, mostly pointing to audit.log. Which hypothesis should precede a limit increase?

Evidence to investigate
clients=300
fd: 500 -> 1500 -> 3000
new references: audit.log
Explore the diagnosis

Investigate whether audit operations close the file and which setting enables that behavior. Client growth does not explain the usage; a larger limit only delays failure. Verify stable fd usage while the gateway performs useful work.

WORKSHOP

Workshop: explain the evidence and choose an action

The limit is unchanged while usage rises after equal request batches. Separate inadequate capacity from resource retention.

Sample evidence
soft limit=1024
after 100 requests: fd=180
after 200 requests: fd=380
after 300 requests: fd=580
  1. Which trend deserves another check?

    Compare your reasoning

    Check whether descriptor usage returns toward baseline after requests finish. Growth can reflect a pool or cache, but no plateau or release strengthens a retention hypothesis.

  2. Which object types should be separated?

    Compare your reasoning

    Regular files, sockets and pipes lead to different investigations. /proc/PID/fd and lsof for the current PID connect the count with a resource type.

  3. Why does increasing the limit not finish verification?

    Compare your reasoning

    It may only postpone failure. Repeat a comparable series and verify a stable usage level, the current process’s effective limit and user responses.

CHECK YOUR UNDERSTANDING

Why does ulimit -n in an SSH session not prove notify-gateway is fixed?

Show explanation

It is a different process and launch context. Inspect the new PID’s limits, descriptor usage and client handling.

Documentation

Guest progress is stored separately in this browser.

TAKE IT TO THE TERMINAL

Practice on an incident

Restore the notification gateway, persist the setting and verify all clients.

DID IT MAKE SENSE?

Was this article helpful?

Rate the article or tell us what needed more explanation. Your feedback helps improve the lessons.

Helpful 0 Not helpful 0

Sign in to rate this article and leave a comment.

Discussion

0

Join the discussion →

No comments yet. Share your first impression or ask a question about this topic.