Sockets, regular files and pipes consume file descriptors. Notifications may stop even with normal CPU and disk readings: the process cannot accept a connection or open a log. Ask how many descriptors are used, which limit applies and why usage grows.
Find the current PID and limit
ulimit -n describes the current shell and its descendants. A service launched earlier by systemd has its own soft and hard limits. A restart changes the PID; retrieve it again before comparing values.
systemctl show notify-gateway -p MainPID -p LimitNOFILE
PID=$(systemctl show -p MainPID --value notify-gateway)
grep "open files" /proc/$PID/limitsMax open files 1024 4096 filesThe example has a working soft limit of 1024. A hard limit of 4096 does not by itself give the process 4096 usable descriptors.
Count and classify usage
/proc/PID/fd contains links for open descriptors. Repeat the count under roughly constant load. Socket usage should relate to clients; thousands of references to one log with an unchanged client count call for a different explanation.
sudo ls /proc/$PID/fd | wc -l
sudo ls -l /proc/$PID/fd | head -n 25
sudo lsof -p "$PID" | head -n 25lsof may be unavailable; /proc is enough for an initial comparison. Small differences between snapshots are normal for a live process.
Separate two causes
A low limit with a predictable number of client connections is a capacity-setting problem. Continuous growth without increasing load suggests a leak. Raising the limit then only postpones failure and increases potential resource use.
In this incident, inspect both resource policy and audit mode: EMFILE alone cannot choose between them.
Make the change persist
For systemd, inspect the unit and drop-in that define LimitNOFILE. Changing limits.conf for PAM sessions does not change the existing service. Unit edits require reloading definitions and restarting, followed by checking the new process.
systemctl cat notify-gateway
systemctl show notify-gateway -p LimitNOFILEChoose a limit from the load profile and headroom, not the largest possible number. If usage leaks, fix its source.
Demonstrate stable client handling
After the fix, wait for the expected clients to connect and check health. Repeat the fd count: it should stabilize rather than merely drop immediately after restart. Another restart tests persistence, but does not replace observation under load.
From symptom to check
| Observation | Next check | How to interpret it |
|---|---|---|
| fd usage nears soft limit as clients increase | Process limit and load profile | The limit may be too low for expected work. |
| fd usage grows with steady clients | Descriptor types and repeated paths | Investigate retained resources or a leak. |
| Limit remains unchanged after restart | Unit, drop-in and new /proc/PID/limits | Check that the source changed and the new process inherited it. |
Common mistakes
- Raising ulimit only in your SSH shell.
- Increasing the limit without explaining repeated references to one file.
- Treating an immediate post-restart fd drop as proof a leak was fixed.
How to prove recovery
- Retrieve the current MainPID and its effective soft limit.
- Wait for the expected clients to connect.
- Check health and repeated fd snapshots under the same load.
- Verify configuration persists across a normal restart.
MINI INCIDENT
A larger limit, or another cause?
Client count stays at 300 with a soft limit of 4096. fd usage grows from 500 to 3000 in two minutes, mostly pointing to audit.log. Which hypothesis should precede a limit increase?
clients=300
fd: 500 -> 1500 -> 3000
new references: audit.logExplore the diagnosis
Investigate whether audit operations close the file and which setting enables that behavior. Client growth does not explain the usage; a larger limit only delays failure. Verify stable fd usage while the gateway performs useful work.
WORKSHOP
Workshop: explain the evidence and choose an action
The limit is unchanged while usage rises after equal request batches. Separate inadequate capacity from resource retention.
soft limit=1024
after 100 requests: fd=180
after 200 requests: fd=380
after 300 requests: fd=580Which trend deserves another check?
Compare your reasoning
Check whether descriptor usage returns toward baseline after requests finish. Growth can reflect a pool or cache, but no plateau or release strengthens a retention hypothesis.
Which object types should be separated?
Compare your reasoning
Regular files, sockets and pipes lead to different investigations. /proc/PID/fd and lsof for the current PID connect the count with a resource type.
Why does increasing the limit not finish verification?
Compare your reasoning
It may only postpone failure. Repeat a comparable series and verify a stable usage level, the current process’s effective limit and user responses.
CHECK YOUR UNDERSTANDING
Why does ulimit -n in an SSH session not prove notify-gateway is fixed?
Show explanation
It is a different process and launch context. Inspect the new PID’s limits, descriptor usage and client handling.
No comments yet. Share your first impression or ask a question about this topic.