A unit defines an execution contract: executable, arguments, user, directory, environment and limits. A drop-in can change part of that contract without editing the original unit. Diagnosis needs the effective configuration and running-process properties.
Read the source unit and its additions
systemctl cat shows the unit files and drop-ins. Pay attention to path comments: the same property can appear in multiple sources. The first matching file is not necessarily the effective setting.
systemctl show adds the manager’s parsed properties. A file’s contents can differ from the manager’s loaded value before daemon-reload.
systemctl cat shop-api
systemctl show shop-api -p FragmentPath -p DropInPaths -p ExecStart -p User -p WorkingDirectoryDo not assume ExecStart is a shell
An ExecStart line is not an ordinary Bash command line. Shell redirections, pipelines and substitutions need an explicitly chosen shell and change the launch contract. The expected executable must exist and be accessible to the unit’s user.
Check executable and working-directory paths first. A configuration lookup error can occur after the program starts, whereas an EXEC error identifies an earlier failure boundary.
systemctl show shop-api -p ExecStart -p WorkingDirectory -p User
sudo journalctl -u shop-api -n 30 --no-pagerCompare declared environment with application expectations
Environment and EnvironmentFile supply process inputs. They need not match your interactive shell. Check the file location and access without publishing secret values.
Inspect limits on the current process PID. Changing the manager’s LimitNOFILE setting does not automatically replace the inherited limit of an already running process.
systemctl show shop-api -p MainPID -p LimitNOFILE
PID=$(systemctl show shop-api -p MainPID --value)
grep "open files" /proc/$PID/limitsSeparate the three application actions
daemon-reload rereads the manager’s unit definitions. Restart creates a new process run. Reload uses a service’s supported reconfiguration mechanism. Choose by what changed rather than the similarity of the words.
After applying a change, obtain the PID, log and user response again. Rapid PID changes are a restart-loop symptom: investigate the initial failure instead of only the growing restart count.
From symptom to check
| Observation | Next check | How to interpret it |
|---|---|---|
| File value changed, show value unchanged | Unit source and daemon-reload | The manager may not have reread the definition. |
| New manager value, old /proc limit | Current PID and launch mechanism | The existing process may have inherited the old limit. |
| Manual run works, unit fails | User, WorkingDirectory, environment | Compare execution conditions, not just the command. |
Common mistakes
- Adding Bash syntax to ExecStart without understanding execution.
- Checking a PID that has already been replaced.
- Treating daemon-reload as an application restart.
How to prove recovery
- The unit source, drop-ins and effective properties are identified.
- The execution conditions explain the manual-run difference.
- The new process has the intended properties and serves the real request.
MINI INCIDENT
A limit increased only on paper
After editing a drop-in, systemctl show reports 4096 while /proc/PID/limits still reports 1024. Explain the discrepancy and the next check before acting.
manager LimitNOFILE=4096
current process soft limit=1024Explore the diagnosis
The manager and an existing process are different state sources. Confirm the current PID, start time and launch mechanism, then plan a controlled new run if the new limit requires inheritance. Recheck /proc and the application afterward.
WORKSHOP
Workshop: explain the evidence and choose an action
A setting occurs in two files. Do not guess the effective value from filenames.
unit: WorkingDirectory=/opt/shop
drop-in: WorkingDirectory=/srv/shop
manager: WorkingDirectory=/srv/shopFrom which directory will the process resolve config.ini?
Compare your reasoning
From /srv/shop under this effective contract. /opt/shop/config.ini does not establish the file sought by this run.
What if the log still refers to the old path?
Compare your reasoning
Check the current PID, log time and whether a new run occurred. An old message can describe a previous process.
How can application be verified without exposing secrets?
Compare your reasoning
Compare safe path properties, the new PID and the application’s configuration-read result. A full environment dump is unnecessary.
CHECK YOUR UNDERSTANDING
When do you need daemon-reload rather than application configuration validation?
Show explanation
The first concerns unit definitions; the second concerns the application’s own configuration format and contents. Both can be necessary, but they answer different questions.
No comments yet. Share your first impression or ask a question about this topic.