Lesson 3 of 10Diagnostics

14 min · theory and practice · Basics

systemd in depth: environment, drop-ins and effective units

Find a setting’s source and separate manager configuration from application configuration.

Explain why a command works manually while its unit fails or ignores a change.

A unit defines an execution contract: executable, arguments, user, directory, environment and limits. A drop-in can change part of that contract without editing the original unit. Diagnosis needs the effective configuration and running-process properties.

Read the source unit and its additions

systemctl cat shows the unit files and drop-ins. Pay attention to path comments: the same property can appear in multiple sources. The first matching file is not necessarily the effective setting.

systemctl show adds the manager’s parsed properties. A file’s contents can differ from the manager’s loaded value before daemon-reload.

Commands
systemctl cat shop-api
systemctl show shop-api -p FragmentPath -p DropInPaths -p ExecStart -p User -p WorkingDirectory

Do not assume ExecStart is a shell

An ExecStart line is not an ordinary Bash command line. Shell redirections, pipelines and substitutions need an explicitly chosen shell and change the launch contract. The expected executable must exist and be accessible to the unit’s user.

Check executable and working-directory paths first. A configuration lookup error can occur after the program starts, whereas an EXEC error identifies an earlier failure boundary.

Commands
systemctl show shop-api -p ExecStart -p WorkingDirectory -p User
sudo journalctl -u shop-api -n 30 --no-pager

Compare declared environment with application expectations

Environment and EnvironmentFile supply process inputs. They need not match your interactive shell. Check the file location and access without publishing secret values.

Inspect limits on the current process PID. Changing the manager’s LimitNOFILE setting does not automatically replace the inherited limit of an already running process.

Commands
systemctl show shop-api -p MainPID -p LimitNOFILE
PID=$(systemctl show shop-api -p MainPID --value)
grep "open files" /proc/$PID/limits

Separate the three application actions

daemon-reload rereads the manager’s unit definitions. Restart creates a new process run. Reload uses a service’s supported reconfiguration mechanism. Choose by what changed rather than the similarity of the words.

After applying a change, obtain the PID, log and user response again. Rapid PID changes are a restart-loop symptom: investigate the initial failure instead of only the growing restart count.

From symptom to check

ObservationNext checkHow to interpret it
File value changed, show value unchangedUnit source and daemon-reloadThe manager may not have reread the definition.
New manager value, old /proc limitCurrent PID and launch mechanismThe existing process may have inherited the old limit.
Manual run works, unit failsUser, WorkingDirectory, environmentCompare execution conditions, not just the command.

Common mistakes

  • Adding Bash syntax to ExecStart without understanding execution.
  • Checking a PID that has already been replaced.
  • Treating daemon-reload as an application restart.

How to prove recovery

  1. The unit source, drop-ins and effective properties are identified.
  2. The execution conditions explain the manual-run difference.
  3. The new process has the intended properties and serves the real request.

MINI INCIDENT

A limit increased only on paper

After editing a drop-in, systemctl show reports 4096 while /proc/PID/limits still reports 1024. Explain the discrepancy and the next check before acting.

Evidence to investigate
manager LimitNOFILE=4096
current process soft limit=1024
Explore the diagnosis

The manager and an existing process are different state sources. Confirm the current PID, start time and launch mechanism, then plan a controlled new run if the new limit requires inheritance. Recheck /proc and the application afterward.

WORKSHOP

Workshop: explain the evidence and choose an action

A setting occurs in two files. Do not guess the effective value from filenames.

Sample evidence
unit: WorkingDirectory=/opt/shop
drop-in: WorkingDirectory=/srv/shop
manager: WorkingDirectory=/srv/shop
  1. From which directory will the process resolve config.ini?

    Compare your reasoning

    From /srv/shop under this effective contract. /opt/shop/config.ini does not establish the file sought by this run.

  2. What if the log still refers to the old path?

    Compare your reasoning

    Check the current PID, log time and whether a new run occurred. An old message can describe a previous process.

  3. How can application be verified without exposing secrets?

    Compare your reasoning

    Compare safe path properties, the new PID and the application’s configuration-read result. A full environment dump is unnecessary.

CHECK YOUR UNDERSTANDING

When do you need daemon-reload rather than application configuration validation?

Show explanation

The first concerns unit definitions; the second concerns the application’s own configuration format and contents. Both can be necessary, but they answer different questions.

Documentation

Guest progress is stored separately in this browser.

TAKE IT TO THE TERMINAL

Practice on an incident

Choose a linked case and write three diagnostic checks before changing configuration. After the repair, repeat the original user operation.

DID IT MAKE SENSE?

Was this article helpful?

Rate the article or tell us what needed more explanation. Your feedback helps improve the lessons.

Helpful 0 Not helpful 0

Sign in to rate this article and leave a comment.

Discussion

0

Join the discussion →

No comments yet. Share your first impression or ask a question about this topic.