Lesson 2 of 10Foundations

12 min · theory and practice · Basics

systemd: a service will not start

Read the failure reason and distinguish restart from daemon-reload.

You will be able to inspect a unit, its journal and actual application behavior after a change.

systemd starts processes using unit files. A failed status is an outcome; the cause is usually in the journal or startup parameters. Repeated restarts without diagnosis only repeat the same failure.

Read the state and journal

systemctl status shows state and recent messages. journalctl -u selects a unit’s journal. Look for the first meaningful application error before its exit message.

Commands
systemctl status shop-api --no-pager
sudo journalctl -u shop-api -n 60 --no-pager
Example output
Main process exited, code=exited, status=1/FAILURE

This line reports an unsuccessful exit. Look earlier in the journal for the cause, such as a missing file.

Inspect the effective unit

systemctl cat shows the unit source and drop-ins. Check ExecStart, User, WorkingDirectory and EnvironmentFile. The application runs as its configured user and may have different permissions and environment variables from your shell.

Commands
systemctl cat shop-api
systemctl show shop-api -p ExecStart -p User -p WorkingDirectory

Do not publish EnvironmentFile contents: they may contain secrets. Checking the path and required parameter is usually sufficient.

Apply the right kind of change

After editing a unit file, daemon-reload reloads systemd definitions. A subsequent restart starts the process again. Editing an application config does not always require daemon-reload; the application defines how that config is reread.

Commands
sudo systemctl daemon-reload
sudo systemctl restart shop-api
systemctl is-active shop-api
curl -sS -i --max-time 5 http://localhost:8080/health

These change commands assume the unit has already been corrected. active describes the unit state; the application request checks the result.

Inspect dependencies and the next start

In the shop-api case, the journal explains the exit, but the repair may be outside the unit, such as its full data filesystem. Repair the confirmed resource, then start the normal service with its existing identity and settings.

After restarting, collect the new PID, state and journal lines. If it exits seconds later, investigate the fresh error: active immediately after startup was only an intermediate observation.

Commands
systemctl show shop-api -p MainPID -p ActiveState -p SubState
sudo journalctl -u shop-api -n 20 --no-pager

From symptom to check

ObservationNext checkHow to interpret it
status=1/FAILUREEarlier journalctl linesThis is the exit outcome; look earlier for the application cause.
Unit file changed, behavior unchangeddaemon-reload, effective unit and PIDThe file edit may not yet be an effective parameter of a new process.
active, but requests failJournal and business requestA live process does not establish useful behavior.

Common mistakes

  • Restarting repeatedly without reading the first error.
  • Inspecting another service’s configuration or an old PID.
  • Switching to root instead of repairing resource access.

How to prove recovery

  1. Inspect the effective unit and intended user.
  2. Retrieve the new PID after restart.
  3. Confirm the service stays active beyond its first seconds.
  4. Repeat the original request and inspect fresh logs.

MINI INCIDENT

Why did restart not apply ExecStart?

The file contains a new command, but systemctl show reports the old one. How do you identify the layer that has not updated?

Evidence to investigate
unit file: ExecStart=/opt/app/new-api
effective ExecStart: /opt/app/old-api
Explore the diagnosis

Reload systemd definitions, inspect cat/show, perform a controlled restart and check the new PID. Also confirm you edited the intended unit rather than a file overridden by a drop-in.

WORKSHOP

Workshop: explain the evidence and choose an action

The application starts and exits one second later. Read the sequence, not only the latest status.

Sample evidence
14:10:00 Started shop-api
14:10:01 app: configuration file not found
14:10:01 Main process exited, status=1
14:10:02 Scheduled restart job
  1. Which line explains the repeated restarts?

    Compare your reasoning

    The initial application error is the missing configuration file. Scheduled restart describes the manager’s reaction to exit rather than the reason for termination.

  2. Which unit properties should be compared with application expectations?

    Compare your reasoning

    Compare ExecStart, arguments, WorkingDirectory, User and configuration path. The file may exist elsewhere, so the error wording alone does not prove the required data is absent.

  3. How can stability be checked after the change?

    Compare your reasoning

    Start again, read fresh logs, verify a stable PID and perform the real request. An active snapshot between restarts does not establish that the loop ended.

CHECK YOUR UNDERSTANDING

You changed ExecStart and ran restart. Why might the old command still be used?

Show explanation

systemd may not have reloaded the unit definition. Run daemon-reload, then restart, and inspect the effective parameters and journal.

Documentation

Guest progress is stored separately in this browser.

TAKE IT TO THE TERMINAL

Practice on an incident

Inspect the shop-api unit and journal, find the cause of failure and restore the application.

DID IT MAKE SENSE?

Was this article helpful?

Rate the article or tell us what needed more explanation. Your feedback helps improve the lessons.

Helpful 0 Not helpful 0

Sign in to rate this article and leave a comment.

Discussion

0

Join the discussion →

No comments yet. Share your first impression or ask a question about this topic.