Lesson 7 of 10Diagnostics

14 min · theory and practice · Basics

Filesystems: mounts, deleted files and free space

Why df and du disagree, and what can block writes despite free space.

Identify the filesystem behind a path and explain space retained by an open file.

“The disk has free space” is incomplete evidence. An application writes to a particular path, which may belong to another filesystem or a read-only mount. Deleting a filename does not necessarily release its allocated blocks.

Tie the observation to a path

Give df the application’s directory. Then use findmnt -T to identify the filesystem serving that path. Compare its source, type and mount options; a single directory tree can combine several filesystems.

A check inside another container or host describes a different environment. Establish where the service runs and where the probe runs; otherwise two correct outputs can answer different questions.

Commands
df -h /var/lib/shop
df -i /var/lib/shop
findmnt -T /var/lib/shop

Separate allocated blocks from filenames

du walks reachable names and totals their storage; df reports filesystem allocation. A process can retain an open deleted file after its ordinary directory entry disappears, leaving allocated space behind.

A large discrepancy does not prove this particular cause: check traversal boundaries, permissions and mountpoints. lsof +L1 tests the open-unlinked-file hypothesis; it is not a universal cleanup command.

Commands
sudo du -xhd1 /var/lib/shop
sudo lsof +L1

Inspect the write failure itself

No space left on device and Read-only file system need different investigations. The first points toward blocks or inodes; the second toward mount options and system messages. More deletion cannot repair a read-only mount.

Do not change mount mode blindly. If a storage error caused the mode change, forcing writes can worsen the situation. In this exercise, identify the cause and justify the next step.

Commands
findmnt -T /var/lib/shop -o TARGET,SOURCE,FSTYPE,OPTIONS
sudo journalctl -k -n 50 --no-pager

Release the resource and verify its growth rate

Once the log’s owning process is known, use its supported rotation or reopening mechanism. A restart can affect users; verify whether an appropriate reload exists and what it actually does.

Verification means more than free blocks. Repeat the user operation, inspect the new log and compare usage after a short interval. Otherwise the change may only postpone the next full disk.

From symptom to check

ObservationNext checkHow to interpret it
High df usage, low du totalCheck the same filesystem and lsof +L1An open deleted file is one testable explanation.
Free blocks, failing writesCompare inodes and mount optionsBlock capacity is only one constraint.
Deletion does not change dfFind the descriptor’s ownerA filename and an open resource have different lifetimes.

Common mistakes

  • Inspecting / when the application writes to a separate mount.
  • Deleting more data without explaining the df/du discrepancy.
  • Forcing writes before understanding why the mount is read-only.

How to prove recovery

  1. The path, filesystem and limiting resource are identified.
  2. The measurements’ disagreement is explained, not just a large file found.
  3. The service can write and does not resume the previous growth pattern.

MINI INCIDENT

The file disappeared, the gigabytes stayed

A 6 GB access.log was removed. du shrank but df barely changed. Which observation connects the discrepancy to a process?

Evidence to investigate
COMMAND PID FD SIZE/OFF NAME
nginx 412 7w 6442450944 /var/log/nginx/access.log (deleted)
Explore the diagnosis

Descriptor 7 of process 412 retains the file. Confirm its owner and supported reopening mechanism, then compare df. The ordinary du walk no longer represents that open object.

WORKSHOP

Workshop: explain the evidence and choose an action

Space is available, but the error identifies mount mode.

Sample evidence
df blocks: 60% free
df inodes: 80% free
write: Read-only file system
mount options: ro
  1. Why do these facts not justify deleting a large file?

    Compare your reasoning

    The observed restriction is read-only mode, not exhausted blocks. Deletion does not repair that write contract.

  2. Where should the cause of ro mode be investigated?

    Compare your reasoning

    Correlate the relevant mount source/options with kernel logs and mount configuration before enabling writes.

  3. What belongs in recovery verification?

    Compare your reasoning

    Verify the intended mode, successful application operation and absence of the recurring error. Free capacity remains a separate measurement.

CHECK YOUR UNDERSTANDING

Why can writes fail despite free blocks?

Show explanation

Inodes may be exhausted, permissions may deny access, or the mount may be read-only. The exact error and path determine the next diagnostic branch.

Documentation

Guest progress is stored separately in this browser.

TAKE IT TO THE TERMINAL

Practice on an incident

Choose a linked case and write three diagnostic checks before changing configuration. After the repair, repeat the original user operation.

DID IT MAKE SENSE?

Was this article helpful?

Rate the article or tell us what needed more explanation. Your feedback helps improve the lessons.

Helpful 0 Not helpful 0

Sign in to rate this article and leave a comment.

Discussion

0

Join the discussion →

No comments yet. Share your first impression or ask a question about this topic.