“The disk has free space” is incomplete evidence. An application writes to a particular path, which may belong to another filesystem or a read-only mount. Deleting a filename does not necessarily release its allocated blocks.
Tie the observation to a path
Give df the application’s directory. Then use findmnt -T to identify the filesystem serving that path. Compare its source, type and mount options; a single directory tree can combine several filesystems.
A check inside another container or host describes a different environment. Establish where the service runs and where the probe runs; otherwise two correct outputs can answer different questions.
df -h /var/lib/shop
df -i /var/lib/shop
findmnt -T /var/lib/shopSeparate allocated blocks from filenames
du walks reachable names and totals their storage; df reports filesystem allocation. A process can retain an open deleted file after its ordinary directory entry disappears, leaving allocated space behind.
A large discrepancy does not prove this particular cause: check traversal boundaries, permissions and mountpoints. lsof +L1 tests the open-unlinked-file hypothesis; it is not a universal cleanup command.
sudo du -xhd1 /var/lib/shop
sudo lsof +L1Inspect the write failure itself
No space left on device and Read-only file system need different investigations. The first points toward blocks or inodes; the second toward mount options and system messages. More deletion cannot repair a read-only mount.
Do not change mount mode blindly. If a storage error caused the mode change, forcing writes can worsen the situation. In this exercise, identify the cause and justify the next step.
findmnt -T /var/lib/shop -o TARGET,SOURCE,FSTYPE,OPTIONS
sudo journalctl -k -n 50 --no-pagerRelease the resource and verify its growth rate
Once the log’s owning process is known, use its supported rotation or reopening mechanism. A restart can affect users; verify whether an appropriate reload exists and what it actually does.
Verification means more than free blocks. Repeat the user operation, inspect the new log and compare usage after a short interval. Otherwise the change may only postpone the next full disk.
From symptom to check
| Observation | Next check | How to interpret it |
|---|---|---|
| High df usage, low du total | Check the same filesystem and lsof +L1 | An open deleted file is one testable explanation. |
| Free blocks, failing writes | Compare inodes and mount options | Block capacity is only one constraint. |
| Deletion does not change df | Find the descriptor’s owner | A filename and an open resource have different lifetimes. |
Common mistakes
- Inspecting / when the application writes to a separate mount.
- Deleting more data without explaining the df/du discrepancy.
- Forcing writes before understanding why the mount is read-only.
How to prove recovery
- The path, filesystem and limiting resource are identified.
- The measurements’ disagreement is explained, not just a large file found.
- The service can write and does not resume the previous growth pattern.
MINI INCIDENT
The file disappeared, the gigabytes stayed
A 6 GB access.log was removed. du shrank but df barely changed. Which observation connects the discrepancy to a process?
COMMAND PID FD SIZE/OFF NAME
nginx 412 7w 6442450944 /var/log/nginx/access.log (deleted)Explore the diagnosis
Descriptor 7 of process 412 retains the file. Confirm its owner and supported reopening mechanism, then compare df. The ordinary du walk no longer represents that open object.
WORKSHOP
Workshop: explain the evidence and choose an action
Space is available, but the error identifies mount mode.
df blocks: 60% free
df inodes: 80% free
write: Read-only file system
mount options: roWhy do these facts not justify deleting a large file?
Compare your reasoning
The observed restriction is read-only mode, not exhausted blocks. Deletion does not repair that write contract.
Where should the cause of ro mode be investigated?
Compare your reasoning
Correlate the relevant mount source/options with kernel logs and mount configuration before enabling writes.
What belongs in recovery verification?
Compare your reasoning
Verify the intended mode, successful application operation and absence of the recurring error. Free capacity remains a separate measurement.
CHECK YOUR UNDERSTANDING
Why can writes fail despite free blocks?
Show explanation
Inodes may be exhausted, permissions may deny access, or the mount may be read-only. The exact error and path determine the next diagnostic branch.
No comments yet. Share your first impression or ask a question about this topic.