A write failure can stop a database, logging or a queue. Start with the filesystem the application writes to. Free space in /home will not help if a separate /var filesystem is full.
Check the relevant mount
df -h shows block usage in readable units. Pass the data directory to identify its filesystem. Inspect Use%, Avail and Mounted on together.
df -h /var/lib/shop
df -i /var/lib/shopFilesystem Size Used Avail Use% Mounted on
/dev/vdb 10G 10G 0 100% /var/lib/shopdf -i checks inodes, the metadata used for files. Many tiny files can exhaust inodes before data blocks.
Locate the growth
du estimates space used by files and directories. Limit the search to the relevant tree to avoid an unnecessary whole-server scan. -x stays on one filesystem.
sudo du -xhd1 /var/lib/shop | sort -h120M /var/lib/shop/uploads
9.8G /var/lib/shop/exportsIn exports, identify the data owner, retention policy and source of growth. A large directory is not permission to delete it.
Verify recovery
Remove only data confirmed disposable under the service policy. If df and du differ greatly, check open deleted files: a process can retain their blocks until it closes the descriptor. lsof is not installed everywhere.
sudo lsof +L1
df -h /var/lib/shop
df -i /var/lib/shopAfter freeing space, test writes and the service request. Fix the growth source, such as broken retention or unbounded file creation.
Fix the growth rate as well as fullness
For a large log, inspect logging level and rotation; for many small cache files, inspect expiry and cleanup. Customer uploads and disposable cache have different value. Establish a directory’s purpose from application configuration before removing data.
Where installed, logrotate -d inspects its planned action without rotating. After the fix, observe growth and write availability. Freeing space once does not establish that the next night’s growth is controlled.
ls -l /etc/logrotate.d
sudo logrotate -d /etc/logrotate.confLog rotation handles logs; for the inode case, investigate cache cleanup separately while preserving uploads.
From symptom to check
| Observation | Next check | How to interpret it |
|---|---|---|
| df -h: Use% 100% | du on the same filesystem | Locate the large block consumer and establish its purpose. |
| df -h: 70% free; df -i: 100% | Directories containing many small files | Inodes are exhausted; one large file’s size is not the issue. |
| du is lower than df after deletion | lsof +L1 | A process may still hold the deleted file open. |
Common mistakes
- Deleting uploads with disposable cache.
- Freeing blocks without checking inodes.
- Clearing a log while retaining debug mode and broken rotation.
How to prove recovery
- Check both df -h and df -i on the data directory.
- Verify the application can write its working data again.
- Preserve customer files and the case’s protected data.
- Observe growth after repairing cleanup or rotation.
MINI INCIDENT
A large file is not always the culprit
The API cannot create a file. The filesystem has 8 GB free but no free inodes. There is a large archive and a directory with a million cache objects. Where do you start?
Avail: 8G
IFree: 0
IUse%: 100%Explore the diagnosis
First establish the small objects’ purpose and cleanup policy. Removing one archive releases few inodes and does not address mass accumulation. Confirm the objects are disposable cache rather than customer uploads, then verify writes recover.
WORKSHOP
Workshop: explain the evidence and choose an action
The directory contains thousands of small files. Compare both capacity measures before choosing a cleanup.
df -h: 45% used
df -i: 100% used
application: No space left on deviceWhy does the error not contradict the free gigabytes?
Compare your reasoning
New filesystem objects need inodes, which are exhausted here. Free blocks do not solve inode exhaustion; investigate file counts and their source.
How can the search avoid unrelated filesystems?
Compare your reasoning
Tie df to the application path and keep the investigation on that filesystem. Compare directories containing many small files rather than only the largest objects by size.
What should change after cleanup?
Compare your reasoning
Free inodes, successful creation of the intended file and the rate of new objects. If the generator keeps the same pace, the recovered capacity will run out again.
CHECK YOUR UNDERSTANDING
There are 5 GB free, but creating a file fails with ENOSPC. What next?
Show explanation
Run df -i on the same filesystem. If inodes are exhausted, freeing one large file does not address the accumulation of tiny files.
No comments yet. Share your first impression or ask a question about this topic.