Lesson 6 of 10Diagnostics

12 min · theory and practice · Basics

Disk space and inodes

Why “No space left on device” can appear even with gigabytes free.

You will be able to locate a full filesystem and check two different limits.

A write failure can stop a database, logging or a queue. Start with the filesystem the application writes to. Free space in /home will not help if a separate /var filesystem is full.

Check the relevant mount

df -h shows block usage in readable units. Pass the data directory to identify its filesystem. Inspect Use%, Avail and Mounted on together.

Commands
df -h /var/lib/shop
df -i /var/lib/shop
Example output
Filesystem  Size  Used Avail Use% Mounted on
/dev/vdb     10G   10G     0 100% /var/lib/shop

df -i checks inodes, the metadata used for files. Many tiny files can exhaust inodes before data blocks.

Locate the growth

du estimates space used by files and directories. Limit the search to the relevant tree to avoid an unnecessary whole-server scan. -x stays on one filesystem.

Commands
sudo du -xhd1 /var/lib/shop | sort -h
Example output
120M /var/lib/shop/uploads
9.8G /var/lib/shop/exports

In exports, identify the data owner, retention policy and source of growth. A large directory is not permission to delete it.

Verify recovery

Remove only data confirmed disposable under the service policy. If df and du differ greatly, check open deleted files: a process can retain their blocks until it closes the descriptor. lsof is not installed everywhere.

Commands
sudo lsof +L1
df -h /var/lib/shop
df -i /var/lib/shop

After freeing space, test writes and the service request. Fix the growth source, such as broken retention or unbounded file creation.

Fix the growth rate as well as fullness

For a large log, inspect logging level and rotation; for many small cache files, inspect expiry and cleanup. Customer uploads and disposable cache have different value. Establish a directory’s purpose from application configuration before removing data.

Where installed, logrotate -d inspects its planned action without rotating. After the fix, observe growth and write availability. Freeing space once does not establish that the next night’s growth is controlled.

Commands
ls -l /etc/logrotate.d
sudo logrotate -d /etc/logrotate.conf

Log rotation handles logs; for the inode case, investigate cache cleanup separately while preserving uploads.

From symptom to check

ObservationNext checkHow to interpret it
df -h: Use% 100%du on the same filesystemLocate the large block consumer and establish its purpose.
df -h: 70% free; df -i: 100%Directories containing many small filesInodes are exhausted; one large file’s size is not the issue.
du is lower than df after deletionlsof +L1A process may still hold the deleted file open.

Common mistakes

  • Deleting uploads with disposable cache.
  • Freeing blocks without checking inodes.
  • Clearing a log while retaining debug mode and broken rotation.

How to prove recovery

  1. Check both df -h and df -i on the data directory.
  2. Verify the application can write its working data again.
  3. Preserve customer files and the case’s protected data.
  4. Observe growth after repairing cleanup or rotation.

MINI INCIDENT

A large file is not always the culprit

The API cannot create a file. The filesystem has 8 GB free but no free inodes. There is a large archive and a directory with a million cache objects. Where do you start?

Evidence to investigate
Avail: 8G
IFree: 0
IUse%: 100%
Explore the diagnosis

First establish the small objects’ purpose and cleanup policy. Removing one archive releases few inodes and does not address mass accumulation. Confirm the objects are disposable cache rather than customer uploads, then verify writes recover.

WORKSHOP

Workshop: explain the evidence and choose an action

The directory contains thousands of small files. Compare both capacity measures before choosing a cleanup.

Sample evidence
df -h: 45% used
df -i: 100% used
application: No space left on device
  1. Why does the error not contradict the free gigabytes?

    Compare your reasoning

    New filesystem objects need inodes, which are exhausted here. Free blocks do not solve inode exhaustion; investigate file counts and their source.

  2. How can the search avoid unrelated filesystems?

    Compare your reasoning

    Tie df to the application path and keep the investigation on that filesystem. Compare directories containing many small files rather than only the largest objects by size.

  3. What should change after cleanup?

    Compare your reasoning

    Free inodes, successful creation of the intended file and the rate of new objects. If the generator keeps the same pace, the recovered capacity will run out again.

CHECK YOUR UNDERSTANDING

There are 5 GB free, but creating a file fails with ENOSPC. What next?

Show explanation

Run df -i on the same filesystem. If inodes are exhausted, freeing one large file does not address the accumulation of tiny files.

Documentation

Guest progress is stored separately in this browser.

TAKE IT TO THE TERMINAL

Practice on an incident

Compare two disk incidents and identify which limit was actually reached.

DID IT MAKE SENSE?

Was this article helpful?

Rate the article or tell us what needed more explanation. Your feedback helps improve the lessons.

Helpful 0 Not helpful 0

Sign in to rate this article and leave a comment.

Discussion

0

Join the discussion →

No comments yet. Share your first impression or ask a question about this topic.