Linux filesystem linux disk storage
Natasha Koval Senior Backend Developer

Linux No space left on device (ENOSPC)

Encountering "No space left on device" (ENOSPC) on Linux means your disk partition has run out of free space, and this guide explains how to identify, diagnose, and fix the issue.

What This Error Means

The "No space left on device" error, often accompanied by the ENOSPC errno (Error NO SPaCe), is a critical system-level message indicating that the disk partition where an application is trying to write data has no remaining free blocks. This isn't just a warning; it means the operation that triggered it—whether creating a file, writing to a log, downloading a package, or even updating system components—has failed.

When you hit ENOSPC, your system's ability to function normally is severely impaired. Applications will crash or fail to start, logs will stop writing, temporary files cannot be created, and you won't be able to install updates or new software. In essence, any operation requiring disk write access will fail. In my experience, this can quickly cascade into system instability if not addressed promptly, particularly on production servers where logs and temporary data accumulate rapidly.

Why It Happens

At its core, ENOSPC occurs because a filesystem has exhausted its allocated resources. This can manifest in two primary ways:

  1. Disk Block Exhaustion: The most common scenario, where the actual amount of free space (measured in bytes or blocks) on a partition has reached zero. This is what df -h typically reports. You've simply filled up the drive with data.
  2. Inode Exhaustion: Less common but equally debilitating, this happens when all available inodes on a filesystem have been used, even if there appears to be free disk space according to df -h. An inode is a data structure that stores information about a file or a directory (like ownership, permissions, and location on disk). Each file or directory requires one inode. If you have millions of tiny files, you can run out of inodes long before you run out of actual disk space. This is a subtle but important distinction that often puzzles engineers new to Linux system administration.

Common Causes

Identifying the root cause is the first step toward a lasting solution. In my career, I've seen ENOSPC triggered by various culprits:

  • Accumulated Log Files: System logs (/var/log), application logs (e.g., Nginx access logs, database logs, custom application logs), or journald entries can grow unchecked, especially on busy systems without proper log rotation policies.
  • Temporary Files and Caches: Uncleaned temporary directories (/tmp, /var/tmp), package manager caches (apt, yum, npm, pip), or build system caches (e.g., Maven, Gradle, Node.js node_modules folders) can consume significant space over time.
  • Large Application Data: Databases growing beyond their allocated space, large media files, user uploads, or data generated by data processing jobs can fill disks quickly.
  • Docker Images and Volumes: Docker can be a significant consumer of disk space. Unused images, stopped containers, and orphan volumes can quickly bloat a system, particularly when developers iterate frequently. I've lost count of the times I've tracked down "missing" disk space to an overabundance of Docker layers.
  • Forgotten Backups: Old backup archives, database dumps, or snapshots that were never deleted can accumulate, especially if automated cleanup routines fail.
  • User-Generated Content: On servers hosting websites or applications that allow user uploads, unrestricted file uploads can quickly consume available storage.
  • Inode Exhaustion (Many Small Files): As mentioned, systems handling a vast number of small files (e.g., caching proxies, mail servers, source control repositories with many small objects) can hit inode limits even with ample free space.

Step-by-Step Fix

When you encounter ENOSPC, approach it systematically. Safety first: be careful with rm -rf. Always verify paths and contents before deleting.

  1. Verify the Error and Affected Partition:
    First, confirm that the ENOSPC error is indeed due to a full disk and identify which partition is full.

    bash df -h df -i

    df -h (disk free, human-readable) shows disk space usage. Look for partitions with 100% Use%.
    df -i (disk free, inodes) shows inode usage. If IUse% is 100%, you have an inode problem, not just a space problem. This is a crucial diagnostic step I always start with.

  2. Identify Large Files/Directories:
    Once you know which partition is full (e.g., / or /var), start looking for large consumers of space. The du (disk usage) command is your best friend here.

    Navigate to the root of the affected partition (or / if it's the root filesystem) and run:

    ```bash

    For current directory, showing top 10 largest items

    du -sh * | sort -rh | head -n 10

    To find large files globally (this can take a long time on a large disk)

    Be mindful of the starting path, e.g., / or /var

    sudo find / -xdev -type f -size +1G -print0 | xargs -0 du -h | sort -rh | head -n 20
    `` Thefindcommand searches for files larger than 1GB (+1G), preventing it from traversing other filesystems (-xdev). Adjust+1Gto+100M` if you're looking for smaller, but numerous, large files.

  3. Check Log Files:
    Log files are a very common culprit.

    ```bash

    Check general log directory size

    sudo du -sh /var/log

    Check journalctl disk usage (for systems using systemd-journald)

    sudo journalctl --disk-usage

    Reduce journald logs to a specific size (e.g., 100MB)

    sudo journalctl --vacuum-size=100M

    Or by time (e.g., keep only logs from the last 7 days)

    sudo journalctl --vacuum-time=7d

    `` For application-specific logs, navigate to their respective directories (e.g.,/var/log/nginx,/var/log/mysql) and usedu -sh *to find large ones. Delete or truncate old log files usingecho > filenamefor active files (to avoid breaking the application) orrm old_log_filefor inactive ones. Implement or verify log rotation (e.g.,logrotate`).

  4. Clear Temporary Files:
    Temporary files can pile up.

    bash sudo rm -rf /tmp/* sudo rm -rf /var/tmp/*
    Be cautious: some applications might be actively using files in /tmp. It's generally safer to reboot if possible, as most /tmp contents are cleared on reboot. If not, consider selectively deleting older files or files owned by non-critical processes.

  5. Manage Docker Resources (if applicable):
    Docker can consume a lot of space.

    ```bash

    Show disk usage of Docker

    docker system df

    Remove all unused Docker data (containers, images, networks, build cache)

    This is often the quickest way to free up significant space on Docker hosts.

    docker system prune -a
    `` I've seendocker system prune -a` free up tens, sometimes hundreds, of gigabytes of disk space on a frequently used CI/CD runner or a developer machine.

  6. Empty Trash and Old Backups:
    If you're on a desktop environment, check your trash can. On servers, look for old backup directories or forgotten archives.

    ```bash

    Example: remove old tarballs or zip files

    sudo find /var/backups -type f -name "*.tar.gz" -mtime +30 -delete
    ```

  7. Delete Unnecessary Files:
    Once you've identified large, unneeded files or directories, delete them.
    bash sudo rm -rf /path/to/large/unnecessary/directory sudo rm /path/to/large/unnecessary/file
    Always double-check the path before hitting enter. There's no undo for rm -rf /!

  8. Extend Partition (If Feasible):
    If cleaning up isn't enough or the disk consistently fills up, you might need to extend the disk partition or add a new one. This typically involves modifying the underlying cloud VM instance, hypervisor, or physical disk configuration, followed by resizing the filesystem. This is a more involved process and often requires a planned outage.

  9. Monitor After Fix:
    After freeing up space, continue to monitor your disk usage to prevent recurrence. Implement automated log rotation, cleanup scripts, and monitoring alerts.

Code Examples

These commands are crucial for troubleshooting and fixing ENOSPC.

1. Check Disk Space and Inode Usage:

# Human-readable disk space usage
df -h

# Inode usage
df -i

2. Find Largest Directories (e.g., in /var):

# Summarize disk usage for immediate subdirectories of /var, sorted largest first
sudo du -sh /var/* | sort -rh | head -n 10

3. Find Large Files Globally (over 500MB):

# Find files larger than 500MB on the root filesystem (exclude other filesystems)
sudo find / -xdev -type f -size +500M -print0 | xargs -0 du -h | sort -rh | head -n 20

4. Manage Systemd Journal Logs:

# Check current journal log disk usage
sudo journalctl --disk-usage

# Vacuum (delete) old journal logs, keeping total size below 200MB
sudo journalctl --vacuum-size=200M

# Vacuum journal logs older than 7 days
sudo journalctl --vacuum-time=7d

5. Clean Docker System:

# Show current Docker disk usage
docker system df

# Remove all unused containers, images, volumes, and networks
docker system prune -a

Environment-Specific Notes

The ENOSPC error can manifest differently or require specific solutions depending on your environment.

  • Cloud (AWS EC2, GCP, Azure VMs):

    • Disk Resizing: Cloud providers offer straightforward ways to increase the size of an attached volume (EBS on AWS, Persistent Disk on GCP). After resizing the underlying volume, you'll still need to extend the filesystem within the OS (e.g., resize2fs for ext4, xfs_growfs for XFS).
    • Ephemeral Storage: Be wary of instances using ephemeral or instance store volumes. These are often smaller and are lost on instance termination, making them unsuitable for persistent data or applications with high disk usage without careful management.
    • Monitoring: Leverage cloud monitoring tools (CloudWatch, Stackdriver, Azure Monitor) to set up alerts for disk utilization. Proactive alerts are critical; I've used these extensively to catch ENOSPC before it impacts users.
    • Log Sinks: Consider sending logs to a dedicated log sink service (e.g., CloudWatch Logs, Stackdriver Logging) rather than storing them locally on the instance, reducing local disk pressure.

  • Docker Containers & Orchestration (Kubernetes):

    • Container Root Filesystem: A common issue in containers is the container's own writable layer filling up. This often points to logs or temporary data being written inside the container instead of to mounted volumes.
    • Volume Management: Ensure your Docker volumes are properly managed. If you're not explicitly cleaning up volumes, they can persist even after containers are removed. docker system prune -a is vital for hosts running many containers.
    • Kubernetes Pods: In Kubernetes, ENOSPC can affect a specific pod (if its container writable layer is full) or the node itself (if /var/lib/docker or /var/lib/kubelet fills up). For pod-level issues, investigate what the application is writing. For node-level, it's often accumulated images or logs. kubectl describe node can offer clues on node disk pressure. Persistent Volume Claims (PVCs) help manage storage, but they too can fill up.

  • Local Development Machines:

    • node_modules and Build Artifacts: For developers, node_modules directories can be enormous. Tools like npx rimraf node_modules or npm cache clean --force can help. Similarly, build artifacts for large projects (e.g., C++ projects, Android builds) can consume tens of GBs.
    • IDE Caches: Integrated Development Environments (IDEs) like IntelliJ, VS Code, or Eclipse create large caches. Periodically clearing these (check your IDE's settings) can free up space.
    • Snap Packages: Ubuntu's Snap packages create large, self-contained bundles. Running sudo snap save and then sudo snap remove --purge <snap-name> for unused snaps, followed by sudo snap set system refresh.retain=2 to limit old revisions, can reclaim space.

Frequently Asked Questions

Q: Can I ignore the "No space left on device" error?
A: Absolutely not. This error will prevent critical system operations, lead to application crashes, data loss, and ultimately system instability. It's a high-priority issue that demands immediate attention.

Q: My df -h shows plenty of free space, but I'm still getting ENOSPC. Why?
A: This is a classic symptom of inode exhaustion. Run df -i to check inode usage. If IUse% is 100%, you have too many small files. You'll need to locate and delete directories containing many tiny files (e.g., old cache directories, temporary files from failed processes) to free up inodes.

Q: How can I prevent this error from happening again?
A: Proactive measures are key:
* Implement robust log rotation (e.g., logrotate).
* Set up disk usage monitoring with alerts (e.g., using Prometheus + Grafana, cloud monitoring, or simple cron jobs checking df).
* Regularly prune Docker resources (docker system prune).
* Review application configurations to ensure they don't store excessive temporary data or logs.
* Consider increasing disk size or using dedicated storage for high-growth data.

Q: Is it safe to run rm -rf / to free up space?
A: NO! ABSOLUTELY NOT! This command recursively deletes everything from the root directory, effectively wiping your entire operating system and all its data. Always double-check your rm commands and use specific paths. When in doubt, use rm -i for interactive confirmation or mv to move suspect files to a temporary location first.

Q: What if I can't delete enough files to free up space?
A: If cleaning up isn't sufficient, you'll need to expand your storage. This involves either growing the existing disk partition (if there's unallocated space or if the underlying volume can be resized) or adding a new disk/volume and migrating data to it. In cloud environments, resizing a volume is often a straightforward process, though it usually requires a filesystem resize operation afterwards.