Skip to main content
A strong digital foundation for your business.

Knowledge Base

Practical guides that make your hosting, domain and virtual server services easier to understand.

Disk full: finding what uses space and cleaning logs and caches

When the disk fills up, the database cannot write, websites return 500 errors and sometimes even logging in over SSH becomes difficult. This guide helps you find what is using the space within minutes and free it up without losing data.

Symptoms and the first check

Typical symptoms are a No space left on device error in commands and logs, MariaDB stopping because it cannot write, a site returning 500 errors because PHP session files cannot be created, and apt or dnf operations failing halfway through. The first step is to tell whether you have run out of space or out of inodes:

Bash
df -h
df -i

If df -h shows a file system at 100%, look for large files using up the space. If df -h shows free space but IUse% in the df -i output is at 100%, you have run out of inodes: the file system uses one inode per file, and millions of small files (PHP sessions, cache files, a mail queue) can exhaust them. If the disk is completely full, first free a little space with the log and package cache cleanup below; most tools also need some free space to work.

Do not delete system directories at random

Deleting files you do not recognize under /var/lib, /usr, /etc or /boot breaks the package manager, the database or the boot process. For example, deleting files in /var/lib/mysql by hand can corrupt the database permanently. Use the cleanup commands of the tools covered in this guide.

Finding what uses the space

Bash
# Size of the top-level directories, on this file system only (-x), smallest to largest
sudo du -xh --max-depth=1 / 2>/dev/null | sort -h
# Repeat inside the largest directory
sudo du -xh --max-depth=1 /var 2>/dev/null | sort -h
# Files larger than 500 MB
sudo find / -xdev -type f -size +500M -exec ls -lh {} + 2>/dev/null
# Interactive browsing (Ubuntu/Debian: apt install ncdu, AlmaLinux: from EPEL)
sudo ncdu -x /

The -x option (-xdev for find) keeps the search on a single file system, so /proc and other mounted disks do not clutter the results. In ncdu you move around with the arrow keys and delete with d; make sure you know what a file is before deleting it. For an inode problem, search by file count rather than size:

Bash
sudo du --inodes -x --max-depth=1 / 2>/dev/null | sort -n | tail
sudo du --inodes -x --max-depth=1 /var 2>/dev/null | sort -n | tail

Repeat the command inside the directory with the most files. On Debian and Ubuntu, PHP sessions are stored under /var/lib/php/sessions and are normally cleaned up by a scheduled job; if this folder keeps growing, check your application's session settings.

The table below summarizes the most common space hogs on servers and the safe cleanup methods covered in this guide:

SourceCheckSafe cleanup
systemd journaljournalctl --disk-usagejournalctl --vacuum-size=200M
Files in /var/logls -lhS /var/logA logrotate rule, truncate -s 0
Package cachedu -sh /var/cache/apt /var/cache/dnfapt clean, dnf clean all
Old kernelsdpkg -l 'linux-image-*', rpm -q kernelapt autoremove --purge, dnf remove --oldinstallonly
Dockerdocker system dfdocker system prune (mind the options)
MySQL/MariaDB binlogSHOW BINARY LOGS;PURGE BINARY LOGS BEFORE ...
Deleted files that are still openlsof +L1Restarting the service concerned

Cleaning up logs safely

The systemd journal

Bash
journalctl --disk-usage
sudo journalctl --vacuum-size=200M
# or keep only the last 14 days
sudo journalctl --vacuum-time=14d

To keep the journal from growing again, set a permanent upper limit:

Bash
sudo mkdir -p /etc/systemd/journald.conf.d
printf '[Journal]\nSystemMaxUse=300M\n' | sudo tee /etc/systemd/journald.conf.d/size.conf
sudo systemctl restart systemd-journald
journalctl --disk-usage

logrotate and large log files

Files under /var/log are rotated by logrotate. Run the configuration in debug mode to check for errors, and find the largest logs:

Bash
sudo logrotate -d /etc/logrotate.conf
sudo ls -lhS /var/log | head
sudo find /var/log -type f -size +100M -exec ls -lh {} +

If your application's own logs (for example /var/www/app/storage/logs) are not covered by logrotate, add a rule in /etc/logrotate.d/app:

Text
/var/www/app/storage/logs/*.log {
    weekly
    rotate 4
    compress
    missingok
    notifempty
    copytruncate
}

copytruncate lets you rotate the log without restarting the application that keeps the file open. Do not delete a large log that a running service is writing to with rm; the space is not freed as long as the process keeps the file open. Empty it instead: sudo truncate -s 0 /var/log/file.log.

Package caches and old kernels

Bash
# Ubuntu/Debian
sudo apt clean
sudo apt autoremove --purge
dpkg -l 'linux-image-*' | grep '^ii'
# AlmaLinux
sudo dnf clean all
sudo dnf remove --oldinstallonly
rpm -q kernel
# The running kernel: never remove this version
uname -r
# Ubuntu: old snap revisions
snap list --all
sudo snap set system refresh.retain=2

apt clean deletes downloaded package archives. apt autoremove --purge removes packages that are no longer needed, old kernels included; it leaves the running kernel and the latest versions alone. On AlmaLinux, dnf remove --oldinstallonly keeps the running kernel and removes older versions beyond the installonly_limit value in /etc/dnf/dnf.conf. If there is a separate /boot partition, old kernels are the most common reason it fills up. On Ubuntu, disabled snap revisions also take up space; refresh.retain limits how many revisions are kept.

Docker, database logs and forgotten backups

Docker

Bash
docker system df
# Size of container logs
sudo sh -c 'du -sh /var/lib/docker/containers/*/*-json.log' 2>/dev/null | sort -h | tail
# Stopped containers, unused networks, dangling images and build cache
docker system prune
Be careful with prune options

docker system prune -a also deletes every image that no container uses, and --volumes also deletes unused volumes. Because the same command removes stopped containers first, the data of a database container you have stopped can be deleted irreversibly from its volume with --volumes. Review what will be deleted with docker system df -v first.

Container logs grow without limit by default. Set a limit in /etc/docker/daemon.json (if the file already exists, merge the settings with its current content) and restart Docker with sudo systemctl restart docker. This also restarts running containers, and the setting applies only to newly created containers, so you need to recreate the existing ones. Details: Docker installation guide.

JSON
{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}

MySQL/MariaDB binary logs (binlog)

If you see many files such as binlog.000123 or mysql-bin.000123 under /var/lib/mysql or /var/log/mysql, these are binary logs. MySQL 8 turns them on by default. Do not delete the files by hand; connect to the database and clean them up with SQL:

SQL
SHOW BINARY LOGS;
PURGE BINARY LOGS BEFORE NOW() - INTERVAL 3 DAY;

For a permanent retention period, use binlog_expire_logs_seconds (for example 259200 = 3 days) on MySQL 8 and MariaDB 10.6 or later, and expire_logs_days on older MariaDB versions. If you use replication, do not purge logs that the replica has not read yet.

Backup archives left on the server

Bash
sudo find / -xdev -type f \( -name '*.tar.gz' -o -name '*.zip' -o -name '*.sql' -o -name '*.sql.gz' \) -size +100M -exec ls -lh {} + 2>/dev/null

Archives made during a site migration or before an update are often forgotten on the server. Before deleting them, move them off the server if you need them: a backup sitting on the same disk as the server is of no use during a disk problem or a reinstall anyway (see the backup strategy).

Why isn't the space being freed?

Deleted files that are still open

If the total you found with du is much lower than the usage shown by df, there are most likely files that were deleted but are still held open by a process. The space is not released until the process closes the file.

Bash
sudo lsof +L1
# if lsof is not installed
sudo find /proc/*/fd -ls 2>/dev/null | grep '(deleted)'

The output shows the process holding the file; when you restart the service with sudo systemctl restart service-name, the space comes back. If you cannot restart the service, you can empty the open file through /proc: sudo truncate -s 0 /proc/PID/fd/FD.

ext4 reserved blocks

By default, ext4 file systems reserve 5% of their space for root; this is why root can keep working when the disk looks full. On a 40 GB disk this is about 2 GB, and in an emergency you can lower it to 1%:

Bash
# Device and type of the root file system
findmnt -no SOURCE,FSTYPE /
sudo tune2fs -l /dev/sda1 | grep -i 'reserved block count'
sudo tune2fs -m 1 /dev/sda1

Replace /dev/sda1 with the device from the findmnt output; with LVM the name looks like /dev/mapper/.... This is an emergency measure: do not set the reserve to zero, and do not treat it as a permanent fix. XFS has no such reserve.

Snapshots do not affect space inside the server

Snapshots in the client area are kept in the storage of the virtualization layer (the datastore), not on the disk inside your server. That is why deleting a snapshot does not increase the free space shown by df. You do need to delete snapshots before growing the disk, though; the disk cannot be grown while a snapshot exists.

The lasting fix: a disk upgrade and a usage alert

Cleanup gives temporary relief; if the disk fills up regularly, consider a package upgrade or an additional disk. After a disk upgrade, the partition and file system are grown automatically while the server is running and VMware Tools is active. A virtual disk cannot be shrunk later. For the steps, see the resource upgrade guide.

To hear about it before the disk fills up again, a small script run hourly by cron is enough. It writes file systems above the threshold to the system log, and also sends an email if mail delivery is configured on the server.

Bash
sudo tee /usr/local/bin/disk-alert.sh > /dev/null <<'EOF'
#!/bin/sh
# Writes file systems above the threshold to the system log and (if available) to email
THRESHOLD=${THRESHOLD:-85}
FULL=$(df -P -x tmpfs -x devtmpfs -x squashfs -x overlay | awk -v t="$THRESHOLD" 'NR>1 && int($5) >= t {print $6 " %" int($5)}')
[ -z "$FULL" ] && exit 0
logger -t disk-alert "Disk usage threshold exceeded: $FULL"
command -v mail >/dev/null && echo "$FULL" | mail -s "$(hostname): disk usage alert" admin@example.com
EOF
sudo chmod +x /usr/local/bin/disk-alert.sh
echo '0 * * * * root /usr/local/bin/disk-alert.sh' | sudo tee /etc/cron.d/disk-alert
# Test: run it with a temporarily lowered threshold, then check the log
sudo env THRESHOLD=1 /usr/local/bin/disk-alert.sh
journalctl -t disk-alert -n 5

Confirm that the cron service is running with systemctl status cron (crond on AlmaLinux). How to check scheduled jobs is explained in the cron guide.

Frequently asked questions

The disk is 100% full and commands fail. Where do I start?

First run the commands that free space quickly and safely: sudo journalctl --vacuum-size=200M and sudo apt clean (sudo dnf clean all on AlmaLinux). If you use ext4, temporarily lowering the reserved block percentage also gives you some breathing room. Once a few hundred MB are free, look for the real cause with du and ncdu.

I deleted files, but df did not change. Why?

Most likely the deleted files are still held open by a process; check with sudo lsof +L1 and restart the service concerned. Deleting snapshots in the client area does not change the df value inside the server either.

/var/lib/docker is very large. Can I delete the folder?

No. This folder holds images, containers and volumes; deleting it by hand breaks Docker and destroys the data in the volumes. See what is taking up space with docker system df, and use Docker's own commands such as docker system prune.

Was this helpful?

Thanks for your feedback!

Still Need Help?

Our support team is here around the clock for anything you can't find above.