# Disk full: finding what uses space and cleaning logs and caches

https://www.hostimor.com/en/knowledgebase/article/disk-full-freeing-up-space

When the disk fills up, the database cannot write, websites return 500 errors and sometimes even logging in over SSH becomes difficult. This guide helps you find what is using the space within minutes and free it up without losing data.

## Symptoms and the first check

Typical symptoms are a `No space left on device` error in commands and logs, MariaDB stopping because it cannot write, a site returning 500 errors because PHP session files cannot be created, and `apt` or `dnf` operations failing halfway through. The first step is to tell whether you have run out of space or out of inodes:

```
df -h
df -i
```

If `df -h` shows a file system at 100%, look for large files using up the space. If `df -h` shows free space but `IUse%` in the `df -i` output is at 100%, you have run out of inodes: the file system uses one inode per file, and millions of small files (PHP sessions, cache files, a mail queue) can exhaust them. If the disk is completely full, first free a little space with the log and package cache cleanup below; most tools also need some free space to work.

> ****Do not delete system directories at random****
> 
> Deleting files you do not recognize under `/var/lib`, `/usr`, `/etc` or `/boot` breaks the package manager, the database or the boot process. For example, deleting files in `/var/lib/mysql` by hand can corrupt the database permanently. Use the cleanup commands of the tools covered in this guide.

## Finding what uses the space

```
# Size of the top-level directories, on this file system only (-x), smallest to largest
sudo du -xh --max-depth=1 / 2>/dev/null | sort -h
# Repeat inside the largest directory
sudo du -xh --max-depth=1 /var 2>/dev/null | sort -h
# Files larger than 500 MB
sudo find / -xdev -type f -size +500M -exec ls -lh {} + 2>/dev/null
# Interactive browsing (Ubuntu/Debian: apt install ncdu, AlmaLinux: from EPEL)
sudo ncdu -x /
```

The `-x` option (`-xdev` for find) keeps the search on a single file system, so `/proc` and other mounted disks do not clutter the results. In `ncdu` you move around with the arrow keys and delete with `d`; make sure you know what a file is before deleting it. For an inode problem, search by file count rather than size:

```
sudo du --inodes -x --max-depth=1 / 2>/dev/null | sort -n | tail
sudo du --inodes -x --max-depth=1 /var 2>/dev/null | sort -n | tail
```

Repeat the command inside the directory with the most files. On Debian and Ubuntu, PHP sessions are stored under `/var/lib/php/sessions` and are normally cleaned up by a scheduled job; if this folder keeps growing, check your application's session settings.

The table below summarizes the most common space hogs on servers and the safe cleanup methods covered in this guide:

| Source | Check | Safe cleanup |
| --- | --- | --- |
| systemd journal | `journalctl --disk-usage` | `journalctl --vacuum-size=200M` |
| Files in `/var/log` | `ls -lhS /var/log` | A logrotate rule, `truncate -s 0` |
| Package cache | `du -sh /var/cache/apt /var/cache/dnf` | `apt clean`, `dnf clean all` |
| Old kernels | `dpkg -l 'linux-image-*'`, `rpm -q kernel` | `apt autoremove --purge`, `dnf remove --oldinstallonly` |
| Docker | `docker system df` | `docker system prune` (mind the options) |
| MySQL/MariaDB binlog | `SHOW BINARY LOGS;` | `PURGE BINARY LOGS BEFORE ...` |
| Deleted files that are still open | `lsof +L1` | Restarting the service concerned |

## Cleaning up logs safely

### The systemd journal

```
journalctl --disk-usage
sudo journalctl --vacuum-size=200M
# or keep only the last 14 days
sudo journalctl --vacuum-time=14d
```

To keep the journal from growing again, set a permanent upper limit:

```
sudo mkdir -p /etc/systemd/journald.conf.d
printf '[Journal]\nSystemMaxUse=300M\n' | sudo tee /etc/systemd/journald.conf.d/size.conf
sudo systemctl restart systemd-journald
journalctl --disk-usage
```

### logrotate and large log files

Files under `/var/log` are rotated by logrotate. Run the configuration in debug mode to check for errors, and find the largest logs:

```
sudo logrotate -d /etc/logrotate.conf
sudo ls -lhS /var/log | head
sudo find /var/log -type f -size +100M -exec ls -lh {} +
```

If your application's own logs (for example `/var/www/app/storage/logs`) are not covered by logrotate, add a rule in `/etc/logrotate.d/app`:

```
/var/www/app/storage/logs/*.log {
    weekly
    rotate 4
    compress
    missingok
    notifempty
    copytruncate
}
```

`copytruncate` lets you rotate the log without restarting the application that keeps the file open. Do not delete a large log that a running service is writing to with `rm`; the space is not freed as long as the process keeps the file open. Empty it instead: `sudo truncate -s 0 /var/log/file.log`.

## Package caches and old kernels

```
# Ubuntu/Debian
sudo apt clean
sudo apt autoremove --purge
dpkg -l 'linux-image-*' | grep '^ii'
# AlmaLinux
sudo dnf clean all
sudo dnf remove --oldinstallonly
rpm -q kernel
# The running kernel: never remove this version
uname -r
# Ubuntu: old snap revisions
snap list --all
sudo snap set system refresh.retain=2
```

`apt clean` deletes downloaded package archives. `apt autoremove --purge` removes packages that are no longer needed, old kernels included; it leaves the running kernel and the latest versions alone. On AlmaLinux, `dnf remove --oldinstallonly` keeps the running kernel and removes older versions beyond the `installonly_limit` value in `/etc/dnf/dnf.conf`. If there is a separate `/boot` partition, old kernels are the most common reason it fills up. On Ubuntu, disabled snap revisions also take up space; `refresh.retain` limits how many revisions are kept.

## Docker, database logs and forgotten backups

### Docker

```
docker system df
# Size of container logs
sudo sh -c 'du -sh /var/lib/docker/containers/*/*-json.log' 2>/dev/null | sort -h | tail
# Stopped containers, unused networks, dangling images and build cache
docker system prune
```

> ****Be careful with prune options****
> 
> `docker system prune -a` also deletes every image that no container uses, and `--volumes` also deletes unused volumes. Because the same command removes stopped containers first, the data of a database container you have stopped can be deleted irreversibly from its volume with `--volumes`. Review what will be deleted with `docker system df -v` first.

Container logs grow without limit by default. Set a limit in `/etc/docker/daemon.json` (if the file already exists, merge the settings with its current content) and restart Docker with `sudo systemctl restart docker`. This also restarts running containers, and the setting applies only to newly created containers, so you need to recreate the existing ones. Details: [Docker installation guide](https://www.hostimor.com/en/knowledgebase/article/install-docker-and-docker-compose).

```
{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}
```

### MySQL/MariaDB binary logs (binlog)

If you see many files such as `binlog.000123` or `mysql-bin.000123` under `/var/lib/mysql` or `/var/log/mysql`, these are binary logs. MySQL 8 turns them on by default. Do not delete the files by hand; connect to the database and clean them up with SQL:

```
SHOW BINARY LOGS;
PURGE BINARY LOGS BEFORE NOW() - INTERVAL 3 DAY;
```

For a permanent retention period, use `binlog_expire_logs_seconds` (for example 259200 = 3 days) on MySQL 8 and MariaDB 10.6 or later, and `expire_logs_days` on older MariaDB versions. If you use replication, do not purge logs that the replica has not read yet.

### Backup archives left on the server

```
sudo find / -xdev -type f \( -name '*.tar.gz' -o -name '*.zip' -o -name '*.sql' -o -name '*.sql.gz' \) -size +100M -exec ls -lh {} + 2>/dev/null
```

Archives made during a site migration or before an update are often forgotten on the server. Before deleting them, move them off the server if you need them: a backup sitting on the same disk as the server is of no use during a disk problem or a reinstall anyway (see the [backup strategy](https://www.hostimor.com/en/knowledgebase/article/server-backup-strategy-restic)).

## Why isn't the space being freed?

### Deleted files that are still open

If the total you found with `du` is much lower than the usage shown by `df`, there are most likely files that were deleted but are still held open by a process. The space is not released until the process closes the file.

```
sudo lsof +L1
# if lsof is not installed
sudo find /proc/*/fd -ls 2>/dev/null | grep '(deleted)'
```

The output shows the process holding the file; when you restart the service with `sudo systemctl restart service-name`, the space comes back. If you cannot restart the service, you can empty the open file through `/proc`: `sudo truncate -s 0 /proc/PID/fd/FD`.

### ext4 reserved blocks

By default, ext4 file systems reserve 5% of their space for root; this is why root can keep working when the disk looks full. On a 40 GB disk this is about 2 GB, and in an emergency you can lower it to 1%:

```
# Device and type of the root file system
findmnt -no SOURCE,FSTYPE /
sudo tune2fs -l /dev/sda1 | grep -i 'reserved block count'
sudo tune2fs -m 1 /dev/sda1
```

Replace `/dev/sda1` with the device from the `findmnt` output; with LVM the name looks like `/dev/mapper/...`. This is an emergency measure: do not set the reserve to zero, and do not treat it as a permanent fix. XFS has no such reserve.

### Snapshots do not affect space inside the server

**Snapshots** in the client area are kept in the storage of the virtualization layer (the datastore), not on the disk inside your server. That is why deleting a snapshot does not increase the free space shown by `df`. You do need to delete snapshots before growing the disk, though; the disk cannot be grown while a snapshot exists.

## The lasting fix: a disk upgrade and a usage alert

Cleanup gives temporary relief; if the disk fills up regularly, consider a package upgrade or an additional disk. After a disk upgrade, the partition and file system are grown automatically while the server is running and VMware Tools is active. A virtual disk cannot be shrunk later. For the steps, see the [resource upgrade guide](https://www.hostimor.com/en/knowledgebase/article/resource-upgrade-cpu-ram-disk).

To hear about it before the disk fills up again, a small script run hourly by cron is enough. It writes file systems above the threshold to the system log, and also sends an email if mail delivery is configured on the server.

```
sudo tee /usr/local/bin/disk-alert.sh > /dev/null <<'EOF'
#!/bin/sh
# Writes file systems above the threshold to the system log and (if available) to email
THRESHOLD=${THRESHOLD:-85}
FULL=$(df -P -x tmpfs -x devtmpfs -x squashfs -x overlay | awk -v t="$THRESHOLD" 'NR>1 && int($5) >= t {print $6 " %" int($5)}')
[ -z "$FULL" ] && exit 0
logger -t disk-alert "Disk usage threshold exceeded: $FULL"
command -v mail >/dev/null && echo "$FULL" | mail -s "$(hostname): disk usage alert" admin@example.com
EOF
sudo chmod +x /usr/local/bin/disk-alert.sh
echo '0 * * * * root /usr/local/bin/disk-alert.sh' | sudo tee /etc/cron.d/disk-alert
# Test: run it with a temporarily lowered threshold, then check the log
sudo env THRESHOLD=1 /usr/local/bin/disk-alert.sh
journalctl -t disk-alert -n 5
```

Confirm that the cron service is running with `systemctl status cron` (`crond` on AlmaLinux). How to check scheduled jobs is explained in the [cron guide](https://www.hostimor.com/en/what-is-cron-a-guide-to-running-scheduled-tasks-reliably).

## Frequently asked questions

### The disk is 100% full and commands fail. Where do I start?

First run the commands that free space quickly and safely: `sudo journalctl --vacuum-size=200M` and `sudo apt clean` (`sudo dnf clean all` on AlmaLinux). If you use ext4, temporarily lowering the reserved block percentage also gives you some breathing room. Once a few hundred MB are free, look for the real cause with `du` and `ncdu`.

### I deleted files, but df did not change. Why?

Most likely the deleted files are still held open by a process; check with `sudo lsof +L1` and restart the service concerned. Deleting snapshots in the client area does not change the `df` value inside the server either.

### /var/lib/docker is very large. Can I delete the folder?

No. This folder holds images, containers and volumes; deleting it by hand breaks Docker and destroys the data in the volumes. See what is taking up space with `docker system df`, and use Docker's own commands such as `docker system prune`.

## Related articles

- [Upgrading CPU, RAM and disk resources](https://www.hostimor.com/en/knowledgebase/article/resource-upgrade-cpu-ram-disk)
- [Resource monitoring and steal time on a Linux server](https://www.hostimor.com/en/knowledgebase/article/linux-server-resource-monitoring-steal-time)
- [Installing Docker and Docker Compose](https://www.hostimor.com/en/knowledgebase/article/install-docker-and-docker-compose)
- [Server backup strategy](https://www.hostimor.com/en/knowledgebase/article/server-backup-strategy-restic)
- [What is cron and how to run scheduled tasks reliably](https://www.hostimor.com/en/what-is-cron-a-guide-to-running-scheduled-tasks-reliably)
