A server backup strategy: the 3-2-1 rule, off-site backups with restic and database dumps
In this guide you set up a tested backup routine for your server: you decide what to keep where according to the 3-2-1 rule, take consistent database dumps, create an encrypted restic repository that lives outside the server, schedule the job and test restores regularly.
Responsibility and the 3-2-1 rule
VDS and VPS services are unmanaged: just like the operating system and applications, backing up your data is your responsibility unless the package explicitly includes it. The details are in the Backup and Data Retention Policy and the Service Scope and Support Terms. The Snapshots tool in the client area gives you a quick way back before a risky change, but it is not a backup: it lives on the same storage, you can keep at most 2, and it disappears along with the server if the server is deleted. It is also crash-consistent: memory is not captured and applications are not quiesced, so you still need a dump for databases (see the snapshot guide).
The 3-2-1 rule offers a simple framework: keep at least 3 copies of your data, on at least 2 different media, with at least 1 copy in another location. For a server, that means the live data, dumps kept on the server for quick recovery, and an encrypted restic repository on another server or on S3-compatible storage.
| Method | What it protects against | What it does not protect against |
|---|---|---|
| Snapshot (Snapshots) | Rolling back within minutes after a bad update or configuration change | Deletion of the server, storage problems, corruption noticed too late |
| Dumps and archives on the server | An accidentally deleted file or table, quick restores | Loss of the server, a compromise, ransomware |
| Off-site restic repository | Loss of the server, going back to older versions, reinstalls | An untested backup, a lost repository password |
What should you back up?
Instead of copying the whole disk, choose the data you would need to bring back after reinstalling the server. The operating system and packages can be reinstalled; the data and configuration you created cannot.
- Configuration:
/etc(web server, PHP, SSH and firewall settings, including the certificate keys under/etc/letsencrypt) and the user crontabs under/var/spool/cron. - Application and site files:
/var/www,/home,/optand/root; application secret files such as.envlive inside these as well. Because restic encrypts the repository, it is safe to include these files in the backup. - Databases: not the data directory of the running database, but the dumps described in the next section.
- Docker: the data in volumes; for database containers, again take a dump (see the example in the Docker guide).
Leave out: /proc, /sys, /dev, /run, /tmp, cache directories, the swap file and Docker images that can be downloaded again (/var/lib/docker/overlay2).
Why a database dump is essential
A running database keeps changing its files. Copying the /var/lib/mysql directory produces a backup made of pieces written at different moments, which may not open when you restore it. --single-transaction takes a consistent view of a single moment without locking InnoDB tables, while --routines, --triggers and --events keep stored procedures, triggers and scheduled events in the dump. On most installations the root user connects over the unix socket without a password; if a password is required, put the credentials in the [client] section of /root/.my.cnf and protect the file with chmod 600.
mkdir -p /var/backups/db && chmod 700 /var/backups/db
# MariaDB / MySQL (use mysqldump instead of mariadb-dump on older versions)
mariadb-dump --single-transaction --routines --triggers --events --all-databases \
| gzip > /var/backups/db/mariadb-all.sql.gz
# PostgreSQL: the whole cluster, including roles
runuser -u postgres -- pg_dumpall | gzip > /var/backups/db/postgres-all.sql.gz
# Check that the dump is intact and complete
gzip -t /var/backups/db/mariadb-all.sql.gz && zcat /var/backups/db/mariadb-all.sql.gz | tail -n 1The last line should contain "Dump completed"; if it does not, the dump was cut short.
An off-site repository with restic
restic is a backup tool that encrypts backups on the client side, stores repeated data only once (deduplication) and sends only the changed pieces on each run. Every backup run creates a "snapshot" record in the repository; do not confuse this with a snapshot of the virtual server. For detailed usage, see the restic documentation.
# Ubuntu / Debian
apt-get update && apt-get install -y restic
# AlmaLinux 9 (from the EPEL repository)
dnf install -y epel-release && dnf install -y restic
restic versionDistribution packages may lag behind the latest restic release; note the output of restic version, and do not try to restore with a client older than the version that created the repository.
An SFTP repository on another server
The most common setup is a user created only for backups on a second Linux server in a different location. The source server connects to this user with a separate SSH key; the restrict prefix in front of the key disables port forwarding and terminal sessions, while SFTP keeps working.
# On the backup server (as root)
useradd -m -s /bin/bash backup-user
install -d -m 700 -o backup-user -g backup-user /srv/restic /home/backup-user/.ssh
# Add the source server's public key with the "restrict" prefix (no forwarding, no terminal)
echo 'restrict ssh-ed25519 AAAA...rest-of-the-key restic@source' >> /home/backup-user/.ssh/authorized_keys
chown backup-user:backup-user /home/backup-user/.ssh/authorized_keys && chmod 600 /home/backup-user/.ssh/authorized_keys# On the source server (as root)
ssh-keygen -t ed25519 -f /root/.ssh/restic_ed25519 -N '' -C restic@source
cat /root/.ssh/restic_ed25519.pub # add this line on the backup server
cat >> /root/.ssh/config <<'EOF'
Host backup-server
HostName backup.example.com
User backup-user
IdentityFile /root/.ssh/restic_ed25519
EOF
chmod 600 /root/.ssh/config
# Accept the host key once by hand; cron cannot do this for you
ssh backup-server trueThen create the repository password and initialize the repository:
openssl rand -base64 32 > /root/.restic-pass
chmod 600 /root/.restic-pass
export RESTIC_REPOSITORY="sftp:backup-server:/srv/restic"
export RESTIC_PASSWORD_FILE="/root/.restic-pass"
restic init
restic backup /etc /var/www --tag first
restic snapshots
restic stats latestA restic repository cannot be opened without this password, and there is no way to recover it. Keep a copy of /root/.restic-pass outside the server, for example in a password manager. If the server is lost completely, it is the only key to your backups.
S3-compatible storage
If you use object storage, the repository address and access keys are passed as environment variables. Keep them in a file that only root can read:
cat > /root/.restic-env <<'EOF'
export RESTIC_REPOSITORY="s3:https://s3.example.com/server-backups"
export AWS_ACCESS_KEY_ID="ACCESS_KEY"
export AWS_SECRET_ACCESS_KEY="SECRET_KEY"
export RESTIC_PASSWORD_FILE="/root/.restic-pass"
EOF
chmod 600 /root/.restic-env
. /root/.restic-env && restic initIf the source server is compromised, an attacker can use the same credentials to delete your backups as well. For stronger protection, set up the backup side so that it does not allow deletion; for example, run restic's rest-server component in append-only mode and run forget and prune from the backup server, or turn on your storage provider's versioning feature.
Scheduling and automation
The script below takes the dump, sends the backup and applies the retention policy. set -euo pipefail is a critical line: without pipefail, gzip finishes successfully even if mariadb-dump fails, and an empty dump looks like a success. The retention policy keeps the last 7 daily, 4 weekly and 6 monthly backups; --prune removes data from the repository that no snapshot uses any more.
#!/usr/bin/env bash
# /usr/local/sbin/backup.sh: database dump + restic backup + retention policy
set -euo pipefail
umask 077
export RESTIC_REPOSITORY="sftp:backup-server:/srv/restic"
export RESTIC_PASSWORD_FILE="/root/.restic-pass"
DUMP_DIR="/var/backups/db"
PING_URL="" # optional: the ping URL of your monitoring service
log() { echo "$(date '+%F %T') $*"; }
on_error() {
log "ERROR: script stopped at line $1"
if [ -n "$PING_URL" ]; then curl -fsS -m 10 "$PING_URL/fail" > /dev/null || true; fi
}
trap 'on_error $LINENO' ERR
mkdir -p "$DUMP_DIR"
log "Database dump"
mariadb-dump --single-transaction --routines --triggers --events --all-databases \
| gzip > "$DUMP_DIR/mariadb-all.sql.gz"
# If you use PostgreSQL:
# runuser -u postgres -- pg_dumpall | gzip > "$DUMP_DIR/postgres-all.sql.gz"
log "restic backup"
restic backup --tag nightly --exclude-caches --exclude '/home/*/.cache' \
/etc /var/www /home /root /opt /var/spool/cron "$DUMP_DIR"
log "Retention policy"
restic forget --tag nightly --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune
log "Done"
if [ -n "$PING_URL" ]; then curl -fsS -m 10 "$PING_URL" > /dev/null || true; fiSave the script as /usr/local/sbin/backup.sh and schedule it to run every night. The weekly restic check --read-data-subset=5% line downloads and verifies a different 5% of the data each time:
chmod 700 /usr/local/sbin/backup.sh
cat > /etc/cron.d/restic <<'EOF'
SHELL=/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
30 2 * * * root /usr/local/sbin/backup.sh >> /var/log/backup.log 2>&1
0 5 * * 0 root restic -r sftp:backup-server:/srv/restic --password-file /root/.restic-pass check --read-data-subset=5\% >> /var/log/restic-check.log 2>&1
EOF
# Run it once by hand, then read the log
/usr/local/sbin/backup.sh >> /var/log/backup.log 2>&1; tail -n 20 /var/log/backup.logIn crontab lines the % character means a newline, so it is written as \% in the cron file. File names under /etc/cron.d must not contain dots, and every line needs the user field (here root). For details, see the cron guide.
If you prefer a systemd timer to cron, create the two unit files and enable them with systemctl enable --now backup.timer; Persistent=true catches up on a run that was missed while the server was off as soon as it boots. systemctl list-timers shows the next run time.
# /etc/systemd/system/backup.service
[Unit]
Description=restic backup
[Service]
Type=oneshot
ExecStart=/usr/local/sbin/backup.sh
# /etc/systemd/system/backup.timer
[Unit]
Description=Nightly restic backup
[Timer]
OnCalendar=*-*-* 02:30:00
RandomizedDelaySec=15m
Persistent=true
[Install]
WantedBy=timers.targetFor notifications, you can put the address of a monitoring service in the script's PING_URL variable. Every successful run sends a signal to the service, and if no signal arrives, the service alerts you. This approach catches both a failing script and a cron job that never ran.
Restore tests and common errors
A backup you cannot restore is not a backup. At least once a month, extract selected directories from a snapshot into a temporary folder, compare the contents, import the database dump into a test database (zcat mariadb-all.sql.gz | mariadb) and confirm that the application starts. This drill shows how long it would take you to get back up in a real incident.
export RESTIC_REPOSITORY="sftp:backup-server:/srv/restic"
export RESTIC_PASSWORD_FILE="/root/.restic-pass"
restic snapshots --compact
restic ls latest /etc/nginx
restic restore latest --target /tmp/restore-test --include /etc/nginx --include /var/backups/db
diff -r /etc/nginx /tmp/restore-test/etc/nginx && echo "Files match"
gzip -t /tmp/restore-test/var/backups/db/mariadb-all.sql.gz && echo "Dump is intact"
rm -rf /tmp/restore-test| Error | Likely cause | Fix |
|---|---|---|
Fatal: unable to open config file | The repository has not been created yet, or the address is wrong | Check the RESTIC_REPOSITORY value; for a new repository, run restic init |
Fatal: wrong password or no key found | The password file is different or has changed | Point to the right file with RESTIC_PASSWORD_FILE; if the password is lost, the repository cannot be opened |
repository is already locked | A previous run stopped halfway, or another restic process is running | Confirm with pgrep -a restic that nothing else is running, then run restic unlock |
Host key verification failed (only in cron) | The backup server's host key was never accepted for root | Run ssh backup-server true once by hand |
Access denied for user 'root'@'localhost' | The database root user requires a password | Put the credentials in /root/.my.cnf and run chmod 600 |
| The cron job never runs | The file name contains a dot, the user field is missing, or % is not escaped | Fix it and check with journalctl -u cron (crond on AlmaLinux) |
A short note for Windows servers
On Windows Server, the built-in Windows Server Backup feature can back up the system state and volumes to a network share or a separate disk. For file-level, encrypted backups kept outside the server, you can use the Windows version of restic; the --use-fs-snapshot flag uses the Volume Shadow Copy Service (VSS) to capture open files consistently and must run with administrator rights. Schedule it with Task Scheduler, and for databases such as SQL Server, again take a local dump first.
# Install the Windows Server Backup feature (adds wbadmin and the management console)
Install-WindowsFeature Windows-Server-Backup
# restic for Windows: a VSS snapshot captures open files consistently too
restic.exe -r sftp:backup-user@backup.example.com:/srv/restic-win --password-file C:\Backup\restic-pass.txt backup --use-fs-snapshot C:\inetpub C:\Backup\dbFrequently asked questions
I take snapshots regularly; do I still need backups?
Yes. A snapshot lives on the same server's storage, you can keep at most 2 at a time, and it is lost if the server is deleted. For a server placed in the deletion queue after an unpaid invoice, recovering its data is not guaranteed either (see what happens if a server invoice is overdue). A backup outside the server covers these risks.
How often should I back up?
Decide the longest stretch of data you can afford to lose, which is your recovery point objective (RPO). For a corporate site that changes once a day, a nightly backup is enough; for an e-commerce site that takes orders, taking an hourly database dump and adding it to restic is a better fit.
How much space does a restic repository take?
The first backup is close to the compressed full size of the data; later backups add only the changed pieces. restic stats --mode raw-data shows the real repository size. Extending the retention policy grows the space slowly, but large dumps that change often can take more room than you expect.
Related articles
Thanks for your feedback!