Tech Notes

A daily copy of my mirror machine with rsync and systemd

How I keep a fresh copy of the /infra folder from my offline mirror box on a spare workstation partition, with rsync over ssh, a read only key, and a systemd timer.

#Linux#Homelab

In my homelab there is one machine which I call mirror.intra. It is a small bare metal box, and it holds everything which I need when the internet is not there. Where I live the power goes out often, and the connection goes away with it, so this box is the reason I can still work on such days.

Everything on it lives in one folder, /infra. Inside there is ui-apt-mirror, which I wrote about before, so Debian and Ubuntu packages are available locally. There is a gogs server with my git repositories. There is a docker registry, so I can pull images without Docker Hub. And there is nginx in front of all of this. Together it is around 750 gigabytes.

The problem with this box is that all of it sits on one nvme disk.

If this disk dies, I lose the packages, the git server and the registry in one moment. And the machine which exists to save me from a bad connection would need a very good connection to be built again.

I already had a copy of that folder on my workstation, on a separate 900G partition. But I made it by hand from time to time. I wanted it to refresh by itself, without me remembering about it.

Why not Proxmox for this one

Most of my homelab runs on Proxmox. For virtual machines and containers there I do not write any scripts at all. Proxmox Backup Server takes the snapshot, keeps only the changed blocks, and when something breaks I restore the whole machine and it boots. This is much better than copying files. A file copy gives you files, and a snapshot gives you a working machine back.

mirror.intra is not on Proxmox. It is bare metal with one job, and I did not want a hypervisor layer under it.

A key which can only read

The copy is a pull. The workstation connects to the mirror and takes the files. The mirror knows nothing and initiates nothing.

I did not want a key which can do everything on the mirror. rsync ships a small wrapper called rrsync, and it exists exactly for this. You put it as a forced command in authorized_keys, and then the key can run rsync and nothing else:

command="/usr/bin/rrsync -ro /infra",restrict ssh-ed25519 AAAA... infra-sync

The -ro means read only. The /infra after it is the root for this key, so every path the client asks for is resolved inside that folder and cannot go above it. The restrict word turns off port forwarding, agent forwarding, X11 and the terminal.

It is easy to check that it works:

ssh -i ~/.ssh/id_infra_sync root@mirror.intra 'id'
# /usr/bin/rrsync error: SSH_ORIGINAL_COMMAND does not run rsync

So even if somebody takes this private key, there is no shell, no write, and no way out of that one folder.

The script
#!/bin/bash
set -euo pipefail

SRC="root@mirror.intra:/"
DST="/infra/"
KEY="/root/.ssh/id_infra_sync"
LOCK="/run/infra-sync.lock"
STATUS="/var/lib/infra-sync/status"

mkdir -p "$(dirname "$STATUS")"

fail() { printf 'FAILED %s %s\n' "$(date -Is)" "$*" > "$STATUS"; echo "FAILED: $*" >&2; exit 1; }

mountpoint -q /infra || fail "/infra is not mounted"

exec 9>"$LOCK"
flock -n 9 || { echo "previous sync still running, skipping"; exit 0; }

printf 'RUNNING %s\n' "$(date -Is)" > "$STATUS"
START=$(date +%s)

nice -n 10 ionice -c2 -n7 \
  rsync -aHAX --numeric-ids \
        --delete --delete-delay \
        --exclude='/lost+found' \
        --human-readable --stats \
        -e "ssh -i $KEY -o IdentitiesOnly=yes -o BatchMode=yes -o ConnectTimeout=15" \
        "$SRC" "$DST" || fail "rsync exited $?"

printf 'OK %s duration=%ss\n' "$(date -Is)" "$(( $(date +%s) - START ))" > "$STATUS"

The mountpoint -q /infra line is the most important one in this file. The copy goes to a separate partition. If that partition is not mounted for any reason, then /infra is just an empty folder on the root filesystem, and rsync would happily write 750 gigabytes into it and fill the system disk. One line prevents this.

--numeric-ids keeps the user and group numbers as they are instead of resolving them through names. The two machines have different accounts with the same numbers, and without this flag the owners would be translated into wrong users.

-H keeps hardlinks. In my folder only 2374 files are hardlinked, but the apt mirror needs them.

--delete-delay does the deletions at the end, after the transfers, and not before.

The delete flag which was silently skipped

In the beginning the sync connected as a normal unprivileged user. The dry run looked almost fine, but there were two lines in the output:

rsync: [sender] opendir "/infra/gogs/data/postgres-data/db-files" failed: Permission denied (13)
IO error encountered -- skipping file deletion

Two folders were not readable for that user. The postgres data of gogs, and the folder where ui-apt-mirror keeps its gpg keys.

The second line is the dangerous one. When rsync meets any IO error, it turns --delete off for the whole run. It does this on purpose, because it cannot know if a file is missing on the source or if it only failed to read the folder, and deleting in that situation would be worse. The problem is what it looks like from outside.

The transfer still runs. New files still arrive. The exit code is 23, which nobody reads when the job is in a timer. And 1129 old files which I deleted on the mirror stay on the copy forever, and their number grows every month. The sync reports success, the copy looks alive, and it slowly becomes something else than the source.

I made the sender root on the mirror side. The key is still the same restricted rrsync -ro /infra key, so root there can read the folder and do nothing else. After that the dry run was clean and the deletions started to happen.

So after you set up any rsync copy, read the whole output of a dry run. Not the exit code, the output. IO error encountered is one quiet line, and it changes what the whole job does.

The timer

I use a systemd timer and not cron, because I want the log of every run and a real unit which I can query.

# /etc/systemd/system/infra-sync.timer
[Unit]
Description=Refresh /infra from mirror.intra (10:00 and 16:15)

[Timer]
OnCalendar=*-*-* 10:00:00
OnCalendar=*-*-* 16:15:00
Persistent=true
AccuracySec=1m
Unit=infra-sync.service

[Install]
WantedBy=timers.target

Two OnCalendar lines simply give two runs per day. Persistent=true runs the job after boot if the machine was off at that time.

The service unit is short:

# /etc/systemd/system/infra-sync.service
[Unit]
Description=Mirror /infra from mirror.intra
After=network-online.target infra.mount
Wants=network-online.target
Requires=infra.mount

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/infra-sync.sh
TimeoutStartSec=20h
ProtectHome=read-only
PrivateTmp=yes
NoNewPrivileges=yes

I first wrote ProtectHome=yes there, and the service could not find the ssh key. This option hides /root too, not only /home, and the key lives in /root/.ssh. With read-only it works and the protection is still there.

Requires=infra.mount ties the job to the partition. If the mount is not there, systemd does not start the service at all, and the check inside the script is the second protection for the same thing.

How it behaves now

The first full run was not needed, because most of the data was already on the partition from the old manual copy. A normal run moves a few hundred megabytes and finishes in fifteen to twenty seconds. The tree walk over 300 thousand files costs more than the transfer itself.

One number surprised me. The service uses up to 1.8G of memory on a run. This is -H, which keeps a table of all files to find the hardlinks, and it grows with the number of files and not with their size. For 300 thousand files this is acceptable. If this folder ever grows to millions of files, -H will be the first thing to remove.

I also dropped --partial from the flags. With it, a transfer which is stopped in the middle leaves a half written file under the real name. Without it, rsync removes its temporary file and the old version stays untouched. My daily difference is small, so resuming a file gives me nothing, and having no broken files is worth more.

For monitoring there is one status file:

$ cat /var/lib/infra-sync/status
OK 2026-08-17T16:15:27+03:00 duration=19s

It is not a real monitoring, and after a power cut in the middle of a run the file stays with RUNNING inside, until the next run writes over it.

The whole setup is one script, two unit files and one line in authorized_keys. It does not replace a backup, because a mirror with --delete repeats my mistakes: if I remove something on the source by accident, it disappears from the copy at 16:15. For the packages and images this is fine, they are all downloadable again. For the git repositories I keep a separate dump, and for everything which runs on Proxmox I did not write anything at all, because there it is already solved better.