How to Monitor restic Backups
restic has no built-in alerting, so the usual way to monitor restic backups is to wrap the backup in a short script that pings a heartbeat monitor: once when the backup starts and once with restic's exit code when it ends. If you schedule restic with resticprofile, its send-before, send-after and send-after-fail hooks do the same without a script. Either way, Watchgoose alerts you when a backup fails, and also when it never runs at all.
Table of contents
- What a restic backup can get wrong without telling you
- Create a check
- Option 1: A wrapper script for plain restic
- Option 2: resticprofile HTTP hooks
- restic exit codes
- Frequently asked questions
- References
What a restic backup can get wrong without telling you
restic prints errors and exits with a non-zero code, but on a scheduled job nobody reads that output. These failures are common:
- The repository is missing or locked, or the password is wrong (restic exits with 10, 11 or 12), or the password file has moved (exit code 1).
- Some source files could not be read, so the snapshot is incomplete (exit code 3).
- The cron entry or systemd timer is gone, or the machine was off, so restic never ran and printed nothing.
A heartbeat check catches all of them. The failures report themselves through the exit code. A missing run is caught because the expected ping does not arrive, but only after the check has received its first success ping: until then it stays New and does not alert about missed runs. A /start or failure ping can still alert before that.
Create a check
Sign up and add a check for the backup. Set its schedule to match the backup, for example 0 2 * * * for a nightly run, and a grace time longer than the slowest backup. Copy the check's ping URL, which looks like https://watchgoose.com/ping/your-uuid-here.
Option 1: A wrapper script for plain restic
Save this as /usr/local/bin/restic-backup.sh and make it executable with chmod +x:
#!/bin/sh
# Back up with restic and report the result to Watchgoose.
URL="${WATCHGOOSE_URL:-https://watchgoose.com/ping/your-uuid-here}"
export RESTIC_REPOSITORY="${RESTIC_REPOSITORY:-/srv/restic-repo}"
export RESTIC_PASSWORD_FILE="${RESTIC_PASSWORD_FILE:-/root/.restic-password}"
curl -fsS -m 10 --retry 3 -o /dev/null "$URL/start" || true
log=$(mktemp)
restic backup --exclude-caches "$@" >"$log" 2>&1
rc=$?
# The exit code becomes the ping: 0 is success, anything else is a failure.
# The last 10000 bytes of restic's output are stored with the ping.
tail -c 10000 "$log" | curl -fsS -m 10 --retry 3 -o /dev/null \
--data-binary @- "$URL/$rc" || true
rm -f "$log"
exit "$rc"
Then schedule it with the paths to back up as arguments, for example in root's crontab:
0 2 * * * /usr/local/bin/restic-backup.sh /etc /home /var/www
Before you rely on it, run the script once by hand with the same arguments, as the same user as the cron job:
/usr/local/bin/restic-backup.sh /etc /home /var/www
In Watchgoose, the check should show a start ping followed by a success ping, and its status should change from New to Up. A new check doesn't alert about missed runs until its first success ping arrives, so a cron entry that never runs would go unnoticed until then. A failure ping alerts right away, and after a /start ping Watchgoose alerts if the run takes longer than the grace time.
How it behaves:
- The
/startping lets Watchgoose measure the run time and alert if the backup runs longer than the grace time. - The final ping goes to
$URL/0on success. Any other code, such as$URL/10for a missing repository, marks the check as failed and sends an alert. See signaling failures. - The end of restic's output is attached to the ping, so the alert shows the actual error.
- A network problem while pinging prints a warning but never changes the backup's exit code.
To keep old snapshots in check, add a restic forget --prune line with your retention policy after the backup, and send the ping after both commands.
Option 2: resticprofile HTTP hooks
resticprofile wraps restic with profiles and schedules, and its HTTP hooks can ping Watchgoose directly. Add the three hooks to the backup section of your profile:
version = "1"
[default]
repository = "local:/srv/restic-repo"
password-file = "/root/.restic-password"
[default.backup]
source = ["/etc", "/home"]
schedule = "02:00"
[default.backup.send-before]
method = "HEAD"
url = "https://watchgoose.com/ping/your-uuid-here/start"
[default.backup.send-after]
method = "HEAD"
url = "https://watchgoose.com/ping/your-uuid-here"
[default.backup.send-after-fail]
method = "POST"
url = "https://watchgoose.com/ping/your-uuid-here/fail"
body = "${ERROR}\n\n${ERROR_STDERR}"
[[default.backup.send-after-fail.headers]]
name = "Content-Type"
value = "text/plain; charset=UTF-8"
On failure, the POST body carries resticprofile's error message and restic's error output, so the alert explains what went wrong. To record the exit code instead of a plain failure, use https://watchgoose.com/ping/your-uuid-here/${ERROR_EXIT_CODE} as the send-after-fail URL.
Run resticprofile backup once by hand and confirm that the check's status changes from New to Up, for the same reason as above: missed-run alerts start after the first success ping.
Two things to know about resticprofile hooks:
- If resticprofile cannot get its lock, no hooks run at all. That run shows up as a missed ping, which still alerts you.
- A failed hook is logged but does not fail the backup, and resticprofile does not retry it.
restic exit codes
Scroll horizontally to see all columns.
| Exit code | Meaning | Watchgoose shows |
|---|---|---|
| 0 | Backup completed | Success |
| 1 | Fatal error, nothing was backed up (for example a missing source folder) | Failure |
| 3 | Backup completed, but some source files could not be read | Failure |
| 10 | Repository does not exist | Failure |
| 11 | Repository is locked | Failure |
| 12 | Wrong password | Failure |
If unreadable files are expected on your system and you want exit code 3 to count as success, change the script to send $URL/0 when rc is 3.
Frequently asked questions
Does restic send alerts when a backup fails?
No. restic prints errors and exits with a non-zero code, but it has no alerting. Wrap the backup in a script that pings a heartbeat monitor with restic's exit code, or use resticprofile's HTTP hooks, so failures and missed runs reach you.
What does restic exit code 3 mean?
Exit code 3 means the backup finished, but some source files could not be read, so the snapshot is incomplete. Exit code 10 means the repository does not exist, 11 that it is locked and 12 that the password is wrong.
How do I monitor resticprofile backups?
Add send-before, send-after and send-after-fail hooks to the backup section of the profile, pointing at the check's start, success and fail URLs. A POST body on send-after-fail can carry the error message and restic's error output.