How to Monitor borgmatic Backups
To monitor borgmatic backups (version 2.1.7 or newer), add a watchgoose: section to your borgmatic configuration with the check's full ping URL. borgmatic then pings Watchgoose when a backup starts, when it finishes, and when it fails, and can attach its own log to the ping. If borgmatic never runs at all, because the timer was lost or the server is down, no ping arrives and Watchgoose alerts you when the backup is overdue.
Table of contents
- Why a successful-looking borgmatic setup still needs monitoring
- Step 1: Create a check
- Step 2: Add the watchgoose hook to borgmatic
- Step 3: Test it
- Which states to send
- Use the full /ping/ URL, not a bare UUID
- Creating checks automatically with slugs
- Frequently asked questions
- References
Why a successful-looking borgmatic setup still needs monitoring
borgmatic is good at running Borg and reporting errors, but it can only report what happens while it runs. Three common failures produce no error message at all:
- The cron entry or systemd timer disappears after a server rebuild or an upgrade.
- The server is off, asleep or out of disk when the backup should start.
- borgmatic hangs on a lock or a slow remote repository and never finishes.
A heartbeat check covers all three. Watchgoose knows when the next backup is due. When the finish ping does not arrive within the schedule plus the grace time, it sends an alert by email, Slack, Microsoft Teams, Telegram or another of its 12+ channels.
Step 1: Create a check
Sign up and add a check for the backup. Set the schedule to match borgmatic's timer, for example the cron expression 0 3 * * * for a nightly run, and choose a grace time longer than your slowest backup. A backup that usually takes 20 minutes might get a 1-hour grace time.
Copy the check's ping URL. It looks like https://watchgoose.com/ping/your-uuid-here.
Step 2: Add the watchgoose hook to borgmatic
Add this section to /etc/borgmatic/config.yaml, or to the configuration file you already use:
watchgoose:
ping_url: https://watchgoose.com/ping/your-uuid-here
send_logs: true
The watchgoose hook was added in borgmatic 2.1.7, so check your version with borgmatic --version first. That is all borgmatic needs. With the default settings it pings Watchgoose:
- start when the backup begins, so Watchgoose can measure the run time;
- finish when it completes successfully;
- fail when any action fails, with borgmatic's log attached.
send_logs: true attaches borgmatic's log output to the finish and fail pings. Watchgoose stores the first 10 kB of each ping body, and borgmatic trims its log to that size by default (ping_body_limit).
Step 3: Test it
Run a backup by hand with verbose output:
borgmatic --verbosity 1
The log shows a line like Pinging Watchgoose start before the backup and Pinging Watchgoose finish after it. In Watchgoose, the check's status changes from New to Up and the event list shows the start and finish pings with borgmatic's log. Do this first run before relying on the check: a new check doesn't alert about missed runs until its first success ping arrives. A fail ping alerts right away, and after a start ping Watchgoose alerts if the backup runs longer than the grace time.
To see a failure, point a test configuration at a repository that does not exist and run it again. The check records a fail ping with the error in the log and sends an alert.
Which states to send
The states option limits which pings borgmatic sends. The default is all four:
watchgoose:
ping_url: https://watchgoose.com/ping/your-uuid-here
send_logs: true
states:
- start
- finish
- fail
Leave out log unless you want borgmatic's log action to post logs on their own. Keep start if you want an alert when a backup runs much longer than usual: with a start ping, Watchgoose alerts when the finish ping is later than the grace time.
Use the full /ping/ URL, not a bare UUID
borgmatic accepts either a full URL or a bare check UUID in ping_url. In borgmatic 2.2.0 and earlier, a bare UUID builds the address https://watchgoose.com/<uuid>, which is not a ping address, so every ping fails with a 404 warning in borgmatic's log and nothing reaches the check. The borgmatic maintainer has fixed this for the next release.
Until you run a fixed version, always paste the full URL, including /ping/. If a check stays grey after a backup, look for Watchgoose error: 404 in borgmatic's output.
Creating checks automatically with slugs
If you manage many servers, you can name checks with slug URLs instead of UUIDs and let borgmatic create them on first use:
watchgoose:
ping_url: https://watchgoose.com/ping/your-ping-key/backup-web-01
send_logs: true
create_slug: true
Your project's ping key is in Project Settings. New checks start with the default schedule, so set each one's schedule and grace time after the first run.
Frequently asked questions
Does borgmatic have built-in monitoring?
Yes. borgmatic has monitoring hooks for several services, including a watchgoose hook. Add a watchgoose section with the check's ping_url to the configuration, and borgmatic pings when a backup starts, finishes and fails. With send_logs: true it also attaches its log to the finish and fail pings.
Why does my borgmatic Watchgoose check never get pings?
Check that ping_url is the full URL including /ping/. In borgmatic 2.2.0 and earlier, a bare check UUID builds an address without /ping/, so every ping fails with a 404 warning in borgmatic's output. The maintainer has fixed this for the next release.
How do I get an alert when a borgmatic backup does not run?
borgmatic cannot report a run that never started, so the alert has to come from outside. Give the Watchgoose check the same schedule as the backup timer and a grace time. If the finish ping does not arrive in time, Watchgoose sends an alert.