A backup job that fails loudly is easy to deal with. A backup job that stops running is harder. The cron entry gets dropped when a server is rebuilt. The disk fills up and the script exits before it writes anything. The credentials for the storage bucket expire. In each case the script that would have sent an error email is the script that never started.
Why silent backup failures are common
Most people find out when they need a restore. By then the last good copy may be days or weeks old, and nobody remembers when the job last ran cleanly.
Silent failures are common because most backup tools only report when they run. If the job does not start, there is nothing to report. A quiet log looks the same as a healthy one.
You need a check that works from the outside. It should expect a signal at a set time and raise an alert when that signal does not arrive.
Ping after the backup command succeeds
A heartbeat monitor works this way. You create a heartbeat monitor and it gives you a unique URL. At the end of your backup job, your script requests that URL. If the request arrives within the window you have set, the monitor stays quiet. If it does not arrive, the monitor raises an incident and alerts the people in your organisation.
Put the ping after the backup command, not at the top of the script. A ping at the start only tells you the script began. It says nothing about whether the backup worked.
Here is a simple example for a nightly database dump:
#!/bin/sh
set -e
pg_dump mydb | gzip > /backups/mydb-$(date +%F).sql.gz
curl -fsS --retry 3 https://YOUR-HEARTBEAT-URL > /dev/nullThe set -e line matters. If pg_dump fails, the script stops before the curl line, so no ping is sent. The monitor then notices the silence and alerts you. Without set -e, the script could carry on and send the ping even though the dump failed, which defeats the check.
Keep the URL private. Anyone who has it can send a ping on your behalf. Store it as an environment variable or in your server's secrets, not in a script that gets committed to a shared repository.
Choose an interval that matches your schedule
The expected interval should match how often the job runs, plus a margin for ordinary variation.
A nightly backup that usually runs at 02:00 and takes twenty minutes needs a window longer than 24 hours, so one slow night does not wake anyone. Set it to 24 hours plus a few hours of margin. A weekly backup needs a week plus a margin. An hourly job needs an hour plus a few minutes.
If the window is too tight, you will get false alarms. People start ignoring alerts, and then a real failure gets ignored too. If the window is too loose, you learn about the failure days after it started, which is the problem you were trying to avoid.
Check the interval after the first few runs. If the job regularly finishes later than you expected, widen the window rather than accepting a stream of noise.
Add a second check that a restore test was done, as a task
A heartbeat proves that the backup ran and finished. It does not prove that the backup can be restored. A file can be created successfully and still be corrupt, incomplete or unreadable.
Once a month, pick a recent backup, restore it to a scratch database or a separate folder, and check that the data opens and looks right. Then record the test as a task. Create it in AperioFlow with a due date for the next restore test, assign it to a named person, and add a short note about what to check.
When the task is marked done, the record shows who did the test and when. If the task sits overdue, that is the signal to act. The heartbeat tells you the job ran. The task tells you someone checked the result.
Keep the restore test separate from the heartbeat in your head. One is automatic and one is a human habit. You need both.
Keep a note of who gets the alert
Alert emails go only to people in your organisation, so the list of who receives them matters. Decide that list before you need it.
Write it down. Put the names in the monitor's notes or in your team's runbook, along with who has the access to fix the server, and who to contact if the storage account itself is the problem. A solo developer might list themselves and one backup contact. An agency might list the person on call for each client site.
Review the list when people join or leave. If someone leaves the organisation and is still on the list, alerts may land in an inbox nobody reads. If nobody is listed for a site, the alert has nowhere useful to go.
A short note costs little. It is the difference between a backup alert that someone acts on and one that sits unread until the next restore request.
If you want to set up a backup heartbeat on the free plan, which includes 10 monitors, you can start free at https://app.aperioflow.co.uk/#/signup. Create the monitor, add the ping to the end of your backup script, and set the window to match your schedule. You will know within a day whether the job is running as expected.