The problem: too many alerts and you stop reading them
Every alert trains you a little. If your phone buzzes six times a week for things that turn out to be nothing, the seventh buzz gets dismissed without a look. That is the real risk with monitoring. It is not that a check misses an outage. It is that the alert which matters arrives in a stream of noise and gets treated like the rest.
The fix is not more monitors. It is fewer alerts, each one of which means something you would act on. This post is a short decision list for choosing which alerts to have. Setup steps and what to do when an alert fires at night are covered elsewhere, so we will stay with the choice itself.
Alert on: the five things that need a person
Keep the list short. For a small web app, five kinds of signal are worth an alert:
- The site is down. The page or endpoint has failed repeatedly and visitors cannot use it. This is the one that always deserves a message.
- Responses are slow over a set threshold. Pick a number, such as several seconds for a page that normally answers in under one, and alert when checks stay above it. Slow is often the first sign of trouble.
- A certificate is close to expiry. An expired certificate stops browsers from loading your site, and it is easy to forget. One warning a few weeks ahead gives you time to renew.
- A heartbeat did not arrive. A heartbeat monitor expects a ping from something that should run on schedule, such as a cron job or a backup. If the ping does not come, the job probably did not run.
- A new error appeared. Not every error, only one you have not seen before. A new error after a deploy is usually the most useful signal you will get.
Each of these answers a question you would otherwise have to check by hand: is it up, is it fast, will it break soon, did the job run, is something new going wrong.
Do not alert on: the noise
Three things should not generate an alert on their own.
A single failed check. Networks drop packets. A check from one location can time out while the site is fine everywhere else. Alert when failures repeat across several checks in a row, not on the first one.
Brief slowness. A page that takes two seconds once is not an incident. Sustained slowness over a set window is.
Errors you have already seen. If an error has been logged a hundred times and you have decided it is harmless, it should not wake anyone. Mark it, fix it, or accept it, and let it stay in the log.
A useful test: if you would ignore the alert at 9am on a Tuesday, do not send it at 3am.
Choosing thresholds: start loose, tighten after a fortnight
Thresholds are guesses at first. Start loose. Set the slow-response limit higher than your normal figure, and set the number of consecutive failures needed before an alert fires at two or three rather than one. You will get fewer messages and you can see what normal looks like.
After about a fortnight, look at what fired. If the slow alert went off every evening when a backup ran, either move the backup or raise the limit a little. If nothing fired at all, tighten the threshold until it sits just above normal behaviour. The aim is an alert that only fires when the site has genuinely changed.
Write the figures down somewhere. When you come back in a month, you will want to know why a number is what it is.
Who should receive each alert in a team of up to 10
In a team, the question is not only what to alert on but who hears it. Send site-down alerts to everyone who can act on them, usually the person on call and one backup. Send certificate warnings to whoever renews them, which is often one person, and make sure that person is on the list. Heartbeat alerts belong to the person who owns the job, since they know what it should do.
Slow-response and new-error alerts are better sent to the developers who deployed the change, not the whole team. A new error after a release is most useful to the person who made the release.
Alert emails go only to people in your organisation. Check the list when someone leaves and when someone joins. The team plan is 29 pounds a month flat for up to 10 people, as on the pricing page, so adding a person does not change the cost.
Reviewing the alerts that fired at the end of each month and removing the noisy ones
Once a month, spend ten minutes looking at what fired. Count the alerts for each monitor. Any alert that fired many times and led to no action is a candidate for removal or a looser threshold. Any real incident that the alerts missed is a reason to add one.
Be willing to delete. An alert that nobody reads is worse than no alert, because it gives the false sense that something is being watched. Tweed Tyne runs its own work on AperioFlow, and the rule there is simple: an alert stays only while someone acts on it.
Keep a note of what you changed. Over a few months, the list gets shorter and more trustworthy.
A simple starter set for a solo developer and for a small agency
For a solo developer with one or two client sites, a starter set could be:
- site down, on the main public page
- slow response over your chosen limit
- a heartbeat for each scheduled backup
- a new error after each deploy
That is four alerts per site. Send them to yourself and nobody else, and review them at the end of the month.
For a small agency running client sites, add the certificate warning for each domain and assign the site-down alerts to the person who owns that client. Keep the heartbeat alerts with whoever maintains the backups. Use the same set across every client so that a new site starts with a known list rather than a blank page.
Whichever set you pick, the test is the same. Each alert should mean something a person can act on. Set up your first monitors on the free plan, which includes 10 monitors, and see which alerts you actually read. Start free at https://app.aperioflow.co.uk/#/signup