A single failed check is often nothing. A page times out once because the server was busy for a few seconds, a deploy briefly restarted the app, or a network hop dropped a request. If every one of those blips sends an alert, you soon stop reading alerts. The one that matters arrives in the same inbox as twenty that did not.
This post explains the failure threshold on an AperioFlow monitor, how it changes the time to your first alert, and how to choose a sensible value for each page you watch.
The problem: one failed check is often a blip
A monitor checks your site or service on a schedule. Each check either passes or fails. Most failures on a healthy site are short. A slow response, a single bad connection, or a brief spike in load can all fail one check and then pass the next.
If you alert on the first failure, you will get a steady trickle of messages that turn out to be nothing. People respond to that by muting the channel, filtering the emails into a folder, or waiting to see whether the problem goes away on its own. That habit is the real risk. When a real outage happens, the alert gets the same treatment as the blips.
The aim is not to hear about fewer problems. The aim is to hear about the problems that persist, and to hear about them in a way you still trust.
What a failure threshold is
Each monitor on AperioFlow has a failure threshold that you set. The monitor alerts only after that number of failed checks in a row. A pass in between resets the count.
So a threshold of 1 means the first failed check raises an alert. A threshold of 3 means the monitor waits until three checks in a row have failed before it alerts. If the second check passes, the count goes back to zero and no alert is sent.
The threshold is set per monitor. A checkout page and a brochure page can have different values, and you can change either one later without affecting the other.
Worked example: a threshold of 1 compared with a threshold of 3
Take a small site, a contact form on a single server. For the arithmetic, assume the monitor checks it once every 60 seconds. This is an example for working out the numbers, not a recommended setting.
Suppose the server stops responding at 10:00:20, between two checks.
With a threshold of 1, the next check at 10:01:00 fails and the alert goes out. You hear about it roughly 40 seconds after the outage starts, and about a minute after the first failed check.
With a threshold of 3, the checks at 10:01:00, 10:02:00 and 10:03:00 all need to fail. The alert goes out at 10:03:00. You hear about it roughly two and a half minutes after the outage starts.
Now take a blip. The server is slow for 20 seconds at 10:00:50 and then recovers. The check at 10:01:00 fails and the check at 10:02:00 passes.
With a threshold of 1, you get an alert at 10:01:00 and another message, or a recovery notice, a minute later. You have spent attention on a problem that was already over.
With a threshold of 3, nothing is sent. The count reset at 10:02:00.
The trade-off is clear. A higher threshold means a longer wait before a real outage reaches you, and fewer messages for short blips. A lower threshold means a faster first alert, and more messages when the page wobbles. Neither is right in every case. It depends on what the page does and what you can do in the first few minutes.
How the check interval and the threshold combine into a delay
The time to your first alert depends on two things: how often the monitor checks, and how many consecutive failures it needs.
A rough rule is that the earliest alert comes about threshold multiplied by the check interval after the first failed check. The first failed check can happen anywhere up to one full interval after the outage begins, so the total delay can be a little longer than that rule suggests.
For a check every 60 seconds:
- Threshold 1: the first alert comes at the first failed check, up to about a minute after the outage starts.
- Threshold 3: the alert comes after the third consecutive failed check, up to about three minutes after the outage starts.
If you shorten the interval, the same threshold gives you a faster alert. If you lengthen it, the delay grows. This is why the interval and the threshold should be set together. A tight interval with a high threshold can still be slower than a loose interval with a threshold of 1, and it is worth working out the sum for each monitor rather than guessing.
For more on choosing how often a monitor checks, see our post on check intervals. For what goes into an alert once it fires, see our post on what to alert on.
Suggested starting points
These are starting points for judgement, not rules. Adjust them after you see how the monitor behaves in practice.
For a checkout page or a login page, a lower threshold is usually worth it. A customer who cannot pay or sign in is a problem now, and a few minutes of delay costs more than an occasional extra message. Keep the threshold low enough that you hear about a real failure while someone can still act on it.
For a brochure page, a contact page or a blog, a higher threshold is usually fine. A short outage on a page that few people need in a hurry is less urgent, and you may prefer fewer messages.
For a backup or a scheduled job, the question is different. A heartbeat monitor checks in each time the job succeeds, and alerts you when a check-in is late. The failure threshold on that kind of monitor works in the same spirit, but the right value depends on how long the job normally takes and how often it runs.
When to lower it, and when to raise it
Lower the threshold after a missed outage. If a real problem reached your customers before you heard about it, the threshold was too high for that page. Drop it by one and watch the next few weeks.
Raise the threshold after a few false alarms. If you have received several alerts that cleared before you could look, and nothing was wrong, the page may be noisy at the current setting. Raise it by one and check whether the quiet period also hides real problems.
Change one value at a time and write down the date. Otherwise you will not know which change made the difference. A simple note in your task tracker is enough.
A short checklist to review thresholds on your first 10 monitors
If you are setting up a few monitors for the first time, or reviewing ones you set up in a hurry, go through this list for each one.
- Name the page or job and say in one sentence what it does for a user or client.
- Note the check interval, so you can work out the delay for each threshold.
- Decide whether a delay of a few minutes is acceptable. If not, use a lower threshold.
- Set the threshold and write it next to the monitor name.
- Look at the alerts for two weeks. Count the false alarms and the late discoveries.
- If you had a false alarm, consider raising the threshold by one.
- If a real problem reached a customer first, consider lowering it by one.
- Check that alert emails go to people in your organisation who can act on them.
Do this for your first 10 monitors and you will have a clearer picture of which pages need fast attention and which can wait a little longer before they interrupt anyone.
The Free plan includes 10 monitors, so you can try this on a real set of pages without paying anything. Set a threshold on each one, then see how quiet your alerts get. You can start at https://app.aperioflow.co.uk/#/signup.