AperioFlow Start free

Blog

A heartbeat alert has arrived: what to check when a cron job checks in late

A late check-in alert tells you one thing: the job did not report success within the window you set. It does not tell you the job crashed. The job may have failed, may have started late, may be running slower than it used to, or may have finished fine while the message to AperioFlow never arrived. Each of those needs a different fix, so work through the checks in order and stop when you find the cause.

The heartbeat monitor works by waiting. Your job calls the check-in URL each time it succeeds. If the call does not arrive before the expected time plus any grace you have set, AperioFlow sends an alert to the people in your organisation. If you have set a failure threshold on the monitor, the alert waits until that many checks have been missed in a row, so a single slow run may not reach you at all.

First check: did the job start?

Start with the simplest question. Did the job run at all?

Look at the scheduler that owns the job. Most schedulers keep a record of each run, with a start time, an end time and an exit status. If there is no entry for the expected run, the job was never triggered. Common reasons are a disabled entry, a changed path, a user account that no longer exists, or a schedule that was edited with a typo.

Next, check the server clock. A job scheduled for 02:00 on a server set to the wrong time zone, or a server whose clock has drifted, will run at a time you did not expect. Compare the server time with a known source and check whether the job ran an hour early or late. If the schedule changed around the clock change, that is the likely cause.

Finally, read the crontab entry, or the equivalent schedule definition, exactly as the system sees it. Check the minute, hour, day and month fields. Check that the command uses full paths to the interpreter and the script, because a job that runs with a minimal environment often fails to find a tool it could find in your shell. If the job never started, fix the schedule and run it once by hand before you wait for the next window.

Second check: did it run but fail?

If the job started, the next question is whether it finished its work. A job that errors out before the check-in call will look exactly like a job that never ran.

Read the job's own log. Look for the last lines written before the run ended. A stack trace, a database connection error or a "file not found" message usually points straight at the problem. If the job writes no log, add one for the next run. A few lines with a timestamp at the start, at each major step and at the end are enough to tell you where it stopped.

Check the exit status too. Many jobs exit with zero on success and a non-zero code on failure. Some scripts swallow errors and exit zero anyway, which can hide a real failure from the scheduler. If your job does this, fix the script so that it exits with a non-zero code when a step fails.

Then check where the check-in call sits in the script. It should come only after the work has succeeded, as the last step before a clean exit. If the call comes first, or sits in a block that runs whether or not the work succeeded, the monitor will report success for a job that failed. Move the call so that a failure stops the script before it reaches the call. A late check-in alert in this case is doing its job, because the call was missing for a reason.

Third check: did it run long?

A job can start on time, succeed and still check in late, because it took longer than the window you gave it. This is common with backups and data jobs, where the volume of work grows over months without anyone changing the schedule.

Compare the run time with the interval you set. Your scheduler or your logs will show how long recent runs took. If the run time has crept up to the length of the interval, the job is close to overlapping with its next run, and the check-in will arrive late some of the time. Look at the start and end times across the last several runs rather than one run, so you can see the trend.

If the job is simply slower now, adjust the expected interval to match. Set it to the new normal with a sensible margin, so that a normal run finishes well inside the window. If the job needs its own grace period, set one on the monitor rather than shortening the work to fit an interval that no longer suits it. A longer window is an honest answer when the job has more to do. It is also worth asking whether the work can be split, or whether an old step can be removed.

Do not widen the window to hide a job that has started to hang. If the run time has jumped sharply in one step, that step needs investigating before you change any setting.

Fourth check: could the check-in request not get out?

The last cause is the one people forget. The job ran, it succeeded, and it tried to send the check-in, but the request did not reach AperioFlow.

Start with the network. Confirm that the server can make outbound HTTPS requests. A firewall rule, a proxy that needs configuring, or a DNS problem on the host can stop the call without any error in the job's own log. From the server, try a manual request to the check-in URL and look at the response. If the request fails at the network layer, the problem is on your side of the connection.

Next, look at the ping URL itself. Check that it has been copied in full, with no line break or missing character, and that the job uses the current URL for this monitor. If you recently recreated the monitor or moved the job to a new server, the old URL may still be in the script. Put the URL in a configuration value or environment variable so that you can change it in one place.

If the request is failing only some of the time, log the response code from the check-in call. A short log line recording the status returned makes intermittent problems much easier to trace across several runs.

After the fix: confirm the next check-in arrives and note the cause in one line

When you have made a change, do not close the alert on trust. Run the job by hand if you can, or wait for the next scheduled run, and confirm that the check-in arrives and the monitor returns to healthy. If the next run is days away, trigger it manually now and check the monitor's history.

Then write down the cause in one line in the monitor's notes or your team's log. Something like "Cron entry had wrong time zone, corrected on the server" or "Backup ran 40 minutes long after the table grew, window raised to 90 minutes" is enough. A single line like this saves the next person from repeating the same search, and it shows you over time which causes come back.

Keep a short runbook next to the job so whoever gets the alert knows where to start

The alert usually goes to the person who set the job up, but not always. It may be a colleague covering leave, or someone who has never seen the server. A short runbook stored next to the job, in the repository or in the same place as the monitor's notes, lets anyone start in the right place.

Keep it brief. Record the job's name, what it does and where it runs. Add the four checks above in the order you would use them, with the exact paths to the logs, the scheduler and the script. List the people who can make a change and how to reach them out of hours. Note the expected run time and the window you set, so that anyone reading an alert can see at once whether a late check-in is unusual.

Review the runbook after the first real incident. The steps that worked, and the ones that did not, will tell you what to add. If you are setting up a heartbeat for the first time, our setup post covers the monitor itself, and the post on expected intervals for long jobs covers how to choose a window before the alerts start.

Heartbeat monitors are included in the free plan's 10 monitors, so adding one to your most important job costs nothing extra. Start free at https://app.aperioflow.co.uk/#/signup

Start free More from the blog

Built for the people who write the code and answer the phone when it breaks.

Run from the UK. Hosted in France.

UK time --:--

Already in?Open the app Start free today
View our policies