Green means it ran

A job is green only when its last success is newer than a threshold you chose on purpose. Enabled, active, scheduled and configured are not green. If nothing recorded a success inside the budget, the tile goes red — and a job that has never succeeded starts red.

This is one rule, it fits on an index card, and it is the difference between a dashboard that reports on your lab and a dashboard that reports on your lab's config files. Most homelab monitoring measures things that are running. Almost none of it measures things that have stopped, which is the failure you actually get: the cron entry that was removed during a rebuild, the container that stopped being scheduled, the backup script that has been exiting 0 without writing anything since March.

Every one of those failures is invisible to a check that asks "is the timer enabled?" All of them are caught by a check that asks "when did this last succeed?"

Why config state can never be green

A systemd timer can be enabled on a machine that is powered off. A cron line can exist on a host whose clock is wrong. A backup job can report FINISHED and transfer nothing. In all three cases every configuration-shaped check passes and the work did not happen.

Freshness has the property configuration lacks: it decays without cooperation. A job that simply stops calling record goes green, then stale, then dark, entirely on its own. Nothing has to notice, nothing has to report the failure, no agent has to survive on the dead host. That is what makes the rule hold in the exact case it was built for — the machine being off.

There is deliberately no fourth way to become green. Not a config file, not a unit being loaded, not a process existing. Something calls record when the work succeeded, or the tile is not green.

Three states, not two

Two states throw away useful information. One missed nightly run and a job that has not succeeded since spring should not look the same:

• Green — last success is inside the budget.
• Stale — budget blown, but by less than the dark multiplier. One missed slot. Look at it in the morning.
• Dark — badly overdue, or has never recorded a success at all.

The last clause matters more than it looks. A job you configured and never verified reports dark, not "unknown". A backup with no successful run on record is not a neutral state waiting for data; it is the worst state, and it should look like it from the first day.

Where the timestamp comes from

Four sources, in the order I trust them:

1. The newest file in the directory the job is supposed to fill. This is the only source that asks about the artifact rather than the job, and it is the one that catches "reports FINISHED, transferred nothing". If you take one thing from this page: point a freshness check at the destination directory of every backup job you have.
2. A stamp file the job touches on success — with one condition, below.
3. A timestamp you already hold, from a log or a database row.
4. Anything that can print a unix timestamp: an API call, a client's snapshot list, a query.

The condition on stamp files is the whole game. The stamp must be written only on success, at the very end of the job, after the last step that can fail. A stamp written at the start, or written unconditionally by a trailing line in the script, is a marker that says "this script ran" — which is the failure mode you were trying to detect, with extra steps and more confidence.

One check, in shell

The pattern is small enough to write out. This is the directory-freshness version, the one worth having on every backup destination:

#!/bin/sh
# Green only if the destination got something recent.
DIR=/mnt/pve/nas-backup/dump
GLOB='vzdump-*'
MAX_AGE_HOURS=30

newest=$(find "$DIR" -maxdepth 1 -name "$GLOB" -printf '%T@\n' 2>/dev/null \
         | sort -n | tail -1 | cut -d. -f1)
[ -z "$newest" ] && { echo "MISSING vzdump: no matching file in $DIR"; exit 1; }

age_h=$(( ( $(date +%s) - newest ) / 3600 ))
[ "$newest" -gt "$(date +%s)" ] && { echo "MISSING vzdump: timestamp in the future (clock skew)"; exit 1; }
[ "$age_h" -gt "$MAX_AGE_HOURS" ] && { echo "STALE vzdump: ${age_h}h old, budget ${MAX_AGE_HOURS}h"; exit 1; }

echo "OK vzdump: ${age_h}h old"

Exit 1 on anything that is not OK, so it drops straight into cron, a wrapper, or a textfile exporter. Two details are worth keeping when you rewrite this in your own style. A timestamp in the future is reported as missing, not as very fresh: future timestamps make every age calculation meaningless and clock skew is worth knowing about on its own. And if you write the result to a file for a scraper to read, write it to a temp file and rename it, so nothing ever reads a half-written document.

Wrapping a job instead of editing it

You rarely want to modify the job itself. Put a wrapper in front of what you already run: it runs the command, times it, records the real exit code, and exits with that same code, so nothing downstream can tell it is there.

0 2 * * *  /opt/liveness/liveness.sh nightly-backup -- /usr/local/bin/backup.sh

For a systemd service, the same wrapper goes in front of ExecStart. For a job you cannot wrap at all — something on an appliance, or a scheduled task on another operating system — have it stamp itself over SSH at the end of its own script. The mechanism does not matter. What matters is that the stamp happens only when the work succeeded.

Picking the threshold

The budget is the interval plus enough slack for a slow run, and no more. Slack for a slow run is not slack for a skipped one — that distinction is the entire art here.

The general rule is 1.25× the interval for anything daily or faster, and interval plus one cycle's slack for anything slower. Never exceed 1.5× on a daily job: past that, a completely missed run hides inside your own tolerance and you have built a check that certifies its own blind spot.

When you get a false alarm, widen the threshold once, and write the reason in the config next to the number. If a job's runtime has grown enough that it needs more than 1.5×, the finding is the runtime, not the threshold.

Run the check somewhere else

A freshness check for a node's backup that runs on that node reports nothing when the node is down — precisely the moment you wanted to hear from it. Run the check from a different host where you can: the NAS, a second small box, the Raspberry Pi in the corner. You do not need both. If you only get one, put it on the other host.

The same logic applies one level up. Whatever exports these results has its own freshness, and a frozen exporter makes a dead job look alive. Add one rule that watches the collector's own age — older than an hour is a problem — and compute ages at query time from the recorded timestamps rather than at write time, so a missed export never turns into a false green.

The label that will bite you

If you export these states to Prometheus, name the label job_name, not job. Prometheus owns job and overwrites it at scrape time with the scrape job's name. Export job="nightly-backup" and it silently becomes job="node" by the time it reaches storage — every per-job panel collapses into one series, and there is no error anywhere. The same rule applies to instance. It is the single most common way a home-grown exporter produces confident nonsense.

Prove the alert path, once a quarter

A freshness check whose notification path is broken is worse than no check, because it is a silence you have learned to trust. Once a quarter, point a check at a stamp file that does not exist, confirm it reports stale and exits 1, and confirm the notification reaches the device you would be holding at 2 a.m. Prefix drill notifications with TEST: so a real one is never mistaken for a drill, and write the date down.

Green means it ran. That is the whole rule, and once every tile on your board obeys it, a green board is finally evidence rather than decoration.

The liveness guard, the exporter, the dashboards and the 25 alert rules are in the Homelab Monitoring Kit ($49 minimum / $59 suggested). The backup-specific version — liveness-check.sh, its four sources and the threshold table — ships in the Proxmox Backup Verification Pack ($39 minimum / $49 suggested). Free updates forever. 30-day 100% refund, no questions.