Alert rules that are not noise

An alert earns its place only if you can finish the sentence "when this fires I will…". Everything else is a dashboard panel. Twenty-five rules you read beats eighty you have trained yourself to dismiss, and the way you get there is a written reason next to every threshold.

The lab this material came from had eleven alerts firing continuously. Not one of them described something anybody was going to do. Everything was, technically, being monitored. Nothing was being noticed. That is the failure worth designing against, and it is not caused by bad rules — it is caused by accumulating rules, each of which looked reasonable on the day it was added.

The one rule: no alert without an action

Before a rule goes in the file, write the sentence that starts "when this fires I will…". If you cannot finish it, you have found a dashboard panel. Put it on a dashboard. A panel you look at when you are curious costs nothing. A notification you have taught yourself to swipe away costs you every real alert that arrives after it.

The corollary is unpopular and correct: an alert nobody acts on gets deleted or fixed, never silenced indefinitely. A permanent silence is a deleted alert that still burns evaluation time and still pads the count so you feel covered.

Write the reason next to the number

Every threshold is a trade between hearing about a problem early and being woken by something that would have fixed itself. That trade is a judgement, and judgements evaporate. Six months later you are looking at for: 15m with no idea whether 15 was reasoning or a default, so you leave it alone — or worse, you widen it during an incident and never narrow it back.

So the reason lives in the rules file as a comment beside the number. Here is what that looks like for the rules I keep, and the reasoning is the point of the table, not the numbers:

Two patterns run through that column. Short windows are for conditions that never self-heal — quorum loss, an OOM kill that already happened. Long windows are for conditions with a normal, boring version that resolves itself: a reboot, a unit restarted by a dependency, a temporary file. Everything else sits between those two poles, and where it sits is the thing you write down.

Four moves, in order

1. Delete

Go through the currently firing list and finish "when this fires I will…" for each one. Delete every rule that fails the sentence. This is usually about half of them, and it is the only step in the whole process with no downside.

2. Deduplicate at the source

One event should produce one notification. The commonest source of duplicates in a homelab is not the alert router — it is a cluster exporter. Every Proxmox node's exporter reports on every guest and every storage in the cluster, so a naive sum() over a three-node cluster gives you three times the truth, and your alert says nine guests are unprotected when three are. Wrap cluster-wide series in max by (id) (...).

Two rules for one condition is the other duplicate: a warning at 85% and a critical at 95% both fire at 96%. That is what inhibition is for, not a reason to drop one tier.

3. Group and delay

route:
  group_by: ['alertname', 'severity']   # one message per alert type, not per host
  group_wait: 30s        # collect the stragglers of a single event
  group_interval: 5m     # a new host joining an existing group waits this long
  repeat_interval: 12h   # do not re-tell me the same thing every four hours

repeat_interval is the setting people leave at four hours and then wonder why they stopped reading notifications. Twelve or twenty-four hours is right for a lab. A problem you have not fixed in twelve hours will not be fixed because you were reminded a fourth time.

Note the direction of the grouping choice: let the router group per-host detail into one message rather than aggregating it away inside the rule. That way one NFS server falling over is one notification that still lists the twelve affected mounts, instead of twelve messages or one message that hides which mounts they were.

4. Inhibit the consequences

Inhibition suppresses alerts caused by another firing alert. It is the difference between "the NFS server is down" and forty pages about mounts.

inhibit_rules:
  # A critical about a thing silences the warning about the same thing.
  - source_matchers: [severity = "critical"]
    target_matchers: [severity = "warning"]
    equal: ['alertname', 'instance']

  # A host that is down is not also interestingly out of memory.
  - source_matchers: [alertname = "InstanceDown"]
    target_matchers: [severity =~ "warning|info"]
    equal: ['instance']

  # A storage that is down explains every stale backup job on it.
  - source_matchers: [alertname = "ProxmoxStorageDown"]
    target_matchers: [alertname =~ "BackupTooOld|JobStale|JobDark"]

Three tiers with three real behaviours

If everything you have is critical, nothing is. A good sign the tiers are working: you have info rules that have never sent you a message and you would still not delete them.

Watch the watcher

Every de-noised system has one hole left. A monitoring stack that is down is perfectly quiet, and quiet is what health looks like. Internal rules cover the internal cases — rule evaluation failing, notifications failing — but nothing inside the stack can report the stack being off.

For that you need a check from outside the failure domain, on a second small host or a free external heartbeat service, that alerts when the monitoring server stops answering and deliberately does not route through your alert manager, because the alert manager is one of the things being watched. Fifteen lines of shell, a counter file so three consecutive failures are required and a restart is not an incident, and it covers the one failure your alerting system can never report on itself.

Fifteen minutes a quarter

Put a recurring review on the calendar. Per rule: how many times did it fire, how many of those were worth acting on, and does it still have an action? A rule at zero out of ten gets its threshold changed or gets deleted — and if you change the threshold, the new reason goes next to the new number, same as the first one did.

Fifteen minutes a quarter is the whole maintenance cost of keeping the list at twenty-five rules you read instead of eighty you have stopped seeing.

The full rule file — 25 rules across seven groups, each with its reasoning as a comment and a pointer into the incident runbook — plus a working router config, the dead-man's-switch script and the quarterly alert-review worksheet are in the Homelab Monitoring Kit ($49 minimum / $59 suggested). Free updates forever. 30-day 100% refund, no questions.