Alertmanager inhibit_rules examples: stop the storm behind the outage

A single host goes down at 02:00. You get the InstanceDown alert, which is the one you needed. You also get the disk alert that stopped reporting, the service alert for each container on that box, the certificate check that could not connect, and the latency alert from the load balancer in front of it. Sixteen notifications, one problem, and the useful one is somewhere in the middle.

inhibit_rules are Alertmanager’s answer to that. They express causal relationships in config: while this is broken, do not tell me about that, because that is just this again wearing a different label.

This assumes you already have Alertmanager running and routing. If not, the setup guide covers that, and the routing tree deep dive covers where notifications go once they survive inhibition.

How inhibition actually works

An inhibition rule has three parts:

inhibit_rules:
  - source_matchers: [severity = "critical"]   # if an alert like this is firing
    target_matchers: [severity = "warning"]    # then mute alerts like this
    equal: [alertname, cluster]                # but only when these labels match

Read it as a sentence: while a critical alert is firing, mute any warning alert that has the same alertname and the same cluster.

Three things determine whether it works:

1. The source must currently be firing. Inhibition is live state, not history. The moment the source resolves, the targets start notifying again on their own.
2. Every label in equal must match on both sides. Not just be present - have the same value.
3. It suppresses notification, not the alert. The target alert is still firing, still visible in the Alertmanager UI and in amtool alert query, still in Prometheus. It is only the notification that is withheld. This is a feature: you can go and look at the full picture during an incident.

source_matchers and target_matchers use the same operators as routing (=, !=, =~, !~). The older source_match, source_match_re, target_match, and target_match_re fields still work but are deprecated.

Pattern 1: host down suppresses everything on that host

The single highest-value rule most teams can add. If the box is down, every check that depends on the box is going to complain, and none of those complaints add information.

inhibit_rules:
  - source_matchers: [alertname = "InstanceDown"]
    target_matchers: [severity =~ "warning|critical"]
    equal: [instance]

equal: [instance] is what keeps this safe. It scopes the suppression to the dead host only - a disk alert on a different, perfectly healthy server has a different instance value and is untouched.

Note the deliberate omission: there is no alertname in equal here, because the whole point is to suppress different alerts that share a host.

Pattern 2: the severity ladder

Many rule sets fire a warning and a critical for the same condition at different thresholds - disk at 80% and disk at 95%. Once you are at 95%, the 80% alert is noise.

inhibit_rules:
  - source_matchers: [severity = "critical"]
    target_matchers: [severity = "warning"]
    equal: [alertname, instance]

Here alertname is in equal, because you only want DiskSpaceCritical to suppress DiskSpaceWarning on the same host - not to suppress an unrelated memory warning that happens to be on the same box.

That contrast between patterns 1 and 2 is the whole skill. What you put in equal defines the blast radius of the suppression.

Pattern 3: cluster or datacentre down

The tier above host-down. When an entire cluster or region is unreachable, individual host alerts are a distraction from the thing you actually need to act on.

inhibit_rules:
  - source_matchers: [alertname = "ClusterUnreachable"]
    target_matchers: [alertname =~ "InstanceDown|NodeNotReady|ServiceDown"]
    equal: [cluster]

This composes with pattern 1, and the layering works in your favour: ClusterUnreachable mutes all the InstanceDown alerts, and each surviving InstanceDown mutes its own host’s children. One notification for a regional outage instead of two hundred.

Pattern 4: planned maintenance as a first-class alert

A neat trick. Emit a maintenance alert from a Prometheus rule driven by a time-based expression or an external metric, then let it inhibit everything in scope:

inhibit_rules:
  - source_matchers: [alertname = "MaintenanceWindow"]
    target_matchers: [severity =~ "warning|critical"]
    equal: [cluster]

This gives you maintenance suppression that is version-controlled and automatic, rather than depending on somebody remembering to run amtool silence add before a deploy and remembering to expire it afterwards. Manual silences are still the right tool for unplanned work - this is for the recurring, predictable windows.

The equal gotcha that breaks rules silently

This is the one that costs people an afternoon.

Every label in equal must be present on both alerts. A label that is missing is treated as an empty string, and Alertmanager compares values, so:

• Source has cluster="eu-1", target has cluster="eu-1" -> match, inhibition works.
• Source has cluster="eu-1", target has no cluster label -> "eu-1" versus "" -> no match, no inhibition, and no error anywhere.

The rule looks correct in the config file. amtool check-config passes. Nothing in the logs complains. You just keep getting the alerts you thought you had suppressed.

Worse, the reverse case is dangerous. If neither alert has the label, both are "", they match, and the inhibition applies far more broadly than you intended - potentially muting alerts across your whole estate.

Before trusting a rule, confirm the labels genuinely exist on both sides:

amtool alert query alertname=InstanceDown -o extended
amtool alert query alertname=DiskSpaceWarning -o extended

Read the label sets from the actual firing alerts, not from the rule file you think produces them. relabel_configs, recording rules, and federation all reshape labels in ways that are easy to forget.

The self-inhibition trap

Nothing stops an alert from matching both source_matchers and target_matchers. Consider:

# Broken: severity is not constrained on the source side
- source_matchers: [alertname = "ServiceDown"]
  target_matchers: [alertname = "ServiceDown"]
  equal: [cluster]

Every ServiceDown alert is now both a source and a target for every other ServiceDown in the same cluster. The behaviour is confusing at best and can suppress the very first alert you needed.

Keep the two matcher sets mutually exclusive. If they share an alertname, make sure something else - usually severity - separates them, as in pattern 2.

Verifying a rule does what you think

There is no dry-run for inhibition the way amtool config routes test works for routing, so verification is empirical:

1. Check the config parses: amtool check-config /etc/alertmanager/alertmanager.yml
2. Reload: systemctl reload alertmanager
3. Trigger the source and target together in a test environment. Stopping node_exporter on a test box is a cheap way to produce a real InstanceDown.
4. Confirm both alerts are firing in amtool alert query while only one notified. The Alertmanager UI marks suppressed alerts explicitly, which is the clearest confirmation.
5. Resolve the source and confirm the target starts notifying again.

Step 4 is the important one. If the target alert is not firing at all, you have not tested inhibition - you have tested that your alert rule did not fire.

What inhibition cannot fix

Inhibit rules are a noise filter, not a triage system. They work on label equality, which means they can only encode relationships you already knew about and thought to write down. They do not help with the alert that fires every Tuesday for a reason nobody has diagnosed, they cannot tell you which of two simultaneous unrelated problems matters more, and they will not tell you the deploy fifteen minutes ago is the reason any of this is happening.

Deduplicating a storm is worth doing, and if you are running Alertmanager these rules will pay for themselves in a week. But the deeper fix is reducing what alerts in the first place and having something correlate an alert with what changed on the box.

That correlation is the part AlertKick builds in: alerts are linked to the events and changes that preceded them, triaged automatically rather than by a label-equality rule you maintain by hand, and delivered through escalation that knows who is on call. If you would rather not hand-maintain a correlation graph in YAML, that is the alternative.

Start with the host-down rule though. It is four lines and it removes more noise than anything else on this page.