Prometheus Alertmanager setup: routing, grouping, and notifications that work

Prometheus decides what is wrong; Alertmanager decides who hears about it and how often. Without it, every firing rule is just a row in a web UI nobody watches. This guide installs Alertmanager, wires it to Prometheus, and builds a config that does the four things that matter - dedupe, group, route, silence - with a complete working alertmanager.yml you can adapt. It assumes Prometheus is installed and you have alert rules worth firing.

1. Install (same pattern as Prometheus)

From prometheus.io/download - substitute the current version for 0.33.1:

cd /tmp
curl -LO https://github.com/prometheus/alertmanager/releases/download/v0.33.1/alertmanager-0.33.1.linux-amd64.tar.gz
tar xvf alertmanager-0.33.1.linux-amd64.tar.gz
cd alertmanager-0.33.1.linux-amd64
sudo cp alertmanager amtool /usr/local/bin/
sudo useradd --system --no-create-home --shell /usr/sbin/nologin alertmanager
sudo mkdir -p /etc/alertmanager /var/lib/alertmanager
sudo chown -R alertmanager:alertmanager /etc/alertmanager /var/lib/alertmanager

# /etc/systemd/system/alertmanager.service
[Unit]
Description=Prometheus Alertmanager
Wants=network-online.target
After=network-online.target

[Service]
User=alertmanager
Group=alertmanager
Type=simple
Restart=on-failure
ExecStart=/usr/local/bin/alertmanager \
  --config.file=/etc/alertmanager/alertmanager.yml \
  --storage.path=/var/lib/alertmanager \
  --web.listen-address=127.0.0.1:9093

[Install]
WantedBy=multi-user.target

Alertmanager listens on 9093 and, like Prometheus, ships without authentication - localhost binding plus a reverse proxy or SSH tunnel is the sane default.

2. A config that does the four jobs

# /etc/alertmanager/alertmanager.yml
global:
  slack_api_url_file: /etc/alertmanager/slack_webhook   # keep secrets out of the yml

route:
  receiver: team-slack          # default: everything lands somewhere
  group_by: [alertname, instance]
  group_wait: 30s               # collect related alerts before first notify
  group_interval: 5m            # batch newcomers into an existing group
  repeat_interval: 4h           # re-notify while still firing
  routes:
    - matchers: [severity = "critical"]
      receiver: oncall-webhook
      repeat_interval: 1h       # criticals nag harder
    - matchers: [severity = "info"]
      receiver: "null"          # rules you want recorded, not announced

receivers:
  - name: team-slack
    slack_configs:
      - channel: "#alerts"
        send_resolved: true
        title: '{{ .CommonLabels.alertname }} ({{ .Status }})'
        text: '{{ range .Alerts }}{{ .Annotations.summary }}{{ "\n" }}{{ end }}'
  - name: oncall-webhook
    webhook_configs:
      - url: "https://example.internal/hooks/oncall"
  - name: "null"

inhibit_rules:
  - source_matchers: [alertname = "InstanceDown"]
    target_matchers: [severity = "warning"]
    equal: [instance]           # host down? suppress its own noisy children

What each piece is doing for you:

• group_by + group_wait turn a 40-server disk-alert storm into one notification listing 40 instances, instead of 40 pings that get the channel muted.
• repeat_interval is the anti-forgetting mechanism: still broken, still notifying - 4h for warnings, 1h for criticals is a humane starting point.
• The routing tree is first-match top-down (with the default as a safety net), so put specific matchers first. Every alert must end at some receiver - a route to nowhere is how alerts silently vanish. (Routing tree in depth: nesting, continue, matcher operators, and testing with amtool.)
• inhibit_rules encode “if the host is down, do not also page about its full disk”. One rule kills entire classes of redundant noise. (More inhibition patterns, including the equal gotcha that breaks rules silently.)
• send_resolved: true closes the loop in the channel - without it, nobody knows whether the 3am alert fixed itself. (Slack config and templates for messages that lead with the summary rather than the label set.)

Validate, then start - amtool check-config catches the receiver-name typos and matcher syntax errors that otherwise fail at 3am:

amtool check-config /etc/alertmanager/alertmanager.yml
sudo systemctl daemon-reload && sudo systemctl enable --now alertmanager

3. Point Prometheus at it

In /etc/prometheus/prometheus.yml:

alerting:
  alertmanagers:
    - static_configs:
        - targets: ["localhost:9093"]

rule_files:
  - /etc/prometheus/rules/*.yml

promtool check config /etc/prometheus/prometheus.yml && sudo systemctl reload prometheus

Verify end-to-end with a deliberately trivial rule (vector(1) as the expression fires instantly), or just stop node_exporter somewhere and watch InstanceDown arrive in Slack: firing in the Prometheus UI -> visible in amtool alert query -> grouped notification in the channel.

4. Silences: planned work without alert spam

Maintenance windows are what separates trusted alerting from muted alerting:

amtool silence add instance="10.0.1.10:9100" \
  --duration 2h --author "you" --comment "kernel upgrade"
amtool silence query          # what is currently muted

Silences match labels, expire automatically, and are visible in the UI - the difference between “we chose not to be paged” and “someone muted the channel and forgot”.

What Alertmanager deliberately does not do

Alertmanager’s scope ends at delivering a payload to a channel or webhook. Three things are explicitly out of scope, and every team hits them:

• On-call schedules. There is no roster; “#alerts” is not a person. Who is on this weekend lives somewhere else.
• Escalation on no-response. If the notification is not acknowledged, nothing happens next. There is no “then phone the secondary after 10 minutes”.
• Phone and SMS. Slack pings do not wake people. Critical paths need a channel that bypasses do-not-disturb.

That layer is a service, not a config file: AlertKick provides on-call rosters, multi-step escalation chains, and delivery over email, Slack, Telegram, WhatsApp, SMS, and phone - on the free plan - alongside the external uptime, SSL, and domain expiry checks your internal stack cannot see, and a heartbeat for the Prometheus host itself, because the monitoring stack is the one thing it can never alert on. Alertmanager for routing metrics alerts, an external layer for reachability and escalation: that pairing is the whole picture.