Alertmanager Slack config and templates: notifications people actually read

Alertmanager’s default Slack output is technically complete and practically useless. It prints the label set, which means the important sentence - what broke and on what - is buried among job, endpoint, prometheus, __name__, and eight other things nobody reads at 3 AM.

The fix is about twenty lines of template. This post covers the slack_configs block itself, then the templating that makes the message scannable.

If Alertmanager is not yet running, start with the setup guide. If alerts are reaching the wrong channel, that is a routing tree problem, not a receiver problem.

Step 1: the webhook, kept out of the config

Create an incoming webhook in Slack (Apps -> Incoming Webhooks -> Add to Slack, pick the channel), then put the URL in a file rather than in alertmanager.yml:

sudo install -m 0600 -o alertmanager -g alertmanager /dev/null /etc/alertmanager/slack_webhook
sudo tee /etc/alertmanager/slack_webhook >/dev/null <<'EOF'
https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX
EOF

Then reference it globally:

global:
  slack_api_url_file: /etc/alertmanager/slack_webhook

Use slack_api_url_file rather than slack_api_url so alertmanager.yml stays committable. A Slack webhook URL is a bearer credential - anyone holding it can post to that channel as your integration - and it is one of the most commonly leaked secrets in public config repos.

Step 2: a receiver worth reading

receivers:
  - name: team-slack
    slack_configs:
      - channel: "#alerts"
        send_resolved: true
        color: '{{ template "slack.color" . }}'
        title: '{{ template "slack.title" . }}'
        title_link: '{{ .ExternalURL }}/#/alerts'
        text: '{{ template "slack.text" . }}'
        short_fields: true

The fields that matter:

• send_resolved: true - without it, nobody learns the 3 AM alert fixed itself, and the channel fills with problems that appear permanent. Turn it on.
• color - accepts good, warning, danger, or a hex value. Templated below.
• title_link - makes the heading clickable. Pointing at your Alertmanager UI gets people one click from silencing.
• channel - is itself a template string, so #alerts-{{ .CommonLabels.team }} works if your labels are reliable. Be careful: if the label is missing you get #alerts- and the post fails.

Step 3: the templates

Tell Alertmanager where templates live:

global:
  slack_api_url_file: /etc/alertmanager/slack_webhook

templates:
  - '/etc/alertmanager/templates/*.tmpl'

Then /etc/alertmanager/templates/slack.tmpl:

{{ define "slack.color" -}}
  {{ if eq .Status "firing" -}}
    {{ if eq .CommonLabels.severity "critical" -}}danger{{ else -}}warning{{ end -}}
  {{ else -}}good{{ end -}}
{{ end }}

{{ define "slack.title" -}}
  [{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}]
  {{ .CommonLabels.alertname }}
  {{ if .CommonLabels.cluster }}({{ .CommonLabels.cluster }}){{ end }}
{{- end }}

{{ define "slack.text" -}}
{{ range .Alerts -}}
*{{ .Annotations.summary }}*
{{ if .Annotations.description }}{{ .Annotations.description }}
{{ end -}}
`{{ .Labels.instance }}`{{ if .Labels.service }} - {{ .Labels.service }}{{ end }}
{{ if .Annotations.runbook_url }}<{{ .Annotations.runbook_url }}|Runbook> - {{ end -}}
<{{ .GeneratorURL }}|Graph> - <{{ $.ExternalURL }}/#/silences/new?filter=%7Balertname%3D%22{{ .Labels.alertname }}%22%7D|Silence>
{{ end -}}
{{ end }}

What that buys you: severity in the colour bar, a count of firing alerts in the heading, the human summary as the first line of each entry, and three links - runbook, the Prometheus graph that produced the alert, and a pre-filled silence form.

Those links are the difference between a notification and a tool. Someone woken at 3 AM should be one click from the graph and one click from silencing, not hunting for a bookmark.

The data you can template against

This is where most template debugging time goes. The object passed to a Slack template is the group, not a single alert:

Field	What it holds
.Status	firing or resolved for the group as a whole
.Alerts	every alert in the group
.Alerts.Firing	just the firing ones (len gives a count)
.Alerts.Resolved	just the resolved ones
.GroupLabels	the labels you grouped by
.CommonLabels	labels identical across every alert in the group
.CommonAnnotations	annotations identical across every alert
.ExternalURL	your Alertmanager’s external URL

Inside {{ range .Alerts }} the context changes to a single alert, giving you .Labels, .Annotations, .StartsAt, .EndsAt, and .GeneratorURL (the link back to the Prometheus expression).

The trap: .CommonLabels.instance is empty whenever a group spans more than one host, because the value is not common to all of them. Templates that rely on .CommonLabels for anything host-specific look perfect in testing with one alert and go blank during the storm you actually built them for. Use .CommonLabels for what you grouped by, and range for everything else.

Note the $ in <{{ $.ExternalURL }}|Silence> above. Inside range, . is the individual alert, so $ reaches back to the top-level group context. Forgetting this is the most common template error.

Different channels for different severities

Combine templates with routing. Criticals somewhere loud, warnings somewhere calm:

route:
  receiver: team-slack
  group_by: [alertname, cluster]
  routes:
    - matchers: [severity = "critical"]
      receiver: critical-slack
      repeat_interval: 1h

receivers:
  - name: team-slack
    slack_configs:
      - channel: "#alerts"
        send_resolved: true
        title: '{{ template "slack.title" . }}'
        text: '{{ template "slack.text" . }}'

  - name: critical-slack
    slack_configs:
      - channel: "#incidents"
        send_resolved: true
        color: danger
        title: '<!here> {{ template "slack.title" . }}'
        text: '{{ template "slack.text" . }}'

Use <!here> sparingly and only in a channel people opted into. An @here on every warning is the fastest route to a muted channel, which is strictly worse than no alerting because it looks like you have alerting.

Testing without breaking things

Templates fail at send time, not at config-check time. amtool check-config parses the file and confirms the templates load, but it does not render them, so a .CommonLabels.nonexistent mistake surfaces only when a real alert arrives.

# Parses config, loads template files, catches syntax errors
amtool check-config /etc/alertmanager/alertmanager.yml
sudo systemctl reload alertmanager

To exercise the real path, push a fake alert straight into the API:

curl -s -XPOST http://localhost:9093/api/v2/alerts -H 'Content-Type: application/json' -d '[
  {
    "labels": {
      "alertname": "TemplateTest",
      "severity": "critical",
      "instance": "test-01:9100",
      "service": "checkout",
      "cluster": "eu-1"
    },
    "annotations": {
      "summary": "Disk will be full in 4 hours",
      "description": "/var at 91%, growing 2.1%/hour",
      "runbook_url": "https://runbooks.internal/disk-full"
    },
    "generatorURL": "http://prometheus:9090/graph"
  }
]'

That renders your actual templates through your actual receiver into a real channel. Send it to a scratch channel first, and send a two-alert payload as well - that is what catches the .CommonLabels mistake described above.

If a message never arrives, check the Alertmanager logs. Template errors are logged at send time and the notification is dropped, which is a genuinely nasty failure mode: the alert fired, Alertmanager tried, and nothing reached Slack.

What Slack notifications are not

A Slack message is a broadcast to a room. It reaches whoever is looking, which at 03:00 is nobody. send_resolved, colour coding, and a pre-filled silence link all make the message better, but none of them make a phone ring, none of them track whether anyone acknowledged, and none of them escalate to a second person when the first is asleep.

That is not a criticism of Alertmanager - delivering payloads is where its scope deliberately ends. It just means Slack is the right destination for warnings and the wrong one for anything that must be acted on within the hour.

AlertKick handles the other half: on-call rotations with overrides, escalation that climbs least to most disruptive until someone acknowledges, and Slack as one channel among several rather than the only one. It works as a webhook receiver alongside the Alertmanager config above, or as the whole stack if you would rather not maintain the templates at all.

Either way, fix the template. Twenty lines is a small price for a message someone can act on half-awake.