Alertmanager - Discard a notification

Alertmanager is a tool using prometheus to handle alerts on metrics.

In some cases it can be useful to ignore some alerts, for example in Kubernetes you may want to ignore some CPUThrotlingHigh alerts.

This alert means you container wants to use more CPU over a period of time than the allowed quota permits. For some low used apps, you may configure a low CPU limit because slowing app is acceptable.

To solve this case we need 2 elements on the AlertManager side:

An empty receiver
A route to this receiver

Here is the needed configuration:

route:
  group_by: [alertname]
  repeat_interval: 12h
  group_wait: 2m
  group_interval: 5m
  receiver: "slack-default"
  routes:
    # blackhole
    - receiver: "blackhole"
      match_re:
        namespace: "prometheus-k8s"
        container: "kube-rbac-proxy"
        alertname: "CPUThrottlingHigh"
      continue: false
    # Other filters
    - receiver: "platform"
      group_interval: 15m
      match_re:
        severity: (critical|warning)
        environment: (test|staging|production|preprod|prod)
        namespace: (kube-system|fluentd|traefik-ingress-controller|prometheus-k8s|opa-gatekeeper).*
        alertname: (KubeCPUQuotaOvercommit|KubeMemoryQuotaOvercommit|KubeQuotaExceeded|CPUThrottlingHigh).*
      continue: false
    

receivers:
  - name: blackhole
  # France
  - name: "platform"
    slack_configs:
      - api_url: 'url'
        send_resolved: true
        channel: "#alerts"
        title: '{{ template "slack.title" . }}'
        text: '{{ template "slack.text" . }}'

Enjoy !