Prometheus配置Alertmanager（钉钉告警）

2026/9/11 13:36:27 拓冰建站浏览量

简介

Alertmanager 主要用于接收 Prometheus 发送的告警信息，它支持丰富的告警通知渠道，例如邮件、微信、钉钉、Slack 等常用沟通工具，而且很容易做到告警信息进行去重，降噪，分组等，是一款很好用的告警通知系统。
下图是Alertmanager与Prometheus的基本架构

一，二进制部署 Alertmanager

**下载地址：**https://github.com/prometheus/alertmanager/releases

本文选择的安装版本为0.24.0

cd /prometheus
#解压
tar -xvzf alertmanager-0.24.0.linux-amd64.tar.gz
mv alertmanager-0.24.0.linux-amd64 alertmanager
cd alertmanager

进行系统service编写

创建alertmanager.service配置文件

cd /usr/lib/systemd/system
vim alertmanager.service

alertmanager.service 文件填入如下内容后保存:wq

[Unit]
Description=https://prometheus.io[Service]
Restart=on-failure
ExecStart=/prometheus/alertmanager/alertmanager --config.file=/prometheus/alertmanager/alertmanager.yml --storage.path=/prometheus/alertmanager/data/[Install]
WantedBy=multi-user.target

查看配置文件

cat alertmanager.service

刷新服务配置并启动服务

systemctl daemon-reload
systemctl start alertmanager.service

查看服务运行状态

systemctl status alertmanager.service

设置开机自启动

systemctl enable alertmanager.service

访问系统

访问系统 http://服务器ip:9093，注意防火墙或安全组开放端口
若看到如上界面则说明alertmanager部署成功

二，配置钉钉机器人

打开钉钉的智能群助手，点击添加机器人
选择自定义机器人
复制webhook地址后点击保存

三，安装钉钉服务(不推荐Docker安装，新版本的安装文档已经很久没更新)

1,二进制安装

**下载地址：**https://github.com/timonwong/prometheus-webhook-dingtalk/releases
本次安装版本为2.1.0

部署包下载完毕，开始安装

cd /prometheus
tar -xvzf prometheus-webhook-dingtalk-2.1.0.linux-amd64.tar.gz
mv prometheus-webhook-dingtalk-2.1.0.linux-amd64 webhook_dingtalk
cd webhook_dingtalk

编写配置文件（复制之后切记删除#的所有注释，否则启动服务时会报错），将上述获取的钉钉webhook地址填写到如下文件

vim dingtalk.yml

timeout: 5stargets:webhook_robot:# 钉钉机器人创建后的webhook地址url: https://oapi.dingtalk.com/robot/send?access_token=xxxxxxxxxxxxwebhook_mention_all:# 钉钉机器人创建后的webhook地址url: https://oapi.dingtalk.com/robot/send?access_token=xxxxxxxxxxxx# 提醒全员mention:all: true

进行系统service编写

创建webhook_dingtalk配置文件

cd /usr/lib/systemd/system
vim webhook_dingtalk.service

webhook_dingtalk.service 文件填入如下内容后保存:wq

[Unit]
Description=https://prometheus.io[Service]
Restart=on-failure
ExecStart=/prometheus/webhook_dingtalk/prometheus-webhook-dingtalk --config.file=/prometheus/webhook_dingtalk/dingtalk.yml --web.listen-address=:8060[Install]
WantedBy=multi-user.target

查看配置文件

cat webhook_dingtalk.service

刷新服务配置并启动服务

systemctl daemon-reload
systemctl start webhook_dingtalk.service

查看服务运行状态

systemctl status webhook_dingtalk.service

设置开机自启动

systemctl enable webhook_dingtalk.service

我们记下 urls=http://localhost:8060/dingtalk/webhook_robot/send 这一段值，接下来的配置会用上

配置Alertmanager

打开 /prometheus/alertmanager/alertmanager.yml，修改为如下内容

global:# 在没有报警的情况下声明为已解决的时间resolve_timeout: 5mroute:# 接收到告警后到自定义分组group_by: ["alertname"]# 分组创建后初始化等待时长group_wait: 10s# 告警信息发送之前的等待时长group_interval: 30s# 重复报警的间隔时长repeat_interval: 5m# 默认消息接收receiver: "dingtalk"receivers:# 钉钉- name: 'dingtalk'webhook_configs:# prometheus-webhook-dingtalk服务的地址- url: http://1xx.xx.xx.7:8060/dingtalk/webhook_robot/sendsend_resolved: true

在prometheus安装文件夹根目录增加alert_rules.yml配置文件，内容如下

groups:- name: alert_rulesrules:- alert: CpuUsageAlertWarningexpr: sum(avg(irate(node_cpu_seconds_total{mode!='idle'}[5m])) without (cpu)) by (instance) > 0.60for: 2mlabels:level: warningannotations:summary: "Instance {{ $labels.instance }} CPU usage high"description: "{{ $labels.instance }} CPU usage above 60% (current value: {{ $value }})"- alert: CpuUsageAlertSerious#expr: sum(avg(irate(node_cpu_seconds_total{mode!='idle'}[5m])) without (cpu)) by (instance) > 0.85expr: (100 - (avg by (instance) (irate(node_cpu_seconds_total{job=~".*",mode="idle"}[5m])) * 100)) > 85for: 3mlabels:level: seriousannotations:summary: "Instance {{ $labels.instance }} CPU usage high"description: "{{ $labels.instance }} CPU usage above 85% (current value: {{ $value }})"- alert: MemUsageAlertWarningexpr: avg by(instance) ((1 - (node_memory_MemFree_bytes + node_memory_Buffers_bytes + node_memory_Cached_bytes) / node_memory_MemTotal_bytes) * 100) > 70for: 2mlabels:level: warningannotations:summary: "Instance {{ $labels.instance }} MEM usage high"description: "{{$labels.instance}}: MEM usage is above 70% (current value is: {{ $value }})"- alert: MemUsageAlertSeriousexpr: (node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes)/node_memory_MemTotal_bytes > 0.90for: 3mlabels:level: seriousannotations:summary: "Instance {{ $labels.instance }} MEM usage high"description: "{{ $labels.instance }} MEM usage above 90% (current value: {{ $value }})"- alert: DiskUsageAlertWarningexpr: (1 - node_filesystem_free_bytes{fstype!="rootfs",mountpoint!="",mountpoint!~"/(run|var|sys|dev).*"} / node_filesystem_size_bytes) * 100 > 80for: 2mlabels:level: warningannotations:summary: "Instance {{ $labels.instance }} Disk usage high"description: "{{$labels.instance}}: Disk usage is above 80% (current value is: {{ $value }})"- alert: DiskUsageAlertSeriousexpr: (1 - node_filesystem_free_bytes{fstype!="rootfs",mountpoint!="",mountpoint!~"/(run|var|sys|dev).*"} / node_filesystem_size_bytes) * 100 > 90for: 3mlabels:level: seriousannotations:summary: "Instance {{ $labels.instance }} Disk usage high"description: "{{$labels.instance}}: Disk usage is above 90% (current value is: {{ $value }})"- alert: NodeFileDescriptorUsageexpr: avg by (instance) (node_filefd_allocated{} / node_filefd_maximum{}) * 100 > 60for: 2mlabels:level: warningannotations:summary: "Instance {{ $labels.instance }} File Descriptor usage high"description: "{{$labels.instance}}: File Descriptor usage is above 60% (current value is: {{ $value }})"- alert: NodeLoad15expr: avg by (instance) (node_load15{}) > 80for: 2mlabels:level: warningannotations:summary: "Instance {{ $labels.instance }} Load15 usage high"description: "{{$labels.instance}}: Load15 is above 80 (current value is: {{ $value }})"- alert: NodeAgentStatusexpr: avg by (instance) (up{}) == 0for: 2mlabels:level: warningannotations:summary: "{{$labels.instance}}: has been down"description: "{{$labels.instance}}: Node_Exporter Agent is down (current value is: {{ $value }})"- alert: NodeProcsBlockedexpr: avg by (instance) (node_procs_blocked{}) > 10for: 2mlabels:level: warningannotations:summary: "Instance {{ $labels.instance }}  Process Blocked usage high"description: "{{$labels.instance}}: Node Blocked Procs detected! above 10 (current value is: {{ $value }})"- alert: NetworkTransmitRate#expr:  avg by (instance) (floor(irate(node_network_transmit_bytes_total{device="ens192"}[2m]) / 1024 / 1024)) > 50expr:  avg by (instance) (floor(irate(node_network_transmit_bytes_total{}[2m]) / 1024 / 1024 * 8 )) > 40for: 1mlabels:level: warningannotations:summary: "Instance {{ $labels.instance }} Network Transmit Rate usage high"description: "{{$labels.instance}}: Node Transmit Rate (Upload) is above 40Mbps/s (current value is: {{ $value }}Mbps/s)"- alert: NetworkReceiveRate#expr:  avg by (instance) (floor(irate(node_network_receive_bytes_total{device="ens192"}[2m]) / 1024 / 1024)) > 50expr:  avg by (instance) (floor(irate(node_network_receive_bytes_total{}[2m]) / 1024 / 1024 * 8 )) > 40for: 1mlabels:level: warningannotations:summary: "Instance {{ $labels.instance }} Network Receive Rate usage high"description: "{{$labels.instance}}: Node Receive Rate (Download) is above 40Mbps/s (current value is: {{ $value }}Mbps/s)"- alert: DiskReadRateexpr: avg by (instance) (floor(irate(node_disk_read_bytes_total{}[2m]) / 1024 )) > 200for: 2mlabels:level: warningannotations:summary: "Instance {{ $labels.instance }} Disk Read Rate usage high"description: "{{$labels.instance}}: Node Disk Read Rate is above 200KB/s (current value is: {{ $value }}KB/s)"- alert: DiskWriteRateexpr: avg by (instance) (floor(irate(node_disk_written_bytes_total{}[2m]) / 1024 / 1024 )) > 20for: 2mlabels:level: warningannotations:summary: "Instance {{ $labels.instance }} Disk Write Rate usage high"description: "{{$labels.instance}}: Node Disk Write Rate is above 20MB/s (current value is: {{ $value }}MB/s)"

修改prometheys.yml,最上方三个节点改为如下配置

global:scrape_interval:     15s evaluation_interval: 15s alerting:alertmanagers:- static_configs:# alertmanager服务地址- targets: ['11x.xx.x.7:9093']rule_files:- "alert_rules.yml"

执行curl -XPOST localhost:9090/-/reload刷新prometheus配置
执行systemctl restart alertmanger.service或docker restart alertmanager刷新alertmanger服务