We have updated the format of the monitoring Slack notifications. You might have already noticed that the monitoring messages in your Slack channels now contain more useful information and are more structured. We’ve already started rolling out the changes in staging clusters and we’ll start rolling them in production clusters during this week.
Notifications should now look something like this:
Most of the items in the notification are self-descriptive. You’ll also notice two emojis:
- :chart_with_upwards_trend: - Link to the Prometheus metric that triggered the alert. Useful to check the progression of that metric and troubleshoot the issue.
- :notebook: - Link to the runbook of the alert. Runbooks contain useful information about the triggered alert that should help the user to troubleshoot them. This icon is only present if the triggered alert contains the
runbook_urlannotation. We highly recommend you to also include arunbook_urlannotation to all your custom alerts so the engineer that handles those alerts has all the needed information.
You can get more information about our infrastructure monitoring and alerting in the documentation repository.