We use Velero for automated backups of Kubernetes resources. Although we were scraping backup-related metrics into Prometheus, we hadn’t defined any automated alerts in case of backup failures.
Our monitoring solution now fires a number of alerts notifying of the backup’s health.
This has already been rolled out everywhere.