As you may know, we define our Kubernetes clusters’ desired state in a yaml file, which is stored in the customer private Git repository. That file is then fed into our CI, which is the one responsible for rolling out the cluster.
That cluster definition file contains some security sensitive data, like GitHub client id and secret, used to authenticate users in the UI dashboards (Grafana, Prometheus), or the OpsGenie token, used to route alerts to our on-call alerting service.
From now on, that security sensitive data will be stored encrypted via AWS KMS, and our CI will decrypt it before using it. This will grant our customers and us an extra level of security.
Next to encrypting those secrets, we’ve also rotated them, generating new tokens and keys.