Our ECS monitoring solution now supports monitoring Elasticsearch clusters using Elasticsearch Exporter, Prometheus and AlertManager, so we can get notified via slack (critical/warnings) and via OpsGenie (critical) for any issues with ES. This is similar functionality which was already available for customers running on the Kubernetes platform.
We’re in the process of removing our kibana deployment from all the Staging clusters and replacing it with the AWS provided kibana setup that comes with the AWS ElasticSearch service. Production clusters will follow.
More …We’ve updated kube2iam to the latest version (0.10.7) on all clusters.
During the comming days, we’ll roll out Concourse version 5.0.1 to all our setups.
More …During the following days we’re going to rollout some changes in how Kubernetes monitoring notifications are delivered. From now on, all notifications comming from the production k8s monitoring system will be shown in our shared slack channel, that is the channel we share with each of our customers. The current notification channels will still work as until now. Here’s an overview of how notifications will work:
More …We are in the process of upgrading our managed Kubernetes clusters from v1.11.6 to v1.11.9.
We’ve made the AWS Service Operator available for deployment on our managed Kubernetes clusters.
More …Update (18-03-2019): We found out there were enough default alerts covering all cases of cronjob failures. The following alerts are covering different failure cases accordingly:
More …We are in the process of upgrading our staging Kubernetes clusters components to the latest stable releases. Production clusters will follow in 1 to 2 weeks (to be announced) after we have confirmed there are no issues with our customer’s workloads.
More …We have updated the format of the monitoring Slack notifications. You might have already noticed that the monitoring messages in your Slack channels now contain more useful information and are more structured. We’ve already started rolling out the changes in staging clusters and we’ll start rolling them in production clusters during this week.
More …We have updated the clusters to have support for mongodb monitoring, alerts and dashboards. If you have a mongodb cluster you will see that there is now a mongodb dashboard in Grafana and that we added specific alert rules for mongodb in prometheus.
We’ve upgraded all the k8s cluster with a new etcd backup implementation. The old backup solution was relying on daily snapshots taken from a service running in the master nodes.
More …Update: Added other affected services next to Kubernetes.
More …We’re rolling out a major update for our Kubernetes etcd clusters to now use encrypted EBS volumes for storing all of the Kubernetes state.
More …We’re updating our Kubernetes staging clusters with CoreDNS, the new dns server that replaces KubeDNS. After an in-depth analysis and tests we’ve verified that the performance and the stability between the two solutions are almost identical. Here you can find more details on why we decided to move to CoreDNS.
More …A Vault upgrade for our setups was long overdue. We’ve upgraded our Vault installation tools from version 0.9.3 to 1.0.1, which is the latest Vault version available at the moment. As Vault is set up as HA, the downtime of the upgrade will be minimal, normally between half a second and a couple of seconds, which is the time the fail-over takes. The upgrade procedure to achieve that minimal downtime is the following:
More …Update: Changed Kubernetes update from 1.10.12 to 1.11.6
More …Update 2 (2018-12-03): Since our last update, the people at Kubernetes updated their documentation to add an important fix in the 1.10.11 changelog:
More …We’ve deployed on all our ECS managed staging clusters a prometheus monitoring system.
More …Following our efforts to improve the overall stability of our Kubernetes clusters, we’ve now set resource reservations for kubelet and other system processes. This will ensure that these critical processes always have enough CPU and memory available to function properly, regardless of what the actual cluster workloads are.
More …