Defaulting to capacity-optimized for Spot nodepools

Most of our EKS clusters leverage Spot instances as a cost-efficient way to provide compute nodes. Historically we’ve been defaulting to the “lowest price” allocation strategy to maximize possible cost savings. However this can lead to quite some more interuptions than we want to tolerate and often a big inbalance between AZ spread if price pressure increases. Therefore we’ve updated our default to use a “capacity optimized” strategy instead for increased stability with (possibly) a marginal higher cost.

[Action required] Upgrades starting for K8s clusters to 1.25

Update 2023-05-04: All clusters have been upgraded to v1.25.

Upgraded EKS cluster add-ons

As part of our regular upgrade cycle, the following EKS cluster components have been updated. We’ve already rolled these out to all non-production clusters. Production upgrades are scheduled to happen in the next few days during business hours. As usual, no workload interuptions are expected.

Deletion prevention on OpenSearch S3 backups

We have made an update to the S3 buckets that are used to take external backups from our managed OpenSearch clusters.

Istio upgraded to version 1.16.3

We have upgraded Istio on all clusters that use it. The version was upgraded from 1.16.1 to 1.16.3. These releases contain bug fixes to improve robustness and security fixes in the underlying Go packages. You can check the full release notes here. We’ve also upgraded Kiali to the latest version, 1.64.0 (changelog).

Vault upgraded to 1.13.0

All Vault setups have been updated to the latest version 1.13.0. Please refer to the upstream changelogs to see what’s changed:

INFO - Empty response for external.metrics.k8s.io/v1beta1 errors with kubectl and helm

We noticed that some of our customers are having questions about the error messages they’re getting when deploying their workload using helm and/or using kubectl:

Post Mortem - Loki log loss

After deploying our Grafana Loki refactor, several issues started popping up, cascading to a loss of logs for a maximum of 12 hours on 22/02 or 23/02. All environments using Loki as main logging provider were affected. Environments logging to other systems like CloudWatch Logs and ElasticSearch were not affected.

Upgraded Teleport to version 12.0.2

We’ve upgraded all Teleport clusters from version 11.1.1 to 12.0.2. Teleport is a tool we mostly use internally to provide secure and auditted access to (EC2) instances, Kubernetes clusters and several dashboards. The nodes will gradually be upgraded to the new version when new instances are launched.

Upgraded cluster add-ons

Update 2023-02-28: These updates have been rolled out to all environments.

Improving Loki performance & scalability

Update 2023-02-15: These changes have now been rolled out everywhere.

Node Termination Handler Slack notifications disabled by default (AWS EKS)

Based on customer feedback, we’ve now disabled posting AWS Node Termination Handler (NTH) notifications to Slack by default. The NTH is responsible for reacting to node state changes, by properly draining a node for example when a Spot Instance interuption is received.

Upgraded K8s clusters to 1.24

Update 2023-02-10: All clusters have been upgraded to v1.24.

Upgraded Teleport to version 11.1.4 for security fixes

We’ve upgraded all Teleport clusters from version 11.1.2 to 11.1.4. This upgrade was done on all Teleport servers to fix potential vulnerabilties:

Upgraded cluster add-ons

As part of our regular upgrade cycle, the following Kubernetes cluster components have been updated. These changes will be rolling out to all clusters soon.

Istio upgraded to version 1.16.1

We have upgraded Istio on all clusters that use it. The version was upgraded from 1.15.2 to 1.16.1. What’s new:

Vault upgraded to 1.12.2

All Vault setups have been updated from 1.12.0 to the latest version 1.12.2. This release brings small improvements and bug fixes. Please refer to the upstream changelogs to see what’s changed:

New feature: Use External-DNS for managing custom DNS records

Whenever you deploy an Ingress resource, external-dns is responsible for creating the matching DNS record. We have now enabled the “CRD” feature of this component, which allows you to manage any DNS records of your choice through external-dns.

Upgraded Teleport to version 11.1.1 for security fix

We’ve upgraded all Teleport clusters from version 11.0.3 to 11.1.1. This upgrade was done on all Teleport servers to fix a potential vulnerabilty:

Fixed issue where an attacker with physical access to user’s computer and raw access to the filesystem could potentially recover the seed QR code.

Upgraded Teleport to version 11.0.3

We’ve upgraded all Teleport clusters from version 10.1.4 to 11.0.3. Teleport is a tool we mostly use internally to provide secure and auditted access to (EC2) instances, Kubernetes clusters and several dashboards. The nodes will gradually be upgraded to the new version when new instances are launched.