Upgraded Teleport to version 13.3.8

We’ve upgraded all Teleport clusters from version 13.3.0 to 13.3.8. Teleport is a tool we mostly use internally to provide secure and auditted access to (EC2) instances, Kubernetes clusters and several dashboards. The nodes will gradually be upgraded to the new version when new instances are launched.

More …

Upgraded Teleport to version 13.3.0

We’ve upgraded all Teleport clusters from version 13.0.3 to13.3.0. Teleport is a tool we mostly use internally to provide secure and auditted access to (EC2) instances, Kubernetes clusters and several dashboards. The nodes will gradually be upgraded to the new version when new instances are launched.

More …

Disable kubernetes-dashboard by default

In an effort to further reduce the footprint of the reference solution we are no longer going to deploy the kubernetes-dashboard* by default. The Skyscrapers team is using k9s as a tool to manage our clusters and we think its a worhty replacement. This tool can just run in your terminal and doesn’t require any deployments on the K8s side. If you have this workload enabled today we will reach out to you to check if you are using this and take action based on your input.

More …

Dedicated system node pool + reduced system component footprint

In order to improve our services we changed the way the Kubernetes nodepools are structured. Previously there was a default nodepool that had a mix of both Kubernetes add-ons and application deployments. This made things more complex than it needed to be. Therefore we created a dedicated system nodepool where all add-ons are scheduled on. During this change we also took a closer look at the requested resources for all add-ons and made adjustments where needed. For most of our customer environment we’ve been able to reduce the cluster size with at least 1 equivalent node. A handful are break-even for now, but we have further optimizations planned as follow-ups.

More …

Upgraded Teleport to version 13.0.3

We’ve upgraded all Teleport clusters from version 12.2.1 to 13.0.3. Teleport is a tool we mostly use internally to provide secure and auditted access to (EC2) instances, Kubernetes clusters and several dashboards. The nodes will gradually be upgraded to the new version when new instances are launched.

More …

Post-mortem - A word on Pods stuck with ContainerCreating status problems

During the past days we’ve had multiple customer reports that launching new Pods got stuck with the ContainerCreating status, mostly affecting CronJobs. Even if you were not directly affected, you might have noticed the messages in your channels that we’ve been rolling out a lot of node updates without prior notice. This small post mortem will explain the reason, what steps we took to mitigate, and what we should have done better.

More …

Istio upgraded to version 1.16.4

We have upgraded Istio on all clusters that use it. The version was upgraded from 1.16.3 to 1.16.4. These release mostly focuses on many security fixes, ranging from moderate to high. You can check the full release notes here. We’ve also upgraded Kiali to the latest version, 1.66.0 (changelog).

More …

Upgraded Teleport to version 12.2.1

We’ve upgraded all Teleport clusters from version to 12.2.1. Teleport is a tool we mostly use internally to provide secure and auditted access to (EC2) instances, Kubernetes clusters and several dashboards. The nodes will gradually be upgraded to the new version when new instances are launched.

More …

Defaulting to capacity-optimized for Spot nodepools

Most of our EKS clusters leverage Spot instances as a cost-efficient way to provide compute nodes. Historically we’ve been defaulting to the “lowest price” allocation strategy to maximize possible cost savings. However this can lead to quite some more interuptions than we want to tolerate and often a big inbalance between AZ spread if price pressure increases. Therefore we’ve updated our default to use a “capacity optimized” strategy instead for increased stability with (possibly) a marginal higher cost.

More …

Upgraded EKS cluster add-ons

As part of our regular upgrade cycle, the following EKS cluster components have been updated. We’ve already rolled these out to all non-production clusters. Production upgrades are scheduled to happen in the next few days during business hours. As usual, no workload interuptions are expected.

More …