Upgraded Vertical Pod Autoscaler to version 1.0.0
We uncovered a bug in our VPA deployments were the VPA was no longer updating its recommendations. As effect the VPA was no longer updating the deployments it was managing.
More …Upgraded Teleport to version 14.0.1
We’ve upgraded all Teleport clusters from version 13.3.8 to 14.0.1. Teleport is a tool we mostly use internally to provide secure and auditted access to (EC2) instances, Kubernetes clusters and several dashboards. The nodes will gradually be upgraded to the new version when new instances are launched.
Upgraded cluster add-ons
Update 2023-10-10: Upgrades have been applied on all clusters.
More …Removed Calico as NetworkPolicies engine - this functionality has merged with the AWS VPC CNI
Historically we’ve been using Calico as controller to provide NetworkPolicies support. This was offered as an optional feature only, considering the resource (and thus possible cost) impact of running this component. As announced in our K8s 1.27 upgrade post, the latest version of the AWS VPC CNI, responsible for providing cluster networking, now has native support for NetworkPolicies built-in.
Support for GPU node pools on AWS (EKS)
We’re adding support for GPU node pools in EKS. GPU nodes are great for compute-intensive workloads such as graphics and visualization workloads, or machine-learning processes. AWS uses the NVIDIA device plugin to make the GPU capacity of a node available to Kubernetes workloads.
More …Upgrading K8s clusters to v1.27
Update 2023-09-28: All clusters have been upgraded to v1.27.
New experimental component Karpenter for cost-effective node scaling
Karpenter is a new big feature that we offer in our AWS reference solution
More …Upgraded Teleport to version 13.3.8
We’ve upgraded all Teleport clusters from version 13.3.0 to 13.3.8. Teleport is a tool we mostly use internally to provide secure and auditted access to (EC2) instances, Kubernetes clusters and several dashboards. The nodes will gradually be upgraded to the new version when new instances are launched.
Upgraded Teleport to version 13.3.0
We’ve upgraded all Teleport clusters from version 13.0.3 to13.3.0. Teleport is a tool we mostly use internally to provide secure and auditted access to (EC2) instances, Kubernetes clusters and several dashboards. The nodes will gradually be upgraded to the new version when new instances are launched.
Disable kubernetes-dashboard by default
In an effort to further reduce the footprint of the reference solution we are no longer going to deploy the kubernetes-dashboard* by default. The Skyscrapers team is using k9s as a tool to manage our clusters and we think its a worhty replacement. This tool can just run in your terminal and doesn’t require any deployments on the K8s side. If you have this workload enabled today we will reach out to you to check if you are using this and take action based on your input.
More …Upgraded cluster add-ons
Update 2023-07-18: These updates have been rolled out to all environments.
More …Upgraded cluster add-ons
Update 2023-07-18: These updates have been rolled out to all environments.
More …Dedicated system node pool + reduced system component footprint
In order to improve our services we changed the way the Kubernetes nodepools are structured. Previously there was a default nodepool that had a mix of both Kubernetes add-ons and application deployments. This made things more complex than it needed to be. Therefore we created a dedicated system nodepool where all add-ons are scheduled on. During this change we also took a closer look at the requested resources for all add-ons and made adjustments where needed. For most of our customer environment we’ve been able to reduce the cluster size with at least 1 equivalent node. A handful are break-even for now, but we have further optimizations planned as follow-ups.
Upgrading K8s clusters to v1.26
We are rolling out AKS and EKS v1.26. This brings our supported AKS platforms to v1.26.3 and EKS to v1.26(.5). Please make sure to update to our recommended client versions matching this upgrade
Vault upgraded to 1.13.3
All Vault setups have been updated to the latest version 1.13.3. Please refer to the upstream changelogs to see what’s changed:
Upgrading OpenSearch clusters to 2.5
We’re starting to gradually upgrade all our managed OpenSearch clusters to the latest supported version in Amazon OpenSearch Service (V2.5).
More …Upgraded Teleport to version 13.0.3
We’ve upgraded all Teleport clusters from version 12.2.1 to 13.0.3. Teleport is a tool we mostly use internally to provide secure and auditted access to (EC2) instances, Kubernetes clusters and several dashboards. The nodes will gradually be upgraded to the new version when new instances are launched.
Post-mortem - A word on Pods stuck with ContainerCreating status problems
During the past days we’ve had multiple customer reports that launching new Pods got stuck with the ContainerCreating status, mostly affecting CronJobs. Even if you were not directly affected, you might have noticed the messages in your channels that we’ve been rolling out a lot of node updates without prior notice. This small post mortem will explain the reason, what steps we took to mitigate, and what we should have done better.
Upgraded fluent-bit and moving to built-in loki output
We have upgraded Fluent Bit form v2.0.1 to v2.1.3 and this is gradually rolling out to all our managed clusters.
More …Upgraded cluster add-ons for monitoring & logging
As part of our regular upgrade cycle, the following Kubernetes cluster components have been updated and our gradually rolling out to all our managed clusters.
More …