Last night Kubernetes sent out a Security Advisory regarding multiple vulnerabilities in the Nginx Ingress Controller, including the critical CVE-2025-1974. If you’re an AWS account owner, you likely also received an email from AWS warning against these vulnerabilities. We are happy to announce we have taken immediate action and upgraded all clusters to the latest version of the Nginx Ingress Controller, mitigating these vulnerabilities.
More …
We are excited to announce that we have added two new controllers to our Flux setup: the Flux Image reflector and the Flux Image automation controller. These work together to update a Git repository when new container images are available.
More …
Update 2025-03-24: These changes have been rolled out to all clusters.
More …
In this post, we’re excited to announce our plans to shift the management of Kubernetes (K8s) add-ons from our existing OpenTofu-based approach (using Terragrunt and Concourse CI) to Flux. This change is part of our broader effort to simplify our platform managemt, improve reliability, and enhance visibility for both our internal teams and our customers. Below, we’ll cover what’s happening, why we’re making these changes, how you’ll benefit, and what to expect during and after the migration.
More …
Since our previous announcement regarding the new DockerHub rate limits starting March 1st, we have updated and made sure that all system workloads managed by Skyscrapers, like ingress controllers, monitoring, istio etc., will not be affected by these new rate limits. We ensured all these components are using different mirrors. If you haven’t taken action regarding your own application worklaods yet, we encourage you to do follow the steps outlined in our documentation (updated) and check your specific GitHub issue with more details. Our colleagues are available to guide you through this process.
Starting March 1, 2025 DockerHub will limit unauthenticated image pulls further from 100 per 6-hours per IP address to 10 per hour per IP address: https://docs.docker.com/docker-hub/usage/. This is a reduction of 40%! Please read this post carefully to understand what it is about and to determine whether you need to take action.
More …
Concourse CI has been upgraded to version 7.12.1. This update brings some small improvements. Relevant Concourse CI changelog: https://github.com/concourse/concourse/releases/tag/v7.12.1
We are rolling out EKS v1.32. Please make sure to update to our recommended client versions matching this upgrade. This upgrade includes an important change in how PersistentVolumeClaims (PVCs) are handled by StatefulSets. When a StatefulSet is deleted, the PVCs created by the StatefulSet will now be automatically removed too. This is a change in behavior from previous versions of Kubernetes.
More …
We are excited to announce that we have completely rewritten our Dex onboarding documentation. This documentation will guide you through the steps required to configure your Identity Provider (like Google, Microsoft or Github) so we can integrate it as authentication mechanism in your platform (eg. for accessing monitoring dashboards). In addition we have added support to inject extra files into the Dex configuration, which for example allows using a Google service account to limit access scopes to specific groups instead of a whole domain.
The following updates have been rolled out to all non-production clusters. Notable updates include a major release for Prometheus, bringing a new UI. As usual there’s also improvements across various other add-ons, ensuring enhanced performance and security. Finally we’d also like to remind you again for Actions to take regarding the Grafana AngularJS deprecation!
More …
We are excited to announce that we have added Kubecost as an optional feature to our Platform. Kubecost is a tool that helps you to manage your Kubernetes costs by providing visibility into your AWS and Kubernetes resource usage and costs. At the moment we are shaping our FinOps offering and Kubecost is part of that. If you are interested in trying out Kubecost, or have questions regarding our FinOps offering please reach out to us through your Skyscrapers lead.
More …
We are rolling out EKS v1.31. Please make sure to update to our recommended client versions matching this upgrade.
More …
As mentioned in our previous post, we removed Teleport from our environments (apart from the ones that were not ready to move away from it). For customers that had nothing else running on the tools environment we also cleaned up the networking. This will also save some costs for those customers.
Concourse CI has been upgraded to version 7.12.0. This update brings some small new features and improvements. Relevant Concourse CI changelog: https://github.com/concourse/concourse/releases/tag/v7.12.0
In order to streamline our colleague’s experiences, we are excited to announce that we have moved to Tailscale for secure remote access to our managed environments. Tailscale is a Zero Trust network that provides a lightweight, seamless yet secure experience for connecting to all the different networks and services we manage. This replaces the use of Teleport and OpenVPN for internal Skyscrapers’ use. Next up, we plan to evaluate replacement options for our customers’ VPN offering in the coming months.
More …
Over the past months, we’ve gathered customer feedback and monitored Loki’s performance within our Kubernetes clusters. This process has highlighted recurring performance challenges. To address these, we are rolling out optimizations designed to enhance stability and performance. This post outlines the changes we’re making and the reasoning behind them. If you have any feedback or questions please don’t hesitate to reach out to us.
More …
The following updates have been rolled out to all clusters. Notable updates include improvements across various add-ons, ensuring enhanced performance and security. This is also a reminder to for Actions to take regarding the Grafana AngularJS deprecation!
More …
We are excited to announce a new version of our Security Policy to reflect the latest changes in our organization, continously improving our security practices. There are some significant changes, with emphasis on the introduction of the “Data Classification and Handling” and “Asset Management” policies. Furthermore, following these new policies, we have made this new version available via our public documentation website.
We’ve upgraded all Teleport clusters to 15.4.21. Teleport is a tool we mostly use internally to provide secure and auditted access to (EC2) instances, Kubernetes clusters and several dashboards. The nodes will gradually be upgraded to the new version when new instances are launched. You can find more information on this release in the Teleport changelog.
More …
Feature: Automated Wiki Creation for Extra Notes
More …
