While we’re migrating customers to the new GitHub maintained gha-runner-scale-set controller, we’ve been noticing several reliability issues with this version. Problems like jobs not being able to schedule or runners being deleted early mid-run. We’ve rolled out a couple of changes to improve the reliability in these situations.
We’ve upgraded all Teleport clusters to 15.2.2. Teleport is a tool we mostly use internally to provide secure and auditted access to (EC2) instances, Kubernetes clusters and several dashboards. The nodes will gradually be upgraded to the new version when new instances are launched. You can find more information on this release in the Teleport changelog.
Update 2024-04-22: These changes have been rolled out to all clusters.
More …Update 2024-04-04: These changes have been rolled out to all clusters.
More …Update 2024-04-04: All clusters have been upgraded to v1.29.
Update 2024-03-21: This change is applied on all clusters.
More …As of today we are supporting the new (officially by GitHub supported) deployment method of GitHub Actions runners, the gha-runner-scale-set-controller. This new controller is a more efficient and scalable way to deploy self-hosted GitHub runners (controlled by a new gha-runner-scale-set) on Kubernetes. Next to improved stability and ongoing development, this new controller adds autoscaling of the runner pool based on the number of pending jobs, resulting in a more scalable and cost-effective solution.
Update 2024-03-07: Upgrades have been applied on all clusters.
More …We’ve upgraded all Teleport clusters from version 14.0.1 to 15.0.1. Teleport is a tool we mostly use internally to provide secure and auditted access to (EC2) instances, Kubernetes clusters and several dashboards. The nodes will gradually be upgraded to the new version when new instances are launched.
As of now as an experimental feature, we added support for Thanos to enable multi-cluster Prometheus monitoring. This enables you to store your Prometheus metrics in a central place, which can be used to query, visualize metrics and write alerts based on data of multiple environments.
More …We have stopped building our own custom EKS AMI. As of now we directly rely on the upstream, AWS-provided image for EKS.
More …Update 2024-01-25: All changes have been rolled out.
More …In September last year we announced the addition of Karpenter as experimental feature. Since then we have been making improvements to our implementation and gradually used some pilots internally and with some customers. We are happy to announce that Karpenter is now deployed by default (via AWS Fargate) on all our EKS clusters and we’ve migrated the system NodePool to use it instead of the standard Cluster Autoscaler.
All Vault setups have been updated to the latest version 1.15.4. Please refer to the upstream changelogs to see what’s changed:
Update 2023-09-28: All clusters have been upgraded to v1.28.
After internal reflection and based on customer feedback, we’re disabling customer Slack notifications for infra level alerts by default. These are alerts which Skyscrapers is responsible for follow-up, and caused confusion with customers whether they needed to take action or not.
As part of our regular upgrade cycle, the following Kubernetes cluster components have been updated and our gradually rolling out to all our managed clusters.
More …We uncovered a bug in our VPA deployments were the VPA was no longer updating its recommendations. As effect the VPA was no longer updating the deployments it was managing.
More …We’ve upgraded all Teleport clusters from version 13.3.8 to 14.0.1. Teleport is a tool we mostly use internally to provide secure and auditted access to (EC2) instances, Kubernetes clusters and several dashboards. The nodes will gradually be upgraded to the new version when new instances are launched.
Update 2023-10-10: Upgrades have been applied on all clusters.
More …