CVE-2021-25742 in ingress-nginx
A security issue was discovered in ingress-nginx where a user that can create or update ingress objects can use the custom snippets feature to obtain all secrets in the cluster.
More …Standardizing on Fluent Bit and Loki updates
For a while we’ve offered Grafana Loki as default logging solution. For shipping logs to Loki we were using the included Promtail. However, more recently, we’ve also supported other logging solutions, like Elasticsearch and Logz.io for customers with more advanced needs. To facilitate this we use the Fluent Bit log processor.
More …Upgraded Teleport to version 8.0.7
We’ve upgraded all Teleport clusters from version 8.0.0 to 8.0.7. This is a minor release, coming with mostly bug and security fixes.
More …VPA enabled for Vault
We have added Vault to the list of autoscaling rules we deploy by default. By doing this we can allow the VPA to set the optimal resource requests and limits within the boundaries that we provide.
More …Module updated for AWS OpenSearch and started upgrades
AWS ElasticSearch Service has been rebranded to AWS OpenSearch for some time now, and thus we’ve decided to rename our Terraform module for managing this service accordingly.
More …Add support for mixed node pools in EKS
We have added support for mixed node pools on AWS.
More …Monitoring for Grafana Loki in case of discarded logs
During a routine monitoring review, we’ve noticed some Promtail pods were using significantly more CPU than the generic request. This pointed us to two issues:
More …Critical CVE-2021-44228 in Log4j - Check your application workloads!
Update 2021-12-16: The patched Log4j 2.15.0 was found to still have a possible vulnerability. We’ve updated the action below to update to (at least) version 2.16.0.
Add support for AWS Secrets Manager in EKS
We’ve added support for using secrets from AWS Secrets Manager in EKS clusters. This support is optional and disabled by default.
More …Upgraded cluster add-ons
As part of our regular upgrade cycle, the following Kubernetes cluster components have been updated. We’ve already rolled these out to all clusters.
More …Upgraded Teleport to version 8.0.0
We’ve upgraded all Teleport clusters to version 8.0.0. This is a major release, coming with many new features:
More …Istio upgraded to version 1.12.0
We have upgraded Istio on all clusters that use it. The version was upgraded from 1.11.2 to 1.12.0.
More …Introducing alerts for Fluent Bit errors
Considering we’re moving more and more log processing to Fluent Bit, it’s important to get notified when logs are not making it to the storage solutions (“outputs”) like Elasticsearch, Logz.io and S3.
More …Concourse upgraded to v7.5.0
We have upgraded our Concourse setups to the latest version 7.5.0.
More …A note on Let's Encrypt chain issues due to DST Root CA X3 expiry
Let’s Encrypt certificates are (usually) cross-signed with the DST Root CA X3 root certificate, however this root certificate expired on September 30th 2021.
More …Making our Terraform helper script public
Every piece of infrastructure we create is managed via Terraform. This is to ensure that everything we deploy is repeatable, follows best practices and is fully tracked.
More …Grafana security patch following High Severity CVE-2021-39226
On the 5th of October a notice for CVE-2021-39226 with a severity of high went out, impacting the Grafana deployments.
More …RDS snapshots cross-account replication available
In some cases, a disaster recovery plan might require RDS snapshots to be replicated / copied over to a different AWS account and region. We can now set up this replication process for the managed RDS instances of our customers. Note that this will work in conjunction of the normal automated daily RDS snapshots that AWS already performs.
More …Upgraded cluster add-ons
As part of our regular upgrade cycle, the following Kubernetes cluster components have been updated. We’ve already rolled these out to all clusters.
More …Guaranteed QoS for all critical system and infrastructure Pods
We’ve seen in multiple occasions that, due to resource starvation in a cluster, the kubelet starts evicting critical infrastructure Pods. This can lead to important downtimes and disruptions in multiple occasions.
More …