As part of our regular upgrade cycle, the following Kubernetes cluster components have been updated. We’ve already rolled these out to all clusters.

Some highlights:

• Max pod density per node increased to 110!
• Major releases for Grafana
• IngressClasses for nginx and nginx-internal controllers. See Actions to take
• Kube-state-metrics v2, which removed deprecated metrics. See Actions to take

Click each link to go to the full upstream changelog.

AKS Core functionality

• aad-pod-identity 1.6.3 -> 1.8.2

EKS Core functionality

• aws_efs_csi_driver 1.2.1 -> 1.3.3
  • Support for cross-account and cross-AZ mount
  • Support for ARM and image is multi-arch
• aws-node-termination-handler 1.13.0 -> 1.13.2
• aws-vpc-cni 1.7.10 -> 1.9.0
  • This version of the CNI supports a much higher Pod density per node by leveraging a new VPC feature to assign IP prefixes to EC2 instances. We have enabled this feature by default and increased the max Pod setting to 110 per node (configurable) for each instance type. To compare, this were the previous limits per instance type: https://github.com/awslabs/amazon-eks-ami/blob/master/files/eni-max-pods.txt.
  • Reduced number of EC2 calls to create & attach ENIs
• metrics-server 0.4.4 -> 0.5.0
  • Big improvements in autoscaling quality and reliability
  • Increased metrics scraping resolution to 15s

Shared Addons (AKS & EKS)

• cert-manager 1.2.0 -> 1.4.3
• dex 2.28.1 -> 2.30.0
• external-dns 0.8.0 -> 0.9.0
• fluent-bit 1.7.7 -> 1.8.3
  • New multiline core feature with built-in support for containerized environments based on Docker or CRI plus stacktrace detectors for different languages such as Go, Ruby, Java, etc.
• grafana 7.5.7 -> 8.1.1
  • Library panels: users can build panels to be used in multiple dashboard
  • Built-in Prometheus metrics browser
  • New visualizations: bar chart, state timeline, status history, histogram
  • Many updated visualizations
  • Improvements to the (Loki) log browser
  • Performance improvements
• ingress-nginx 0.46.0 -> 0.48.1
  • Our default Ingress controllers now come with an IngressClass resource. The networking.k8s.io/v1 spec allows for setting and ingressClassName field as replacement for the deprecated kubernetes.io/ingress.class annotation.
• kubernetes-dashboard 2.2.0 -> 2.3.1
• kube-prometheus-stack chart 15.4.6 -> 17.1.3
  • kube-state-metrics has been updated to v2.0, which removes several deprecated metrics. Make sure to update these metrics in your alerts or Grafana dashboards if you are still using them
• prometheus 2.27.1 -> 2.28.1
• prometheus-elasticsearch-exporter 1.1.0 -> 1.2.1
• prometheus-operator 0.48.1 -> 0.49.0
• velero 1.6.0 -> 1.6.2

Actions to take

Replaced metrics

Non-generic resource metrics for pods and nodes were marked deprecated and are now removed. Make sure to update your alerting rules and/or Grafana dashboards if you were using these metrics.

For Pod metrics: kube_pod_container_resource_requests and kube_pod_container_resource_limits are the replacements with resource labels representing the resource name and unit labels representing the resource unit.

• kube_pod_container_resource_requests_cpu_cores becomes kube_pod_container_resource_requests{resource="cpu", unit="core"}
• kube_pod_container_resource_limits_cpu_cores becomes kube_pod_container_resource_limits{resource="cpu", unit="core"}
• kube_pod_container_resource_requests_memory_bytes becomes kube_pod_container_resource_requests{resource="memory", unit="byte"}
• kube_pod_container_resource_limits_memory_bytes becomes kube_pod_container_resource_limits{resource="memory", unit="byte"}

For node metrics: kube_node_status_capacity and kube_node_status_allocatable are the replacements with resource labels representing the resource name and unit labels representing the resource unit.

• kube_node_status_capacity_pods becomes kube_node_status_capacity{resource="pod"}
• kube_node_status_capacity_cpu_cores becomes kube_node_status_capacity{resource="cpu", unit="core"}
• kube_node_status_capacity_memory_bytes becomes kube_node_status_capacity{resource="memory", unit="byte"}
• kube_node_status_allocatable_pods becomes kube_node_status_allocatable{resource="pod"}
• kube_node_status_allocatable_cpu_cores becomes kube_node_status_allocatable{resource="cpu", unit="core"}
• kube_node_status_allocatable_memory_bytes becomes kube_node_status_allocatable{resource="memory", unit="byte"}

Use IngressClass

If you haven’t already, you should move your Ingresses to the new GA apiVersion of networking.k8s.io/v1. It’s important to note that apiVersion extensions/v1beta1 for Ingress has been deprecated for a while and will be removed in Kubernetes 1.22! Check out the deprecated API migration guide for more details. The upstream Ingress page is also an interesting source.

The networking.k8s.io/v1 spec also comes with an ingressClassName field as replacement for the deprecated kubernetes.io/ingress.class annotation. By default, on our platform you can now set ingressClassName: nginx or ingressClassName: nginx-internal respectively.

For example, a simple Ingress resource would look like this:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/tls-acme: "true"
  name: marketing-website
  namespace: production
spec:
  ingressClassName: nginx
  rules:
    - host: www.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: my-service
                port:
                  name: http
  tls:
    - secretName: www-example-com-tls
      hosts:
        - www.example.com