Upgraded cluster components & increased Pod density. Actions to take!

As part of our regular upgrade cycle, the following Kubernetes cluster components have been updated. We’ve already rolled these out to all clusters.

Some highlights:

  • Max pod density per node increased to 110!
  • Major releases for Grafana
  • IngressClasses for nginx and nginx-internal controllers. See Actions to take
  • Kube-state-metrics v2, which removed deprecated metrics. See Actions to take

Click each link to go to the full upstream changelog.

AKS Core functionality

EKS Core functionality

Shared Addons (AKS & EKS)

Actions to take

Replaced metrics

Non-generic resource metrics for pods and nodes were marked deprecated and are now removed. Make sure to update your alerting rules and/or Grafana dashboards if you were using these metrics.

For Pod metrics: kube_pod_container_resource_requests and kube_pod_container_resource_limits are the replacements with resource labels representing the resource name and unit labels representing the resource unit.

  • kube_pod_container_resource_requests_cpu_cores becomes kube_pod_container_resource_requests{resource="cpu", unit="core"}
  • kube_pod_container_resource_limits_cpu_cores becomes kube_pod_container_resource_limits{resource="cpu", unit="core"}
  • kube_pod_container_resource_requests_memory_bytes becomes kube_pod_container_resource_requests{resource="memory", unit="byte"}
  • kube_pod_container_resource_limits_memory_bytes becomes kube_pod_container_resource_limits{resource="memory", unit="byte"}

For node metrics: kube_node_status_capacity and kube_node_status_allocatable are the replacements with resource labels representing the resource name and unit labels representing the resource unit.

  • kube_node_status_capacity_pods becomes kube_node_status_capacity{resource="pod"}
  • kube_node_status_capacity_cpu_cores becomes kube_node_status_capacity{resource="cpu", unit="core"}
  • kube_node_status_capacity_memory_bytes becomes kube_node_status_capacity{resource="memory", unit="byte"}
  • kube_node_status_allocatable_pods becomes kube_node_status_allocatable{resource="pod"}
  • kube_node_status_allocatable_cpu_cores becomes kube_node_status_allocatable{resource="cpu", unit="core"}
  • kube_node_status_allocatable_memory_bytes becomes kube_node_status_allocatable{resource="memory", unit="byte"}

Use IngressClass

If you haven’t already, you should move your Ingresses to the new GA apiVersion of networking.k8s.io/v1. It’s important to note that apiVersion extensions/v1beta1 for Ingress has been deprecated for a while and will be removed in Kubernetes 1.22! Check out the deprecated API migration guide for more details. The upstream Ingress page is also an interesting source.

The networking.k8s.io/v1 spec also comes with an ingressClassName field as replacement for the deprecated kubernetes.io/ingress.class annotation. By default, on our platform you can now set ingressClassName: nginx or ingressClassName: nginx-internal respectively.

For example, a simple Ingress resource would look like this:

apiVersion: networking.k8s.io/v1
kind: Ingress
    kubernetes.io/tls-acme: "true"
  name: marketing-website
  namespace: production
  ingressClassName: nginx
    - host: www.example.com
          - path: /
            pathType: Prefix
                name: my-service
                  name: http
    - secretName: www-example-com-tls
        - www.example.com