We have started rolling out AKS and EKS 1.19. This brings AKS on Kubernetes v1.19.7 and EKS on Kubernetes v1.19.6-eks-49a6c0.

In the process of upgrading EKS the following components have also been upgraded:

• KubeProxy from 1.18.9 to 1.19.6
• CoreDNS from 1.7.0 to 1.8.0
• Cluster Autoscaler from 1.18.3 to 1.19.1

Upon writing upgrades of non-production clusters have already been rolled out. Production clusters will follow in the next week(s) after some extra validation, so you can expect to be contacted by an engineer to determine an upgrade window.

Important changes between K8s 1.18 and 1.19

For more detailed info on what’s new and changed, please make sure to check the Kubernetes 1.19 release announcement and full Kubernetes 1.19.x changelog.

Here’s a small list of some major themes:

• Ingress graduates to General Availability

  In short, this means you should start moving all your Ingresses to the networking.k8s.io/v1 apiVersion.

  You can find the Ingress documentation here: https://kubernetes.io/docs/concepts/services-networking/ingress/.

  Important: We don’t setup an IngressClass for our default nginx (and nginx-internal) Ingress controllers yet and thus still rely on the kubernetes.io/ingress.class annotation. We plan to implement this in a future update.

• Deprecation warnings

  Requests to a deprecated API are returned with a warning containing a target removal release and any replacement API.

• CLI support for debugging workloads

  You can use kubectl alpha debug as a convenient command to debug workloads:

  • Insert a debug container in clusters that don’t have ephemeral containers enabled
  • Modify a crashing container for easier debugging by changing its image, for example to busybox, or its command, for example, to sleep 1d so you have time to kubectl exec
  • Inspect configuration files on a node’s host filesystem

  For more info on this enhancement, check the debug running pods documentation.

• EndpointSlices are enabled by default

  EndpointSlices are an exciting new API that provides a scalable and extensible alternative to the Endpoints API. EndpointSlices track IP addresses, ports, readiness, and topology information for Pods backing a Service.

  For more information, check the EndpointSlices docs

• Immutable Secrets and ConfigMaps (beta)

  Secret and ConfigMap volumes can be marked as immutable, which significantly reduces load on the API server if there are many Secret and ConfigMap volumes in the cluster. See the ConfigMap and Secret documentation for more information.

• seccomp graduates to General Availability

  The support for the seccomp.security.alpha.kubernetes.io/pod and container.seccomp.security.alpha.kubernetes.io/... annotations are now deprecated and have been replaced by a first class seccompProfile field for Pod and Container securityContext objects.

  For more information, check the seccomp docs.

• Pod Topology Spread has reached stable status

  You can use topology spread constraints to control how pods are spread across your cluster among failure-domains such as regions, zones, nodes, and other user-defined topology domains. This can help to achieve high availability, as well as efficient resource utilization.

  For more information, check the Pod Topology Spread Constraints docs.

And there’s much more to be found in the full Kubernetes 1.19.x changelog!

AWS EKS specific changes

• You’re no longer required to provide a security context for non-root containers that need to access the web identity token file used with IRSA.
• Amazon EKS optimized Amazon Linux 2 AMIs include the Linux kernel version 5.4. This brings important patches regarding conntrack race conditions that can be encountered and eg. reflected in failing DNS resolution. For more info, check out https://github.com/awslabs/amazon-eks-ami/issues/357.
• The ExtendedResourceToleration admission controller is enabled. This admission controller automatically adds tolerations for taints to pods requesting extended resources, such as GPUs, so you don’t have to manually add the tolerations. For more information, see ExtendedResourceToleration in the Kubernetes documentation.

Actions to take

You should start moving your Ingresses to the new GA apiVersion of networking.k8s.io/v1. It’s important to note that apiVersion extensions/v1beta1 for Ingress has been deprecated for a while and will be removed in K8s 1.22!

If necessary for your workloads, a Skyscrapers engineer will get in contact in the coming days to plan an upgrade window for production.