Skyscrapers Changelog

This changelog lists all updates, improvements and new features our Engineering team develops for our Reference Solutions. These are rolled out automatically to all SRE-as-a-Service customers.

[Action required] Update external-dns behavior for private Route53 zones

10 May 2023 • Kubernetes
 kubernetes  k8s  external-dns  dns  route53  aws  eks  action-required

We’re updating the behavior of external-dns when managing records in private Route53 zones.

Note that this new behavior is still in the works and will be rolled out after 2023-06-09. Since it brings breaking changes for domains in private zones, we’re publishing this changelog in advance to give customers reaction time to update their manifests. See Actions to take below.

Also note that this only applies to EKS clusters, not AKS.

Current behavior

Up until now, we’ve been using a single external-dns deployment that manages all DNS records defined in various resources in the cluster, regardless of whether they are set up in public or private Route53 zones. In other words, this single external-dns deployment didn’t make any distinction between public and private Route53 zones. When it found a record to add or update, it considered any matching public and private zone for it. So if there was both a public and private zone that matched a specific DNS domain, it added a record to both zones.

This behavior is of course not always desired. A developer might want to create an internal ingress with a private-only DNS domain, or you might need two separate ingresses, a public and an internal one, with the same DNS domain pointing to different cluster endpoints, which would require separate records in both a public and a private Route53 zone, with a different value each. These scenarios were not possible with the current external-dns setup.

New behavior

To overcome this, we’re adding an optional second external-dns deployment, to be able to manage public and private DNS domains separately. To be able to make use of this new behavior, developers will now need to specify if a domain is intended to be private-only with an annotation (external-dns.alpha.kubernetes.io/zone-type). This annotation will control which external-dns deployment will take ownership of the DNS record and in which type of Route53 zone will be used. The annotation can be added to any of the external-dns configured sources (Service, Ingress, Gateway, VirtualService, DNSEndpoint).

The annotation can have the following values:

  • public (default): with an annotation value of public, or without the annotation, any domains extracted from the resource will be treated as public-only, and only public Route53 zones will be considered in external-dns. This is the default behavior.
  • private: only private Route53 zones will be considered, even if there’s matching public zones for the domains defined in the resource.
  • all: both public and private Route53 zones will be considered when creating or updating any DNS domain extracted from the given resource. If there’s both a public and private zone matching a domain, a record will be created in both. This is equivalent to the current behavior.

Example

When the new external-dns behavior is rolled out to your clusters, the following Ingress will trigger external-dns to add or update the DNS record foo.internal.staging.skyscrape.rs to any Route53 private zones that match (if any). But won’t add the record to any public zones. Up until now the record would have been added to both public and private zones.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    external-dns.alpha.kubernetes.io/zone-type: private
    kubernetes.io/tls-acme: "true"
  name: foo
  namespace: default
spec:
  ingressClassName: nginx
  rules:
    - host: foo.internal.staging.skyscrape.rs
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: foo
                port:
                  name: http
  tls:
    - secretName: foo-staging-tls
      hosts:
        - foo.internal.staging.skyscrape.rs

Actions to take

This is a breaking change of the external-dns behavior for all those customers that use private Route53 zones. If you are not using any private zone you don’t have to take any further action.

If you have DNS domains in private Route53 zones, which are currently managed by external-dns, you’ll need to set the annotation mentioned above to the resources from which those domains are extracted. The annotation value will depend on the intent of those domains. Most likely they will need to be private only, in which case you should set external-dns.alpha.kubernetes.io/zone-type: private. On the other hand if you want to keep the current behavior of setting the same DNS record in both public and private zones, you should set external-dns.alpha.kubernetes.io/zone-type: all.

It is safe to already set the annotation, the current external-dns deployment will just ignore it.

Please make sure to update any affected resource with the necessary annotation value before the new behavior is rolled out. We will also post a GitHub issue in your repository if we have detected such use of private Route53 zones.

Related Posts