Previously we used Project Calico as networking plugin (CNI) on all our Kubernetes clusters (KOPS & EKS). However with our move to EKS as base for our reference solution, we will be defaulting to AWS’ own VPC CNI.
Using a custom CNI with EKS, like Calico, comes with some restrictions. More concretely this blocking us from using more advanced features relying on communication between the control plane and Pods running on the K8s workers (eg. kubectl proxy, Admission Webhooks, etc.) because these run on different networks.
Using the VPC CNI however comes also with a downside: currently the maximum amount of pods per instance is limited per EC2 instance type, since Pod IPs are directly provided via instance ENIs from the VPC pool. An easy overview of max Pods per instance type is available in the EKS AMI repository.
We feel this is only a minor downside for our customer’s setups, compared to the benefits, and can be easily tackled through proper capacity planning. We’re also eagerly watching progress on the next generation of the AWS VPC CNI plugin.
All new EKS clusters will be setup in new, large subnets using the VPC CNI. Calico is still kept as an option for new clusters.
Calico can still be used for network policy enforcement on top of the VPC CNI. This can be enabled optionally upon request.