If a Kubernetes Service had no active Endpoints, for example when a deployment is scaled to 0, then requests to that Service were timing out. Instead it’s supposed to reject traffic with the appropriate ICMP response.
The reason this was happening is related to the Calico NetworkPolicies engine we deployed a while back. You can read more upon why this is happening in the relevant upstream bug reports:
- https://github.com/kubernetes/kubernetes/issues/78994
- https://github.com/projectcalico/calico/issues/1055
We have fixed this issue by making sure the Calico rules RETURN back to the rest of the iptables chain instead of ACCEPTing the traffic.
This fix has been rolled out to all clusters already and no further action is required.