If a Kubernetes Service had no active Endpoints, for example when a deployment is scaled to 0, then requests to that Service were timing out. Instead it’s supposed to reject traffic with the appropriate ICMP response.
The reason this was happening is related to the Calico NetworkPolicies engine we deployed a while back. You can read more upon why this is happening in the relevant upstream bug reports:
We have fixed this issue by making sure the Calico rules
RETURN back to the rest of the
iptables chain instead of
ACCEPTing the traffic.
This fix has been rolled out to all clusters already and no further action is required.