Rate limit for image pulling

[stevefan1999-personal]

Since DockerHub imposed a download limit for each IP, and under the NAT (AWS is infamous for this recently since it started taking money for dedicated IPv4 address, so we have to use NAT gateway of some sort) we share the same IP.

This means there could be multiple pulls under a same IP address for all the nodes in the cluster and will contest for image pulling backoff, which is under the category of Thundering herd problem.

Until DockerHub provides a stable IPv6 service, we can't possibly give each node a dedicated IPv4.

In order to mitigate this we will need to make the node to download the image one by one at a suitable interval. Otherwise, it will take at least quadratic time for all nodes to finish downloading images.

[stevefan1999-personal]

At the very least, I advise we have a cluster level lease lock to implement leader locking: Lease | Kubernetes

We can use an init container to contest for locks, and when the init container and downloader finishes, the lock should automatically be released, and is available for other nodes to contest with.

Another approach is to use a stateful set and rollout the deployment to each node one by one.