Parallel image pulling within a pod

[fidencio]

Enhancement Description

• One-line enhancement description (can be used as a release note): Allow parallel image pulling for a multi-containers pod, as it reduce the pod start-up time by a significant amount.
• Kubernetes Enhancement Proposal: TBD
• Discussion Link: TBD
• PRs by stage and milestone:
  • Alpha - v1.xx
    • KEP (k/enhancements) update PR(s):
    • Code (k/k) update PR(s):
    • Docs (k/website) update PR(s):

Please keep this description up to date. This will help the Enhancement Team to track the evolution of the enhancement efficiently.

[fidencio]

Right now this is just an idea to be discussed before opening a KEP, mostly in order to understand the drawbacks and corner-cases of such approach.

Currently the serial behaviour can hurt quite a lot snapshotters that are already slow(er) (such as devmapper or blockfile), mainly in cases where users are not pinning a specfic version of the containers.

Potential drawbacks that my limited knowledge can think of are:

• Changing the pod lifecycle in order to pass this information down to the runtimes in a non-sequential way
• Complicating the calculation done in order to detect a disk pressure on the node
• Maybe causing complications on error propagation when pulling multi images

Although this may be similar to https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/3673-kubelet-parallel-image-pull-limit/README.md, #3673 (AFAIU) allows multiple images, for different pods, to be downloaded at the same time ... but does NOT change the behaviour of images being pulled in sequence for a multi-containers pod.

[SergeyKanzhelev]

This was discussed before and this KEP can help in some cases. But there are a few important considerations, I will list off the top of my head, you can find conversations in various issues:

1. The change of behavior (pull all images of the Pod) is a breaking change. There are Pods out in the wild that would use init containers to configure images for the main container by running things like ctr tag. Since it was always the promise that the image pull of the "next" container is not starting before the previous container run to completion, that scenario worked. Breaking it will require a way to return the previous behavior OR convincing community that it is not important. Other things init container might have been doing are configuring registry mirrors, proxies, and other network settings, as well as delaying the image pull of the main container, especially if it is "heavy" and needs to be sequenced with other operations on the node.
2. kubelet has the limit on parallel image pulling. Vendors may have this value set to lower values like 2 or 3 just because the higher value is not helping to speed up downloads - disk IO is a bottleneck. With these limits, the first Pod with 4 containers will effectively monopolize image pulling. It is especially bad if the Pod with 4 images has a long running init container. So 3 of these images will only be needed later and other Pods should be downloading their images for the init containers.

I was thinking some sort of new CRI API that allows to pause, cancel, or resume image pulling as well as declaring "high" and "low" priority pulling with the ability to increase priority, may be a solution here. Also on API change - we may need to introduce this as a per-node or per-pod field.

[fidencio]

The change of behavior (pull all images of the Pod) is a breaking change. There are Pods out in the wild that would use init containers to configure images for the main container by running things like ctr tag. Since it was always the promise that the image pull of the "next" container is not starting before the previous container run to completion, that scenario worked. Breaking it will require a way to return the previous behavior OR convincing community that it is not important.

Although I understand this behaviour, this seems very anti-pattern. Still, a reasonable compromise would be that initContainers are not changed, while containers themselves could be as those already do not have any guaranteed order of execution, meaning:

• Images for initContainers are still pulled in serial
• images for main containers are parallelised

Other things init container might have been doing are configuring registry mirrors, proxies, and other network settings, as well as delaying the image pull of the main container, especially if it is "heavy" and needs to be sequenced with other operations on the node.

This could be worked around by leaving initContainers as they are, which is my suggestion above.

kubelet has the limit on parallel image pulling. Vendors may have this value set to lower values like 2 or 3 just because the higher value is not helping to speed up downloads - disk IO is a bottleneck.

This is exactly a case where the time to mature the feature becomes very important, and I do think that documentation plays the game here more than anything else.

@SergeyKanzhelev, what's your take on what I mentioned above and am repeating here:

Still, a reasonable compromise would be that initContainers are not changed, while containers themselves could be as those already do not have any guaranteed order of execution, meaning:

• Images for initContainers are still pulled in serial
• images for main containers are parallelised

[SergeyKanzhelev]

Although I understand this behaviour, this seems very anti-pattern.

it may be seen as antipattern. But the reality is that it is hard to implement many of scenarios I listed any other way. Do you see any alternatives that are possible?

• Images for initContainers are still pulled in serial
• images for main containers are parallelised

Couple of comments here. Will this limited scope still be helpful? My understanding is that downloading images WHILE init container is running is the most common request. Second comment - regular container are also sequenced. And the pull of the next container's image will not start before the PostStart of a first container completed. So there is also sequential execution this KEP may break.

I think the KEP has legs. However simply ignoring breaking changes by limiting scope and not providing an alternative is not a good way forward.

[fidencio]

But the reality is that it is hard to implement many of scenarios I listed any other way. Do you see any alternatives that are possible?

Partially. Considering the user knows what they're doing (and I assume this is true on this cases), having a way to set "blocking" / "non-blocking" per pod would be quite beneficial. With blocking being the default behaviour we ensure nothing breaks, but this is me talking without actually looking at the code.

Will this limited scope still be helpful? My understanding is that downloading images WHILE init container is running is the most common request

It'd partially help. If I'm running a pod with 3 containers, apart from the init, pulling those 3 images in parallel would already be helpful. But, ideally, as you mentioned, I'd prefer having the images pulled in parallel WHILE the init container is running.

rregular container are also sequenced. And the pull of the next container's image will not start before the PostStart of a first container completed.

This has to change, IMHO. :-). I understand the reasoning of not changing it by default, but not giving the users a flag that changes the behaviour is not exactly optimal. Again, without looking at the code, maybe the blocking / non-blocking would be a potential path to take. You can see that in my first comment I added as a drawback "Changing the pod lifecycle in order to pass this information down to the runtimes in a non-sequential way", and here is where I would like to hear from node maintainers how do-able it'd be.

[SergeyKanzhelev]

I think the change here will not be small scoped. Some API change will likely be needed, plus the new CRI API will be introduced. And with this, we may need to fix more issues than the parallel download. Like maybe another KEP - download progress notifications will fit into this "refactoiring".

All this said it may be lower priority than improving how kubelet works with streaming-based image pulls. Like https://github.com/containerd/stargz-snapshotter. With the streaming based image pulls, the parallel pull is effectively not an issue, pull time is close to zero and all the costs are shifted to execution time.

[pacoxu]

Although this may be similar to https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/3673-kubelet-parallel-image-pull-limit/README.md, #3673 (AFAIU) allows multiple images, for different pods, to be downloaded at the same time ... but does NOT change the behaviour of images being pulled in sequence for a multi-containers pod.

Is this proposal compatible with maxParallelImagePulls and serializeImagePulls? Something like: If maxParallelImagePulls=3 and serializeImagePulls=false, with this proposal, a pod with 4 images to pull will finally pull 3 of the 4 in parallel.

[SergeyKanzhelev]

a pod with 4 images to pull will finally pull 3 of the 4 in parallel.

yes, this is a proposal. However I believe we need to be careful and make sure there is a fairness between Pods on pull ordering.

[pacoxu]

/sig node

[HirazawaUi]

/cc