Does zstd:chunked support prefetching?

[viluon]

Some of the zstd:chunked PRs mention eStargz support and that the format is generally based on stargz. An important feature of eStargz for my side-project is prefetching, i.e., eagerly pulling certain files. Does zstd:chunked support prioritized files? Where can I learn more about the format?

[Luap99]

@giuseppe PTAL

[giuseppe]

@viluon no. zstd:chunked doesn't allow yet to specify a priority for files.

What change would we need to perform to allow it? Would it be enough to add a new field to the files index?

[viluon]

I believe so. eStargz prioritizes all files up to a "prefetch landmark" according to their order in the layer. The files index could have a field indicating the number of files to be prefetched. Conforming implementations of zstd:chunked could then order the files when building the layer (filling in the prefetch field accordingly) and pull the prefetch subset eagerly.

[giuseppe]

instead of a boolean, I'd prefer implementing it as two float values, that specifies what is the likelihood that a file is going to be needed at startup and at runtime by the container. so we can profile the container and see with different configurations when a file is needed before the container gets into a ready state, and when a file is needed before the container terminates.

With this flexibility, we can tweak the remote pull to sort the files, or define a threshold to decide whether a file must be preloaded or not.

[viluon]

I think that's a good idea, but I also think that eStargz supports prefetching files with a single range request (I think I heard that somewhere but I'm not sure). Of course, this isn't that much of a concern for the specification, but maybe the docs should point out that builders/pullers should optimise for this common case and order the files accordingly.