/var/lib/docker mount considerations

[shaikatz]

Hi,

I've been trying to use the volumeClaimTemplate ability with RunnerSet, to give my ephemeral runners docker caching support.

Currently, I'm seeing an issue where my next job automatically brings up containers from the previous job.

I guess the reason is that when mounting the whole /var/lib/docker folder we also mount the containers folder which holds the running containers data.

What will be the right solution to take advantage of PVC but only for the docker layer caching and not to persist in running containers between jobs? any other methods you guys use to speed up docker build on ephemeral containers?

Thanks!

[mumoshu]

@shaikatz Hey! There seems like no single solution to your problem, so you need to figure out how yourself mostly.

But recently, awesome @TheDarkula has contributed an example config and his associated experience to set up a cache for rust builds. One of the essential learnings we have along the way was that you need to be sure which container within the runner pod you need to mount volumes, depending on whether you run dind within the runner or not, and whether you use containerized job steps or not.

I think you can extend @TheDarkula's work to /var/lib/docker for your goal.

Also note that you'd need this patch #1340 to let ARC correctly clean and reuse PVs for RunnerSet-managed pods.

[TheDarkula]

@mumoshu You are too sweet :)

I actually found a huge problem while sharing /var/lib/docker, as it can easily be corrupted, which will then fail to bring up a container at all.
I am working on a much more elegant solution for a docker image cache, and I will create documentation on it when I have completed it.

[bquenin]

Hi @TheDarkula!

I'm interested in your more elegant solution for a docker image cache. Did you have the chance to make any progress on this?

Thanks!

[TheDarkula]

@bquenin I did a lot of reseach, documented over here.

Essentially, I wanted to use something nice, like kube-fledged, but unfortunately, it does not really seem possible with ARC.

[bquenin]

thanks @TheDarkula, I'll have a look at it