[CI] First build Base and Build images, then Intel Drivers

[lukaszstolarczuk]

and fix building "Intel Drivers" images, as they currently can't be built because of "No space left on device" error.

[lukaszstolarczuk]

example CI failure in nightly job last weekend: https://github.com/intel/llvm/actions/runs/19381290080/job/55460338609#step:3:914

extra cleanup step upgrades from 19G free to 52G free space available

[lukaszstolarczuk]

Finally extra cleanup takes 1m 21s and gives extra 27GB of free space.
Currently cleanup seems to be required only for "Intel Drivers" images, but I kept it in both jobs for unity...?

@intel/dpcpp-devops-reviewers please review.

[aelovikov-intel]

Isn't it a deal breaker for us? @sarnex , @uditagarwal97

[sarnex]

Probably yes, but if I understand correctly that the problem is hard drive space can't we just move the job from the GitHub hosted runners to a self-hosted one which has enough space?

[uditagarwal97]

+1 to just moving docker container build jobs to self-hosted runners

[lukaszstolarczuk]

@aelovikov-intel, what is the deal breaker?

Just to be on the same page, there are 2 issues here. While I tried fixing issue that "intel drivers" are based on old Base/Build images I noticed the issue that "intel drivers" images can't be build (because they are too big/too litle space is available).

[lukaszstolarczuk]

if you want to move building dockers to self-hosted runners, pls let me know which label should I used, at best.

[aelovikov-intel]

While I tried fixing issue that "intel drivers" are based on old Base/Build images

Are you saying that it's the same limitation when running this workflow in pre-commit (whenever image-related tasks/code are being updated) before/after PR?

[sarnex]

I just created the docker-builder label and put it on some random Linux machine, can you try using that? Probably CI will fail because the OS isn't set up right but I'll set it up once I see the errors.

[lukaszstolarczuk]

docker-builder used.

Are you saying that it's the same limitation when running this workflow in pre-commit (whenever image-related tasks/code are being updated) before/after PR?

I actually only checked the nightly build - we fixed a dependency for benchmarks, but it was still failing on "nigthly" image, which is based on "intel drivers". Once I re-built "intel drivers" and "nightly" it all worked.

Anyway, I believe if we use only a single runner for building dockers the issue may be gone, because we have these dockers cached in the system...?

side note: we'd have to regularly clean/prune old images from that runner (to free up the disk space).

[sarnex]

side note: we'd have to regularly clean/prune old images from that runner (to free up the disk space).

I already have to do this lol