[ci.jenkins.io] Use of `s390x` permanent agent

[lemeurherve]

For 2 weeks, ci.jenkins.io's s390x agent has been used to build Jenkins Core controller images in order to optimise their multiarch build:

• Optimise multiarch sanity check duration jenkinsci/docker#2154
• chore(pipeline/multiarch): don't rebuild images of architectures already built elsewhere jenkinsci/docker#2155
• [ci.jenkins.io] Ensure s390x permanent agent has every required tools to build docker images #4928

But as since then its hard drive has been full at least 3 times and as this architecture build wasn't the bottleneck, this usage has been reverted:

• Optimise multiarch sanity check duration jenkinsci/docker#2154 (comment)
• chore(pipeline): don't use s390x permanent agent jenkinsci/docker#2185

Opened this helpdesk to trace those elements, and to mention that it would be nice to improve this agent monitoring.

[lemeurherve]

since then its hard drive has been full at least 3 times

@MarkEWaite comment from https://matrix.to/#/!JLUOInpEYmxJIYXlzs:matrix.org/$cYoWLiiliCEzDAJlu7mIuPzAZ73DKeKpHMgbdqgXJyQ?via=matrix.org&via=gitter.im&via=g4v.dev:

As far as I can tell, the buildkit on s390x does not exit when the job finishes that was using it. We accumulate buildkit volumes and they need to be deleted.

[dduportal]

For audit (and eventually runbook procedure if we reinstate the Docker builds on this agent):

In order to "clean up" the disk from Docker (images, containers and volumes leftovers), I ran the following commands:

# Ensure all running containers are forced-kill. Dangerous: might fail running builds.
docker ps -q | xargs docker kill
# Remove all stopped containers and associated volumes/resources. Safe to run at any moments.
docker system prune --volumes --force
# Remove all Docker scanner runs. Safe to run at any moments.
docker scout cache prune --force;
# Delete all Docker images, even if used. Dangerous: might fail running builds.
docker images ls -q | xargs docker rmi -f

[dduportal]

Closing as we audited all tasks. Thanks!