GHA runners are running out of space

[tamuri]

GHA jobs are regularly failing, on both Github runners and our self-hosted runners, due to insufficient space. The job doesn't have enough space to create the Python environment with requirements. This has been triggered by the recent addition of sdv, which pulls in the pytorch package (~400MB). Our CI VMs have 8 self-hosted runners on a single machine, and the additional usage requirements has filled up the drive.

Short-term solution: removed GHA upgrades and rebooted the machine to remove temp files etc. That's been done, so jobs are running again (apart from the Check workflow which runs on a Github runner)

Long-term solution:

• Replace current CI VMs with ones having larger volume size (200GB to 300GB). Will require unregistering existing self-hosted runners on Github and provisioning GHA runners on the new VMs
• Move Check workflow to run on our self-hosted runner Don't cache tox in checks and docs workflows

[tamuri]

Another option to try: clear out space on the Ubuntu runner and then re-enable caching.

https://github.com/jlumbroso/free-disk-space