Skip to content

CI: Fix "No space left on device" errors in ci-basic workflow#91

Merged
dmidem merged 1 commit intozsa1from
ci-basic-disk-cleanup
Oct 27, 2025
Merged

CI: Fix "No space left on device" errors in ci-basic workflow#91
dmidem merged 1 commit intozsa1from
ci-basic-disk-cleanup

Conversation

@dmidem
Copy link
Copy Markdown
Collaborator

@dmidem dmidem commented Oct 13, 2025

Problem

The ci-basic.yml workflow started failing with "No space left on device" errors across all PRs. This happened even for previously passing PRs that hadn't been modified (#89, for example) when I tried to re-run the CI workflow, for example: https://github.com/QED-it/zebra/actions/runs/17938984393/job/52561489949?pr=89

Root Cause Investigation

Initially suspected two possible causes:

  1. Dependency version changes - Missing --locked flag in cargo commands could allow dependencies to update and consume more disk space during compilation
  2. GitHub runner disk space reduction - Changes to ubuntu-latest VM resources (reduced total space or increased pre-installed packages)

Results of the Investigation

  1. Adding --locked alone didn't resolve the issue, confirming the problem was insufficient available disk space on the GitHub runner. However, I kept the --locked flag in this PR as it's the proper way to ensure reproducible builds.
  2. I couldn't find information about recent changes to ubuntu-latest VM resources on the Internet, and couldn't verify from our previous successful PR runs since we didn't log resource information. I tried cleaning up unused disk space, which resolved the issue.

Solution

This PR implements disk space optimization for the CI workflow in the existing ubuntu-latest instance (continuing to use it), and also adds simple VM resource logging and additional improvements:

  1. Disk Space Cleanup

    • Remove unused pre-installed Android SDK (/usr/local/lib/android)
    • Remove GitHub Actions tool cache ($AGENT_TOOLSDIRECTORY)
    • Result: Freed ~14 GB of disk space (from 23 GB to 37 GB available)
  2. Resource Monitoring

    • Added system resource logging (disk, RAM, CPU) before and after cleanup to track space usage
  3. Additional Improvements

    • Added --locked flag to all cargo commands to ensure reproducible builds
    • Added --no-deps flag to cargo doc and cargo clippy to avoid building dependencies unnecessarily
    • Simplified matrix configuration to explicitly use ubuntu-latest (removed redundant matrix syntax)

@dmidem dmidem requested a review from PaulLaux October 13, 2025 09:02
@dmidem dmidem force-pushed the ci-basic-disk-cleanup branch from 32c9d4d to 0a96bc0 Compare October 13, 2025 09:44
Copy link
Copy Markdown

@PaulLaux PaulLaux left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

approved with minor comments

Comment on lines +18 to +22
- name: Show system resource summary (before cleanup)
run: |
df -h
free -h
lscpu | egrep 'Model name|Socket|Thread|Core|CPU\(s\)'
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove it, we don't need it twice.

# Remove heavy preinstalled SDKs and toolchains
sudo rm -rf /usr/local/lib/android || true
sudo rm -rf "$AGENT_TOOLSDIRECTORY" || true # preinstalled tool caches
df -h
Copy link
Copy Markdown

@PaulLaux PaulLaux Oct 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not needed (the df -h)

@dmidem dmidem merged commit 5438017 into zsa1 Oct 27, 2025
55 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants