Skip to content

Fix CI out of memory errors #632

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ElliottKasoar merged 4 commits into from
Dec 9, 2025
Merged

Fix CI out of memory errors #632

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
6a02dd2
Clear disk space to build docker images
ElliottKasoar Nov 28, 2025
32dd153
Move cache clean in CI earlier
ElliottKasoar Nov 28, 2025
54f1bb1
Update CI actions
ElliottKasoar Nov 28, 2025
1d9136a
Add additional uv cache cleans to CI
ElliottKasoar Dec 9, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/ci-dev.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,10 +14,10 @@ jobs:
python-version: ["3.12"]

steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6

- name: Install uv
uses: astral-sh/setup-uv@v6
uses: astral-sh/setup-uv@v7
with:
python-version: ${{ matrix.python-version }}

Expand Down
18 changes: 10 additions & 8 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,14 +12,18 @@ jobs:
python-version: ["3.10","3.11","3.12"]

steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6

- name: Install uv
uses: astral-sh/setup-uv@v6
uses: astral-sh/setup-uv@v7
with:
version: "0.6.2"
python-version: ${{ matrix.python-version }}

- name: Create space in cache
run: |
rm -rf ~/.cache/*
uv cache clean

- name: Install "all" dependencies
run: uv sync --extra all

Expand All @@ -42,6 +46,7 @@ jobs:

- name: Install updated e3nn dependencies
run: |
uv cache clean
uv sync --extra mattersim --extra fairchem --extra sevennet --extra nequip --extra d3
uv pip install --reinstall pynvml
uv pip install fairchem-core[torch-extras] --no-build-isolation
Expand All @@ -55,6 +60,7 @@ jobs:

- name: Install dgl dependencies
run: |
uv cache clean
uv sync --extra mace --extra m3gnet --extra alignn --extra d3
uv pip install --reinstall pynvml

Expand All @@ -64,13 +70,9 @@ jobs:
PYTEST_ADDOPTS: "--durations=0"
run: uv run --no-sync pytest tests/test_{mlip_calculators,single_point,eos}.py

- name: Create space in cache
run: |
rm -rf ~/.cache/*
uv cache clean

- name: Install UMA
run: |
uv cache clean
uv sync --extra uma
uv pip install --reinstall pynvml
uv pip install fairchem-core[torch-extras] --no-build-isolation
Expand Down
89 changes: 87 additions & 2 deletions .github/workflows/docker-jupyter-image.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,50 @@ jobs:
packages: write
contents: read
steps:

# Free space from https://stackoverflow.com/questions/75536771/github-runner-out-of-disk-space-after-building-docker-image
- name: Check disk space
run: df . -h

- name: Free disk space
run: |
sudo docker rmi $(docker image ls -aq) >/dev/null 2>&1 || true
sudo rm -rf \
/usr/share/dotnet /usr/local/lib/android /opt/ghc \
/usr/local/share/powershell /usr/share/swift /usr/local/.ghcup \
/usr/lib/jvm || true
echo "some directories deleted"
sudo apt install aptitude -y >/dev/null 2>&1
sudo aptitude purge aria2 ansible azure-cli shellcheck rpm xorriso zsync \
esl-erlang firefox gfortran google-chrome-stable \
google-cloud-sdk imagemagick \
libmagickcore-dev libmagickwand-dev libmagic-dev ant ant-optional kubectl \
mercurial apt-transport-https mono-complete libmysqlclient \
unixodbc-dev yarn chrpath libssl-dev libxft-dev \
libfreetype6 libfreetype6-dev libfontconfig1 libfontconfig1-dev \
snmp pollinate libpq-dev postgresql-client powershell ruby-full \
sphinxsearch subversion mongodb-org azure-cli microsoft-edge-stable \
temurin-11-jdk temurin-8-jdk temurin-17-jdk temurin-21-jdk \
llvm-16-dev llvm-17-dev llvm-18-dev \
-y -f >/dev/null 2>&1
sudo aptitude purge google-cloud-sdk -f -y >/dev/null 2>&1
sudo aptitude purge microsoft-edge-stable -f -y >/dev/null 2>&1 || true
sudo apt purge microsoft-edge-stable -f -y >/dev/null 2>&1 || true
sudo aptitude purge '~n ^mysql' -f -y >/dev/null 2>&1
sudo aptitude purge '~n ^php' -f -y >/dev/null 2>&1
sudo aptitude purge '~n ^dotnet' -f -y >/dev/null 2>&1
sudo apt-get autoremove -y >/dev/null 2>&1
sudo apt-get autoclean -y >/dev/null 2>&1
echo "some packages purged"

- name: Check disk space
run: |
sudo dpkg-query -Wf '${Installed-Size}\t${Package}\n' | sort -nr | head
df . -h
sudo du /usr/ -hx -d 4 --threshold=1G | sort -hr | head

- name: 'clone the repo'
uses: actions/checkout@v4
uses: actions/checkout@v6
- name: 'login to ghcr'
uses: docker/login-action@v3
with:
Expand All @@ -31,15 +73,58 @@ jobs:
tags: |
ghcr.io/stfc/janus-core/jupyter:amd64-${{ github.sha }}
ghcr.io/stfc/janus-core/jupyter:amd64-latest

build-arm64:
runs-on: ubuntu-24.04-arm
if: github.repository == 'stfc/janus-core'
permissions:
packages: write
contents: read
steps:

# Free space from https://stackoverflow.com/questions/75536771/github-runner-out-of-disk-space-after-building-docker-image
- name: Check disk space
run: df . -h

- name: Free disk space
run: |
sudo docker rmi $(docker image ls -aq) >/dev/null 2>&1 || true
sudo rm -rf \
/usr/share/dotnet /usr/local/lib/android /opt/ghc \
/usr/local/share/powershell /usr/share/swift /usr/local/.ghcup \
/usr/lib/jvm || true
echo "some directories deleted"
sudo apt install aptitude -y >/dev/null 2>&1
sudo aptitude purge aria2 ansible azure-cli shellcheck rpm xorriso zsync \
esl-erlang firefox gfortran google-chrome-stable \
google-cloud-sdk imagemagick \
libmagickcore-dev libmagickwand-dev libmagic-dev ant ant-optional kubectl \
mercurial apt-transport-https mono-complete libmysqlclient \
unixodbc-dev yarn chrpath libssl-dev libxft-dev \
libfreetype6 libfreetype6-dev libfontconfig1 libfontconfig1-dev \
snmp pollinate libpq-dev postgresql-client powershell ruby-full \
sphinxsearch subversion mongodb-org azure-cli microsoft-edge-stable \
temurin-11-jdk temurin-8-jdk temurin-17-jdk temurin-21-jdk \
llvm-16-dev llvm-17-dev llvm-18-dev \
-y -f >/dev/null 2>&1
sudo aptitude purge google-cloud-sdk -f -y >/dev/null 2>&1
sudo aptitude purge microsoft-edge-stable -f -y >/dev/null 2>&1 || true
sudo apt purge microsoft-edge-stable -f -y >/dev/null 2>&1 || true
sudo aptitude purge '~n ^mysql' -f -y >/dev/null 2>&1
sudo aptitude purge '~n ^php' -f -y >/dev/null 2>&1
sudo aptitude purge '~n ^dotnet' -f -y >/dev/null 2>&1
sudo apt-get autoremove -y >/dev/null 2>&1
sudo apt-get autoclean -y >/dev/null 2>&1
echo "some packages purged"

- name: Check disk space
run: |
sudo dpkg-query -Wf '${Installed-Size}\t${Package}\n' | sort -nr | head
df . -h
sudo du /usr/ -hx -d 4 --threshold=1G | sort -hr | head

- name: 'clone the repo'
uses: actions/checkout@v4
uses: actions/checkout@v6
- name: 'login to ghcr'
uses: docker/login-action@v3
with:
Expand Down
46 changes: 45 additions & 1 deletion .github/workflows/docker-marimo-image.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,8 @@ jobs:
packages: write
contents: read
steps:
# free space from https://stackoverflow.com/questions/75536771/github-runner-out-of-disk-space-after-building-docker-image

# Free space from https://stackoverflow.com/questions/75536771/github-runner-out-of-disk-space-after-building-docker-image
- name: Check disk space
run: df . -h

Expand Down Expand Up @@ -52,6 +52,7 @@ jobs:
sudo dpkg-query -Wf '${Installed-Size}\t${Package}\n' | sort -nr | head
df . -h
sudo du /usr/ -hx -d 4 --threshold=1G | sort -hr | head

- name: 'clone the repo'
uses: actions/checkout@v4
- name: 'login to ghcr'
Expand All @@ -72,13 +73,56 @@ jobs:
tags: |
ghcr.io/stfc/janus-core/marimo:amd64-${{ github.sha }}
ghcr.io/stfc/janus-core/marimo:amd64-latest

build-arm64:
runs-on: ubuntu-24.04-arm
if: github.repository == 'stfc/janus-core'
permissions:
packages: write
contents: read
steps:

# Free space from https://stackoverflow.com/questions/75536771/github-runner-out-of-disk-space-after-building-docker-image
- name: Check disk space
run: df . -h

- name: Free disk space
run: |
sudo docker rmi $(docker image ls -aq) >/dev/null 2>&1 || true
sudo rm -rf \
/usr/share/dotnet /usr/local/lib/android /opt/ghc \
/usr/local/share/powershell /usr/share/swift /usr/local/.ghcup \
/usr/lib/jvm || true
echo "some directories deleted"
sudo apt install aptitude -y >/dev/null 2>&1
sudo aptitude purge aria2 ansible azure-cli shellcheck rpm xorriso zsync \
esl-erlang firefox gfortran google-chrome-stable \
google-cloud-sdk imagemagick \
libmagickcore-dev libmagickwand-dev libmagic-dev ant ant-optional kubectl \
mercurial apt-transport-https mono-complete libmysqlclient \
unixodbc-dev yarn chrpath libssl-dev libxft-dev \
libfreetype6 libfreetype6-dev libfontconfig1 libfontconfig1-dev \
snmp pollinate libpq-dev postgresql-client powershell ruby-full \
sphinxsearch subversion mongodb-org azure-cli microsoft-edge-stable \
temurin-11-jdk temurin-8-jdk temurin-17-jdk temurin-21-jdk \
llvm-16-dev llvm-17-dev llvm-18-dev \
-y -f >/dev/null 2>&1
sudo aptitude purge google-cloud-sdk -f -y >/dev/null 2>&1
sudo aptitude purge microsoft-edge-stable -f -y >/dev/null 2>&1 || true
sudo apt purge microsoft-edge-stable -f -y >/dev/null 2>&1 || true
sudo aptitude purge '~n ^mysql' -f -y >/dev/null 2>&1
sudo aptitude purge '~n ^php' -f -y >/dev/null 2>&1
sudo aptitude purge '~n ^dotnet' -f -y >/dev/null 2>&1
sudo apt-get autoremove -y >/dev/null 2>&1
sudo apt-get autoclean -y >/dev/null 2>&1
echo "some packages purged"

- name: Check disk space
run: |
sudo dpkg-query -Wf '${Installed-Size}\t${Package}\n' | sort -nr | head
df . -h
sudo du /usr/ -hx -d 4 --threshold=1G | sort -hr | head

- name: 'clone the repo'
uses: actions/checkout@v4
- name: 'login to ghcr'
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/windows.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,10 @@ jobs:
python-version: ["3.10","3.11","3.12"]

steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6

- name: Install uv
uses: astral-sh/setup-uv@v6
uses: astral-sh/setup-uv@v7
with:
python-version: ${{ matrix.python-version }}
prune-cache: false
Expand Down