Skip to content

Commit 050818c

Browse files
booxterbooxter
committed
ci: Constrain all dependencies
Eventually we should adopt (some of this code) from ci-actions that should host reusable parts of the constraints infra. The reusable doesn't exist at the moment and is being cooked, so in the meantime, it's ok to constrain it per repo. (It also allows us to validate some CUDA specific quirks around flash-attn before we commit to the shape of the reusable workflow action.) Signed-off-by: Ihar Hrachyshka <ihar.hrachyshka@gmail.com>
1 parent 5136461 commit 050818c

File tree

10 files changed

+313
-30
lines changed

10 files changed

+313
-30
lines changed

‎.github/workflows/constraints-update.yml‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
name: Update constraints-dev.txt
2+
3+
on:
4+
schedule:
5+
- cron: '0 3 * * 1' # Every Monday at 03:00 UTC
6+
workflow_dispatch:
7+
8+
jobs:
9+
update-constraints:
10+
runs-on: ubuntu-latest
11+
permissions:
12+
contents: write
13+
pull-requests: write
14+
15+
steps:
16+
- name: Checkout repository
17+
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
18+
19+
- name: Set up Python
20+
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
21+
with:
22+
python-version: '3.11'
23+
24+
- name: Install tox
25+
run: python -m pip install --upgrade pip tox -c constraints-dev.txt
26+
27+
- name: Run tox constraints
28+
run: tox -e constraints
29+
30+
- name: git diff
31+
run: |
32+
git diff constraints-dev.txt
33+
34+
- name: set release date
35+
run: |
36+
echo "date=$(date --rfc-3339=date)" >> "${GITHUB_ENV}"
37+
38+
- name: Create Pull Request
39+
uses: peter-evans/create-pull-request@271a8d0340265f705b14b6d32b9829c1cb33d45e # v7.0.8
40+
with:
41+
signoff: true
42+
commit-message: "chore: update constraints-dev.txt (${{ env.date }})"
43+
title: "chore: update constraints-dev.txt (${{ env.date }})"
44+
body: |
45+
This PR updates the constraints-dev.txt file using `tox -e constraints`.
46+
branch: update-constraints
47+
token: ${{ secrets.GH_PERSONAL_ACCESS_TOKEN }}

‎.github/workflows/e2e-nvidia-l40s-x4.yml‎

Lines changed: 5 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -177,27 +177,15 @@ jobs:
177177
- name: Install ilab
178178
working-directory: ./instructlab
179179
run: |
180-
export CUDA_HOME="/usr/local/cuda"
181-
export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64"
182-
export PATH="$PATH:$CUDA_HOME/bin"
183-
python3.11 -m venv --upgrade-deps venv
184-
. venv/bin/activate
185-
nvidia-smi
186-
python3.11 -m pip cache remove llama_cpp_python
187-
188-
CMAKE_ARGS="-DLLAMA_CUDA=on" python3.11 -m pip install . -c constraints-dev.txt
189-
190-
# https://github.com/instructlab/instructlab/issues/1821
191-
# install with Torch and build dependencies installed
192-
python3.11 -m pip install packaging wheel setuptools-scm
193-
python3.11 -m pip install .[cuda] -r requirements-vllm-cuda.txt
180+
./scripts/install-ilab-with-cuda.sh
194181
195182
- name: Update instructlab-training library
196183
working-directory: ./training
197184
run: |
198185
. ../instructlab/venv/bin/activate
199-
pip install .
200-
pip install .[cuda]
186+
pip_install="pip install -c constraints-dev.txt"
187+
$pip_install .
188+
$pip_install .[cuda]
201189
202190
- name: Check disk before tests
203191
if: always()
@@ -380,7 +368,7 @@ jobs:
380368
working-directory: ./training
381369
run: |
382370
python -m pip install --upgrade pip
383-
pip install -r requirements-dev.txt
371+
pip install -r requirements-dev.txt -c constraints-dev.txt
384372
385373
- name: Try to upload Phase 1 to s3
386374
id: phase-1-upload-s3

‎.github/workflows/lint.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -80,7 +80,7 @@ jobs:
8080
- name: Install tox
8181
run: |
8282
python -m pip install --upgrade pip
83-
python -m pip install tox tox-gh
83+
python -m pip install tox tox-gh -c constraints-dev.txt
8484
8585
- name: "${{ matrix.lint.name }}"
8686
run: |

‎.github/workflows/smoke.yaml‎

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -135,7 +135,7 @@ jobs:
135135
run: |
136136
python3.11 -m venv --upgrade-deps venv
137137
. venv/bin/activate
138-
pip install tox
138+
pip install tox -c constraints-dev.txt
139139
140140
# flash-attn has a bug in the setup.py that causes pip to attempt
141141
# installing it before torch is installed. This is a bug because their
@@ -146,19 +146,20 @@ jobs:
146146
run: |
147147
source venv/bin/activate
148148
# The list is taken from the pull request linked above
149-
pip install torch packaging setuptools wheel psutil ninja
149+
pip install torch packaging setuptools wheel psutil ninja -c constraints-dev.txt
150150
151151
- name: "Install tox-current-env to reuse the venv with pre-installed build dependencies"
152152
run: |
153153
source venv/bin/activate
154-
pip install tox-current-env
154+
pip install tox-current-env -c constraints-dev.txt
155155
156156
- name: "Install dependencies from tox.ini in the current venv, using current venv installed deps"
157157
run: |
158158
source venv/bin/activate
159159
tox -e py3-smoke --print-deps-to-file=./deps.txt
160-
pip install -r ./deps.txt --no-build-isolation
161-
pip install .
160+
pip_install="pip install -c constraints-dev.txt"
161+
$pip_install -r ./deps.txt --no-build-isolation
162+
$pip_install .
162163
163164
- name: "Show disk utilization BEFORE tests"
164165
if: always()

‎.github/workflows/unit.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -72,7 +72,7 @@ jobs:
7272
run: |
7373
python -m venv --upgrade-deps venv
7474
. venv/bin/activate
75-
pip install tox
75+
pip install tox -c constraints-dev.txt
7676
7777
- name: "Show disk utilization BEFORE tests"
7878
if: always()

‎constraints-dev.txt‎

Lines changed: 195 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,195 @@
1+
accelerate==1.0.1 # via -r requirements-cuda.txt, peft, trl
2+
aiofiles==24.1.0 # via -r requirements.txt
3+
aiohappyeyeballs==2.6.1 # via aiohttp
4+
aiohttp==3.11.18 # via fsspec
5+
aiosignal==1.3.2 # via aiohttp
6+
annotated-types==0.7.0 # via pydantic
7+
anyio==4.9.0 # via httpx, jupyter-server
8+
argon2-cffi==23.1.0 # via jupyter-server
9+
argon2-cffi-bindings==21.2.0 # via argon2-cffi
10+
arrow==1.3.0 # via isoduration
11+
astroid==3.3.10 # via pylint
12+
asttokens==3.0.0 # via stack-data
13+
async-lru==2.0.5 # via jupyterlab
14+
attrs==25.3.0 # via aiohttp, jsonschema, referencing
15+
babel==2.17.0 # via jupyterlab-server
16+
beautifulsoup4==4.13.4 # via nbconvert
17+
bitsandbytes==0.45.5 # via -r requirements-cuda.txt
18+
bleach==6.2.0 # via nbconvert
19+
cachetools==5.5.2 # via tox
20+
certifi==2025.4.26 # via httpcore, httpx, requests
21+
cffi==1.17.1 # via argon2-cffi-bindings
22+
cfgv==3.4.0 # via pre-commit
23+
chardet==5.2.0 # via tox
24+
charset-normalizer==3.4.2 # via requests
25+
colorama==0.4.6 # via tox
26+
comm==0.2.2 # via ipykernel, ipywidgets
27+
contourpy==1.3.2 # via matplotlib
28+
cycler==0.12.1 # via matplotlib
29+
datasets==3.6.0 # via -r requirements.txt, trl
30+
debugpy==1.8.14 # via ipykernel
31+
decorator==5.2.1 # via ipython
32+
defusedxml==0.7.1 # via nbconvert
33+
dill==0.3.8 # via datasets, multiprocess, pylint
34+
distlib==0.3.9 # via virtualenv
35+
einops==0.8.1 # via flash-attn
36+
executing==2.2.0 # via stack-data
37+
fastjsonschema==2.21.1 # via nbformat
38+
filelock==3.18.0 # via datasets, huggingface-hub, torch, tox, transformers, virtualenv
39+
flash-attn==2.7.4.post1 # via -r requirements-cuda.txt
40+
fonttools==4.58.0 # via matplotlib
41+
fqdn==1.5.1 # via jsonschema
42+
frozenlist==1.6.0 # via aiohttp, aiosignal
43+
fsspec==2025.3.0 # via datasets, huggingface-hub, torch
44+
h11==0.16.0 # via httpcore
45+
httpcore==1.0.9 # via httpx
46+
httpx==0.28.1 # via jupyterlab
47+
huggingface-hub==0.31.4 # via -r requirements-dev.txt, accelerate, datasets, peft, tokenizers, transformers
48+
identify==2.6.10 # via pre-commit
49+
idna==3.10 # via anyio, httpx, jsonschema, requests, yarl
50+
iniconfig==2.1.0 # via pytest
51+
instructlab-dolomite==0.2.0 # via -r requirements.txt
52+
ipykernel==6.29.5 # via -r requirements-dev.txt, jupyter, jupyter-console, jupyterlab
53+
ipython==9.2.0 # via -r requirements-dev.txt, ipykernel, ipywidgets, jupyter-console
54+
ipython-pygments-lexers==1.1.1 # via ipython
55+
ipywidgets==8.1.7 # via jupyter
56+
isoduration==20.11.0 # via jsonschema
57+
isort==6.0.1 # via -r requirements-dev.txt, pylint
58+
jedi==0.19.2 # via ipython
59+
jinja2==3.1.6 # via jupyter-server, jupyterlab, jupyterlab-server, nbconvert, torch
60+
json5==0.12.0 # via jupyterlab-server
61+
jsonpointer==3.0.0 # via jsonschema
62+
jsonschema==4.23.0 # via jupyter-events, jupyterlab-server, nbformat
63+
jsonschema-specifications==2025.4.1 # via jsonschema
64+
jupyter==1.1.1 # via -r requirements-dev.txt
65+
jupyter-client==8.6.3 # via ipykernel, jupyter-console, jupyter-server, nbclient
66+
jupyter-console==6.6.3 # via jupyter
67+
jupyter-core==5.7.2 # via ipykernel, jupyter-client, jupyter-console, jupyter-server, jupyterlab, nbclient, nbconvert, nbformat
68+
jupyter-events==0.12.0 # via jupyter-server
69+
jupyter-lsp==2.2.5 # via jupyterlab
70+
jupyter-server==2.16.0 # via jupyter-lsp, jupyterlab, jupyterlab-server, notebook, notebook-shim
71+
jupyter-server-terminals==0.5.3 # via jupyter-server
72+
jupyterlab==4.4.2 # via jupyter, notebook
73+
jupyterlab-pygments==0.3.0 # via nbconvert
74+
jupyterlab-server==2.27.3 # via jupyterlab, notebook
75+
jupyterlab-widgets==3.0.15 # via ipywidgets
76+
kiwisolver==1.4.8 # via matplotlib
77+
liger-kernel==0.5.9 # via -r requirements-cuda.txt
78+
llvmlite==0.44.0 # via numba
79+
markdown-it-py==3.0.0 # via rich
80+
markupsafe==3.0.2 # via jinja2, nbconvert
81+
matplotlib==3.10.3 # via -r requirements-dev.txt
82+
matplotlib-inline==0.1.7 # via ipykernel, ipython
83+
mccabe==0.7.0 # via pylint
84+
mdurl==0.1.2 # via markdown-it-py
85+
mistune==3.1.3 # via nbconvert
86+
mpmath==1.3.0 # via sympy
87+
multidict==6.4.4 # via aiohttp, yarl
88+
multiprocess==0.70.16 # via datasets
89+
mypy==1.15.0 # via -r requirements-dev.txt
90+
mypy-extensions==1.1.0 # via mypy
91+
nbclient==0.10.2 # via nbconvert
92+
nbconvert==7.16.6 # via jupyter, jupyter-server
93+
nbformat==5.10.4 # via jupyter-server, nbclient, nbconvert
94+
nest-asyncio==1.6.0 # via ipykernel
95+
networkx==3.4.2 # via torch
96+
nodeenv==1.9.1 # via pre-commit
97+
notebook==7.4.2 # via jupyter
98+
notebook-shim==0.2.4 # via jupyterlab, notebook
99+
numba==0.61.2 # via -r requirements.txt
100+
numpy==1.26.4 # via -r requirements-dev.txt, -r requirements.txt, accelerate, bitsandbytes, contourpy, datasets, matplotlib, numba, pandas, peft, transformers
101+
nvidia-cublas-cu12==12.6.4.1 # via nvidia-cudnn-cu12, nvidia-cusolver-cu12, torch
102+
nvidia-cuda-cupti-cu12==12.6.80 # via torch
103+
nvidia-cuda-nvrtc-cu12==12.6.77 # via torch
104+
nvidia-cuda-runtime-cu12==12.6.77 # via torch
105+
nvidia-cudnn-cu12==9.5.1.17 # via torch
106+
nvidia-cufft-cu12==11.3.0.4 # via torch
107+
nvidia-cufile-cu12==1.11.1.6 # via torch
108+
nvidia-curand-cu12==10.3.7.77 # via torch
109+
nvidia-cusolver-cu12==11.7.1.2 # via torch
110+
nvidia-cusparse-cu12==12.5.4.2 # via nvidia-cusolver-cu12, torch
111+
nvidia-cusparselt-cu12==0.6.3 # via torch
112+
nvidia-nccl-cu12==2.26.2 # via torch
113+
nvidia-nvjitlink-cu12==12.6.85 # via nvidia-cufft-cu12, nvidia-cusolver-cu12, nvidia-cusparse-cu12, torch
114+
nvidia-nvtx-cu12==12.6.77 # via torch
115+
overrides==7.7.0 # via jupyter-server
116+
packaging==25.0 # via -r requirements.txt, accelerate, datasets, huggingface-hub, ipykernel, jupyter-events, jupyter-server, jupyterlab, jupyterlab-server, matplotlib, nbconvert, peft, pyproject-api, pytest, tox, transformers
117+
pandas==2.2.3 # via datasets
118+
pandocfilters==1.5.1 # via nbconvert
119+
parso==0.8.4 # via jedi
120+
peft==0.15.2 # via -r requirements.txt
121+
pexpect==4.9.0 # via ipython
122+
pillow==11.2.1 # via matplotlib
123+
platformdirs==4.3.8 # via jupyter-core, pylint, tox, virtualenv
124+
pluggy==1.6.0 # via pytest, tox
125+
pre-commit==4.2.0 # via -r requirements-dev.txt
126+
prometheus-client==0.22.0 # via jupyter-server
127+
prompt-toolkit==3.0.51 # via ipython, jupyter-console
128+
propcache==0.3.1 # via aiohttp, yarl
129+
psutil==7.0.0 # via accelerate, ipykernel, peft
130+
ptyprocess==0.7.0 # via pexpect, terminado
131+
pure-eval==0.2.3 # via stack-data
132+
py-cpuinfo==9.0.0 # via -r requirements.txt
133+
pyarrow==20.0.0 # via datasets
134+
pycparser==2.22 # via cffi
135+
pydantic==2.11.4 # via -r requirements.txt, pylint-pydantic
136+
pydantic-core==2.33.2 # via pydantic
137+
pygments==2.19.1 # via ipython, ipython-pygments-lexers, jupyter-console, nbconvert, rich
138+
pylint==3.3.7 # via -r requirements-dev.txt, pylint-plugin-utils, pylint-pydantic
139+
pylint-plugin-utils==0.8.2 # via pylint-pydantic
140+
pylint-pydantic==0.3.5 # via -r requirements-dev.txt
141+
pyparsing==3.2.3 # via matplotlib
142+
pyproject-api==1.9.1 # via tox
143+
pytest==8.3.5 # via -r requirements-dev.txt
144+
python-dateutil==2.9.0.post0 # via arrow, jupyter-client, matplotlib, pandas
145+
python-json-logger==3.3.0 # via jupyter-events
146+
pytz==2025.2 # via pandas
147+
pyyaml==6.0.2 # via -r requirements.txt, accelerate, datasets, huggingface-hub, jupyter-events, peft, pre-commit, transformers
148+
pyzmq==26.4.0 # via ipykernel, jupyter-client, jupyter-console, jupyter-server
149+
referencing==0.36.2 # via jsonschema, jsonschema-specifications, jupyter-events
150+
regex==2024.11.6 # via transformers
151+
requests==2.32.3 # via datasets, huggingface-hub, jupyterlab-server, transformers
152+
rfc3339-validator==0.1.4 # via jsonschema, jupyter-events
153+
rfc3986-validator==0.1.1 # via jsonschema, jupyter-events
154+
rich==14.0.0 # via -r requirements.txt, trl
155+
rpds-py==0.25.1 # via jsonschema, referencing
156+
ruff==0.11.10 # via -r requirements-dev.txt
157+
safetensors==0.5.3 # via accelerate, instructlab-dolomite, peft, transformers
158+
send2trash==1.8.3 # via jupyter-server
159+
six==1.17.0 # via python-dateutil, rfc3339-validator
160+
sniffio==1.3.1 # via anyio
161+
soupsieve==2.7 # via beautifulsoup4
162+
stack-data==0.6.3 # via ipython
163+
sympy==1.14.0 # via torch
164+
terminado==0.18.1 # via jupyter-server, jupyter-server-terminals
165+
tinycss2==1.4.0 # via bleach
166+
tokenizers==0.21.1 # via transformers
167+
tomlkit==0.13.2 # via pylint
168+
torch==2.7.0 # via -r requirements.txt, accelerate, bitsandbytes, flash-attn, instructlab-dolomite, liger-kernel, peft
169+
tornado==6.5 # via ipykernel, jupyter-client, jupyter-server, jupyterlab, notebook, terminado
170+
tox==4.26.0 # via -r requirements-dev.txt, tox-current-env
171+
tox-current-env==0.0.16 # via -r requirements-dev.txt
172+
tqdm==4.67.1 # via datasets, huggingface-hub, peft, transformers
173+
traitlets==5.14.3 # via comm, ipykernel, ipython, ipywidgets, jupyter-client, jupyter-console, jupyter-core, jupyter-events, jupyter-server, jupyterlab, matplotlib-inline, nbclient, nbconvert, nbformat
174+
transformers==4.52.1 # via -r requirements.txt, instructlab-dolomite, peft, trl
175+
triton==3.3.0 # via liger-kernel, torch
176+
trl==0.17.0 # via -r requirements.txt
177+
types-python-dateutil==2.9.0.20250516 # via arrow
178+
types-pyyaml==6.0.12.20250516 # via -r requirements-dev.txt
179+
types-requests==2.32.0.20250515 # via types-tqdm
180+
types-tqdm==4.67.0.20250516 # via -r requirements-dev.txt
181+
typing-extensions==4.13.2 # via anyio, beautifulsoup4, huggingface-hub, ipython, mypy, pydantic, pydantic-core, referencing, torch, typing-inspection
182+
typing-inspection==0.4.0 # via pydantic
183+
tzdata==2025.2 # via pandas
184+
uri-template==1.3.0 # via jsonschema
185+
urllib3==2.4.0 # via requests, types-requests
186+
virtualenv==20.31.2 # via pre-commit, tox
187+
wcwidth==0.2.13 # via prompt-toolkit
188+
webcolors==24.11.1 # via jsonschema
189+
webencodings==0.5.1 # via bleach, tinycss2
190+
websocket-client==1.8.0 # via jupyter-server
191+
wheel==0.45.1 # via -r requirements.txt
192+
widgetsnbextension==4.0.14 # via ipywidgets
193+
xxhash==3.5.0 # via datasets
194+
yarl==1.20.0 # via aiohttp
195+
setuptools==80.8.0 # via jupyterlab, triton

‎constraints-dev.txt.in‎

Whitespace-only changes.

‎requirements-dev.txt‎

Lines changed: 11 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,15 +2,23 @@
22

33
-r requirements.txt
44

5+
isort
56
matplotlib
67
numpy
7-
pre-commit>=3.0.4,<5.0
8-
pylint>=2.16.2,<4.0
8+
pre-commit>=3.0.4
9+
pylint>=2.16.2
910
pylint-pydantic
10-
tox>=4.4.2,<5
11+
ruff
12+
tox>=4.4.2
13+
tox-current-env
1114

1215
ipython
1316
ipykernel
1417
jupyter
1518

1619
huggingface_hub
20+
21+
mypy>=1.10.0
22+
types-tqdm
23+
types-PyYAML
24+
pytest

‎scripts/generate_constraints.sh‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
#!/bin/sh
2+
set -e
3+
4+
# Require Linux; at least until we generate constraints files for each platform
5+
if [ "$(uname)" != "Linux" ]; then
6+
echo "This script is only supported on Linux."
7+
exit 1
8+
fi
9+
10+
CONSTRAINTS_FILE=constraints-dev.txt
11+
12+
pip-compile \
13+
--no-header \
14+
--annotate \
15+
--annotation-style line \
16+
--allow-unsafe \
17+
--strip-extras \
18+
--output-file=$CONSTRAINTS_FILE \
19+
--constraint constraints-dev.txt.in \
20+
requirements{,-dev,-cuda}.txt
21+
22+
# clean up empty lines and comments
23+
sed '/^#.*/d' -i constraints-dev.txt
24+
sed '/^$/d' -i constraints-dev.txt
25+
26+
# pip-compile lists -r requirements.txt twice for some reason: once with
27+
# relative path and once with absolute. Clean it up.
28+
sed -E 's/-r \/[^ ]+\/[^,]+, *//' -i $CONSTRAINTS_FILE

0 commit comments

Comments
 (0)