Skip to content

resolve failing tests with pt-2.1 #21130

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Borda merged 13 commits into from
Aug 29, 2025
Merged

resolve failing tests with pt-2.1 #21130

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
747bb19
debug: missing `nvcc` on images with pt-2.1
Borda Aug 29, 2025
5050c0c
drop deepspeed
Borda Aug 29, 2025
312778f
drop nvcc
Borda Aug 29, 2025
8c3329b
pip uninstall -y deepspeed
Borda Aug 29, 2025
a023307
TORCH_VER
Borda Aug 29, 2025
4f6859a
TORCH_VER
Borda Aug 29, 2025
eb2ff32
update
Borda Aug 29, 2025
f31b89d
Apply suggestions from code review
Borda Aug 29, 2025
b3cbd5d
A100_X_2
Borda Aug 29, 2025
e50826b
A100 & L4
Borda Aug 29, 2025
7e6ff69
typo
Borda Aug 29, 2025
abf4203
Empty-Commit
Borda Aug 29, 2025
fa770db
machine
Borda Aug 29, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 11 additions & 6 deletions .lightning/workflows/fabric.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,19 +4,19 @@ trigger:
pull_request:
branches: ["master"]

timeout: "75" # minutes
machine: "L4_X_2"
timeout: "55" # minutes
machine: "T4_X_4"
parametrize:
matrix: {}
include:
# note that this is setting also all oldest requirements which is linked to Torch == 2.0
# note that this is setting also all oldest requirements which is linked to Torch == 2.1
- image: "pytorchlightning/pytorch_lightning:base-cuda12.1.1-py3.10-torch2.1"
PACKAGE_NAME: "fabric"
- image: "pytorchlightning/pytorch_lightning:base-cuda12.6.3-py3.12-torch2.7"
- image: "pytorchlightning/pytorch_lightning:base-cuda12.6.3-py3.12-torch2.8"
PACKAGE_NAME: "fabric"
# - image: "pytorchlightning/pytorch_lightning:base-cuda12.6.3-py3.12-torch2.7"
# PACKAGE_NAME: "fabric"
- image: "pytorchlightning/pytorch_lightning:base-cuda12.6.3-py3.12-torch2.7"
- image: "pytorchlightning/pytorch_lightning:base-cuda12.6.3-py3.12-torch2.8"
PACKAGE_NAME: "lightning"
exclude: []

Expand All @@ -30,6 +30,7 @@ run: |
python --version
pip --version
pip install -q fire wget packaging
pip list
set -ex

CUDA_VERSION="${image##*cuda}" # Remove everything up to and including "cuda"
Expand All @@ -40,12 +41,15 @@ run: |
echo "Torch URL: ${TORCH_URL}"
COVERAGE_SOURCE=$(python -c 'n = "$(PACKAGE_NAME)" ; print(dict(fabric="lightning_fabric").get(n, n))')
echo "collecting coverage for: ${COVERAGE_SOURCE}"
TORCH_VER=$(python -c "import torch; print(torch.__version__.rsplit('.', 1)[0])")

if [ "${TORCH_VER}" == "2.1" ]; then
echo "Set oldest versions"
cd requirements/fabric
pip uninstall -y deepspeed
pip install -U "lightning-utilities[cli]"
cd requirements/fabric
python -m lightning_utilities.cli requirements set-oldest --req_files "['base.txt', 'strategies.txt']"
python -m lightning_utilities.cli requirements prune-pkgs --packages deepspeed --req_files strategies.txt
cd ../..
pip install "cython<3.0" wheel # for compatibility
fi
Expand Down Expand Up @@ -92,6 +96,7 @@ run: |
export PL_RUN_STANDALONE_TESTS=1
wget https://raw.githubusercontent.com/Lightning-AI/utilities/main/scripts/run_standalone_tests.sh
bash ./run_standalone_tests.sh "tests_fabric"
export PL_RUN_STANDALONE_TESTS=0

# echo "Reporting coverage" # todo
# python -m coverage report
Expand Down
17 changes: 11 additions & 6 deletions .lightning/workflows/pytorch.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,19 +4,19 @@ trigger:
pull_request:
branches: ["master"]

timeout: "75" # minutes
machine: "L4_X_2"
timeout: "55" # minutes
machine: "T4_X_4"
parametrize:
matrix: {}
include:
# note that this is setting also all oldest requirements which is linked to Torch == 2.0
# note that this is setting also all oldest requirements which is linked to Torch == 2.1
- image: "pytorchlightning/pytorch_lightning:base-cuda12.1.1-py3.10-torch2.1"
PACKAGE_NAME: "pytorch"
- image: "pytorchlightning/pytorch_lightning:base-cuda12.6.3-py3.12-torch2.7"
- image: "pytorchlightning/pytorch_lightning:base-cuda12.6.3-py3.12-torch2.8"
PACKAGE_NAME: "pytorch"
# - image: "pytorchlightning/pytorch_lightning:base-cuda12.6.3-py3.12-torch2.7"
# PACKAGE_NAME: "pytorch"
- image: "pytorchlightning/pytorch_lightning:base-cuda12.6.3-py3.12-torch2.7"
- image: "pytorchlightning/pytorch_lightning:base-cuda12.6.3-py3.12-torch2.8"
PACKAGE_NAME: "lightning"
exclude: []

Expand All @@ -30,6 +30,7 @@ run: |
python --version
pip --version
pip install -q fire wget packaging
pip list
set -ex

CUDA_VERSION="${image##*cuda}" # Remove everything up to and including "cuda"
Expand All @@ -40,12 +41,15 @@ run: |
echo "Torch URL: ${TORCH_URL}"
COVERAGE_SOURCE=$(python -c 'n = "$(PACKAGE_NAME)" ; print(dict(fabric="pytorch_lightning").get(n, n))')
echo "collecting coverage for: ${COVERAGE_SOURCE}"
TORCH_VER=$(python -c "import torch; print(torch.__version__.rsplit('.', 1)[0])")

if [ "${TORCH_VER}" == "2.1" ]; then
recho "Set oldest versions"
cd requirements/pytorch
pip uninstall -y deepspeed
pip install -U "lightning-utilities[cli]"
cd requirements/pytorch
python -m lightning_utilities.cli requirements set-oldest --req_files "['base.txt', 'extra.txt', 'strategies.txt', 'examples.txt']"
python -m lightning_utilities.cli requirements prune-pkgs --packages deepspeed --req_files strategies.txt
cd ../..
pip install "cython<3.0" wheel # for compatibility
fi
Expand Down Expand Up @@ -108,6 +112,7 @@ run: |
export PL_RUN_STANDALONE_TESTS=1
wget https://raw.githubusercontent.com/Lightning-AI/utilities/main/scripts/run_standalone_tests.sh
bash ./run_standalone_tests.sh "tests_pytorch"
export PL_RUN_STANDALONE_TESTS=0

echo "Testing: PyTorch standalone tasks"
cd tests_pytorch/
Expand Down
2 changes: 1 addition & 1 deletion dockers/base-cuda/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
# limitations under the License.

ARG UBUNTU_VERSION=22.04
ARG CUDA_VERSION=11.7.1
ARG CUDA_VERSION=12.1.1


FROM nvidia/cuda:${CUDA_VERSION}-runtime-ubuntu${UBUNTU_VERSION}
Expand Down
Loading