Merge branch 'master' into ci/lit

Borda · web-flow · commit 2c2a76640aeb · 2025-08-04T21:00:41.000+03:00
diff --git a/.github/CONTRIBUTING.md b/.github/CONTRIBUTING.md
@@ -3,7 +3,7 @@
 Welcome to the PyTorch Lightning community! We're building the most advanced research platform on the planet to implement the latest, best practices
 and integrations that the amazing PyTorch team and other research organization rolls out!
 
-If you are new to open source, check out [this blog to get started with your first Open Source contribution](https://devblog.pytorchlightning.ai/quick-contribution-guide-86d977171b3a).
+If you are new to open source, check out [this blog to get started with your first Open Source contribution](https://medium.com/pytorch-lightning/quick-contribution-guide-86d977171b3a).
 
 ## Main Core Value: One less thing to remember
 
diff --git a/.github/markdown-links-config.json b/.github/markdown-links-config.json
@@ -22,5 +22,9 @@
         "Accept-Encoding": "zstd, br, gzip, deflate"
       }
     }
-  ]
+  ],
+  "timeout": "20s",
+  "retryOn429": true,
+  "retryCount": 5,
+  "fallbackRetryDelay": "20s"
 }
diff --git a/.github/workflows/call-clear-cache.yml b/.github/workflows/call-clear-cache.yml
@@ -23,7 +23,7 @@ on:
 jobs:
   cron-clear:
     if: github.event_name == 'schedule' || github.event_name == 'pull_request'
-    uses: Lightning-AI/utilities/.github/workflows/cleanup-caches.yml@v0.14.3
+    uses: Lightning-AI/utilities/.github/workflows/cleanup-caches.yml@v0.15.0
     with:
       scripts-ref: v0.14.3
       dry-run: ${{ github.event_name == 'pull_request' }}
@@ -32,7 +32,7 @@ jobs:
 
   direct-clear:
     if: github.event_name == 'workflow_dispatch' || github.event_name == 'pull_request'
-    uses: Lightning-AI/utilities/.github/workflows/cleanup-caches.yml@v0.14.3
+    uses: Lightning-AI/utilities/.github/workflows/cleanup-caches.yml@v0.15.0
     with:
       scripts-ref: v0.14.3
       dry-run: ${{ github.event_name == 'pull_request' }}
diff --git a/.github/workflows/ci-schema.yml b/.github/workflows/ci-schema.yml
@@ -8,7 +8,7 @@ on:
 
 jobs:
   check:
-    uses: Lightning-AI/utilities/.github/workflows/check-schema.yml@v0.14.3
+    uses: Lightning-AI/utilities/.github/workflows/check-schema.yml@v0.15.0
     with:
       # skip azure due to the wrong schema file by MSFT
       # https://github.com/Lightning-AI/lightning-flash/pull/1455#issuecomment-1244793607
diff --git a/README.md b/README.md
@@ -55,6 +55,12 @@ ______________________________________________________________________
 
 &nbsp;
 
+# Why PyTorch Lightning?   
+
+Training models in plain PyTorch is tedious and error-prone - you have to manually handle things like backprop, mixed precision, multi-GPU, and distributed training, often rewriting code for every new project. PyTorch Lightning organizes PyTorch code to automate those complexities so you can focus on your model and data, while keeping full control and scaling from CPU to multi-node without changing your core code. But if you want control of those things, you can still opt into more DIY.   
+
+Fun analogy: If PyTorch is Javascript, PyTorch Lightning is ReactJS or NextJS.
+
 # Lightning has 2 core packages
 
 [PyTorch Lightning: Train and deploy PyTorch at scale](#why-pytorch-lightning).
diff --git a/_notebooks b/_notebooks
@@ -1 +1 @@
-Subproject commit fd70f5114b21f7f970bd5587b1d3def689507069
+Subproject commit 99ba2b407843a846210b64a8312bbf4ee33e51a5
diff --git a/docs/source-fabric/advanced/model_parallel/tp_fsdp.rst b/docs/source-fabric/advanced/model_parallel/tp_fsdp.rst
@@ -9,7 +9,7 @@ The :doc:`Tensor Parallelism documentation <tp>` and a general understanding of
 
 .. raw:: html
 
-    <a target="_blank" href="https://lightning.ai/lightning-ai/studios/tensor-parallelism-supercharging-large-model-training-with-lightning-fabric">
+    <a target="_blank" href="https://lightning.ai/lightning-ai/studios/pretrain-an-llm-with-pytorch-lightning">
       <img src="https://pl-bolts-doc-images.s3.us-east-2.amazonaws.com/app-2/studio-badge.svg" alt="Open In Studio" style="width: auto; max-width: none;"/>
     </a>
 
diff --git a/docs/source-pytorch/advanced/model_parallel/tp.rst b/docs/source-pytorch/advanced/model_parallel/tp.rst
@@ -8,7 +8,7 @@ This method is most effective for models with very large layers, significantly e
 
 .. raw:: html
 
-    <a target="_blank" href="https://lightning.ai/lightning-ai/studios/tensor-parallelism-supercharging-large-model-training-with-pytorch-lightning">
+    <a target="_blank" href="https://lightning.ai/lightning-ai/studios/pretrain-an-llm-with-pytorch-lightning">
       <img src="https://pl-bolts-doc-images.s3.us-east-2.amazonaws.com/app-2/studio-badge.svg" alt="Open In Studio" style="width: auto; max-width: none;"/>
     </a>
 
diff --git a/requirements/ci.txt b/requirements/ci.txt
@@ -1,6 +1,6 @@
 setuptools <80.9.1
 wheel <0.46.0
-awscli >=1.30.0, <1.42.0
+awscli >=1.30.0, <1.43.0
 twine ==6.1.0
 importlib-metadata <9.0.0
 wget
diff --git a/requirements/docs.txt b/requirements/docs.txt
@@ -3,7 +3,7 @@ myst-parser >=0.18.1, <4.0.0
 nbsphinx >=0.8.5, <=0.9.7
 nbconvert >7.14, <7.17
 pandoc >=1.0, <=2.4
-docutils >=0.16, <0.22
+docutils>=0.18.1,<=0.22
 sphinxcontrib-fulltoc >=1.0, <=1.2.0
 sphinxcontrib-mockautodoc
 sphinx-autobuild
@@ -17,7 +17,7 @@ sphinx-rtd-dark-mode
 sphinxcontrib-video ==0.4.1
 jinja2 <3.2.0
 
-lightning-utilities >=0.11.1, <0.15.0
+lightning-utilities >=0.11.1, <0.16.0
 
 # installed from S3 location and fetched in advance
 lai-sphinx-theme
diff --git a/requirements/fabric/base.txt b/requirements/fabric/base.txt
@@ -2,7 +2,7 @@
 #  in case you want to preserve/enforce restrictions on the latest compatible version, add "strict" as an in-line comment
 
 torch >=2.1.0, <2.8.0
-fsspec[http] >=2022.5.0, <2025.6.0
+fsspec[http] >=2022.5.0, <2025.8.0
 packaging >=20.0, <=25.0
 typing-extensions >=4.5.0, <4.15.0
-lightning-utilities >=0.10.0, <0.15.0
+lightning-utilities >=0.10.0, <0.16.0
diff --git a/requirements/fabric/examples.txt b/requirements/fabric/examples.txt
@@ -2,4 +2,4 @@
 #  in case you want to preserve/enforce restrictions on the latest compatible version, add "strict" as an in-line comment
 
 torchvision >=0.16.0, <0.23.0
-torchmetrics >=0.10.0, <1.8.0
+torchmetrics >=0.10.0, <1.9.0
diff --git a/requirements/fabric/test.txt b/requirements/fabric/test.txt
@@ -1,4 +1,4 @@
-coverage ==7.9.2
+coverage ==7.10.2
 numpy >=1.17.2, <1.27.0
 pytest ==8.4.1
 pytest-cov ==6.2.1
diff --git a/requirements/pytorch/base.txt b/requirements/pytorch/base.txt
@@ -1,11 +1,11 @@
 # NOTE: the upper bound for the package version is only set for CI stability, and it is dropped while installing this package
 #  in case you want to preserve/enforce restrictions on the latest compatible version, add "strict" as an in-line comment
 
-torch >=2.1.0, <2.8.0
+torch >=2.1.0, <=2.8.0
 tqdm >=4.57.0, <4.68.0
 PyYAML >5.4, <6.1.0
-fsspec[http] >=2022.5.0, <2025.6.0
-torchmetrics >0.7.0, <1.8.0
+fsspec[http] >=2022.5.0, <2025.8.0
+torchmetrics >0.7.0, <1.9.0
 packaging >=20.0, <=25.0
 typing-extensions >=4.5.0, <4.15.0
-lightning-utilities >=0.10.0, <0.15.0
+lightning-utilities >=0.10.0, <0.16.0
diff --git a/requirements/pytorch/examples.txt b/requirements/pytorch/examples.txt
@@ -4,4 +4,4 @@
 requests <2.33.0
 torchvision >=0.16.0, <0.23.0
 ipython[all] <8.19.0
-torchmetrics >=0.10.0, <1.8.0
+torchmetrics >=0.10.0, <1.9.0
diff --git a/requirements/pytorch/extra.txt b/requirements/pytorch/extra.txt
@@ -6,6 +6,6 @@ matplotlib>3.1, <3.10.0
 omegaconf >=2.2.3, <2.4.0
 hydra-core >=1.2.0, <1.4.0
 jsonargparse[signatures,jsonnet] >=4.39.0, <4.41.0
-rich >=12.3.0, <14.1.0
+rich >=12.3.0, <14.2.0
 tensorboardX >=2.2, <2.7.0  # min version is set by torch.onnx missing attribute
 bitsandbytes >=0.45.2,<0.47.0; platform_system != "Darwin"
diff --git a/requirements/pytorch/test.txt b/requirements/pytorch/test.txt
@@ -1,4 +1,4 @@
-coverage ==7.9.2
+coverage ==7.10.2
 pytest ==8.4.1
 pytest-cov ==6.2.1
 pytest-timeout ==2.4.0
@@ -16,4 +16,4 @@ pandas >2.0, <2.4.0  # needed in benchmarks
 fastapi  # for `ServableModuleValidator`  # not setting version as re-defined in App
 uvicorn  # for `ServableModuleValidator`  # not setting version as re-defined in App
 
-tensorboard >=2.9.1, <2.20.0  # for `TensorBoardLogger`
+tensorboard >=2.9.1, <2.21.0  # for `TensorBoardLogger`
diff --git a/requirements/typing.txt b/requirements/typing.txt
@@ -1,4 +1,4 @@
-mypy==1.16.1
+mypy==1.17.1
 torch==2.7.1
 
 types-Markdown
diff --git a/src/lightning/pytorch/CHANGELOG.md b/src/lightning/pytorch/CHANGELOG.md
@@ -25,7 +25,7 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
 
 ### Fixed
 
--
+- fix progress bar console clearing for Rich `14.1+` ([#21016](https://github.com/Lightning-AI/pytorch-lightning/pull/21016))
 
 
 ---
diff --git a/src/lightning/pytorch/callbacks/model_checkpoint.py b/src/lightning/pytorch/callbacks/model_checkpoint.py
@@ -133,9 +133,15 @@ class ModelCheckpoint(Checkpoint):
             will only save checkpoints at epochs 0 < E <= N
             where both values for ``every_n_epochs`` and ``check_val_every_n_epoch`` evenly divide E.
         save_on_train_epoch_end: Whether to run checkpointing at the end of the training epoch.
-            If this is ``False``, then the check runs at the end of the validation.
+            If ``True``, checkpoints are saved at the end of every training epoch.
+            If ``False``, checkpoints are saved at the end of validation.
+            If ``None`` (default), checkpointing behavior is determined based on training configuration.
+            If ``check_val_every_n_epoch != 1``, checkpointing will not be performed at the end of
+            every training epoch. If there are no validation batches of data, checkpointing will occur at the
+            end of the training epoch. If there is a non-default number of validation runs per training epoch
+            (``val_check_interval != 1``), checkpointing is performed after validation.
         enable_version_counter: Whether to append a version to the existing file name.
-            If this is ``False``, then the checkpoint files will be overwritten.
+            If ``False``, then the checkpoint files will be overwritten.
 
     Note:
         For extra customization, ModelCheckpoint includes the following attributes:
diff --git a/src/lightning/pytorch/callbacks/progress/rich_progress.py b/src/lightning/pytorch/callbacks/progress/rich_progress.py
@@ -331,7 +331,19 @@ def _init_progress(self, trainer: "pl.Trainer") -> None:
             self._reset_progress_bar_ids()
             reconfigure(**self._console_kwargs)
             self._console = get_console()
-            self._console.clear_live()
+
+            # Compatibility shim for Rich >= 14.1.0:
+            if hasattr(self._console, "_live_stack"):
+                # In recent Rich releases, the internal `_live` variable was replaced with `_live_stack` (a list)
+                # to support nested Live displays. This broke our original call to `clear_live()`,
+                # because it now only pops one Live instance instead of clearing them all.
+                # We check for `_live_stack` and clear it manually for compatibility across
+                # both old and new Rich versions.
+                if len(self._console._live_stack) > 0:
+                    self._console.clear_live()
+            else:
+                self._console.clear_live()
+
             self._metric_component = MetricsTextColumn(
                 trainer,
                 self.theme.metrics,
@@ -447,6 +459,11 @@ def _add_task(self, total_batches: Union[int, float], description: str, visible:
             visible=visible,
         )
 
+    def _initialize_train_progress_bar_id(self) -> None:
+        total_batches = self.total_train_batches
+        train_description = self._get_train_description(self.trainer.current_epoch)
+        self.train_progress_bar_id = self._add_task(total_batches, train_description)
+
     def _update(self, progress_bar_id: Optional["TaskID"], current: int, visible: bool = True) -> None:
         if self.progress is not None and self.is_enabled:
             assert progress_bar_id is not None
@@ -531,6 +548,9 @@ def on_train_batch_end(
         batch: Any,
         batch_idx: int,
     ) -> None:
+        if not self.is_disabled and self.train_progress_bar_id is None:
+            # can happen when resuming from a mid-epoch restart
+            self._initialize_train_progress_bar_id()
         self._update(self.train_progress_bar_id, batch_idx + 1)
         self._update_metrics(trainer, pl_module)
         self.refresh()
diff --git a/src/lightning/pytorch/core/module.py b/src/lightning/pytorch/core/module.py
@@ -466,10 +466,10 @@ def log(
         )
 
         # make sure user doesn't introduce logic for multi-dataloaders
-        if "/dataloader_idx_" in name:
+        if add_dataloader_idx and "/dataloader_idx_" in name:
             raise MisconfigurationException(
                 f"You called `self.log` with the key `{name}`"
-                " but it should not contain information about `dataloader_idx`"
+                " but it should not contain information about `dataloader_idx` when `add_dataloader_idx=True`"
             )
 
         value = apply_to_collection(value, (Tensor, numbers.Number), self.__to_tensor, name)
diff --git a/src/lightning/pytorch/trainer/connectors/accelerator_connector.py b/src/lightning/pytorch/trainer/connectors/accelerator_connector.py
@@ -453,10 +453,11 @@ def _check_strategy_and_fallback(self) -> None:
 
         if (
             strategy_flag in FSDPStrategy.get_registered_strategies() or type(self._strategy_flag) is FSDPStrategy
-        ) and self._accelerator_flag not in ("cuda", "gpu"):
+        ) and not (self._accelerator_flag in ("cuda", "gpu") or isinstance(self._accelerator_flag, CUDAAccelerator)):
             raise ValueError(
-                f"The strategy `{FSDPStrategy.strategy_name}` requires a GPU accelerator, but got:"
-                f" {self._accelerator_flag}"
+                f"The strategy `{FSDPStrategy.strategy_name}` requires a GPU accelerator, but received "
+                f"`accelerator={self._accelerator_flag!r}`. Please set `accelerator='cuda'`, `accelerator='gpu'`,"
+                " or pass a `CUDAAccelerator()` instance to use FSDP."
             )
         if strategy_flag in _DDP_FORK_ALIASES and "fork" not in torch.multiprocessing.get_all_start_methods():
             raise ValueError(
diff --git a/src/pytorch_lightning/README.md b/src/pytorch_lightning/README.md
@@ -370,7 +370,7 @@ The PyTorch Lightning community is maintained by
 - [10+ core contributors](https://lightning.ai/docs/pytorch/stable/community/governance.html) who are all a mix of professional engineers, Research Scientists, and Ph.D. students from top AI labs.
 - 680+ active community contributors.
 
-Want to help us build Lightning and reduce boilerplate for thousands of researchers? [Learn how to make your first contribution here](https://devblog.pytorchlightning.ai/quick-contribution-guide-86d977171b3a)
+Want to help us build Lightning and reduce boilerplate for thousands of researchers? [Learn how to make your first contribution here](https://medium.com/pytorch-lightning/quick-contribution-guide-86d977171b3a)
 
 PyTorch Lightning is also part of the [PyTorch ecosystem](https://pytorch.org/ecosystem/) which requires projects to have solid testing, documentation and support.
 
diff --git a/tests/README.md b/tests/README.md
@@ -26,7 +26,7 @@ Additionally, for testing backward compatibility with older versions of PyTorch
 bash .actions/pull_legacy_checkpoints.sh
 ```
 
-Note: These checkpoints are generated to set baselines for maintaining backward compatibility with legacy versions of PyTorch Lightning. Details of checkpoints for back-compatibility can be found [here](https://github.com/Lightning-AI/pytorch-lightning/blob/master/tests/legacy/README.md).
+Note: These checkpoints are generated to set baselines for maintaining backward compatibility with legacy versions of PyTorch Lightning. Details of checkpoints for back-compatibility can be found [here](https://github.com/Lightning-AI/pytorch-lightning/tree/master/tests/legacy/README.md).
 
 You can run the full test suite in your terminal via this make script:
 
diff --git a/tests/tests_pytorch/trainer/connectors/test_accelerator_connector.py b/tests/tests_pytorch/trainer/connectors/test_accelerator_connector.py
@@ -582,6 +582,11 @@ class AcceleratorSubclass(CPUAccelerator):
     Trainer(accelerator=AcceleratorSubclass(), strategy=FSDPStrategySubclass())
 
 
+@RunIf(min_cuda_gpus=1)
+def test_check_fsdp_strategy_and_fallback_with_cudaaccelerator():
+    Trainer(strategy="fsdp", accelerator=CUDAAccelerator())
+
+
 @mock.patch.dict(os.environ, {}, clear=True)
 def test_unsupported_tpu_choice(xla_available, tpu_available):
     # if user didn't set strategy, _Connector will choose the SingleDeviceXLAStrategy or XLAStrategy

Original file line number	Diff line number	Diff line change
`@@ -22,5 +22,9 @@`
`22`	`22`	`"Accept-Encoding": "zstd, br, gzip, deflate"`
`23`	`23`	`}`
`24`	`24`	`}`
`25`		`- ]`
	`25`	`+ ],`
	`26`	`+ "timeout": "20s",`
	`27`	`+ "retryOn429": true,`
	`28`	`+ "retryCount": 5,`
	`29`	`+ "fallbackRetryDelay": "20s"`
`26`	`30`	`}`