Releasing/1.8.4 extra2 (#15960)

* [docs] Include all components in the API reference (#15805)

* Update docs

Co-authored-by: Jirka Borovec <[email protected]>
(cherry picked from commit e6f4c84)

* Bump playwright from 1.27.1 to 1.28.0 in /requirements (#15903)

* Bump playwright from 1.27.1 to 1.28.0 in /requirements

Bumps [playwright](https://github.com/Microsoft/playwright-python) from 1.27.1 to 1.28.0.
- [Release notes](https://github.com/Microsoft/playwright-python/releases)
- [Commits](microsoft/playwright-python@v1.27.1...v1.28.0)

---
updated-dependencies:
- dependency-name: playwright
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

* 1.28

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Jirka <[email protected]>
(cherry picked from commit 73a6dbe)

* [App] Add `configure_layout` method for works (#15926)

* Add `configure_layout` method for works
* Check for api access availability
* Updates from review
* Update CHANGELOG.md
* Apply suggestions from code review

Co-authored-by: Sherin Thomas <[email protected]>
(cherry picked from commit d5b9c67)

* Don't try to aggregate `requirements/__pycache__/base.txt` in setuptools (#15775)

Exlucde __pycache__ in setuptools

(cherry picked from commit ca5ca0e)

* [App] Multiprocessing-safe work pickling (#15836)

(cherry picked from commit df67833)

* Multinode on MPS (#15748)

* Fix restarting attribute for lr finder
* update lite executor
* update trainer executor
* update spawn executor
* add multinode component tests
* add testing helpers
* add lite tests
* add trainer tests
* update changelog
* update trainer
* update workflow
* update tests
* debug
* add reason for skipif
* Apply suggestions from code review
* switch skipif

Co-authored-by: Jirka <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Carlos Mocholí <[email protected]>
Co-authored-by: Adrian Wälchli <[email protected]>
Co-authored-by: Jirka Borovec <[email protected]>

(cherry picked from commit 36aecde)

* [App] Resolve PythonServer on M1 (#15949)

Co-authored-by: thomas <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

(cherry picked from commit 904323b)

* Lite: Fix DataLoader shuffling when using DistributedSampler (#15931)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

(cherry picked from commit 3004f13)

* [App] Temporarily disable ready (#15958)

(cherry picked from commit d0b101c)

* update date

* Fix restarting attribute for lr finder (#15620)

(cherry picked from commit 15184c6)

* [App] Improve pdb for multiprocessing (#15950)

Co-authored-by: thomas <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
(cherry picked from commit 482b279)

* [App] Improve debug triggering (#15951)

(cherry picked from commit 772d121)

* [App] Add automatic conversion to structures (#15961)

(cherry picked from commit 67a47d4)

* Make LightningModule torch.jit.script-able again (#15947)

* Make LightningModule torch.jit.script-able again
* remove skip

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
(cherry picked from commit b5fa896)

* Fix ImportErrors on Multinode if package not present (#15963)

(cherry picked from commit cbd4dd6)

* Fix typo in definition of world size in docs (#15954)

(cherry picked from commit 7a1e0e8)

* [App] Enable running an app from the Gallery (#15941)

Co-authored-by: thomas <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Ethan Harris <[email protected]>
Co-authored-by: Jirka <[email protected]>

(cherry picked from commit 4983083)

Co-authored-by: Akihiro Nitta <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Ethan Harris <[email protected]>
Co-authored-by: Sherin Thomas <[email protected]>
Co-authored-by: Justus Schock <[email protected]>
Co-authored-by: thomas chaton <[email protected]>
Co-authored-by: Adrian Wälchli <[email protected]>

Author: 8 people

.actions/setup_tools.py

@@ -179,7 +179,7 @@ def _load_aggregate_requirements(req_dir: str = "requirements", freeze_requireme
         load_requirements(d, file_name="base.txt", unfreeze=not freeze_requirements)
         for d in glob.glob(os.path.join(req_dir, "*"))
         # skip empty folder as git artefacts, and resolving Will's special issue
-        if os.path.isdir(d) and len(glob.glob(os.path.join(d, "*"))) > 0
+        if os.path.isdir(d) and len(glob.glob(os.path.join(d, "*"))) > 0 and "__pycache__" not in d
     ]
     if not requires:
         return None

.azure/app-cloud-e2e.yml

@@ -51,7 +51,7 @@ jobs:
   - job: App_cloud_e2e_testing
     pool: azure-cpus
     container:
-      image: mcr.microsoft.com/playwright/python:v1.27.1-focal
+      image: mcr.microsoft.com/playwright/python:v1.28.0-focal
       options: "--shm-size=4gb"
     strategy:
       matrix:

.github/workflows/ci-app-tests.yml

@@ -99,7 +99,7 @@ jobs:
 
       - name: Adjust tests
         if: ${{ matrix.pkg-name == 'lightning' }}
-        run: python .actions/assistant.py copy_replace_imports --source_dir="./tests" --source_import="lightning_app" --target_import="lightning.app"
+        run: python .actions/assistant.py copy_replace_imports --source_dir="./tests" --source_import="lightning_app,lightning_lite,pytorch_lightning" --target_import="lightning.app,lightning.lite,lightning.pytorch"
 
       - name: Adjust examples
         if: ${{ matrix.pkg-name != 'lightning' }}

docs/source-app/api_references.rst

@@ -32,11 +32,19 @@ ___________________
     :nosignatures:
     :template: classtemplate_no_index.rst
 
+    ~database.client.DatabaseClient
+    ~database.server.Database
     ~python.popen.PopenPythonScript
     ~python.tracer.TracerPythonScript
     ~training.LightningTrainerScript
     ~serve.gradio.ServeGradio
     ~serve.serve.ModelInferenceAPI
+    ~serve.python_server.PythonServer
+    ~serve.streamlit.ServeStreamlit
+    ~multi_node.base.MultiNode
+    ~multi_node.lite.LiteMultiNode
+    ~multi_node.pytorch_spawn.PyTorchSpawnMultiNode
+    ~multi_node.trainer.LightningTrainerMultiNode
     ~auto_scaler.AutoScaler
 
 ----

docs/source-pytorch/clouds/cluster_intermediate_1.rst

@@ -24,7 +24,7 @@ PyTorch Lightning follows the design of `PyTorch distributed communication packa
 
 - *MASTER_PORT* - required; has to be a free port on machine with NODE_RANK 0
 - *MASTER_ADDR* - required (except for NODE_RANK 0); address of NODE_RANK 0 node
-- *WORLD_SIZE* - required; how many nodes are in the cluster
+- *WORLD_SIZE* - required; the total number of GPUs/processes that you will use
 - *NODE_RANK* - required; id of the node in the cluster
 
 .. _training_script_setup:

examples/app_dag/requirements.txt

@@ -1,2 +1,2 @@
-sklearn
+scikit-learn
 pandas

pyproject.toml

@@ -30,14 +30,12 @@ files = [
     "src/lightning_lite",
     "src/lightning_app",
 ]
+# This section is for folders with "-" as they are not valid python modules
 exclude = [
+    "src/lightning_app/cli/app-template",
     "src/lightning_app/cli/component-template",
     "src/lightning_app/cli/pl-app-template",
     "src/lightning_app/cli/react-ui-template",
-    "src/lightning_app/cli/app-template",
-    "src/lightning_app/components/database",
-    "src/lightning_app/components/multi_node",
-    "src/lightning_app/frontend/just_py/just_py",
 ]
 install_types = "True"
 non_interactive = "True"
@@ -67,7 +65,9 @@ module = [
     "lightning_app.api.request_types",
     "lightning_app.cli.commands.app_commands",
     "lightning_app.cli.commands.connection",
-    "lightning_app.cli.react-ui-template.example_app",
+    "lightning_app.cli.commands.lightning_cli",
+    "lightning_app.cli.commands.cmd_install",
+    "lightning_app.cli.cmd_install",
     "lightning_app.components.database.client",
     "lightning_app.components.database.server",
     "lightning_app.components.database.utilities",
@@ -94,6 +94,7 @@ module = [
     "lightning_app.frontend.streamlit_base",
     "lightning_app.frontend.utils",
     "lightning_app.frontend.web",
+    "lightning_app.perf.pdb",
     "lightning_app.runners.backends.__init__",
     "lightning_app.runners.backends.backend",
     "lightning_app.runners.backends.cloud",

requirements/app/test.txt

@@ -4,7 +4,7 @@ pytest==7.2.0
 pytest-timeout==2.1.0
 pytest-cov==4.0.0
 pytest-doctestplus>=0.9.0
-playwright==1.27.1
+playwright==1.28.0
 httpx
 trio<0.22.0
 pympler

requirements/app/ui.txt

@@ -1,2 +1,2 @@
-streamlit>=1.3.1, <=1.11.1
+streamlit>=1.0.0, <=1.15.2
 panel>=0.12.7, <=0.13.1

src/lightning_app/CHANGELOG.md

@@ -4,18 +4,21 @@ All notable changes to this project will be documented in this file.
 
 The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
 
-## [1.8.4] - 2022-12-06
+## [1.8.4] - 2022-12-08
 
 ### Added
 
 - Add `code_dir` argument to tracer run ([#15771](https://github.com/Lightning-AI/lightning/pull/15771))
 - Added the CLI command `lightning run model` to launch a `LightningLite` accelerated script ([#15506](https://github.com/Lightning-AI/lightning/pull/15506))
 - Added the CLI command `lightning delete app` to delete a lightning app on the cloud ([#15783](https://github.com/Lightning-AI/lightning/pull/15783))
 - Added a CloudMultiProcessBackend which enables running a child App from within the Flow in the cloud ([#15800](https://github.com/Lightning-AI/lightning/pull/15800))
+- Utility for pickling work object safely even from a child process ([#15836](https://github.com/Lightning-AI/lightning/pull/15836))
 - Added `AutoScaler` component ([#15769](https://github.com/Lightning-AI/lightning/pull/15769))
 - Added the property `ready` of the LightningFlow to inform when the `Open App` should be visible ([#15921](https://github.com/Lightning-AI/lightning/pull/15921))
 - Added private work attributed `_start_method` to customize how to start the works ([#15923](https://github.com/Lightning-AI/lightning/pull/15923))
-
+- Added a `configure_layout` method to the `LightningWork` which can be used to control how the work is handled in the layout of a parent flow ([#15926](https://github.com/Lightning-AI/lightning/pull/15926))
+- Added the ability to run a Lightning App or Component directly from the Gallery using `lightning run app organization/name` ([#15941](https://github.com/Lightning-AI/lightning/pull/15941))
+- Added automatic conversion of list and dict of works and flows to structures ([#15961](https://github.com/Lightning-AI/lightning/pull/15961))
 
 ### Changed
 
@@ -37,13 +40,16 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
 - Fixed SSH CLI command listing stopped components ([#15810](https://github.com/Lightning-AI/lightning/pull/15810))
 - Fixed bug when launching apps on multiple clusters ([#15484](https://github.com/Lightning-AI/lightning/pull/15484))
 - Fixed Sigterm Handler causing thread lock which caused KeyboardInterrupt to hang ([#15881](https://github.com/Lightning-AI/lightning/pull/15881))
+- Fixed MPS error for multinode component (defaults to cpu on mps devices now as distributed operations are not supported by pytorch on mps) ([#15748](https://github.com/Ligtning-AI/lightning/pull/15748))
 - Fixed the work not stopped when successful when passed directly to the LightningApp ([#15801](https://github.com/Lightning-AI/lightning/pull/15801))
 - Fixed the PyTorch Inference locally on GPU ([#15813](https://github.com/Lightning-AI/lightning/pull/15813))
 - Fixed the `enable_spawn` method of the `WorkRunExecutor` ([#15812](https://github.com/Lightning-AI/lightning/pull/15812))
 - Fixed require/import decorator ([#15849](https://github.com/Lightning-AI/lightning/pull/15849))
-
 - Fixed a bug where using `L.app.structures` would cause multiple apps to be opened and fail with an error in the cloud ([#15911](https://github.com/Lightning-AI/lightning/pull/15911))
-
+- Fixed PythonServer generating noise on M1 ([#15949](https://github.com/Lightning-AI/lightning/pull/15949))
+- Fixed multiprocessing breakpoint ([#15950](https://github.com/Lightning-AI/lightning/pull/15950))
+- Fixed detection of a Lightning App running in debug mode ([#15951](https://github.com/Lightning-AI/lightning/pull/15951))
+- Fixed `ImportError` on Multinode if package not present ([#15963](https://github.com/Lightning-AI/lightning/pull/15963))
 
 ## [1.8.3] - 2022-11-22