Releasing/1.8.4.post (#15988)

* Apply dynamo to training_step, validation_step, test_step, predict_step (#15957)

* Apply dynamo to training_step, validation_step, test_step, predict_step

* Add entry to CHANGELOG.md

(cherry picked from commit edc9986)

* [App] Resolve run installation (#15974)

(cherry picked from commit dd83587)

* App: Move AutoScaler dependency to extra requirements (#15971)

* Make autoscaler dependency optional
* update chglog
* dont directly import aiohttp

(cherry picked from commit 346e936)

# Conflicts:
#	requirements/app/base.txt
#	src/lightning_app/CHANGELOG.md

* Avoid using the same port number for autoscaler works (#15966)

* dont hardcode port in python server
* add another chglog

(cherry picked from commit a72d268)

* Fix `action_name` usage in `XLAProfiler` (#15886)

* Fix `action_name` usage in `XLAProfiler`
* add changelog
* Update src/pytorch_ligh
* Update xla.py

Co-authored-by: awaelchli <[email protected]>
Co-authored-by: Jirka Borovec <[email protected]>

(cherry picked from commit c748f82)

* Fix multinode cloud component (#15965)

* fix multinode cloud component
* add tests

(cherry picked from commit d21b899)

* ci: update signaling (#15981)

* ci: update signaling
* config

(cherry picked from commit e56e7f1)

* Fix cloudcomputes registration for structures (#15964)

* fix cloudcomputes
* updates cloudcompute registration
* changelog

(cherry picked from commit 90a4c02)

* Document running dev lightning on the cloud (#15962)

* document running dev lightning on the cloud

* document running dev lightning on the cloud

* Update .github/CONTRIBUTING.md

Co-authored-by: Noha Alon <[email protected]>

* document running dev lightning on the cloud

* git clone & pip install -e

* Update .github/CONTRIBUTING.md

Co-authored-by: Jirka Borovec <[email protected]>

Co-authored-by: Noha Alon <[email protected]>
Co-authored-by: Jirka Borovec <[email protected]>
(cherry picked from commit cfd00d3)

* [App] Install exact version whn upgrading and not when testing (#15984)

* [App] Install exact version whn upgrading and not when testing
* Update CHANGELOG.md

Co-authored-by: Jirka Borovec <[email protected]>

(cherry picked from commit 1657ea8)

* releasing 1.8.4.post0

Co-authored-by: Luca Antiga <[email protected]>
Co-authored-by: thomas chaton <[email protected]>
Co-authored-by: Akihiro Nitta <[email protected]>
Co-authored-by: Liyang90 <[email protected]>
Co-authored-by: Justus Schock <[email protected]>
Co-authored-by: Ethan Harris <[email protected]>

Author: 7 people

.github/CONTRIBUTING.md

@@ -309,6 +309,20 @@ and the last true master commit is `ccc111` and your first commit is `mmm222`.
   git push -f
   ```
 
+#### How to run an app on the cloud with a local version of lightning
+
+The lightning cloud uses the latest release by default. However, you might want to run your app with some local changes you've made to the lightning framework. To use your local version of lightning on the cloud, set the following environment variable:
+
+```bash
+git clone https://github.com/Lightning-AI/lightning.git
+cd lightning
+pip install -e .
+export PACKAGE_LIGHTNING=1  # <- this is the magic to use your version (not mainstream PyPI)!
+lightning run app app.py --cloud
+```
+
+By seting `PACKAGE_LIGHTNING=1`, lightning packages the lightning source code in your local directory in addition to your app source code and uploads them to the cloud.
+
 ### Bonus Workflow Tip
 
 If you don't want to remember all the commands above every time you want to push some code/setup a Lightning Dev environment on a new VM, you can set up bash aliases for some common commands. You can add these to one of your `~/.bashrc`, `~/.zshrc`, or `~/.bash_aliases` files.

.github/workflows/release-pypi.yml

@@ -12,6 +12,7 @@ defaults:
     shell: bash
 
 jobs:
+
   init:
     runs-on: ubuntu-20.04
     steps:
@@ -23,6 +24,7 @@ jobs:
         name: dist-packages-${{ github.sha }}
         path: dist
 
+
   build-packages:
     needs: init
     runs-on: ubuntu-20.04
@@ -40,22 +42,20 @@ jobs:
     - uses: actions/setup-python@v4
       with:
         python-version: 3.9
-
     - name: Install dependencies
       run: pip install -U setuptools wheel
-
     - name: Build packages
       env:
         PACKAGE_NAME: ${{ matrix.pkg-name }}
       run: |
         python setup.py sdist bdist_wheel
         ls -lh dist/
-
     - uses: actions/upload-artifact@v3
       with:
         name: dist-packages-${{ github.sha }}
         path: dist
 
+
   upload-packages:
     runs-on: ubuntu-20.04
     needs: build-packages
@@ -73,6 +73,7 @@ jobs:
         files: 'dist/*'
         repo-token: ${{ secrets.GITHUB_TOKEN }}
 
+
   release-version:
     runs-on: ubuntu-20.04
     outputs:
@@ -87,6 +88,7 @@ jobs:
       id: lai-package
       run: python -c "import lightning as L; print(f'version={L.__version__}')" >> $GITHUB_OUTPUT
 
+
   signaling:
     runs-on: ubuntu-20.04
     needs: [release-version]
@@ -100,12 +102,6 @@ jobs:
       with:
         repository: gridai/base-images
         token: ${{ secrets.PAT_GHOST }}
-        ref: main
-    - uses: fregante/setup-git-token@v1
-      with:
-        token: ${{ secrets.PAT_GHOST }}
-        name: PL Ghost
-        email: [email protected]
     - name: Update lightning version
       run: |
         import json, os
@@ -115,20 +111,21 @@ jobs:
         with open("versions.json", "w") as fw:
             json.dump(vers, fw)
       shell: python
-    - name: GIT Commit
+    - run: cat versions.json
+    - name: GIT commit & push
+      env:
+        BRANCH_NAME: "trigger/lightning-${{ env.TAG }}"
       run: |
+        git config --global user.name "PL Ghost"
+        git config --global user.email [email protected]
+        git checkout -b ${BRANCH_NAME}
         git add versions.json
-        git commit -m "bumping lightning version -> ${TAG}"
-        cat versions.json
-    - name: GIT Push
-      run: |
         git status
-        # force push is not very nice
-        # but so far the push is rejected even with exception for this user
-        git push -f
+        git commit -m "bumping lightning version -> ${TAG}"
+        git push -u origin ${BRANCH_NAME} -f
+
 
   waiting:
-    # TODO: replace with back signal from build images/ loop checking for a specific branch?
     runs-on: ubuntu-20.04
     needs: [release-version, signaling]
     env:
@@ -152,6 +149,7 @@ jobs:
           time.sleep(60)
       shell: python
 
+
   pre-publish-packages:
     runs-on: ubuntu-20.04
     needs: build-packages
@@ -181,6 +179,7 @@ jobs:
         pkg-pattern: "*"
         pypi-test-token: ${{ secrets.PYPI_TEST_TOKEN_LAI }}
 
+
   publish-packages:
     runs-on: ubuntu-20.04
     needs: [build-packages, waiting]
@@ -210,6 +209,7 @@ jobs:
         pkg-pattern: "*"
         pypi-token: ${{ secrets.PYPI_TOKEN_LAI }}
 
+
   legacy-checkpoints:
     needs: [build-packages]
     uses: ./.github/workflows/legacy-checkpoints.yml

examples/app_server_with_auto_scaler/app.py

@@ -1,3 +1,4 @@
+# ! pip install torch torchvision
 from typing import Any, List
 
 import torch
@@ -22,10 +23,10 @@ class BatchResponse(BaseModel):
 class PyTorchServer(L.app.components.PythonServer):
     def __init__(self, *args, **kwargs):
         super().__init__(
-            port=L.app.utilities.network.find_free_network_port(),
             input_type=BatchRequestModel,
             output_type=BatchResponse,
-            cloud_compute=L.CloudCompute("gpu"),
+            *args,
+            **kwargs,
         )
 
     def setup(self):
@@ -57,30 +58,32 @@ def scale(self, replicas: int, metrics: dict) -> int:
         """The default scaling logic that users can override."""
         # scale out if the number of pending requests exceeds max batch size.
         max_requests_per_work = self.max_batch_size
-        pending_requests_per_running_or_pending_work = metrics["pending_requests"] / (
-            replicas + metrics["pending_works"]
-        )
-        if pending_requests_per_running_or_pending_work >= max_requests_per_work:
+        pending_requests_per_work = metrics["pending_requests"] / (replicas + metrics["pending_works"])
+        if pending_requests_per_work >= max_requests_per_work:
             return replicas + 1
 
         # scale in if the number of pending requests is below 25% of max_requests_per_work
         min_requests_per_work = max_requests_per_work * 0.25
-        pending_requests_per_running_work = metrics["pending_requests"] / replicas
-        if pending_requests_per_running_work < min_requests_per_work:
+        pending_requests_per_work = metrics["pending_requests"] / replicas
+        if pending_requests_per_work < min_requests_per_work:
             return replicas - 1
 
         return replicas
 
 
 app = L.LightningApp(
     MyAutoScaler(
+        # work class and args
         PyTorchServer,
-        min_replicas=2,
+        cloud_compute=L.CloudCompute("gpu"),
+        # autoscaler specific args
+        min_replicas=1,
         max_replicas=4,
         autoscale_interval=10,
         endpoint="predict",
         input_type=RequestModel,
         output_type=Any,
         timeout_batching=1,
+        max_batch_size=8,
     )
 )

requirements/app/base.txt

@@ -12,5 +12,3 @@ beautifulsoup4>=4.8.0, <4.11.2
 inquirer>=2.10.0
 psutil<5.9.4
 click<=8.1.3
-s3fs>=2022.5.0, <2022.8.3
-aiohttp>=3.8.0, <=3.8.3

requirements/app/cloud.txt

@@ -3,3 +3,5 @@
 redis>=4.0.1, <=4.2.4
 docker>=5.0.0, <=5.0.3
 # setuptools==59.5.0
+s3fs>=2022.5.0, <2022.8.3
+aiohttp>=3.8.0, <=3.8.3

requirements/app/components.txt

@@ -1,2 +1,3 @@
 # deps required by components in the lightning app repository (src/lightning_app/components)
 lightning_api_access>=0.0.3
+aiohttp>=3.8.0, <=3.8.3

src/lightning/__version__.py

@@ -1 +1 @@
-version = "1.8.4"
+version = "1.8.4.post0"

src/lightning_app/CHANGELOG.md

@@ -13,7 +13,11 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
 - Added the CLI command `lightning delete app` to delete a lightning app on the cloud ([#15783](https://github.com/Lightning-AI/lightning/pull/15783))
 - Added a CloudMultiProcessBackend which enables running a child App from within the Flow in the cloud ([#15800](https://github.com/Lightning-AI/lightning/pull/15800))
 - Utility for pickling work object safely even from a child process ([#15836](https://github.com/Lightning-AI/lightning/pull/15836))
-- Added `AutoScaler` component ([#15769](https://github.com/Lightning-AI/lightning/pull/15769))
+- Added `AutoScaler` component (
+   [#15769](https://github.com/Lightning-AI/lightning/pull/15769),
+   [#15971](https://github.com/Lightning-AI/lightning/pull/15971),
+   [#15966](https://github.com/Lightning-AI/lightning/pull/15966)
+)
 - Added the property `ready` of the LightningFlow to inform when the `Open App` should be visible ([#15921](https://github.com/Lightning-AI/lightning/pull/15921))
 - Added private work attributed `_start_method` to customize how to start the works ([#15923](https://github.com/Lightning-AI/lightning/pull/15923))
 - Added a `configure_layout` method to the `LightningWork` which can be used to control how the work is handled in the layout of a parent flow ([#15926](https://github.com/Lightning-AI/lightning/pull/15926))
@@ -50,6 +54,10 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
 - Fixed multiprocessing breakpoint ([#15950](https://github.com/Lightning-AI/lightning/pull/15950))
 - Fixed detection of a Lightning App running in debug mode ([#15951](https://github.com/Lightning-AI/lightning/pull/15951))
 - Fixed `ImportError` on Multinode if package not present ([#15963](https://github.com/Lightning-AI/lightning/pull/15963))
+- Fixed MultiNode Component to use separate cloud computes ([#15965](https://github.com/Lightning-AI/lightning/pull/15965))
+- Fixed Registration for CloudComputes of Works in `L.app.structures` ([#15964](https://github.com/Lightning-AI/lightning/pull/15964))
+- Fixed a bug where auto-upgrading to the latest lightning via the CLI could get stuck in a loop ([#15984](https://github.com/Lightning-AI/lightning/pull/15984))
+
 
 ## [1.8.3] - 2022-11-22
 

src/lightning_app/__version__.py

@@ -1 +1 @@
-version = "1.8.4"
+version = "1.8.4.post0"

src/lightning_app/cli/cmd_install.py

@@ -101,7 +101,7 @@ def gallery_apps_and_components(
     except Exception:
         return None
 
-    entry, kind = _resolve_entry(app_or_component, version_arg)
+    entry, kind = _resolve_entry(name, version_arg)
 
     if kind == "app":
         # give the user the chance to do a manual install
@@ -111,16 +111,19 @@ def gallery_apps_and_components(
         # run installation if requested
         _install_app_from_source(source_url, git_url, folder_name, cwd=cwd, overwrite=overwrite, git_sha=git_sha)
 
-        return os.path.join(os.getcwd(), folder_name, entry["appEntrypointFile"])
+        return os.path.join(os.getcwd(), *entry["appEntrypointFile"].split("/"))
 
     elif kind == "component":
         # give the user the chance to do a manual install
-        git_url = _show_install_component_prompt(entry, app_or_component, org, yes_arg)
-
+        source_url, git_url, folder_name, git_sha = _show_install_app_prompt(
+            entry, app_or_component, org, yes_arg, resource_type="component"
+        )
+        if "@" in git_url:
+            git_url = git_url.split("git+")[1].split("@")[0]
         # run installation if requested
-        _install_component_from_source(git_url)
+        _install_app_from_source(source_url, git_url, folder_name, cwd=cwd, overwrite=overwrite, git_sha=git_sha)
 
-        return os.path.join(os.getcwd(), entry["appEntrypointFile"])
+        return os.path.join(os.getcwd(), *entry["entrypointFile"].split("/"))
 
     return None