Add ML environment lock file, update README files

dashboard/README.md

@@ -1,121 +1,104 @@
-## Dashboard
-
-Here are a few how-to guides on how to develop and use the dashboard.
-
-### Prerequisites
-- Ensure you have Conda installed.
-- Ensure you have Docker installed (if you plan to use Docker).
-
-### How to create a new conda environment lock file
-
-1. Activate the `base` conda environment:
-    ```console
-    conda activate base
-    ```
-
-2. Install `conda-lock` (if not already installed):
-    ```console
-    conda install -c conda-forge conda-lock
-    ```
-
-3. Create the lock file starting from the existing minimal environment file:
-    ```console
-    conda-lock --file environment.yml --lockfile environment-lock.yml
-    ```
-
-### How to set up the conda environment
-
-1. Activate the `base` conda environment:
-    ```console
-    conda activate base
-    ```
-
-2. Install `conda-lock` (if not already installed):
-    ```console
-    conda install -c conda-forge conda-lock
-    ```
-
-3. Create the `synapse-gui` conda environment from the lock file:
-    ```console
-    conda-lock install --name synapse-gui environment-lock.yml
-    ```
-
-### How to run the GUI
-
-1. Activate the `synapse-gui` conda environment:
-    ```console
-    conda activate synapse-gui
-    ```
-
-2. Set the database settings (read only):
-    ```console
-    export SF_DB_HOST='127.0.0.1'
-    export SF_DB_READONLY_PASSWORD='your_password_here'  # Use SINGLE quotes around the password!
-    ```
-
-3. For local development, open a separate terminal and keep it open while SSH forwarding the database connection:
-    ```console
-    ssh -L 27017:mongodb05.nersc.gov:27017 <username>@dtn03.nersc.gov -N
-    ```
-
-4. Run the GUI from the `dashboard/` folder:
-    - Via the web browser interface:
-    ```console
-    python -u app.py --port 8080
-    ```
-    - As a desktop application:
-    ```console
-    python -u app.py --app
-    ```
-    If you run the GUI as a desktop application, make sure to set the following environment variable first:
-    ```console
-    python -m pip install pywebview[qt]
-    export PYWEBVIEW_GUI=qt
-    ```
-
-5. Terminate the GUI via `Ctrl` + `C`.
-
-### How to build and run the Docker container
+# Table of Contents
+* [Overview](#Overview)
+* [Run the Dashboard Locally](#Run-the-Dashboard-Locally)
+    * [Without Docker](#Without-Docker)
+    * [With Docker](#With-Docker)
+* [Run the Dashboard at NERSC](#Run-the-Dashboard-at-NERSC)
+* [Get the Superfacility API Credentials](#Get-the-Superfacility-API-Credentials)
+* [For Maintainers](#For-Maintainers)
+    * [Generate the conda environment lock file](#Generate-the-conda-environment-lock-file)
+    * [Build and push the Docker container to NERSC](#Build-and-push-the-Docker-container-to-NERSC)
+* [References](#References)
 
-1. Move to the root directory of the repository.
+# Overview
+
+The Synapse dashboard provides a web interface for working with data from experiments, simulations, and ML models.
+
+The dashboard can be run in two distinct ways:
+
+1. Locally on your computer.
+
+2. At NERSC through Spin.
+
+# Run the Dashboard Locally
+
+This section describes how to develop and use the dashboard locally.
+
+## Without Docker
+
+### Prepare the conda environment
+
+1. Move to the [dashboard/](./) directory.
+
+2. Activate the conda environment `base`:
+```bash
+conda activate base
+```
+
+3. Install `conda-lock` if not installed yet:
+```bash
+conda install -c conda-forge conda-lock
+```
+
+4. Create the conda environment `synapse-gui`:
+```bash
+conda-lock install --name synapse-gui environment-lock.yml
+```
+
+### Run the dashboard
+
+1. Create an SSH tunnel to access the MongoDB database at NERSC (in a separate terminal):
+   ```bash
+   ssh -L 27017:mongodb05.nersc.gov:27017 <username>@dtn03.nersc.gov -N
+   ```
+
+2. Move to the [dashboard/](./) directory.
+
+3. Set up the database settings (read-only):
+   ```bash
+   export SF_DB_HOST='127.0.0.1'
+   export SF_DB_READONLY_PASSWORD='your_password_here'  # Use SINGLE quotes around the password!
+   ```
+
+4. Activate the conda environment `synapse-gui`:
+   ```bash
+   conda activate synapse-gui
+   ```
+
+5. Run the dashboard as a web application:
+   ```bash
+   python -u app.py --port 8080
+   ```
+
+## With Docker
+
+### Run the dashboard
+
+1. Create an SSH tunnel to access the MongoDB database at NERSC (in a separate terminal):
+   ```bash
+   ssh -L 27017:mongodb05.nersc.gov:27017 <username>@dtn03.nersc.gov -N
+   ```
+
+2. Move to the root directory of the repository.
+
+3. Build the Docker image as described [below](#build-the-docker-image).
+
+4. Run the Docker container:
+   ```bash
+   docker run --network=host -v /etc/localtime:/etc/localtime -v $PWD/ml:/app/ml -e SF_DB_HOST='127.0.0.1' -e SF_DB_READONLY_PASSWORD='your_password_here' synapse-gui
+   ```
+   For debugging, you can enter the container without starting the app:
+   ```bash
+   docker run --network=host -v /etc/localtime:/etc/localtime -v $PWD/ml:/app/ml -e SF_DB_HOST='127.0.0.1' -e SF_DB_READONLY_PASSWORD='your_password_here' -it synapse-gui bash
+   ```
+   Note that `-v /etc/localtime:/etc/localtime` is necessary to synchronize the time zone in the container with the host machine.
 
-2. Build the Docker image based on `Dockerfile`:
-    ```console
-    docker build --platform linux/amd64 -t synapse-gui -f dashboard.Dockerfile .
-    ```
-
-3. Run the Docker container from the `dashboard/` folder:
-    ```console
-    docker run --network=host -v /etc/localtime:/etc/localtime -v $PWD/ml:/app/ml -e SF_DB_HOST='127.0.0.1' -e SF_DB_READONLY_PASSWORD='your_password_here' synapse-gui
-    ```
-    For debugging, you can also enter the container without starting the app:
-    ```console
-    docker run --network=host -v /etc/localtime:/etc/localtime -v $PWD/ml:/app/ml -e SF_DB_HOST='127.0.0.1' -e SF_DB_READONLY_PASSWORD='your_password_here' -it synapse-gui bash
-    ```
-    Note that `-v /etc/localtime:/etc/localtime` is necessary to synchronize the time zone in the container with the host machine.
-
-4. Optional: Publish the container privately to NERSC registry (https://registry.nersc.gov):
-    ```console
-    docker login registry.nersc.gov
-    # Username: your NERSC username
-    # Password: your NERSC password without 2FA
-    ```
-    ```console
-    docker tag synapse-gui:latest registry.nersc.gov/m558/superfacility/synapse-gui:latest
-    docker tag synapse-gui:latest registry.nersc.gov/m558/superfacility/synapse-gui:$(date "+%y.%m")
-    docker push -a registry.nersc.gov/m558/superfacility/synapse-gui
-    ```
-    This has been also automated through the Python script [publish_container.py](https://github.com/BLAST-AI-ML/synapse/blob/main/publish_container.py), which can be executed via
-    ```console
-    python publish_container.py --gui
-    ```
-
-5. Optional: From time to time, as you develop the container, you might want to prune old, unused images to get back GBytes of storage on your development machine:
-    ```console
-    docker system prune -a
-    ```
-
-### How to get the Superfacility API credentials
+# Run the Dashboard at NERSC
+
+Connect to the [dashboard](https://bellasuperfacility.lbl.gov/) deployed at NERSC through Spin and play around!
+Remember that you need to upload valid Superfacility API credentials in order to launch simulations or train ML models directly from the dashboard.
+
+# Get the Superfacility API Credentials
 
 Following the instructions at [docs.nersc.gov/services/sfapi/authentication/#client](https://docs.nersc.gov/services/sfapi/authentication/#client):
 
@@ -127,15 +110,89 @@ Following the instructions at [docs.nersc.gov/services/sfapi/authentication/#cli
 
 4. Enter a client name (e.g., "Synapse"), choose `sf558` for the user, choose "Red" security level, and select either "Your IP" or "Spin" from the "IP Presets" menu, depending on whether the key will be used from a local computer or from Spin.
 
-5. Download the private key file (in pem format) and save it as `priv_key.pem` in the root directory of the GUI.
-   Each time the GUI is launched, it will automatically find the existing key file and load the corresponding credentials.
+5. Download the private key file (in pem format) and save it as `priv_key.pem` in the root directory of the dashboard.
+   Each time the dashboard is launched, it will automatically find the existing key file and load the corresponding credentials.
 
 6. Copy your client ID and add it on the first line of your private key file as described in the instructions at [nersc.github.io/sfapi_client/quickstart/#storing-keys-in-files](https://nersc.github.io/sfapi_client/quickstart/#storing-keys-in-files):
-    ```
-    randmstrgz
-    -----BEGIN RSA PRIVATE KEY-----
-    ...
-    -----END RSA PRIVATE KEY-----
-    ```
+   ```
+   randmstrgz
+   -----BEGIN RSA PRIVATE KEY-----
+   ...
+   -----END RSA PRIVATE KEY-----
+   ```
 
 7. Run `chmod 600 priv_key.pem` to change the permissions of your private key file to read/write only.
+
+# For Maintainers
+
+## Generate the conda environment lock file
+
+1. Move to the directory [dashboard/](.).
+
+2. Activate the conda environment `base`:
+   ```bash
+   conda activate base
+   ```
+
+3. Install `conda-lock` if not installed yet:
+   ```bash
+   conda install -c conda-forge conda-lock
+   ```
+
+4. Generate the conda environment lock file:
+   ```bash
+   conda-lock --file environment.yml --lockfile environment-lock.yml
+   ```
+
+## Build and push the Docker container to NERSC
+
+> [!WARNING]
+> Pushing a new Docker container affects the production dashboard deployed at NERSC through Spin.
+
+> [!TIP]
+> Run this workflow automatically with the Python script [publish_container.py](../publish_container.py):
+> ```bash
+> python publish_container.py --gui
+> ```
+
+> [!TIP]
+> Prune old, unused images periodically in order to free up space on your machine:
+> ```bash
+> docker system prune -a
+> ```
+
+### Build the Docker image
+
+1. Move to the root directory of the repository.
+
+2. Build the Docker image:
+   ```bash
+   docker build --platform linux/amd64 -t synapse-gui -f dashboard.Dockerfile .
+   ```
+
+### Push the Docker container
+
+1. Move to the root directory of the repository.
+
+2. Login to the [NERSC registry](https://registry.nersc.gov):
+   ```bash
+   docker login registry.nersc.gov
+   # Username: your NERSC username
+   # Password: your NERSC password without 2FA
+   ```
+
+3. Tag the Docker image:
+   ```bash
+   docker tag synapse-gui:latest registry.nersc.gov/m558/superfacility/synapse-gui:latest
+   docker tag synapse-gui:latest registry.nersc.gov/m558/superfacility/synapse-gui:$(date "+%y.%m")
+   ```
+
+4. Push the Docker container:
+   ```bash
+   docker push -a registry.nersc.gov/m558/superfacility/synapse-gui
+   ```
+
+# References
+
+* [Using NERSC's `registry.nersc.gov`](https://docs.nersc.gov/development/containers/registry/)
+* [Superfacility API authentication](https://docs.nersc.gov/services/sfapi/authentication/#client)

ml.Dockerfile

@@ -10,10 +10,11 @@ ENV PIP_ROOT_USER_ACTION=ignore
 # Install any needed packages specified in the environment file
 # Match the CUDA 12.4.0 on Perlmutter (NERSC):
 #   https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#specifying-a-different-target-platform-for-an-environment
-COPY ml/environment.yml /app/ml/environment.yml
+COPY ml/environment-lock.yml /app/ml/environment-lock.yml
 ENV CONDA_OVERRIDE_CUDA=12.4.0
 RUN conda info
-RUN conda env create -y -f environment.yml \
+RUN conda install -c conda-forge conda-lock \
+    && conda-lock install --name synapse-ml environment-lock.yml \
     && conda clean --all -y
 
 # Configure an exectuable entrypoint script