Merge pull request #27 from jacquelinegarrahan/main

Final syncs before v0.1 release

Author: jacquelinegarrahan

docs/config.md

@@ -37,22 +37,6 @@ Configuration of the job environment must respect hostnames available to their o
 ## Overriding configuration values
 
 
-```yaml
-model:
-  author: Jaqueline Garrhan
-  laboratory: SLAC
-  facility: LCLS
-  beampath: cu_hxr
-  description: Example model
-
-
-deployment:
-  version: v0.0
-  source: https://github.com/jacquelinegarrahan/my-model-repo
-  variables: my-model-repo/my_model_repo/files/variables.yaml
-  flow_name: example_cu_hxr_0.0
-```
-
 ```yaml
 input_variables:
   input_variable_1:

docs/demo.md

@@ -1,13 +1,17 @@
 # Demo
 
 This demo walks through the creation of a model compatible with [LUME-services](https://slaclab.github.io/lume-services/) tooling. You will:
-1. Create a GitHub repository for the demo model.
+1. Create a GitHub repository for the demo model
 2. Build a templated project using [`lume-services-model-template`](https://github.com/slaclab/lume-services-model-template)
-3. Register your model using the LUME-services API to Prefect orchestration tools, the MySQL model registry, and store results in a MongoDB service
+3. Register your model using the LUME-services API and store deployment information into a MySQL database
+4. Run your model, storing results in MongoDB database
+5. Retrieve model results using LUME-services
 
 
-prerequisites:
-- dockerhub or stanford container registry
+Requirements:
+* [Docker desktop](https://www.docker.com/products/docker-desktop/) for launching the development environment
+* [Conda installation](https://docs.conda.io/en/latest/miniconda.html) for the management of  Python environments
+* Either a [DockerHub](https://hub.docker.com/) account or Stanford account to use the [Stanford Container Registry](https://itcommunity.stanford.edu/unconference/sessions/2018/introducing-scr-stanford-container-registry) provided by the [code.stanford.edu](https://code.stanford.edu/SiteDocs/faq) effort.
 
 
 ## Package a model
@@ -20,7 +24,7 @@ Using your GitHub account, create an repository named `my-model` (feel free to s
 
 Clone `lume-model-services-template` and navigate to repository:
 ```
-git clone https://github.com/jacquelinegarrahan/lume-services-model-template
+git clone https://github.com/slaclab/lume-services-model-template
 cd lume-services-model-template
 ```
 
@@ -57,7 +61,7 @@ Select container_registry:
 2 - Stanford Container Registry
 Choose from 1, 2 [1]:
 ```
-If you plan to use the Stanford Container Registry, enter `2`. Otherwise, enter `1` for DockerHub.
+If you plan to use the Stanford Container Registry, enter `2`. Otherwise, enter `1` for DockerHub. Now, enter the username you use for the contaner registry:
 
 ```
 container_username: YOUR REGISTRY USERNAME
@@ -304,7 +308,7 @@ The release will trigger a GitHub action workflow for your project, which you ca
 The below steps mimic a production deployment workflow.
 
 
-### 14. Start services with docker-compose
+### 12. Start services with docker-compose
 
 LUME-services is packaged with a command line utility for launching the development environment, a docker-compose application with all services packaged and configurable via environment variables.
 
@@ -327,9 +331,12 @@ Once the console logs a message about passed health checks, you've started all s
 
 And access the UI using your browser at http://localhost:8080.
 
+## 13. Pull image (Stanford Container Registry)
+If using the Stanford Container Registry, you'll have to pull the image before kicking off workflows so that Docker can find the image.
 
 
-### 15. Run the notebook and register your model
+
+### 14. Run the notebook and register your model
 
 Create a new console window. Activate lume-services-dev and configure the environment variables:
 ```

docs/developer/todo.md

@@ -6,6 +6,7 @@
     - [x] Remove tests in pip installation
     - [ ] Docs!
     - [ ] Set up all loggers
+    - [ ] Set collection at the system level.
     - [ ] Move environments, flows, tasks, and results under model directory
     - [ ] Fix the workflow tests to use templated repos with docker build
     - [x] Fix image issue... Build a designated image in deployment using a lume-services base image

docs/examples/Demo.ipynb

@@ -0,0 +1 @@
+3

docs/examples/test_file.txt

@@ -10,7 +10,7 @@
 [![Issues][issues-shield]][issues-url]
 [![Issues Closed][issues-closed-shield]][issues-closed-url]
 [![Tests][tests-workflow]][tests-workflow-url]
-![Coverage][coverage]
+[![Coverage]][coverage]
 [![Docs][docs-workflow]][docs-workflow-url]
 </div>
 <br />

docs/index.md

@@ -144,35 +144,32 @@ services:
 
   # agent
   agent:
-    image: prefecthq/prefect:1.2.3-python3.10
+    image: prefecthq/prefect:1.4.0-python3.10
     # The environment variables are passed from agent to job pods
     # This configures access to services
     command: >
       bash -c "prefect server create-tenant --name default --slug default &>/dev/null ;
-      prefect agent docker start --label lume-services --agent-address http://localhost:5000/ --show-flow-logs --log-level DEBUG --network service-net
-      --env LUME_BACKEND=$LUME_BACKEND
-      --env LUME_MOUNTED_FILESYSTEM__IDENTIFIER=$LUME_MOUNTED_FILESYSTEM__IDENTIFIER
-      --env LUME_MOUNTED_FILESYSTEM__MOUNT_PATH=$LUME_MOUNTED_FILESYSTEM__MOUNT_PATH
-      --env LUME_MOUNTED_FILESYSTEM__MOUNT_ALIAS=$LUME_MOUNTED_FILESYSTEM__MOUNT_ALIAS
-      --env LUME_MOUNTED_FILESYSTEM__MOUNT_TYPE=$LUME_MOUNTED_FILESYSTEM__MOUNT_TYPE
-      --env LUME_ENVIRONMENT__LOCAL_PIP_REPOSITORY=$LUME_ENVIRONMENT__LOCAL_PIP_REPOSITORY
-      --env LUME_ENVIRONMENT__LOCAL_CONDA_CHANNEL_DIRECTORY=$LUME_ENVIRONMENT__LOCAL_CONDA_CHANNEL_DIRECTORY
-      --env LUME_MODEL_DB__HOST=$LUME_MODEL_DB__HOST
-      --env LUME_MODEL_DB__PORT=$LUME_MODEL_DB__PORT
-      --env LUME_MODEL_DB__USER=$LUME_MODEL_DB__USER
-      --env LUME_MODEL_DB__PASSWORD=$LUME_MODEL_DB__PASSWORD
-      --env LUME_MODEL_DB__DATABASE=$LUME_MODEL_DB__DATABASE
-      --env LUME_RESULTS_DB__USERNAME=$LUME_RESULTS_DB__USERNAME
-      --env LUME_RESULTS_DB__PASSWORD=$LUME_RESULTS_DB__PASSWORD
-      --env LUME_RESULTS_DB__PORT=$LUME_RESULTS_DB__PORT
-      --env LUME_RESULTS_DB__HOST=$LUME_RESULTS_DB__HOST
-      --env LUME_RESULTS_DB__DATABASE=$LUME_RESULTS_DB__DATABASE
-      --env LUME_PREFECT__SERVER__HOST=$LUME_PREFECT__SERVER__HOST
-      --env LUME_PREFECT__SERVER__HOST_PORT=$LUME_PREFECT__SERVER__HOST_PORT
-      --env LUME_PREFECT__HOME_DIR=$LUME_PREFECT__HOME_DIR
-      --env LUME_PREFECT__DEBUG=$LUME_PREFECT__DEBUG
-      --env LUME_PREFECT__BACKEND=$LUME_PREFECT__BACKEND
-      --volume ${LUME_MOUNTED_FILESYSTEM__MOUNT_PATH:-PWD}:${LUME_MOUNTED_FILESYSTEM__MOUNT_ALIAS:-/lume-services/data}"
+      prefect agent docker start --label lume-services --agent-address http://localhost:5000/ --show-flow-logs --log-level DEBUG --network service-net --env LUME_BACKEND=$LUME_BACKEND
+      --env LUME_MOUNTED_FILESYSTEM__IDENTIFIER=${LUME_MOUNTED_FILESYSTEM__IDENTIFIER:-mounted}
+      --env LUME_MOUNTED_FILESYSTEM__MOUNT_PATH=${LUME_MOUNTED_FILESYSTEM__MOUNT_PATH}
+      --env LUME_MOUNTED_FILESYSTEM__MOUNT_ALIAS=${LUME_MOUNTED_FILESYSTEM__MOUNT_ALIAS:-/lume-services/data}
+      --env LUME_MOUNTED_FILESYSTEM__MOUNT_TYPE=${LUME_MOUNTED_FILESYSTEM__MOUNT_TYPE:-DirectoryOrCreate}
+      --env LUME_MODEL_DB__HOST=mysql
+      --env LUME_MODEL_DB__PORT=3306
+      --env LUME_MODEL_DB__USER=${LUME_MODEL_DB__USER:-root}
+      --env LUME_MODEL_DB__PASSWORD=${LUME_MODEL_DB__PASSWORD:-password}
+      --env LUME_MODEL_DB__DATABASE=${LUME_MODEL_DB__DATABASE:-lume_services_models}
+      --env LUME_RESULTS_DB__USERNAME=${LUME_RESULTS_DB__USERNAME:-root}
+      --env LUME_RESULTS_DB__PASSWORD=${LUME_RESULTS_DB__PASSWORD:-password}
+      --env LUME_RESULTS_DB__PORT=27017
+      --env LUME_RESULTS_DB__HOST=mongodb
+      --env LUME_RESULTS_DB__DATABASE=${LUME_RESULTS_DB__DATABASE:-lume_services_results}
+      --env LUME_PREFECT__SERVER__HOST=http://apollo
+      --env LUME_PREFECT__SERVER__HOST_PORT=4200
+      --env LUME_PREFECT__HOME_DIR=~/.prefect
+      --env LUME_PREFECT__DEBUG=${LUME_PREFECT__DEBUG:-false}
+      --env LUME_PREFECT__BACKEND=server
+      --volume ${LUME_MOUNTED_FILESYSTEM__MOUNT_PATH}:${LUME_MOUNTED_FILESYSTEM__MOUNT_ALIAS:-/lume-services/data}"
     volumes:
         - type: bind
           source: ./config.toml
@@ -183,25 +180,22 @@ services:
           target: /var/run/docker.sock
           read_only: true
         - type: bind
-          source: ${LUME_MOUNTED_FILESYSTEM__MOUNT_PATH:-PWD}
+          source: ${LUME_MOUNTED_FILESYSTEM__MOUNT_PATH}
           target: ${LUME_MOUNTED_FILESYSTEM__MOUNT_ALIAS:-/lume-services/data}
           read_only: false
     environment:
       LUME_MOUNTED_FILESYSTEM__IDENTIFIER: ${LUME_MOUNTED_FILESYSTEM__IDENTIFIER:-mounted}
-      LUME_MOUNTED_FILESYSTEM__MOUNT_PATH: ${LUME_MOUNTED_FILESYSTEM__MOUNT_PATH:-PWD}
+      LUME_MOUNTED_FILESYSTEM__MOUNT_PATH: ${LUME_MOUNTED_FILESYSTEM__MOUNT_PATH}
       LUME_MOUNTED_FILESYSTEM__MOUNT_ALIAS: ${LUME_MOUNTED_FILESYSTEM__MOUNT_ALIAS:-/lume-services/data}
       LUME_MOUNTED_FILESYSTEM__MOUNT_TYPE: ${LUME_MOUNTED_FILESYSTEM__MOUNT_TYPE:-DirectoryOrCreate}
-      LOCAL_CHANNEL_ONLY: ${LUME_PREFECT__ISOLATED:-false}
-      LUME_ENVIRONMENT__LOCAL_PIP_REPOSITORY: ${LUME_EVIRONMENT__LOCAL_PIP_REPOSITORY}
-      LUME_ENVIRONMENT__LOCAL_CONDA_CHANNEL_DIRECTORY: ${LUME_EVIRONMENT__LOCAL_CONDA_CHANNEL_DIRECTORY}
-      LUME_MODEL_DB__HOST: http://mysql
+      LUME_MODEL_DB__HOST: mysql
       LUME_MODEL_DB__PORT: 3306
       LUME_MODEL_DB__USER: ${LUME_MODEL_DB__USER:-root}
       LUME_MODEL_DB__PASSWORD: ${LUME_MODEL_DB__PASSWORD:-password}
       LUME_MODEL_DB__DATABASE: ${LUME_MODEL_DB__DATABASE:-lume_services_models}
-      LUME_RESULTS_DB__DATABASE: ${LUME_MODEL_DB__DATABASE:-results}
+      LUME_RESULTS_DB__DATABASE: ${LUME_MODEL_DB__DATABASE:-lume_service_results}
       LUME_RESULTS_DB__USERNAME: ${LUME_RESULTS_DB__USERNAME:-root}
-      LUME_RESULTS_DB__HOST: http://mongodb
+      LUME_RESULTS_DB__HOST: mongodb
       LUME_RESULTS_DB__PASSWORD: ${LUME_RESULTS_DB__PASSWORD:-password}
       LUME_RESULTS_DB__PORT: 27017
       LUME_PREFECT__SERVER__HOST: http://apollo

lume_services/docker/files/docker-compose.yml

@@ -2,27 +2,28 @@
 CREATE TABLE model (
 	model_id INTEGER NOT NULL AUTO_INCREMENT,
 	created DATETIME DEFAULT now(),
-	author VARCHAR(255) NOT NULL,
-	laboratory VARCHAR(255) NOT NULL,
-	facility VARCHAR(255) NOT NULL,
-	beampath VARCHAR(255) NOT NULL,
+	author VARCHAR(50) NOT NULL,
+	laboratory VARCHAR(50) NOT NULL,
+	facility VARCHAR(50) NOT NULL,
+	beampath VARCHAR(50) NOT NULL,
 	description VARCHAR(255) NOT NULL,
-	PRIMARY KEY (model_id)
+	PRIMARY KEY (model_id),
+	CONSTRAINT _model_entry UNIQUE (author, laboratory, facility, beampath, description)
 );
 CREATE TABLE project (
-	project_name VARCHAR(255) NOT NULL,
+	project_name VARCHAR(50) NOT NULL,
 	description VARCHAR(255) NOT NULL,
 	PRIMARY KEY (project_name)
 );
 CREATE TABLE deployment (
 	deployment_id INTEGER NOT NULL AUTO_INCREMENT,
-	version VARCHAR(255) NOT NULL,
+	version VARCHAR(10) NOT NULL,
 	deploy_date DATETIME DEFAULT now(),
-	package_import_name VARCHAR(255) NOT NULL,
+	package_import_name VARCHAR(50) NOT NULL,
 	asset_dir VARCHAR(255),
 	source VARCHAR(255) NOT NULL,
-	sha256 VARCHAR(255) NOT NULL,
-	image VARCHAR(255),
+	sha256 VARCHAR(64) NOT NULL,
+	image VARCHAR(100),
 	is_live BOOL NOT NULL,
 	model_id INTEGER NOT NULL,
 	PRIMARY KEY (deployment_id),
@@ -31,8 +32,8 @@ CREATE TABLE deployment (
 );
 CREATE TABLE flow (
 	flow_id VARCHAR(255) NOT NULL,
-	flow_name VARCHAR(255) NOT NULL,
-	project_name VARCHAR(255) NOT NULL,
+	flow_name VARCHAR(50) NOT NULL,
+	project_name VARCHAR(50) NOT NULL,
 	deployment_id INTEGER NOT NULL,
 	PRIMARY KEY (flow_id),
 	FOREIGN KEY(project_name) REFERENCES project (project_name),

lume_services/docker/files/model-db-init.sql

@@ -292,9 +292,27 @@ def __init__(self):
         super().__init__("CONDA_PREFIX environment variabe is not set.")
 
 
-class NoFlowFoundError(Exception):
-
+class NoFlowFoundInPackageError(Exception):
     def __init__(self, source_path):
         self.source_path = source_path
         self.message = "No flow entrypoint found for the distribution at %s"
         super().__init__(self.message, self.source_path)
+
+
+class PathNotInMount(Exception):
+    def __init__(
+        self, filesystem_identifier: str, path: str, mount_path: str, mount_alias: str
+    ):
+        self.filesystem_identifier = filesystem_identifier
+        self.path = (path,)
+        self.mount_path = mount_path
+        self.mount_alias = mount_alias
+        self.message = "Path %s not in mount for mounted filesystem identifier: %s, \
+            Mount path: %s, Mount alias: %s"
+        super().__init__(
+            self.message,
+            self.path,
+            self.filesystem_identifier,
+            self.mount_path,
+            self.mount_alias,
+        )

lume_services/errors.py

@@ -1,12 +1,11 @@
 from datetime import datetime, timedelta
-from pydantic import BaseModel, validator, Field, root_validator
+from pydantic import BaseModel, validator, Field
 from prefect import Parameter
 from prefect.run_configs import RunConfig
 from typing import List, Optional, Dict, Literal, Any
 from prefect import Flow as PrefectFlow
 from dependency_injector.wiring import Provide, inject
 from lume_services.config import Context
-from lume_services.utils import get_callable_from_string
 
 from lume_services.services.scheduling import SchedulingService
 from lume_services.services.scheduling.backends.local import LocalBackend
@@ -252,7 +251,7 @@ def run_and_return(
             if self.prefect_flow is None:
                 self.load_flow()
 
-            scheduling_service.run_and_return(
+            return scheduling_service.run_and_return(
                 parameters=parameters,
                 flow=self.prefect_flow,
                 task_name=task_name,
@@ -261,7 +260,7 @@ def run_and_return(
             )
 
         elif isinstance(scheduling_service.backend, (ServerBackend,)):
-            scheduling_service.run_and_return(
+            return scheduling_service.run_and_return(
                 parameters=parameters,
                 flow_id=self.flow_id,
                 task_name=task_name,