Merge branch 'develop' into feature/dynamic-pipelines

Author: schustmi

docs/book/component-guide/deployers/local.md

@@ -52,6 +52,7 @@ For additional configuration of the Local deployer, you can pass the following `
   * `port_range`: The range of ports to search for a free port. Defaults to `(8000, 65535)`.
   * `address`: The address that the deployment server will listen on. Defaults to `127.0.0.1`.
   * `blocking`: Whether to run the deployment in the current process instead of running it as a daemon process. Defaults to False. Use this if you want to debug issues with the deployment ASGI application itself.
+  * `auto_reload`: Whether to enable auto-reload for the uvicorn server. This is useful to speed up local development by automatically restarting the server when code changes are detected without requiring a re-provisioning of the entire deployment. Defaults to False. NOTE: the `auto_reload` setting has no effect on changes in the pipeline configuration, step configuration or stack configuration.
 
 Check out [this docs page](https://docs.zenml.io/concepts/steps_and_pipelines/configuration) for more information on how to specify settings.
 

docs/book/getting-started/deploying-zenml/deploy-with-docker.md

@@ -193,9 +193,15 @@ These configuration options are only relevant if you're using Hashicorp Vault as
 
 * **ZENML\_SECRETS\_STORE\_TYPE:** Set this to `hashicorp` in order to set this type of secret store.
 * **ZENML\_SECRETS\_STORE\_VAULT\_ADDR**: The URL of the HashiCorp Vault server to connect to. NOTE: this is the same as setting the `VAULT_ADDR` environment variable.
-* **ZENML\_SECRETS\_STORE\_VAULT\_TOKEN**: The token to use to authenticate with the HashiCorp Vault server. NOTE: this is the same as setting the `VAULT_TOKEN` environment variable.
 * **ZENML\_SECRETS\_STORE\_VAULT\_NAMESPACE**: The Vault Enterprise namespace. Not required for Vault OSS. NOTE: this is the same as setting the `VAULT_NAMESPACE` environment variable.
 * **ZENML\_SECRETS\_STORE\_MOUNT\_POINT**: The mount point to use for the HashiCorp Vault secrets store. If not set, the default value of `secret` will be used.
+* **ZENML\_SECRETS\_STORE\_VAULT\_AUTH_METHOD**: The authentication method to use to authenticate with the HashiCorp Vault server. One of: `token`, `app_role`, `aws`. Defaults to `token` if not set.
+* **ZENML\_SECRETS\_STORE\_VAULT\_AUTH\_MOUNT\_POINT**: The mount point to use for the authentication method. If not set, the default value specific to the authentication method will be used.
+* **ZENML\_SECRETS\_STORE\_VAULT\_TOKEN**: The token to use to authenticate with the HashiCorp Vault server. Mandatory if the authentication method is `token`. NOTE: this is the same as setting the `VAULT_TOKEN` environment variable.
+* **ZENML\_SECRETS\_STORE\_VAULT\_APP\_ROLE\_ID**: The role ID to use for the app role authentication method. Mandatory if the authentication method is `app_role`.
+* **ZENML\_SECRETS\_STORE\_VAULT\_APP\_SECRET\_ID**: The secret ID to use for the app role authentication method. Mandatory if the authentication method is `app_role`.
+* **ZENML\_SECRETS\_STORE\_VAULT\_AWS\_ROLE**: The AWS role to use for the AWS authentication method. Only relevant if the authentication method is `aws`.
+* **ZENML\_SECRETS\_STORE\_VAULT\_AWS\_HEADER\_VALUE**: The AWS header value to use for the AWS authentication method. Only relevant if the authentication method is `aws`.
 * **ZENML\_SECRETS\_STORE\_MAX\_VERSIONS**: The maximum number of secret versions to keep for each Vault secret. If not set, the default value of 1 will be used (only the latest version will be kept).
 {% endtab %}
 

docs/book/getting-started/zenml-pro/secrets-stores.md

@@ -0,0 +1,239 @@
+---
+icon: lock
+description: >-
+  Learn how to link your own secrets store backend to your ZenML Pro workspace.
+---
+
+# Secrets Stores
+
+The secrets you configure in your ZenML Pro workspaces are by default stored in the same database as your other workspace resources. However, you have the option to link your own backend to your workspace and store the secrets in your own infrastructure. This functionality is powered by the same [ZenML Secrets Store functionality](https://docs.zenml.io/deploying-zenml/deploying-zenml/secret-management) that is available in ZenML OSS and several options are available for you to choose from: AWS Secrets Manager, GCP Secret Manager, Azure Key Vault and HashiCorp Vault.
+
+## How to configure a secrets store
+
+This operation has two main stages:
+
+1. first, you prepare the authentication credentials and necessary permissions for the secrets store. This varies depending on the secrets store backend and the authentication method you want to use (see following sections for more details).
+2. then, you communicate these credentials to the ZenML Pro support team, who will update your workspace to use the new secrets store and also migrate all your existing secrets in the process.
+
+## AWS Secrets Manager
+
+The authentication used by the AWS secrets store is built on the [ZenML Service Connector](https://docs.zenml.io/stacks/service-connectors/overview) of the same type as the secrets store. This means that you can use any of the [authentication methods supported by the Service Connector](https://docs.zenml.io/stacks/service-connectors/connector-types/aws-service-connector#authentication-methods) to authenticate with the secrets store.
+
+The recommended authentication method documented here is to use the [implicit authentication method](https://docs.zenml.io/stacks/service-connectors/connector-types/aws-service-connector#implicit-authentication), because this doesn't need any sensitive credentials to be exchanged with the ZenML Pro support team.
+
+The process is as follows:
+
+1. Identify the AWS IAM role of your ZenML Pro workspace. Every ZenML Pro workspace is associated with a particular AWS IAM role that bears all the AWS permissions granted to the workspace. The ARN of this role is formed as follows: `arn:aws:iam::715803424590:role/zenml-<workspace-uuid>`. For example, if your workspace UUID is `123e4567-e89b-12d3-a456-426614174000`, the ARN of the role is `arn:aws:iam::715803424590:role/zenml-123e4567-e89b-12d3-a456-426614174000`.
+
+2. Create an AWS IAM role in your AWS account that will be assumed by the ZenML Pro workspace role:
+
+  * use the following trust relationship to allow the ZenML Pro workspace role to assume the new role:
+
+    ```json
+    {
+      "Version": "2012-10-17",
+      "Statement": [
+        {
+          "Effect": "Allow",
+          "Principal": {
+            "AWS": "arn:aws:iam::715803424590:role/zenml-<workspace-uuid>"
+          }
+        }
+      ]
+    }
+    ```
+
+   * attach the following custom IAM policy to the new role to allow it to access the AWS Secrets Manager service:
+
+    ```json
+    {
+      "Version": "2012-10-17",
+      "Statement": [
+        {
+          "Effect": "Allow",
+          "Action": [
+            "secretsmanager:CreateSecret",
+            "secretsmanager:GetSecretValue",
+            "secretsmanager:DescribeSecret",
+            "secretsmanager:PutSecretValue",
+            "secretsmanager:UpdateSecret",
+            "secretsmanager:TagResource",
+            "secretsmanager:DeleteSecret"
+          ],
+          "Resource": "arn:aws:secretsmanager:<AWS-region>:<AWS-account-id>:secret:zenml/*"
+        }
+      ]
+    }
+    ```
+
+3. Contact the ZenML Pro support team to update your ZenML Pro workspace to use the new secrets store. You will need to provide the ARN of the new role you created in step 2 and the region where the AWS Secrets Manager service is located. After your workspace is updated, you will see the following changes in the workspace configuration:
+
+```json
+{
+  "id": "...",
+  "name": "...",
+  "zenml_service": {
+    "configuration": {
+      "version": "...",
+      "secrets_store": {
+        "type": "aws",
+        "settings": {
+          "auth_method": "implicit",
+          "auth_config": {
+            "region": "<AWS-region>",
+            "role_arn": "arn:aws:iam::<AWS-account-id>:role/<IAM-role-name>"
+          }
+        }
+      }
+    }
+  }
+}
+```
+
+Here is an example Terraform code to create the new role and attach the custom policy:
+
+```terraform
+terraform {
+  required_providers {
+    aws = {
+      source  = "hashicorp/aws"
+      version = "~> 4.0"
+    }
+  }
+}
+
+data "aws_region" "current" {}
+data "aws_caller_identity" "current" {}
+
+resource "aws_iam_role" "zenml_pro_workspace_role" {
+  name = "zenml-${var.workspace_uuid}"
+  assume_role_policy = jsonencode(
+    {
+      Version = "2012-10-17"
+      Statement = [
+        {
+          Effect = "Allow"
+          Principal = {
+            AWS = "arn:aws:iam::715803424590:role/zenml-${var.workspace_uuid}"
+          }
+        }
+      ]
+    }
+  )
+}
+
+resource "aws_iam_role_policy" "zenml_pro_workspace_policy" {
+  name = "zenml-${var.workspace_uuid}"
+  role = aws_iam_role.zenml_pro_workspace_role.id
+  policy = jsonencode(
+    {
+      Version = "2012-10-17"
+      Statement = [
+        {
+          Effect = "Allow"
+          Action = [
+            "secretsmanager:CreateSecret",
+            "secretsmanager:GetSecretValue",
+            "secretsmanager:DescribeSecret",
+            "secretsmanager:PutSecretValue",
+            "secretsmanager:UpdateSecret",
+            "secretsmanager:TagResource",
+            "secretsmanager:DeleteSecret"
+          ]
+          Resource = "arn:aws:secretsmanager:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:secret:zenml/*"
+        }
+      ]
+    }
+  )
+}
+
+output "zenml_pro_secrets_store_role_arn" {
+  value = aws_iam_role.zenml_pro_secrets_store_role.arn
+}
+
+output "zenml_pro_secrets_store_region" {
+  value = data.aws_region.current.name
+}
+```
+
+If you choose a different authentication method, your will need to provide different credentials. See the [AWS Secrets Manager](https://docs.zenml.io/stacks/service-connectors/connector-types/aws-service-connector#authentication-methods) documentation on the available authentication methods and their configuration options for more details.
+
+## HashiCorp Vault
+
+The HashiCorp Vault secrets store supports the following authentication methods:
+
+* [Token authentication](https://python-hvac.org/en/stable/usage/auth_methods/token.html) - authentication using a static token
+* [App Role authentication](https://python-hvac.org/en/stable/usage/auth_methods/approle.html) - authentication using a Vault App Role (app role ID and secret ID)
+* [AWS authentication](https://python-hvac.org/en/stable/usage/auth_methods/aws.html) - implicit authentication using an AWS IAM role (IAM role ARN)
+
+The recommended authentication method documented here is to use the implicit AWS authentication, because this doesn't need any sensitive credentials to be exchanged with the ZenML Pro support team.
+
+The process is as follows:
+
+1. Identify the AWS IAM role of your ZenML Pro workspace. Every ZenML Pro workspace is associated with a particular AWS IAM role that bears all the AWS permissions granted to the workspace. The ARN of this role is formed as follows: `arn:aws:iam::715803424590:role/zenml-<workspace-uuid>`. For example, if your workspace UUID is `123e4567-e89b-12d3-a456-426614174000`, the ARN of the role is `arn:aws:iam::715803424590:role/zenml-123e4567-e89b-12d3-a456-426614174000`.
+
+2. Enable the AWS authentication method for your HashiCorp Vault:
+
+```shell
+vault auth enable aws
+```
+
+3. Enable the AWS authentication method for your HashiCorp Vault and configure an AWS role to use for authentication, e.g.:
+
+
+```shell
+vault auth enable aws
+
+vault write auth/aws/config/client \
+  iam_server_id_header_value="<workspace-uuid>" \
+  sts_region="eu-central-1"
+
+vault write auth/aws/role/zenml-<workspace-uuid> \
+  auth_type=iam \
+  bound_iam_principal_arn=arn:aws:iam::715803424590:role/zenml-<workspace-uuid> \
+  resolve_aws_unique_ids=false \
+  policies="zenml-<workspace-uuid>" \
+  ttl=1h max_ttl=24h
+```
+
+A few points to note:
+
+* use the IAM role ARN of your ZenML Pro workspace as the bound IAM principal ARN.
+* it's recommended to use a header value to further secure the authentication process. Use a value that is unique to your workspace.
+* configuring `resolve_aws_unique_ids` to `false` is required for the authentication to work.
+* you can point to a custom policy to further restrict the permissions of the authenticated role to a particular mount point.
+
+4. Contact the ZenML Pro support team to update your ZenML Pro workspace to use the new secrets store. You will need to provide the following information:
+
+* the URL of the HashiCorp Vault server
+* the name of the AWS Hashicorp Vault role you created in step 2 (e.g. `zenml-<workspace-uuid>`)
+* the header value you used for the authentication process (e.g. `<workspace-uuid>`)
+* the namespace of the HashiCorp Vault server (if applicable)
+* the mount point to use (if applicable)
+
+After your workspace is updated, you will see the following changes in the workspace configuration:
+
+```json
+{
+  "id": "...",
+  "name": "...",
+  "zenml_service": {
+    "configuration": {
+      "version": "...",
+      "secrets_store": {
+        "type": "hashicorp",
+        "settings": {
+          "auth_method": "aws",
+          "auth_config": {
+            "vault_addr": "https://vault.example.com",
+            "vault_namespace": "zenml",
+            "mount_point": "secrets-<workspace-uuid>",
+            "aws_role": "zenml-<workspace-uuid>",
+            "aws_header_value": "<workspace-uuid>"
+          }
+        }
+      }
+    }
+  }
+}
+```

docs/book/getting-started/zenml-pro/toc.md

@@ -15,3 +15,4 @@
 * [Teams](teams.md)
 * [Roles & Permissions](roles.md)
 * [Service Accounts](service-accounts.md)
+* [Secrets Stores](secrets-stores.md)

docs/book/how-to/deployment/deployment_settings.md

@@ -241,6 +241,27 @@ A rudimentary playground dashboard is included with the ZenML python package tha
 When supplying your own custom dashboard, you may also need to [customize the security headers](./deployment_settings#secure-headers) to allow the dashboard to access various resources. For example, you may want to tweak the `Content-Security-Policy` header to allow the dashboard to access external javascript libraries, images, etc.
 {% endhint %}
 
+#### Jinja2 templates
+
+You can use a Jinja2 template to dynamically generate the `index.html` file that hosts the single-page application. This is useful if you want to dynamically generate the dashboard files based on the pipeline configuration, step configuration or stack configuration. A `service_info` variable is passed to the template that contains the service information, such as the service name, version, and description. This variable has the same structure as the `zenml.deployers.server.models.ServiceInfo` model.
+
+Example:
+
+```jinja2
+<html>
+<head>
+    <title>Pipeline: {{ service_info.pipeline.pipeline_name }}</title>
+    <meta charset="UTF-8">
+    <meta name="viewport" content="width=device-width, initial-scale=1.0">
+    <link rel="stylesheet" href="https://unpkg.com/mvp.css">
+</head>
+<body>
+    <h1>Pipeline: {{ service_info.pipeline.pipeline_name }}</h1>
+    <p>Deployment: {{ service_info.deployment.name }}</p>
+</body>
+</html>
+```
+
 ### CORS
 
 Fine-tune cross-origin access:
@@ -363,6 +384,17 @@ settings:
 
 Tune server runtime parameters for performance and topology:
 
+The following settings are available for tuning the uvicorn server:
+* `thread_pool_size`: the size of the thread pool for CPU-bound work offload.
+* `uvicorn_host`: the host to bind the uvicorn server to.
+* `uvicorn_port`: the port to bind the uvicorn server to.
+* `uvicorn_workers`: the number of workers to use for the uvicorn server.
+* `log_level`: the log level to use for the uvicorn server.
+* `uvicorn_reload`: whether to enable auto-reload for the uvicorn server. This is useful when using [the local Deployer stack component](https://docs.zenml.io/stacks/stack-components/deployers/docker) to speed up local development by automatically restarting the server when code changes are detected. NOTE: the `uvicorn_reload` setting has no effect on changes in the pipeline configuration, step configuration or stack configuration.
+* `uvicorn_kwargs`: a dictionary of keyword arguments to pass to the uvicorn server.
+
+The following settings are available:
+
 ```python
 from zenml.config import DeploymentSettings
 from zenml.enums import LoggingLevels

examples/weather_agent/requirements.txt

@@ -0,0 +1,5 @@
+# Core dependencies for the weather agent example
+zenml
+
+# OpenAI client for LLM integration
+openai

helm/Chart.yaml

@@ -1,6 +1,6 @@
 apiVersion: v2
 name: zenml
-version: "0.91.0"
+version: "0.91.0-dev.3"
 description: "ZenML: MLOps for Reliable AI: from Classical AI to Agents."
 keywords:
 - mlops

helm/templates/_environment.tpl

@@ -331,15 +331,35 @@ project_id: {{ .SecretsStore.gcp.project_id | quote }}
 auth_method: {{ .SecretsStore.azure.authMethod | quote }}
 key_vault_name: {{ .SecretsStore.azure.key_vault_name | quote }}
 {{- else if eq .SecretsStore.type "hashicorp" }}
+auth_method: {{ .SecretsStore.hashicorp.authMethod | quote }}
+{{- if .SecretsStore.hashicorp.authConfig.auth_mount_point }}
+auth_mount_point: {{ .SecretsStore.hashicorp.authConfig.auth_mount_point | quote }}
+{{- end }}
+{{- if .SecretsStore.hashicorp.vault_addr }}
 vault_addr: {{ .SecretsStore.hashicorp.vault_addr | quote }}
+{{- else }}
+vault_addr: {{ .SecretsStore.hashicorp.authConfig.vault_addr | quote }}
+{{- end }}
 {{- if .SecretsStore.hashicorp.vault_namespace }}
 vault_namespace: {{ .SecretsStore.hashicorp.vault_namespace | quote }}
+{{- else if .SecretsStore.hashicorp.authConfig.vault_namespace }}
+vault_namespace: {{ .SecretsStore.hashicorp.authConfig.vault_namespace | quote }}
 {{- end }}
 {{- if .SecretsStore.hashicorp.mount_point }}
 mount_point: {{ .SecretsStore.hashicorp.mount_point | quote }}
+{{- else if .SecretsStore.hashicorp.authConfig.mount_point }}
+mount_point: {{ .SecretsStore.hashicorp.authConfig.mount_point | quote }}
 {{- end }}
 {{- if .SecretsStore.hashicorp.max_versions }}
 max_versions: {{ .SecretsStore.hashicorp.max_versions | quote }}
+{{- else if .SecretsStore.hashicorp.authConfig.max_versions }}
+max_versions: {{ .SecretsStore.hashicorp.authConfig.max_versions | quote }}
+{{- end }}
+{{- if eq .SecretsStore.hashicorp.authMethod "app_role" }}
+app_role_id: {{ .SecretsStore.hashicorp.authConfig.app_role_id | quote }}
+{{- else if eq .SecretsStore.hashicorp.authMethod "aws" }}
+aws_role: {{ .SecretsStore.hashicorp.authConfig.aws_role | quote }}
+aws_header_value: {{ .SecretsStore.hashicorp.authConfig.aws_header_value | quote }}
 {{- end }}
 {{- else if eq .SecretsStore.type "custom" }}
 class_path: {{ .SecretsStore.custom.class_path | quote }}
@@ -439,8 +459,14 @@ auth_config: {{ include "zenml.legacyGCPSecretsStoreAuthConfig" . | fromYaml | t
 auth_config: {{ .SecretsStore.gcp.authConfig | toJson | quote }}
 {{- end }}
 {{- else if eq .SecretsStore.type "hashicorp" }}
+{{- if eq .SecretsStore.hashicorp.authMethod "token" }}
 {{- if .SecretsStore.hashicorp.vault_token }}
 vault_token: {{ .SecretsStore.hashicorp.vault_token | quote }}
+{{- else }}
+vault_token: {{ .SecretsStore.hashicorp.authConfig.vault_token | quote }}
+{{- end }}
+{{- else if eq .SecretsStore.hashicorp.authMethod "app_role" }}
+app_secret_id: {{ .SecretsStore.hashicorp.authConfig.app_secret_id | quote }}
 {{- end }}
 {{- end }}
 {{- end }}