Merge branch 'develop' into feature/dynamic-pipelines

Author: schustmi

AGENTS.md

@@ -62,6 +62,9 @@ Use filesystem navigation tools to explore the codebase structure as needed.
   - It resolves dependencies more quickly and reliably than pip
   - It can resolve dependency conflicts that pip sometimes struggles with or takes a long time to resolve
 
+### Documentation Access via MCP
+ZenML documentation is available via a built-in GitBook MCP server: https://docs.zenml.io/~gitbook/mcp. IDE agents like Cursor and Claude Code can add this as an HTTP MCP server named 'ZenML Docs' to answer questions directly from the docs while you code. This enables live, source-of-truth lookups with fewer hallucinations and faster feature discovery. Note that the MCP server indexes the latest released docs, not the develop branch. For full setup details and examples, see docs/book/reference/llms-txt.md.
+
 ### Environment Variables
 - Several environment variables are useful during ZenML development:
   - `ZENML_DEBUG=true`: Enables verbose debug logging

CLAUDE.md

@@ -12,6 +12,16 @@ This document provides guidance for Claude Code when working with the ZenML code
 
 Use filesystem navigation tools to explore the codebase structure as needed.
 
+## Use ZenML Docs via MCP
+Claude Code can query ZenML documentation via the built-in GitBook MCP server: https://docs.zenml.io/~gitbook/mcp. This enables real-time, source-of-truth lookups from the docs while you code, reducing hallucinations and speeding up feature discovery.
+
+Quick setup (CLI):
+```bash
+claude mcp add zenmldocs --transport http https://docs.zenml.io/~gitbook/mcp
+```
+
+Note: The MCP server indexes the latest released docs, not the develop branch. For full setup details and editor alternatives, see docs/book/reference/llms-txt.md.
+
 ## Code Style & Quality Standards
 
 ### Formatting and Linting

README.md

@@ -127,7 +127,7 @@ The MCP (Model Context Protocol) integration transforms your ZenML metadata into
 
 The best way to learn about ZenML is through our comprehensive documentation and tutorials:
 
-- **[Your First AI Pipeline](https://docs.zenml.io/your-first-ai-pipeline)** - Build and evaluate an AI service in minutes
+- **[Your First AI Pipeline](https://docs.zenml.io/getting-started/your-first-ai-pipeline)** - Build and evaluate an AI service in minutes
 - **[Starter Guide](https://docs.zenml.io/user-guides/starter-guide)** - From zero to production in 30 minutes
 - **[LLMOps Guide](https://docs.zenml.io/user-guides/llmops-guide)** - Specific patterns for LLM applications
 - **[SDK Reference](https://sdkdocs.zenml.io/)** - Complete SDK reference

RELEASE_NOTES.md

@@ -1,5 +1,81 @@
 <!-- markdown-link-check-disable -->
 
+# 0.91.0
+
+This release brings major deployment enhancements (Local Deployer, fully customizable deployment server, and deployment visualizations), powerful caching controls (file/object-based cache invalidation, cache expiration, and custom cache functions), Python 3.13 support, an MLX array materializer, and a new option to customize image tags during builds. It also deprecates legacy model deployers and drops Python 3.9 support.
+
+## ⚠️ Breaking Changes
+
+- Drop Python 3.9 support
+
+## Deprecations
+
+- Model deployers are deprecated; use deployers to deploy entire pipelines instead
+
+## New Features
+
+### Deployment
+- Local Deployer for deploying pipelines locally
+- Deployment server is fully customizable via settings
+- Attach custom visualizations to deployments
+
+### Caching
+- Specify files or Python objects that invalidate a step's cache
+- Cache expiration to bound cache lifetime
+- Custom cache function for advanced invalidation logic
+
+### Other
+- MLX array materializer
+- Python 3.13 support
+- Customize the image tag of built Docker images
+
+## Bug Fixes
+
+- Fix incompatibility of print capturing when using `numba`
+- Fix the mount point configuration in the Hashicorp Vault secrets store
+
+## What's Changed
+* Add version 0.85.0 to legacy docs by @github-actions[bot] in https://github.com/zenml-io/zenml/pull/4029
+* Add 0.90.0 to the migration tests by @github-actions[bot] in https://github.com/zenml-io/zenml/pull/4031
+* Document pipeline deployments and deployers by @stefannica in https://github.com/zenml-io/zenml/pull/4017
+* Snapshot docs by @schustmi in https://github.com/zenml-io/zenml/pull/4023
+* Update examples for deployed pipelines by @safoinme in https://github.com/zenml-io/zenml/pull/3960
+* Quote helm chart description by @schustmi in https://github.com/zenml-io/zenml/pull/4032
+* Add MLX array materializer by @nicholasjng in https://github.com/zenml-io/zenml/pull/4027
+* Better CLI error messages by @schustmi in https://github.com/zenml-io/zenml/pull/4035
+* Prevent MLX integration install on Intel Macs by @strickvl in https://github.com/zenml-io/zenml/pull/4039
+* Fixes for timestamps in logs by @bcdurak in https://github.com/zenml-io/zenml/pull/4052
+* Fix breaking `mypy` checks by @Json-Andriopoulos in https://github.com/zenml-io/zenml/pull/4051
+* Extend list pipeline filters by @Json-Andriopoulos in https://github.com/zenml-io/zenml/pull/3990
+* Add image tag docker setting by @Json-Andriopoulos in https://github.com/zenml-io/zenml/pull/4025
+* Small docs improvements by @schustmi in https://github.com/zenml-io/zenml/pull/4041
+* Fix secret tags and logs for the AWS deployer by @stefannica in https://github.com/zenml-io/zenml/pull/4056
+* Fix snapshot exclusive tagging by @schustmi in https://github.com/zenml-io/zenml/pull/4034
+* Removing 3.9 & Adding 3.13 by @bcdurak in https://github.com/zenml-io/zenml/pull/4053
+* Fix for inconsistencies between numba and the wrapped print by @bcdurak in https://github.com/zenml-io/zenml/pull/4060
+* Fix the user avatar URL field length by @stefannica in https://github.com/zenml-io/zenml/pull/4065
+* Additional sorting options for snapshots and deployments by @schustmi in https://github.com/zenml-io/zenml/pull/4033
+* Document how to use Pro API keys to access the Pro workspace API by @stefannica in https://github.com/zenml-io/zenml/pull/4066
+* Upgrading mypy by @bcdurak in https://github.com/zenml-io/zenml/pull/4062
+* Update link for Your First AI Pipeline in README.md by @htahir1 in https://github.com/zenml-io/zenml/pull/4069
+* Add snapshot delete command to CLI by @strickvl in https://github.com/zenml-io/zenml/pull/4059
+* Deprecate model deployer in docs by @htahir1 in https://github.com/zenml-io/zenml/pull/4076
+* Additional caching configurations by @schustmi in https://github.com/zenml-io/zenml/pull/4040
+* Log metadata changes by @Json-Andriopoulos in https://github.com/zenml-io/zenml/pull/4049
+* Customizable Deployment server settings by @stefannica in https://github.com/zenml-io/zenml/pull/4064
+* Add GitBook MCP server documentation by @strickvl in https://github.com/zenml-io/zenml/pull/4080
+* Make sure image and user parameters are propagated by @zavidnyi in https://github.com/zenml-io/zenml/pull/4081
+* Fix image building bug introduced by deployment requirements by @stefannica in https://github.com/zenml-io/zenml/pull/4082
+* Local Deployer implementation by @stefannica in https://github.com/zenml-io/zenml/pull/4085
+* Fix the mount point configuration in the Hashicorp Vault secrets store and add it to helm chart by @stefannica in https://github.com/zenml-io/zenml/pull/4088
+* Deployment custom visualizations by @safoinme in https://github.com/zenml-io/zenml/pull/4016
+
+## New Contributors
+* @zavidnyi made their first contribution in https://github.com/zenml-io/zenml/pull/4081
+
+**Full Changelog**: https://github.com/zenml-io/zenml/compare/0.90.0...0.91.0
+
+
 # 0.90.0
 
 This release introduces pipeline snapshots and pipeline deployments, refactors the base package dependencies, and adds runtime environment variable support.

docker/zenml-server-hf-spaces.Dockerfile

@@ -59,6 +59,7 @@ ENV ZENML_SERVER_SECURE_HEADERS_CSP="frame-ancestors *;"
 # ENV ZENML_SECRETS_STORE_VAULT_ADDR=""
 # ENV ZENML_SECRETS_STORE_VAULT_TOKEN=""
 # ENV ZENML_SECRETS_STORE_VAULT_NAMESPACE=""
+# ENV ZENML_SECRETS_STORE_MOUNT_POINT=""
 # ENV ZENML_SECRETS_STORE_MAX_VERSIONS=""
 
 ENTRYPOINT ["uvicorn", "zenml.zen_server.zen_server_api:app", "--log-level", "debug", "--no-server-header"]

docs/book/component-guide/deployers/README.md

@@ -25,10 +25,11 @@ Use deployers when you need request-response patterns, and orchestrators for sch
 
 ### Deployer Flavors
 
-ZenML provides deployer implementations for different deployment environments:
+Out of the box, ZenML comes with a `local` deployer already part of the default stack that deploys pipelines on your local machine in the form of background processes. Additional Deployers are provided by integrations:
 
 | Deployer                           | Flavor    | Integration   | Notes                                                                        |
 |------------------------------------|-----------|---------------|------------------------------------------------------------------------------|
+| [Local](local.md)                  | `local`  | _built-in_  | This is the default Deployer. It deploys pipelines on your local machine in the form of background processes. Should be used only for running ZenML locally. |
 | [Docker](docker.md)                | `docker`   | Built-in      | Deploys pipelines as locally running Docker containers                                |
 | [GCP Cloud Run](gcp-cloud-run.md)            | `gcp`     | `gcp`         | Deploys pipelines to Google Cloud Run for serverless execution             |
 | [AWS App Runner](aws-app-runner.md)           | `aws`     | `aws`         | Deploys pipelines to AWS App Runner for serverless execution                       |
@@ -43,9 +44,15 @@ zenml deployer flavor list
 
 You don't need to directly interact with the ZenML deployer stack component in your code. As long as the deployer that you want to use is part of your active [ZenML stack](../../user-guide/production-guide/understand-stacks.md), you can simply deploy a pipeline or snapshot using the ZenML CLI or the ZenML SDK. The resulting deployment can be managed using the ZenML CLI or the ZenML SDK.
 
-Example:
+Examples:
 
-* set up a stack with a deployer:
+* just use the default stack - it has a default local deployer that will deploy the pipeline on your local machine in the form of a background process:
+
+```bash
+zenml stack set default
+```
+
+* or set up a new stack with a deployer in it:
 
 ```bash
 zenml deployer register docker --flavor=local
@@ -129,7 +136,7 @@ def weather_pipeline(city: str = "Paris", temperature: float = 20.0) -> str:
     return analysis
 ```
 
-For more information, see the [Deployable Pipeline Requirements](../../how-to/deployment/deployment.md#deployable-pipeline-requirements) section of the tutorial.
+For more information, see the [Deployable Pipeline Requirements](https://docs.zenml.io/concepts/deployment#deployable-pipeline-requirements) section of the tutorial.
 
 #### Deployment Lifecycle Management
 

docs/book/component-guide/deployers/aws-app-runner.md

@@ -138,7 +138,7 @@ ZenML will build a Docker image called `<CONTAINER_REGISTRY_URI>/zenml:<PIPELINE
 You can now [deploy any ZenML pipeline](https://docs.zenml.io/concepts/deployment) using the AWS App Runner deployer:
 
 ```shell
-zenml pipeline deploy my_module.my_pipeline
+zenml pipeline deploy --name my_deployment my_module.my_pipeline
 ```
 
 ### Additional configuration

docs/book/component-guide/deployers/docker.md

@@ -25,22 +25,22 @@ To use the Docker deployer, you can register it and use it in your active stack:
 ```shell
 zenml deployer register docker --flavor=docker
 
-# Register and activate a stack with the new orchestrator
+# Register and activate a stack with the new deployer
 zenml stack register docker-deployer -D docker -o default -a default --set
 ```
 {% hint style="info" %}
 ZenML will build a local Docker image called `zenml:<PIPELINE_NAME>` and use it to deploy your pipeline as a Docker container. Check out [this page](https://docs.zenml.io/how-to/customize-docker-builds/) if you want to learn more about how ZenML builds these images and how you can customize them.
 {% endhint %}
 
-You can now deploy your ZenML pipeline using the Docker deployer:
+You can now [deploy any ZenML pipeline](https://docs.zenml.io/concepts/deployment) using the Docker deployer:
 
 ```shell
 zenml pipeline deploy my_module.my_pipeline
 ```
 
 ### Additional configuration
 
-For additional configuration of the Local Docker orchestrator, you can pass the following `DockerDeployerSettings` attributes defined in the `zenml.deployers.docker.docker_deployer` module when configuring the deployer or defining or deploying your pipeline:
+For additional configuration of the Docker deployer, you can pass the following `DockerDeployerSettings` attributes defined in the `zenml.deployers.docker.docker_deployer` module when configuring the deployer or defining or deploying your pipeline:
 
 * Basic settings common to all Deployers:
 
@@ -71,7 +71,7 @@ def greet(name: str) -> str:
 
 settings = {
     "deployer": DockerDeployerSettings(
-        run_args={"port": 8000}
+        port=8000
     )
 }
 

docs/book/component-guide/deployers/gcp-cloud-run.md

@@ -114,7 +114,7 @@ ZenML will build a Docker image called `<CONTAINER_REGISTRY_URI>/zenml:<PIPELINE
 You can now [deploy any ZenML pipeline](https://docs.zenml.io/concepts/deployment) using the GCP Cloud Run deployer:
 
 ```shell
-zenml pipeline deploy my_module.my_pipeline
+zenml pipeline deploy --name my_deployment my_module.my_pipeline
 ```
 
 ### Additional configuration

docs/book/component-guide/deployers/local.md

@@ -0,0 +1,79 @@
+---
+description: Deploying pipelines on your local machine as background processes.
+---
+
+# Local Deployer
+
+The local deployer is a [deployer](./) flavor that comes built-in with ZenML and deploys pipelines on your local machine as background processes.
+
+### When to use it
+
+The local deployer is part of your default stack when you're first getting started with ZenML. Due to it running locally on your machine, it requires no additional setup and is easy to use and debug.
+
+You should use the local deployer if:
+
+* you're just getting started with ZenML and want to deploy pipelines without setting up any cloud infrastructure.
+* you're writing a new pipeline and want to experiment and debug quickly
+
+### How to deploy it
+
+The local deployer comes with ZenML and works without any additional setup.
+
+### How to use it
+
+To use the local deployer, you can register it and use it in your active stack:
+
+```shell
+zenml deployer register <DEPLOYER_NAME> --flavor=local
+
+# Register and activate a stack with the new deployer
+zenml stack register <STACK_NAME> -D <DEPLOYER_NAME> ... --set
+```
+
+You can now [deploy any ZenML pipeline](https://docs.zenml.io/concepts/deployment) using the local deployer:
+
+```shell
+zenml pipeline deploy --name my_deployment my_module.my_pipeline
+```
+### Additional configuration
+
+For additional configuration of the Local deployer, you can pass the following `LocalDeployerSettings` attributes defined in the `zenml.deployers.local.local_deployer` module when configuring the deployer or defining or deploying your pipeline:
+
+* Basic settings common to all Deployers:
+
+  * `auth_key`: A user-defined authentication key to use to authenticate with deployment API calls.
+  * `generate_auth_key`: Whether to generate and use a random authentication key instead of the user-defined one.
+  * `lcm_timeout`: The maximum time in seconds to wait for the deployment lifecycle management to complete.
+
+* Settings specific to the Local deployer:
+
+  * `port`: A custom port that the deployment server will listen on. Not set by default.
+  * `allocate_port_if_busy`: If True, allocate a free port if the configured `port` is busy or not set. Defaults to True.
+  * `port_range`: The range of ports to search for a free port. Defaults to `(8000, 65535)`.
+  * `address`: The address that the deployment server will listen on. Defaults to `127.0.0.1`.
+  * `blocking`: Whether to run the deployment in the current process instead of running it as a daemon process. Defaults to False. Use this if you want to debug issues with the deployment ASGI application itself.
+
+Check out [this docs page](https://docs.zenml.io/concepts/steps_and_pipelines/configuration) for more information on how to specify settings.
+
+For example, if you wanted to specify the port to use for the deployment, you would configure settings as follows:
+
+```python
+from zenml import step, pipeline
+from zenml.deployers.local.local_deployer import LocalDeployerSettings
+
+
+@step
+def greet(name: str) -> str:
+    return f"Hello {name}!"
+
+
+settings = {
+    "deployer": LocalDeployerSettings(
+        port=8000
+    )
+}
+
+@pipeline(settings=settings)
+def greet_pipeline(name: str = "John"):
+    greet(name=name)
+```