Merge branch 'develop' into feature/dynamic-pipelines

Author: schustmi

.cursor/rules/comments.mdc

@@ -0,0 +1,30 @@
+---
+description: Project-wide commenting policy — explain WHY the code is the way it is; avoid change-log comments.
+alwaysApply: true
+---
+
+# Commenting Policy: Explain **Why**, Not **What**
+
+**Goal:** Comments should help a future maintainer understand intent and constraints months later—not narrate edits.
+
+## Use comments to explain
+- Non-obvious design decisions / trade-offs and invariants
+- Complex logic or algorithms (what makes them tricky)
+- Business rules and constraints that drive the design
+- Purpose and contract of functions/classes (especially public APIs)
+- Edge cases and gotchas being handled and why they matter
+
+## Avoid change-tracking comments
+- “Updated from previous version”
+- “New implementation”
+- “Changed to use X instead of Y”
+- “Refactored this section”
+
+## Avoid simple explanatory comments
+- Avoid comments that are obvious descriptions of what the code does when it's
+  already clear from the code itself.
+
+## Guidance
+- Prefer self-explanatory code; add comments only where extra context is genuinely needed.
+- Write for “future you” 6+ months from now.
+- In Python, prefer docstrings for API intent and inline comments only for truly non-obvious lines.

.gitignore

@@ -189,6 +189,16 @@ test_terraform/
 # feast registry database
 examples/feast_feature_store/feast_feature_repo/data/registry.db
 
+# Computer vision example
+examples/computer_vision/*.jpg
+examples/computer_vision/*.jpeg
+examples/computer_vision/*.pt
+examples/computer_vision/data
+examples/computer_vision/runs
+examples/computer_vision/predictions
+examples/computer_vision/test_fixed_training
+examples/computer_vision/test_runs
+
 #neptune
 *.neptune/
 
@@ -200,5 +210,4 @@ zenml_tutorial/
 .claude/
 
 .local/
-# PLEASE KEEP THIS LINE AT THE EOF: never include here src/zenml/zen_server/dashboard, since it is affecting release flow
-
+# PLEASE KEEP THIS LINE AT THE EOF: never include here src/zenml/zen_server/dashboard, since it is affecting release flow

.typos.toml

@@ -38,6 +38,7 @@ cachable = "cachable"
 OT = "OT"
 cll = "cll"
 ser = "ser"
+fo = "fo"
 
 [default]
 locale = "en-us"

AGENTS.md

@@ -14,6 +14,12 @@ Use filesystem navigation tools to explore the codebase structure as needed.
 
 ## Code Style & Quality Standards
 
+### Commenting policy — explain why, not what
+- Use comments to document intent, trade‑offs, constraints, invariants, and tricky edge cases—i.e., why the code is this way—rather than narrating changes. Prefer self‑explanatory code; add comments only where extra context is needed. Write for a reader 6+ months later.
+- Use for: complex logic/algorithms, non‑obvious design decisions, business rules/constraints, API purpose/contracts, edge cases.
+- Avoid: change‑tracking comments (“Updated from previous version”, “New implementation”, “Changed to use X instead of Y”, “Refactored this section”).
+- Avoid simple explanatory comments, where it is already clear from the code itself.
+
 ### Formatting and Linting
 - Format code with: `bash scripts/format.sh` (requires Python environment with dev dependencies)
   - Run this before every commit to ensure proper formatting

CLAUDE.md

@@ -24,6 +24,12 @@ Note: The MCP server indexes the latest released docs, not the develop branch. F
 
 ## Code Style & Quality Standards
 
+### Commenting policy — explain why, not what
+- Use comments to document intent, trade‑offs, constraints, invariants, and tricky edge cases—i.e., why the code is this way—rather than narrating changes. Prefer self‑explanatory code; add comments only where extra context is needed. Write for a reader 6+ months later.
+- Use for: complex logic/algorithms, non‑obvious design decisions, business rules/constraints, API purpose/contracts, edge cases.
+- Avoid: change‑tracking comments (“Updated from previous version”, “New implementation”, “Changed to use X instead of Y”, “Refactored this section”).
+- Avoid simple explanatory comments, where it is already clear from the code itself.
+
 ### Formatting and Linting
 - Format code with: `bash scripts/format.sh` (requires Python environment with dev dependencies)
   - Run this before every commit to ensure proper formatting
@@ -297,4 +303,4 @@ When tackling complex tasks:
 ---
 
 *This document is maintained to help Claude Code work effectively with the
-ZenML codebase. For human contributors, see CONTRIBUTING.md.*
+ZenML codebase. For human contributors, see CONTRIBUTING.md.*

docs/book/api-docs/oss-api/oss-api/getting-started.md

@@ -38,9 +38,10 @@ Choosing a method:
 
 ### Using a short-lived API token
 
-You can generate a short-lived (1 hour) API token from your ZenML dashboard. This is useful when you need a fast way to make authenticated HTTP requests to the ZenML API endpoints and you don't need a long-term solution.
+You can generate a short-lived API token using the CLI or the ZenML UI. This is useful when you need a fast way to make authenticated HTTP requests to the ZenML API endpoints and you don't need a long-term solution.
+
+1. Generate a short-lived API token through the API Tokens page under your ZenML UI server settings, or the `zenml token` CLI command, as documented in the [Using an API token](../../../how-to/manage-zenml-server/connecting-to-zenml/connect-with-an-api-token.md) guide.
 
-1. Generate a short-lived API token through the API Tokens page under your ZenML dashboard server settings, as documented in the [Using an API token](../../../how-to/manage-zenml-server/connecting-to-zenml/connect-with-an-api-token.md) guide.
 2. Use the API token as the bearer token in your HTTP requests. For example, you can use the following command to check your current user:
    *   using `curl`:
 
@@ -67,24 +68,63 @@ You can generate a short-lived (1 hour) API token from your ZenML dashboard. Thi
 {% hint style="info" %}
 **Important Notes**
 
-* API tokens expire after 1 hour and cannot be retrieved after the initial generation
-* Tokens are scoped to your user account and inherit your permissions
-* For long-term programmatic access, it is instead recommended to [set up a service account and an API key](./#using-a-service-account-and-an-api-key)
+* API tokens expire after the configured expiration time (default 1 hour) and need to be renewed periodically.
+* individual API tokens cannot be revoked after they are generated. If a token is compromised, you may need to lock the user account or service account to prevent further access.
+* Tokens are scoped to the user account or service account that was used to generate them and inherit their permissions.
+* For long-term programmatic access, it is instead recommended to [set up a service account API key](./#using-a-service-account-and-an-api-key).
 {% endhint %}
 
 ### Using a service account and an API key
 
-You can use a service account's API key to obtain short-lived API tokens for programmatic access to the ZenML server's REST API. This is particularly useful when you need a long-term, secure way to make authenticated HTTP requests to the ZenML API endpoints.
+You can use a service account's API key to authenticate to the ZenML server's REST API programmatically. This is particularly useful when you need a long-term, secure way to make authenticated HTTP requests to the ZenML API endpoints.
 
-1.  Create a [service account](https://docs.zenml.io/how-to/manage-zenml-server/connecting-to-zenml/connect-with-a-service-account):
+Start by [creating a service account and an API key](https://docs.zenml.io/how-to/manage-zenml-server/connecting-to-zenml/connect-with-a-service-account), e.g.:
 
     ```shell
     zenml service-account create myserviceaccount
     ```
 
-This will print out the `<ZENML_API_KEY>`, you can use in the next steps.
+Then, there are two methods to authenticate with the API using the API key - one is simpler but less secure, the other is secure and recommended but more complex:
+
+{% tabs %}
+{% tab title="Direct API key authentication" %}
+
+{% hint style="warning" %}
+This approach, albeit simple, is not recommended because the long-lived API key is exposed with every API request, which makes it easier to be compromised. Use it only in low-risk circumstances.
+{% endhint %}
+
+Use the API key directly to authenticate your API requests by including it in the `Authorization` header. For example, you can use the following command to check your current user:
 
-2. To obtain an API token using your API key, send a POST request to the `/api/v1/login` endpoint. Here are examples using common HTTP clients:
+   *   using curl:
+
+       ```bash
+       curl -H "Authorization: Bearer YOUR_API_KEY" https://your-zenml-server/api/v1/current-user
+       ```
+   *   using wget:
+
+       ```bash
+       wget -qO- --header="Authorization: Bearer YOUR_API_KEY" https://your-zenml-server/api/v1/current-user
+       ```
+   *   using python:
+
+       ```python
+       import requests
+
+       response = requests.get(
+           "https://your-zenml-server/api/v1/current-user",
+           headers={"Authorization": f"Bearer {YOUR_API_KEY}"}
+       )
+
+       print(response.json())
+       ```
+
+{% endtab %}
+
+{% tab title="Token exchange authentication" %}
+
+Reduce the risk of API key exposure by periodically exchanging the API key for a short-lived API access token.
+
+1. To obtain an API access token using your API key, send a POST request to the `/api/v1/login` endpoint. Here are examples using common HTTP clients:
    *   using curl:
 
        ```bash
@@ -112,7 +152,7 @@ This will print out the `<ZENML_API_KEY>`, you can use in the next steps.
        print(response.json())
        ```
 
-This will return a response like this:
+This will return a response like this (the API token is the `access_token` field):
 
 ```json
 {
@@ -124,7 +164,7 @@ This will return a response like this:
 }
 ```
 
-3. Once you have obtained an API token, you can use it to authenticate your API requests by including it in the `Authorization` header. For example, you can use the following command to check your current user:
+2. Once you have obtained an API token, you can use it to authenticate your API requests by including it in the `Authorization` header. When the token expires, simply repeat the steps above to obtain a new token. For example, you can use the following command to check your current user:
    *   using curl:
 
        ```bash
@@ -148,6 +188,9 @@ This will return a response like this:
        print(response.json())
        ```
 
+{% endtab %}
+{% endtabs %}
+
 {% hint style="info" %}
 **Important notes**
 

docs/book/api-docs/pro-api/pro-api/getting-started.md

@@ -53,10 +53,10 @@ Similar to [short-lived tokens for OSS and Workspace servers](../../oss-api/oss-
 To generate a new API token for the ZenML Pro API:
 
 1. Navigate to the organization settings page in your ZenML Pro dashboard
-2.  Select "API Tokens" from the left sidebar
+2. Select "API Tokens" from the left sidebar
 
     ![API Tokens](../../../.gitbook/assets/zenml-pro-api-token-01.png)
-3.  Click the "Create new token" button. Once generated, you'll see a dialog showing your new API token.
+3. Click the "Create new token" button. Once generated, you'll see a dialog showing your new API token.
 
     ![API Tokens](../../../.gitbook/assets/zenml-pro-api-token-02.png)
 4. Simply use the API token as the bearer token in your HTTP requests. For example, you can use the following command to check your current user:
@@ -85,8 +85,9 @@ To generate a new API token for the ZenML Pro API:
 {% hint style="info" %}
 **Important Notes**
 
-* API tokens expire after 1 hour and cannot be retrieved after initial generation
-* Tokens are scoped to your user account and inherit your permissions
+* API tokens expire after 1 hour and need to be renewed periodically.
+* individual API tokens cannot be revoked after they are generated. If a token is compromised, you may need to lock the user account or service account to prevent further access.
+* Tokens are scoped to the user account or service account that was used to generate them and inherit their permissions.
 * As an alternative to API tokens, you can use [service accounts and API keys](./#programmatic-access-with-service-accounts-and-api-keys) for long-term access.
 {% endhint %}
 
@@ -100,22 +101,9 @@ To generate a new API token for the ZenML Pro API:
 * Follow the migration guide in [Migration of workspace level service accounts](https://docs.zenml.io/pro/core-concepts/service-accounts#migration-of-workspace-level-service-accounts) to learn how to migrate from your existing service accounts.
 {% endhint %}
 
-Organization‑level service accounts and API keys can be used to authenticate to the ZenML Pro API and to obtain short‑lived tokens for the Workspace API across all workspaces. See [Service Accounts](https://docs.zenml.io/pro/core-concepts/service-accounts) for setup and examples.
+Organization‑level service accounts and API keys can be used to authenticate to the ZenML Pro API and the Workspace API across all workspaces in your organization. See [Service Accounts](https://docs.zenml.io/pro/core-concepts/service-accounts) for setup and examples.
 
-{% hint style="warning" %}
-Which credentials should you use?
-
-- For organization/workspace/user management on the Pro management API (`cloudapi.zenml.io`), use short‑lived Pro API tokens (no service accounts).
-- For pipeline/run‑template operations on your Workspace API (your workspace URL), use temporary user tokens or—recommended for automation—service accounts + API keys.
-
-See the high‑level overview: [Connect to a server](https://docs.zenml.io/how-to/manage-zenml-server/connecting-to-zenml#choose-how-to-connect).
-{% endhint %}
-
-## Workspace API Authentication
-
-The **Workspace API** is different from the ZenML Pro API and supports different authentication methods. This is the API you'll use for running snapshots and other pipeline operations.
-
-### When to Use Which API
+## When to Use Which API
 
 | Task | Use This API | Authentication Method |
 |------|-------------|----------------------|
@@ -124,6 +112,12 @@ The **Workspace API** is different from the ZenML Pro API and supports different
 | Running snapshots, pipeline operations | **Workspace API** (your workspace URL) | Service accounts + API keys (recommended for automation) |
 | Pipeline development, artifact management | **Workspace API** (your workspace URL) | Temporary API tokens or service accounts |
 
+See the high‑level overview: [Connect to a server](https://docs.zenml.io/how-to/manage-zenml-server/connecting-to-zenml#choose-how-to-connect).
+
+## Workspace API Authentication
+
+The **Workspace API** is different from the ZenML Pro API and supports different authentication methods. This is the API you'll use for running snapshots and other pipeline operations.
+
 ### Workspace API Authentication Methods
 
 Programmatic access to the ZenML Pro workspace API is achieved mostly the same way as the ZenML OSS server API. This is because the Workspace API in ZenML Pro is an extension of the OSS API with some additional endpoints. The only exception is that workspace level service accounts and API keys are disabled and organization level service accounts and API keys are used instead (see [Service Accounts](https://docs.zenml.io/pro/core-concepts/service-accounts)).

docs/book/getting-started/zenml-pro/secrets-stores.md

@@ -17,7 +17,7 @@ This operation has two main stages:
 
 ## AWS Secrets Manager
 
-The authentication used by the AWS secrets store is built on the [ZenML Service Connector](https://docs.zenml.io/stacks/service-connectors/overview) of the same type as the secrets store. This means that you can use any of the [authentication methods supported by the Service Connector](https://docs.zenml.io/stacks/service-connectors/connector-types/aws-service-connector#authentication-methods) to authenticate with the secrets store.
+The authentication used by the AWS secrets store is built on the [ZenML Service Connector](https://docs.zenml.io/stacks/service-connectors/auth-management) of the same type as the secrets store. This means that you can use any of the [authentication methods supported by the Service Connector](https://docs.zenml.io/stacks/service-connectors/connector-types/aws-service-connector#authentication-methods) to authenticate with the secrets store.
 
 The recommended authentication method documented here is to use the [implicit authentication method](https://docs.zenml.io/stacks/service-connectors/connector-types/aws-service-connector#implicit-authentication), because this doesn't need any sensitive credentials to be exchanged with the ZenML Pro support team.
 
@@ -236,4 +236,4 @@ After your workspace is updated, you will see the following changes in the works
     }
   }
 }
-```
+```