Merge branch 'develop' into feature/dynamic-pipelines

Author: schustmi

README.md

@@ -7,7 +7,7 @@
     </a>
   <br />
   <div align="center">
-    <h3 align="center">Your unified toolkit for shipping everything from decision trees to complex AI agents.</h3>
+    <h3 align="center">One AI Platform From Pipelines to Agents </h3>
   </div>
 
   [![PyPi][pypi-shield]][pypi-url]
@@ -36,6 +36,7 @@
     <a href="https://github.com/zenml-io/zenml/issues">Report Bug</a> •
     <a href="https://zenml.io/pro">Sign up for ZenML Pro</a> •
     <a href="https://www.zenml.io/blog">Blog</a> •
+    <a href="https://docs.zenml.io/">Docs</a>
     <br />
     <br />
     🎉 For the latest release, see the <a href="https://github.com/zenml-io/zenml/releases">release notes</a>.
@@ -78,19 +79,9 @@ ZenML is used by thousands of companies to run their AI workflows. Here are some
 
 ## 🚀 Get Started (5 minutes)
 
-### 🏗️ Architecture Overview
-
-ZenML uses a [**client-server architecture**](https://docs.zenml.io/getting-started/system-architectures) with an integrated web dashboard ([zenml-io/zenml-dashboard](https://github.com/zenml-io/zenml-dashboard)):
-
-- **Local Development**: `pip install "zenml[server]"` - runs both client and server locally
-- **Production**: Deploy server separately, connect with `pip install zenml` + `zenml login <server-url>`
-
 ```bash
 # Install ZenML with server capabilities
-pip install "zenml[server]"
-
-# Install required dependencies
-pip install scikit-learn openai numpy
+pip install "zenml[server]"  # pip install zenml will install a slimmer client
 
 # Initialize your ZenML repository
 zenml init
@@ -99,9 +90,40 @@ zenml init
 zenml login
 ```
 
-Here is a brief demo:
+You can then explore any of the [examples](examples/) in this repo. We recommend starting with the [quickstart](examples/quickstart/), which demonstrates core ZenML concepts: pipelines, steps, artifacts, snapshots, and deployments.
+
+### 🏗️ Architecture Overview
+
+ZenML uses a [**client-server architecture**](https://docs.zenml.io/getting-started/system-architectures) with an integrated web dashboard ([zenml-io/zenml-dashboard](https://github.com/zenml-io/zenml-dashboard)):
+
+- **Local Development**: `pip install "zenml[local]"` - runs both client and server locally
+- **Production**: Deploy server separately, connect with `pip install zenml` + `zenml login <server-url>`
+
+## 🎮 Demo
+
+Here is a short demo:
+
+ [![Watch the video](https://img.youtube.com/vi/rzWmaHMaI88/0.jpg)](https://youtu.be/rzWmaHMaI88)
+
+## 🖼️ Resources
+
+The best way to learn about ZenML is through our comprehensive documentation and tutorials:
+
+- **[Documentation](https://docs.zenml.io/)** - Complete product documentation
+- **[Your First AI Pipeline](https://docs.zenml.io/getting-started/your-first-ai-pipeline)** - Build and evaluate an AI service in minutes
+- **[Starter Guide](https://docs.zenml.io/user-guides/starter-guide)** - From zero to production in 30 minutes
+- **[LLMOps Guide](https://docs.zenml.io/user-guides/llmops-guide)** - Specific patterns for LLM applications
+- **[SDK Reference](https://sdkdocs.zenml.io/)** - Complete SDK reference
+
+ ## 📚 More examples
 
-https://github.com/user-attachments/assets/edeb314c-fe07-41ba-b083-cd9ab11db4a7
+1. **[Agent Architecture Comparison](examples/agent_comparison/)** - Compare AI agents with LangGraph workflows, LiteLLM integration, and automatic visualizations via custom materializers
+2. **[Deploying ML Models](examples/deploying_ml_model/)** - Deploy classical ML models as production endpoints with monitoring and versioning
+3. **[Deploying Agents](examples/deploying_agent/)** - Document analysis service with pipelines, evaluation, and embedded web UI
+4. **[E2E Batch Inference](examples/e2e/)** - Complete MLOps pipeline with feature engineering
+5. **[LLM RAG Pipeline](https://github.com/zenml-io/zenml-projects/tree/main/llm-complete-guide)** - Production RAG with evaluation loops
+6. **[Agentic Workflow (Deep Research)](https://github.com/zenml-io/zenml-projects/tree/main/deep_research)** - Orchestrate your agents with ZenML
+7. **[Fine-tuning Pipeline](https://github.com/zenml-io/zenml-projects/tree/main/gamesense)** - Fine-tune and deploy LLMs
 
 ## 🗣️ Chat With Your Pipelines: ZenML MCP Server
 
@@ -121,26 +143,6 @@ Stop clicking through dashboards to understand your ML workflows. The **[ZenML M
 
 The MCP (Model Context Protocol) integration transforms your ZenML metadata into conversational insights, making pipeline debugging and analysis as easy as asking a question. Perfect for teams who want to democratize access to ML operations without requiring dashboard expertise.
 
-## 📚 Learn More
-
-### 🖼️ Getting Started Resources
-
-The best way to learn about ZenML is through our comprehensive documentation and tutorials:
-
-- **[Your First AI Pipeline](https://docs.zenml.io/getting-started/your-first-ai-pipeline)** - Build and evaluate an AI service in minutes
-- **[Starter Guide](https://docs.zenml.io/user-guides/starter-guide)** - From zero to production in 30 minutes
-- **[LLMOps Guide](https://docs.zenml.io/user-guides/llmops-guide)** - Specific patterns for LLM applications
-- **[SDK Reference](https://sdkdocs.zenml.io/)** - Complete SDK reference
-
-### 📖 Production Examples
-
-1. **[Agent Architecture Comparison](examples/agent_comparison/)** - Compare AI agents with LangGraph workflows, LiteLLM integration, and automatic visualizations via custom materializers
-2. **[Minimal Agent Production](examples/minimal_agent_production/)** - Document analysis service with pipelines, evaluation, and web UI
-3. **[E2E Batch Inference](examples/e2e/)** - Complete MLOps pipeline with feature engineering
-4. **[LLM RAG Pipeline](https://github.com/zenml-io/zenml-projects/tree/main/llm-complete-guide)** - Production RAG with evaluation loops
-5. **[Agentic Workflow (Deep Research)](https://github.com/zenml-io/zenml-projects/tree/main/deep_research)** - Orchestrate your agents with ZenML
-6. **[Fine-tuning Pipeline](https://github.com/zenml-io/zenml-projects/tree/main/gamesense)** - Fine-tune and deploy LLMs
-
 ### 🎓 Books & Resources
 
 <div align="center">
@@ -152,7 +154,7 @@ The best way to learn about ZenML is through our comprehensive documentation and
   </a>
 </div>
 
-ZenML is featured in these comprehensive guides to production AI systems.
+[ZenML](https://zenml.io) is featured in these comprehensive guides to production AI systems.
 
 ## 🤝 Join ML Engineers Building the Future of AI
 

docker/zenml-quickstart-dev.Dockerfile

@@ -11,8 +11,6 @@ ARG CLOUD_PROVIDER
 # Install the Python requirements
 RUN pip install uv
 
-RUN uv pip install "git+https://github.com/zenml-io/zenml.git@$ZENML_BRANCH" notebook pyarrow datasets transformers transformers[torch] torch sentencepiece
-
 RUN echo "Cloud Provider: $CLOUD_PROVIDER";
 # Install cloud-specific ZenML integrations
 RUN if [ "$CLOUD_PROVIDER" = "aws" ]; then \

docker/zenml-quickstart.Dockerfile

@@ -14,8 +14,6 @@ ARG CLOUD_PROVIDER
 # Install the Python requirements
 RUN pip install uv
 
-RUN uv pip install zenml${ZENML_VERSION:+==$ZENML_VERSION} notebook pyarrow datasets transformers transformers[torch] torch sentencepiece
-
 RUN echo "Cloud Provider: $CLOUD_PROVIDER";
 # Install cloud-specific ZenML integrations
 RUN if [ "$CLOUD_PROVIDER" = "aws" ]; then \

docs/book/.gitbook/assets/deploying_agent_web_ui.png

@@ -39,76 +39,131 @@ def basic_step() -> str:
 
 
 @pipeline
-def basic_pipeline():
+def basic_pipeline() -> str:
     """A simple pipeline with just one step."""
-    basic_step()
+    greeting = basic_step()
+    return greeting
 
 
 if __name__ == "__main__":
 <strong>    basic_pipeline()
 </strong></code></pre>
 
-{% hint style="success" %}
-Run this pipeline locally with `python run.py`. ZenML automatically tracks the execution and stores artifacts.
-{% endhint %}
+Run this pipeline in batch mode locally:
+
+```bash
+python run.py
+```
+ 
+You will see ZenML automatically tracks the execution and stores artifacts. View these on the CLI or on the dashboard.
+
 {% endstep %}
 
 {% step %}
-#### Create your ZenML account
+#### Create a Pipeline Snapshot (Optional but Recommended)
 
-Create a [ZenML Pro account](https://zenml.io/pro) with a 14-day free trial (no payment information required). It will provide you with a dashboard to visualize pipelines, manage infrastructure, and collaborate with team members.
+Before deploying, you can create a **snapshot** - an immutable, reproducible version of your pipeline including code, configuration, and container images:
 
-<figure><img src="../.gitbook/assets/dcp_walkthrough.gif" alt="ZenML Pro Dashboard"><figcaption><p>The ZenML Pro Dashboard</p></figcaption></figure>
+```bash
+# Create a snapshot of your pipeline
+zenml pipeline snapshot create run.basic_pipeline --name my_snapshot
+```
 
-First-time users will need to set up a workspace and project. This process might take a few minutes. In the meanwhile, feel free to check out the [Core Concepts](core-concepts.md) page to get familiar with the main ideas ZenML is built on. Once ready, connect your local environment:
+Snapshots are powerful because they:
+- **Freeze your pipeline state** - Ensure the exact same pipeline always runs
+- **Enable parameterization** - Run the same snapshot with different inputs
+- **Support team collaboration** - Share ready-to-use pipeline configurations
+- **Integrate with automation** - Trigger from dashboards, APIs, or CI/CD systems
+
+[Learn more about Snapshots](../how-to/snapshots/snapshots.md)
+{% endstep %}
+
+{% step %}
+#### Deploy your pipeline as a real-time service
+
+ZenML can deploy your pipeline (or snapshot) as a persistent HTTP service for real-time inference:
 
 ```bash
-# Log in and select your workspace
-zenml login
+# Deploy your pipeline directly
+zenml pipeline deploy run.basic_pipeline --name my_deployment
+
+# OR deploy a snapshot (if you created one above)
+zenml pipeline snapshot deploy my_snapshot --deployment my_deployment
+```
+
+Your pipeline now runs as a production-ready service! This is perfect for serving predictions to web apps, powering AI agents, or handling real-time requests.
+
+**Key insight**: When you deploy a pipeline directly with `zenml pipeline deploy`, ZenML automatically creates an implicit snapshot behind the scenes, ensuring reproducibility.
 
-# Activate your project
+[Learn more about Pipeline Deployments](../how-to/deployment/deployment.md)
+{% endstep %}
+
+{% step %}
+#### Set up a ZenML Server (For Remote Infrastructure)
+
+To use remote infrastructure (cloud deployers, orchestrators, artifact stores), you need to deploy a ZenML server to manage your pipelines centrally. You can use [ZenML Pro](https://zenml.io/pro) (managed, 14-day free trial) or [deploy it yourself](../getting-started/deploying-zenml/README.md) (self-hosted, open-source).
+
+Connect your local environment:
+
+```bash
+zenml login
 zenml project set <PROJECT_NAME>
 ```
+
+Once connected, you'll have a centralized dashboard to manage infrastructure, collaborate with team members, and schedule pipeline runs.
 {% endstep %}
 
 {% step %}
-#### Create your first remote stack
+#### Create your first remote stack (Optional)
 
-A "stack" in ZenML represents the infrastructure where your pipelines run. Moving from local to cloud resources is where ZenML truly shines.
+A "stack" in ZenML represents the infrastructure where your pipelines run. You can now scale from local development to cloud infrastructure without changing any code.
 
 <figure><img src="../.gitbook/assets/stack-deployment-options.png" alt="ZenML Stack Deployment Options"><figcaption><p>Stack deployment options</p></figcaption></figure>
 
-The fastest way to create a cloud stack is through the **Infrastructure-as-Code** option. This uses Terraform to deploy cloud resources and register them as a ZenML stack.
+Remote stacks can include:
+- **[Remote Deployers](https://docs.zenml.io/stacks/stack-components/deployers)** ([AWS App Runner](https://docs.zenml.io/stacks/stack-components/deployers/aws-app-runner), [GCP Cloud Run](https://docs.zenml.io/stacks/stack-components/deployers/gcp-cloud-run), [Azure Container Instances](https://docs.zenml.io/stacks/stack-components/container-registries/azure)) - for deploying your pipelines as scalable HTTP services on the cloud
+- **[Remote Orchestrators](https://docs.zenml.io/stacks/stack-components/orchestrators)** ([Kubernetes](https://docs.zenml.io/stacks/stack-components/orchestrators/kubernetes), [GCP Vertex AI](https://docs.zenml.io/stacks/stack-components/orchestrators/vertex), [AWS SageMaker](https://docs.zenml.io/stacks/stack-components/orchestrators/sagemaker)) - for running batch pipelines at scale
+- **[Remote Artifact Stores](https://docs.zenml.io/stacks/stack-components/artifact-stores)** ([S3](https://docs.zenml.io/stacks/stack-components/artifact-stores/s3), [GCS](https://docs.zenml.io/stacks/stack-components/artifact-stores/gcp), [Azure Blob](https://docs.zenml.io/stacks/stack-components/artifact-stores/azure)) - for storing and versioning pipeline artifacts
+
+The fastest way to create a cloud stack is through the **Infrastructure-as-Code** option, which uses Terraform to deploy cloud resources and register them as a ZenML stack.
 
 You'll need:
 
 * [Terraform](https://developer.hashicorp.com/terraform/install) version 1.9+ installed locally
 * Authentication configured for your preferred cloud provider (AWS, GCP, or Azure)
 * Appropriate permissions to create resources in your cloud account
 
-The deployment wizard will guide you through each step.
+```bash
+# Create a remote stack using the deployment wizard
+zenml stack register <STACK_NAME> \
+  --deployer <DEPLOYER_NAME> \
+  --orchestrator <ORCHESTRATOR_NAME> \
+  --artifact-store <ARTIFACT_STORE_NAME>
+```
+
+The wizard will guide you through each step.
 {% endstep %}
 
 {% step %}
-#### Run your pipeline on the remote stack
-
-Now run your pipeline in the cloud without changing any code.
+#### Deploy and run on remote infrastructure
 
-First, activate your new stack:
+Once you have a remote stack, you can:
 
+1. **Deploy your service to the cloud** - Your deployment runs on managed cloud infrastructure:
 ```bash
-zenml stack set <NAME_OF_YOUR_NEW_STACK>
+zenml stack set <REMOTE_STACK_NAME>
+zenml pipeline deploy run.basic_pipeline --name my_production_deployment
 ```
 
-Then run the exact same script:
-
+2. **Run batch pipelines at scale** - Use the same code with a cloud orchestrator:
 ```bash
-python run.py
+zenml stack set <REMOTE_STACK_NAME>
+python run.py  # Automatically runs on cloud infrastructure
 ```
 
-ZenML handles packaging code, building containers, orchestrating execution, and tracking artifacts automatically.
+ZenML handles packaging code, building containers, orchestrating execution, and tracking artifacts automatically across all cloud providers.
 
-<figure><img src="../.gitbook/assets/pipeline-run-on-the-dashboard.png" alt="Pipeline Run in ZenML Dashboard"><figcaption><p>Your pipeline in the ZenML dashboard</p></figcaption></figure>
+<figure><img src="../.gitbook/assets/pipeline-run-on-the-dashboard.png" alt="Pipeline Run in ZenML Dashboard"><figcaption><p>Your pipeline in the ZenML Pro Dashboard</p></figcaption></figure>
 {% endstep %}
 
 {% step %}
@@ -131,6 +186,8 @@ To continue your ZenML journey, explore these key topics:
 
 **For LLMs and AI Agents:**
 * **LLMOps Guide**: Write your [first AI pipeline](your-first-ai-pipeline.md) for agent development patterns
+* **Deploying Agents**: To see an example of a deployed document extraction agent, see the [deploying agents](https://github.com/zenml-io/zenml/tree/main/examples/deploying_agent) example
+* **Agent Outer Loop**: See the [Agent Outer Loop](https://github.com/zenml-io/zenml/tree/main/examples/agent_outer_loop) example to learn about training classifiers and improving agents through feedback loops
 * **Agent Evaluation**: Learn to [systematically evaluate](https://github.com/zenml-io/zenml/tree/main/examples/agent_comparison) and compare different agent architectures
 * **Prompt Management**: Version and track prompts, tools, and agent configurations as [artifacts](../how-to/artifacts/artifacts.md)