Prepare release 0.84.0 (#3825)

* Adding the new version to the necessary files.

* Add release notes

* Add github urls to csp connect source

* Remove csp connect src

---------

Co-authored-by: ZenML GmbH <[email protected]>
Co-authored-by: Michael Schuster <[email protected]>

Author: 3 people

RELEASE_NOTES.md

@@ -1,5 +1,104 @@
 <!-- markdown-link-check-disable -->
 
+# 0.84.0
+
+The `0.84.0` release delivers significant **architectural improvements** and **orchestration enhancements** that boost ZenML's reliability, performance, and developer experience. This release focuses on pipeline execution resilience with step retries and early stopping capabilities, enhanced Kubernetes orchestrator features, and improved server-side processing for better scalability.
+
+## 🚀 Orchestration Enhancements
+
+### Pipeline Control & Reliability
+- **Early Pipeline Stopping**: Added ability to stop pipelines early with Kubernetes orchestrator implementation for better resource management
+- **Step Retries**: Implemented configurable step retry mechanisms for improved pipeline resilience
+- **Step Status Refresh**: Enhanced step status monitoring with real-time refresh capabilities
+
+### Kubernetes Orchestrator Improvements
+- **Run Steps using Jobs**: Leverage Kubernetes jobs for better retry behavior when running steps
+- **Enhanced Pod Management**: Always build pipeline images for Kubernetes orchestrator and improved pod caching
+- **Pod Logs Access**: Added orchestrator pod logs functionality for better debugging and monitoring
+- **Kubernetes Model Validation**: Added warnings for invalid Kubernetes model data
+
+## ⚡ Performance Improvements
+- **Threadsafe RestZenStore**: Implemented thread-safe operations for improved concurrent access
+- **Server-side Processing**: Moved parent step computation and cascading tags processing to server-side for better performance
+- **Improved Fetching**: Enhanced pipeline and step run fetching mechanisms for faster response times
+
+## 🔧 Developer Experience
+
+### CLI & Login Improvements
+- **New ZenML Login**: Redesigned login experience with improved CLI styling and user interface
+
+### Configuration & Compatibility
+- **Easier Configuration**: Simplified step operator and experiment tracker configuration for steps
+- **Unified Imports**: Standardized config imports to match documentation examples
+
+## 🔒 Infrastructure & Security
+- **Helm Chart Improvements**: Added PVC creation option for local configuration path
+- **Service Connector Security**: Hidden service connector secrets as internal implementation details
+
+## 🛠️ Fixes
+- **Model Version Fetching**: Fixed fetching model versions by UUID passed as string
+- **Visualization Handling**: Ensured visualizations are committed before potential transaction rollbacks
+- **SageMaker Configuration**: Made `SagemakerOrchestratorSettings.processor_tags` optional
+- **Data Artifact Access**: Fixed data artifact fetching issues
+- **Pydantic Validation**: Added support for step/parameter names with leading underscores in run templates
+- **Path Sanitization**: Added missing remote path and Docker tag sanitizations
+
+## 📚 Documentation
+- **Best Practices**: Updated documentation with minimum permissions for cloud stack components
+- **Concurrent Execution**: Added guidance for handling concurrent pipeline execution in separate containers
+- **User Guides**: Enhanced user guides with `llm-complete-guide` project and improved examples
+- **ZenML Pro**: Updated URLs in documentation to use zenml.io/pro
+- **VSCode Integration**: Added VSCode tutorial pipeline testing to CI
+
+## What's Changed
+* Update issue template configurations and URLs by @htahir1 in https://github.com/zenml-io/zenml/pull/3756
+* Add new release to migration tests by @schustmi in https://github.com/zenml-io/zenml/pull/3764
+* Split run submission and monitoring by @schustmi in https://github.com/zenml-io/zenml/pull/3723
+* Made --no-verify-ssl work for pro tenants as well by @AlexejPenner in https://github.com/zenml-io/zenml/pull/3767
+* Threadsafe RestZenStore by @schustmi in https://github.com/zenml-io/zenml/pull/3758
+* Server-side parent step computation by @schustmi in https://github.com/zenml-io/zenml/pull/3762
+* Unified config imports to match docs by @AlexejPenner in https://github.com/zenml-io/zenml/pull/3743
+* Fixed misleading error message by @AlexejPenner in https://github.com/zenml-io/zenml/pull/3759
+* Adjusted settings to include submodules by @AlexejPenner in https://github.com/zenml-io/zenml/pull/3757
+* Switch from pkg_resources to importlib by @schustmi in https://github.com/zenml-io/zenml/pull/3722
+* Helm chart: Add option to create PVC for local configuration path by @jsuchome in https://github.com/zenml-io/zenml/pull/3761
+* Hide service connector secrets as internal implementation details by @stefannica in https://github.com/zenml-io/zenml/pull/3770
+* Fix fetching model version by UUID passed as string by @schustmi in https://github.com/zenml-io/zenml/pull/3777
+* Ensure visualizations are committed before potential transaction rollback by @schustmi in https://github.com/zenml-io/zenml/pull/3769
+* Make SagemakerOrchestratorSettings.processor_tags optional by @schustmi in https://github.com/zenml-io/zenml/pull/3773
+* Improved pipeline/step run fetching by @schustmi in https://github.com/zenml-io/zenml/pull/3776
+* Misc CI fixes by @schustmi in https://github.com/zenml-io/zenml/pull/3780
+* Improved Kubernetes orchestrator pod caching by @schustmi in https://github.com/zenml-io/zenml/pull/3719
+* Compute cascading tags server-side by @schustmi in https://github.com/zenml-io/zenml/pull/3781
+* Add `llm-complete-guide` project to user guides by @strickvl in https://github.com/zenml-io/zenml/pull/3788
+* Docs: Handle concurrent pipeline execution in separate containers by @htahir1 in https://github.com/zenml-io/zenml/pull/3799
+* Ability to stop pipelines early + Kubernetes Orchestrator implementation by @bcdurak in https://github.com/zenml-io/zenml/pull/3716
+* Updated docs with minimum permissions sets for cloud stack components by @htahir1 in https://github.com/zenml-io/zenml/pull/3796
+* Easier option to enable a step operator/experiment tracker for a step by @schustmi in https://github.com/zenml-io/zenml/pull/3774
+* Always build pipeline image for Kubernetes orchestrator by @schustmi in https://github.com/zenml-io/zenml/pull/3797
+* Add VSCode tutorial pipeline testing to CI by @strickvl in https://github.com/zenml-io/zenml/pull/3737
+* Update best practices for upgrading ZenML servers by @htahir1 in https://github.com/zenml-io/zenml/pull/3795
+* Orchestrator pod logs for Kubernetes by @bcdurak in https://github.com/zenml-io/zenml/pull/3778
+* Warn if passing invalid kubernetes model data by @schustmi in https://github.com/zenml-io/zenml/pull/3805
+* Update ZenML Pro URLs in documentation to use zenml.io/pro by @htahir1 in https://github.com/zenml-io/zenml/pull/3813
+* Update `README.md` by @strickvl in https://github.com/zenml-io/zenml/pull/3783
+* Fix data artifact fetching by @schustmi in https://github.com/zenml-io/zenml/pull/3811
+* Step retries by @schustmi in https://github.com/zenml-io/zenml/pull/3789
+* Use pydantic validation aliases to support step/parameter names with leading underscores by @schustmi in https://github.com/zenml-io/zenml/pull/3814
+* New zenml login and CLI style improvements by @znegrin in https://github.com/zenml-io/zenml/pull/3790
+* Onboarding changes by @schustmi in https://github.com/zenml-io/zenml/pull/3817
+* Add ignore list for tutorial pipeline testing script by @strickvl in https://github.com/zenml-io/zenml/pull/3808
+* Add missing remote path sanitizations by @schustmi in https://github.com/zenml-io/zenml/pull/3816
+* Tiny README fixes by @strickvl in https://github.com/zenml-io/zenml/pull/3818
+* Add docker tag sanitization by @schustmi in https://github.com/zenml-io/zenml/pull/3820
+* Add field descriptions to config for forms by @AlexejPenner in https://github.com/zenml-io/zenml/pull/3779
+* Step Status Refresh Functionality + Kubernetes Orchestrator Implementation by @bcdurak in https://github.com/zenml-io/zenml/pull/3735
+* Upper limit datasets version by @schustmi in https://github.com/zenml-io/zenml/pull/3824
+
+
+**Full Changelog**: https://github.com/zenml-io/zenml/compare/0.83.1...0.84.0
+
+
 # 0.83.1
 
 The `0.83.1` release delivers significant **performance improvements** and optimizations that enhance ZenML's efficiency and responsiveness. This release focuses on architectural improvements including step configuration separation from deployment, idempotent POST requests with request caching, and enhanced artifact loading performance, alongside important bug fixes and documentation updates.

examples/quickstart/configs/training_aws.yaml

@@ -1,7 +1,7 @@
 # Environment configuration
 settings:
   docker:
-    parent_image: "715803424590.dkr.ecr.eu-central-1.amazonaws.com/zenml-public-pipelines:quickstart-0.83.1-py3.11-aws"
+    parent_image: "715803424590.dkr.ecr.eu-central-1.amazonaws.com/zenml-public-pipelines:quickstart-0.84.0-py3.11-aws"
     skip_build: True  # If you switch this to False remove the parent_image
     requirements: requirements.txt
     environment:

examples/quickstart/configs/training_azure.yaml

@@ -1,7 +1,7 @@
 # Environment configuration
 settings:
   docker:
-    parent_image: "zenmldocker/zenml-public-pipelines:quickstart-0.83.1-py3.11-azure"
+    parent_image: "zenmldocker/zenml-public-pipelines:quickstart-0.84.0-py3.11-azure"
     skip_build: True
     requirements: requirements.txt
     environment:

examples/quickstart/configs/training_gcp.yaml

@@ -1,7 +1,7 @@
 # Environment configuration
 settings:
   docker:
-    parent_image: "zenmldocker/zenml-public-pipelines:quickstart-0.83.1-py3.11-gcp"
+    parent_image: "zenmldocker/zenml-public-pipelines:quickstart-0.84.0-py3.11-gcp"
     skip_build: True
     requirements: requirements.txt
     environment:

examples/quickstart/quickstart.ipynb

@@ -499,7 +499,7 @@
         "# Common imports and setup\n",
         "if CLOUD_PROVIDER.lower() == \"gcp\":\n",
         "    parent_image = (\n",
-        "        \"zenmldocker/zenml-public-pipelines:quickstart-0.83.1-py3.11-gcp\"\n",
+        "        \"zenmldocker/zenml-public-pipelines:quickstart-0.84.0-py3.11-gcp\"\n",
         "    )\n",
         "    skip_build = True\n",
         "\n",
@@ -508,7 +508,7 @@
         "        SagemakerOrchestratorSettings,\n",
         "    )\n",
         "\n",
-        "    parent_image = \"339712793861.dkr.ecr.eu-central-1.amazonaws.com/zenml-public-pipelines:quickstart-0.83.1-py3.11-aws\"\n",
+        "    parent_image = \"339712793861.dkr.ecr.eu-central-1.amazonaws.com/zenml-public-pipelines:quickstart-0.84.0-py3.11-aws\"\n",
         "    skip_build = True  # if you switch this to False, you need to remove the parent image\n",
         "\n",
         "    settings[\"orchestrator.sagemaker\"] = SagemakerOrchestratorSettings(\n",
@@ -517,7 +517,7 @@
         "\n",
         "elif CLOUD_PROVIDER.lower() == \"azure\":\n",
         "    parent_image = (\n",
-        "        \"zenmldocker/zenml-public-pipelines:quickstart-0.83.1-py3.11-azure\"\n",
+        "        \"zenmldocker/zenml-public-pipelines:quickstart-0.84.0-py3.11-azure\"\n",
         "    )\n",
         "    skip_build = True\n",
         "\n",

examples/quickstart/requirements.txt

@@ -1,4 +1,4 @@
-zenml[server]==0.83.1
+zenml[server]==0.84.0
 notebook
 pyarrow
 datasets<4.0.0

examples/quickstart/requirements_aws.txt

@@ -1,4 +1,4 @@
-zenml[server]==0.83.1
+zenml[server]==0.84.0
 notebook
 pyarrow
 datasets<4.0.0

examples/quickstart/requirements_azure.txt

@@ -1,4 +1,4 @@
-zenml[server]==0.83.1
+zenml[server]==0.84.0
 notebook
 pyarrow
 datasets<4.0.0

examples/quickstart/requirements_gcp.txt

@@ -1,4 +1,4 @@
-zenml[server]==0.83.1
+zenml[server]==0.84.0
 notebook
 pyarrow
 datasets<4.0.0

helm/Chart.yaml

@@ -1,6 +1,6 @@
 apiVersion: v2
 name: zenml
-version: "0.83.1"
+version: "0.84.0"
 description: Open source MLOps framework for portable production ready ML pipelines
 keywords:
 - mlops