Merge branch 'main' into bugfix/fixgame

Author: strickvl

.github/workflows/staging_run_complete_llm.yml

@@ -45,6 +45,18 @@ jobs:
         run: |
           zenml init
 
+      - name: List and describe ZenML projects
+        working-directory: ./llm-complete-guide
+        run: |
+          zenml project list || echo "Could not list projects"
+          zenml project describe || echo "Failed to describe project"
+
+      - name: Register Set ZenML project
+        working-directory: ./llm-complete-guide
+        run: |
+          zenml project register llm-complete-guide || echo "Failed to register project"
+          zenml project set llm-complete-guide || echo "Failed to set project"
+
       - name: Set stack (Staging)
         working-directory: ./llm-complete-guide
         run: |

.typos.toml

@@ -5,6 +5,7 @@ extend-exclude = [
     "*.ipynb",
     "llm-finetuning/*",
     "end-to-end-computer-vision/*",
+    "research-radar/data/*.jsonl",
 ]
 
 [default.extend-identifiers]
@@ -17,6 +18,8 @@ MDEyOk9yZ2FuaXphdGlvbjg4Njc2OTU1 = "MDEyOk9yZ2FuaXphdGlvbjg4Njc2OTU1"
 
 [default.extend-words]
 # Don't correct the surname "Teh"
+preprocesser = "preprocesser"
+Preprocesser = "Preprocesser"
 aks = "aks"
 hashi = "hashi"
 womens = "womens"
@@ -39,6 +42,12 @@ colours = "colours"
 initialised = "initialised"
 visualisation = "visualisation"
 customise = "customise"
+modernbert = "modernbert"
+deepseek = "deepseek"
+huggingface = "huggingface"
+answerdotai = "answerdotai"
+preprocessor = "preprocessor"
+logits = "logits"
 
 [default]
-locale = "en-us"
+locale = "en-us"

README.md

@@ -1,5 +1,5 @@
 <!-- PROJECT LOGO -->
-<img referrerpolicy="no-referrer-when-downgrade" src="https://static.scarf.sh/a.png?x-pxid=03d804c9-e44a-471e-b56d-81085bc925ec" />
+<img referrerpolicy="no-referrer-when-downgrade" src="https://static.scarf.sh/a.png?x-pxid=03d804c9-e44a-471e-b56d-81685bc925ec" />
 
 <br />
 <div align="center">
@@ -55,23 +55,20 @@ you can adapt for your application. We maintain a growing list of projects
 from various ML domains including time-series, tabular data, computer vision, 
 etc.
 
-# 🧱 Project List
-
-A list of updated and maintained projects by the ZenML team and the community:
-
-| Project | Tags | Tools |
-|---------|------|-------|
-| [LLM Agents](llm-agents) | `NLP` `LLM Agents` `Conversational AI` `RAG` `Vector Stores` `Production MLOps` | `langchain` `llama_index` `faiss` `openai` |
-| [LLM Finetuning](llm-finetuning) | `NLP` `LLM` `Model Fine-tuning` `Transfer Learning` `Parameter Optimization` | `huggingface` `pytorch` `wandb` |
-| [Complete Guide to LLMs](llm-complete-guide) | `NLP` `LLM` `RAG` `Fine-tuning` `Model Evaluation` `Embeddings` `Synthetic Data` | `openai` `supabase` `huggingface` `argilla` `gradio` `anthropic` `litellm` |
-| [LLM LoRA Finetuning](llm-lora-finetuning) | `NLP` `Parameter-Efficient Fine-tuning` `LoRA` `LLM` `Distributed Training` | `huggingface` `pytorch` `accelerate` `peft` `phi-2` |
-| [End-to-end Computer Vision](end-to-end-computer-vision) | `Computer Vision` `Object Detection` `Data Labeling` `Human-in-the-Loop` | `pytorch` `label_studio` `fiftyone` `vertex-ai` `gcp` `yolov8` |
-| [Flux Dreambooth](flux-dreambooth) | `Image Generation` `Fine-tuning` `Stable Diffusion` `LoRA` `Video Generation` | `modal` `kubernetes` `huggingface` `flux` `stable-video-diffusion` |
-| [Huggingface to Sagemaker](huggingface-sagemaker) | `Model Deployment` `NLP` `Sentiment Analysis` `Model Training` `CI/CD` | `pytorch` `mlflow` `huggingface` `aws` `sagemaker` `s3` `kubeflow` `slack` `github` |
-| [Databricks Production QA Demo](databricks-production-qa-demo) | `Quality Assurance` `CI/CD` `Model Monitoring` `Model Explainability` `Data Drift` | `databricks` `mlflow` `evidently` `shap` `slack` |
-| [ECB Interest Rate Prediction with GCP Cloud Composer](airflow-cloud-composer-etl-feature-train) | `ETL` `Time Series` `Feature Engineering` `Regression` `Workflow Orchestration` | `cloud-composer` `airflow` `vertex-ai` `bigquery` `xgboost` `gcp` |
-| [Supabase OpenAI Summary](supabase-openai-summary) | `NLP` `Text Summarization` `Database Integration` `LLM` `Automated Reporting` | `openai` `supabase` `slack` `github-actions` `gcp` |
-| [Sign Language Detection with YOLOv5](sign-language-detection-yolov5) | `Computer Vision` `Object Detection` `Real-time Processing` `Model Deployment` | `mlflow` `gcp` `bentoml` `vertex-ai` `docker` |
+| Project | Domain | Key Features | Core Technologies |
+|---------|--------|-------------|-------------------|
+| [ZenML Support Agent](zenml-support-agent) | 🤖 LLMOps | 🔍 RAG, 📊 Vector DB, 💬 Conversational | langchain, llama_index, openai |
+| [ZenCoder](zencoder) | 🤖 LLMOps | 🧠 Fine-tuning, 📈 Transfer Learning | huggingface, pytorch, wandb |
+| [Complete Guide to LLMs](llm-complete-guide) | 🤖 LLMOps | 🔍 RAG, 🧠 Fine-tuning, 📊 Evaluation | openai, huggingface, anthropic |
+| [Gamesense](gamesense) | 🤖 LLMOps | 🧠 LoRA, ⚡ Efficient Training | pytorch, peft, phi-2 |
+| [Nightwatch AI](nightwatch-ai) | 🤖 LLMOps | 📝 Summarization, 📊 Reporting | openai, supabase, slack |
+| [ResearchRadar](research-radar) | 🤖 LLMOps | 📝 Classification, 📊 Comparison | anthropic, huggingface, transformers |
+| [End-to-end Computer Vision](end-to-end-computer-vision) | 👁️ Vision | 🎯 Object Detection, 🏷️ Labeling | pytorch, label_studio, yolov8 |
+| [Magic Photobooth](magic-photobooth) | 👁️ Vision | 🖼️ Image Gen, 🎬 Video Gen | stable-diffusion, huggingface |
+| [Sign Language Detection](sign-language-detection-yolov5) | 👁️ Vision | 🎯 Object Detection, ⚡ Real-time | mlflow, bentoml, vertex-ai |
+| [Huggingface to Sagemaker](huggingface-sagemaker) | 🚀 MLOps | 🔄 CI/CD, 📦 Deployment | mlflow, sagemaker, kubeflow |
+| [Databricks Production QA](databricks-production-qa-demo) | 🚀 MLOps | 📊 Monitoring, 🔍 Quality Assurance | databricks, evidently, shap |
+| [Eurorate Predictor](eurorate-predictor) | 📊 Data | ⏱️ Time Series, 🔄 ETL | airflow, bigquery, xgboost |
 
 # 💻 System Requirements
 
@@ -80,7 +77,7 @@ Read [our docs](https://docs.zenml.io/getting-started/installation) for
 installation details.
 
 - Linux or macOS.
-- Python 3.7, 3.8, 3.9 or 3.10
+- Python >=3.9
 
 # 🪃 Contributing
 
@@ -131,21 +128,19 @@ the Apache License Version 2.0.
 | 🗳 **[Vote for Features]**  | Pick what we work on next!                                              |
 | 📓 **[Docs]**               | Full documentation for creating your own ZenML pipelines.               |
 | 📒 **[API Reference]**      | Detailed reference on ZenML's API.                                      |
-| 👨‍🍳 **[MLStacks]**        | Terraform-based infrastructure recipes for pre-made ZenML stacks.       |
 | ⚽️ **[Examples]**           | Learn best through examples where ZenML is used. We've got you covered. |
 | 📬 **[Blog]**               | Use cases of ZenML and technical deep dives on how we built it.         |
 | 🔈 **[Podcast]**            | Conversations with leaders in ML, released every 2 weeks.               |
 | 💬 **[Join Slack]**         | Need help with your specific use case? Say hi on Slack!                 |
 | 🗺 **[Roadmap]**            | See where ZenML is working to build new features.                       |
 | 🙋‍♀️ **[Contribute]**      | How to contribute to the ZenML project and code base.                   |
 
-[ZenML 101]: https://docs.zenml.io/user-guide/starter-guide
+[ZenML 101]: https://docs.zenml.io/user-guides/starter-guide
 [Core Concepts]: https://docs.zenml.io/getting-started/core-concepts
 [Our latest release]: https://github.com/zenml-io/zenml/releases
 [Vote for Features]: https://zenml.io/discussion
 [Docs]: https://docs.zenml.io/
 [API Reference]: https://apidocs.zenml.io/
-[MLStacks]: https://github.com/zenml-io/mlops-stacks
 [Examples]: https://github.com/zenml-io/zenml/tree/main/examples
 [Blog]: https://blog.zenml.io/
 [Podcast]: https://podcast.zenml.io/

databricks-production-qa-demo/steps/deployment/deployment_deploy.py

@@ -45,7 +45,7 @@ def deployment_deploy() -> Annotated[
     In this example, the step can be configured to use different input data.
     See the documentation for more information:
 
-        https://docs.zenml.io/how-to/build-pipelines/use-pipeline-step-parameters
+        https://docs.zenml.io/how-to/pipeline-development/use-configuration-files
 
     Args:
         dataset_inf: The inference dataset.

databricks-production-qa-demo/steps/etl/train_data_splitter.py

@@ -41,7 +41,7 @@ def train_data_splitter(
     In this example, the step can be configured to use different test
     set sizes. See the documentation for more information:
 
-        https://docs.zenml.io/how-to/build-pipelines/use-pipeline-step-parameters
+        https://docs.zenml.io/how-to/pipeline-development/use-configuration-files
 
     Args:
         dataset: Dataset read from source.

databricks-production-qa-demo/steps/hp_tuning/hp_tuning_single_search.py

@@ -50,7 +50,7 @@ def hp_tuning_single_search(
     to use different input datasets and also have a flag to fall back to default
     model architecture. See the documentation for more information:
 
-        https://docs.zenml.io/how-to/build-pipelines/use-pipeline-step-parameters
+        https://docs.zenml.io/how-to/pipeline-development/use-configuration-files
 
     Args:
         model_package: The package containing the model to use for hyperparameter tuning.

databricks-production-qa-demo/steps/inference/inference_predict.py

@@ -43,7 +43,7 @@ def inference_predict(
     In this example, the step can be configured to use different input data.
     See the documentation for more information:
 
-        https://docs.zenml.io/how-to/build-pipelines/use-pipeline-step-parameters
+        https://docs.zenml.io/how-to/pipeline-development/use-configuration-files
 
     Args:
         dataset_inf: The inference dataset.

databricks-production-qa-demo/steps/promotion/compute_performance_metrics_on_current_data.py

@@ -44,7 +44,7 @@ def compute_performance_metrics_on_current_data(
     and target environment stage for promotion.
     See the documentation for more information:
 
-        https://docs.zenml.io/how-to/build-pipelines/use-pipeline-step-parameters
+        https://docs.zenml.io/how-to/pipeline-development/use-configuration-files
 
     Args:
         dataset_tst: The test dataset.

databricks-production-qa-demo/steps/promotion/promote_with_metric_compare.py

@@ -46,7 +46,7 @@ def promote_with_metric_compare(
     and target environment stage for promotion.
     See the documentation for more information:
 
-        https://docs.zenml.io/how-to/build-pipelines/use-pipeline-step-parameters
+        https://docs.zenml.io/how-to/pipeline-development/use-configuration-files
 
     Args:
         latest_metric: Recently trained model metric results.

databricks-production-qa-demo/steps/training/model_trainer.py

@@ -72,7 +72,7 @@ def model_trainer(
     hyperparameters to the model constructor. See the documentation for more
     information:
 
-        https://docs.zenml.io/how-to/build-pipelines/use-pipeline-step-parameters
+        https://docs.zenml.io/how-to/pipeline-development/use-configuration-files
 
     Args:
         dataset_trn: The preprocessed train dataset.