Skip to content

Commit 0e2fddf

Browse files
author
Jill Grant
authored
Merge pull request #275909 from schaffererin/userstory255690
New scenario doc: Deploy data and ML pipelines on AKS with Flyte
2 parents 6e5aa29 + 8c347ba commit 0e2fddf

File tree

2 files changed

+160
-0
lines changed

2 files changed

+160
-0
lines changed

‎articles/aks/TOC.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -883,6 +883,8 @@
883883
href: open-ai-secure-access-quickstart.md
884884
- name: Deploy an AI model with the AI toolchain operator
885885
href: ai-toolchain-operator.md
886+
- name: Deploy data and ML pipelines with Flyte
887+
href: use-flyte.md
886888
- name: DevOps
887889
items:
888890
- name: Azure DevOps Project

‎articles/aks/use-flyte.md

Lines changed: 158 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,158 @@
1+
---
2+
title: Build and deploy data and machine learning pipelines with Flyte on Azure Kubernetes Service (AKS)
3+
titleSuffix: Azure Kubernetes Service
4+
description: Learn about Flyte, an open-source platform for building and deploying data and machine learning pipelines on Azure Kubernetes Service (AKS).
5+
ms.topic: how-to
6+
ms.date: 06/06/2024
7+
author: schaffererin
8+
ms.author: schaffererin
9+
ms.service: azure-kubernetes-service
10+
---
11+
12+
# Build and deploy data and machine learning pipelines with Flyte on Azure Kubernetes Service (AKS)
13+
14+
This article shows you how to use Flyte on Azure Kubernetes Service (AKS). Flyte is an open-source workflow orchestrator that unifies machine learning, data engineering, and data analytics stacks to help you build robust and reliable applications. When using Flyte as a Kubernetes-native workflow automation tool, you can focus on experimentation and providing business value without increasing your scope to infrastructure and resource management. Keep in mind that Flyte isn't officially supported by Microsoft, so use it at your own discretion.
15+
16+
For more information, see [Introduction to Flyte][flyte].
17+
18+
## Flyte use cases
19+
20+
Flyte can be used for a variety of use cases, including:
21+
22+
* Deliver models for streamlined profit and loss financial calculations.
23+
* Process petabytes of data to efficiently conduct 3D mapping of new areas.
24+
* Quickly rollback to previous versions and minimize impact of bugs in your pipelines.
25+
26+
For more information, see [Core Flyte use cases](https://docs.flyte.org/en/latest/core_use_cases/index.html).
27+
28+
## Prerequisites
29+
30+
* An Azure subscription. If you don't have an Azure subscription, you can create a [free account][azure-free].
31+
* If you have multiple subscriptions, make sure you select the correct one using the `az account set --subscription <subscription-id>` command.
32+
* The Azure CLI installed and configured. Check your version using the `az --version` command. If you need to install or upgrade, see [Install the Azure CLI][install-azure-cli].
33+
* The Helm CLI installed and updated. Check your version using the `helm version` command. If you need to install or upgrade, see [Install Helm][install-helm].
34+
* The `kubectl` CLI installed and updated. Install it locally using the `az aks install-cli` command or using [Install kubectl][install-kubectl].
35+
* A local Docker development environment. For more information, see [Get Docker][get-docker].
36+
* `flytekit` and `flytectl` installed. For more information, see [Flyte installation][flyte-install].
37+
38+
> [!NOTE]
39+
> If you're using the Azure Cloud Shell, the Azure CLI, Helm, and kubectl are already installed.
40+
41+
### Set environment variables
42+
43+
* Set environment variables for use throughout the article. Replace the placeholder values with your own values.
44+
45+
```bash
46+
export RESOURCE_GROUP="<resource-group-name>"
47+
export LOCATION="<location>"
48+
export CLUSTER_NAME="<cluster-name>"
49+
export DNS_NAME_PREFIX="<dns-name-prefix>"
50+
```
51+
52+
## Create an AKS cluster
53+
54+
1. Create an Azure resource group for the AKS cluster using the [`az group create`][az-group-create] command.
55+
56+
```azurecli-interactive
57+
az group create --name $RESOURCE_GROUP --location $LOCATION
58+
```
59+
60+
2. Create an AKS cluster using the [`az aks create`][az-aks-create] command with the `--enable-azure-rbac`, `--enable-managed-identity`, `--enable-aad`, and `--dns-name-prefix` parameters.
61+
62+
```azurecli-interactive
63+
az aks create --resource-group $RESOURCE_GROUP --name $CLUSTER_NAME --enable-azure-rbac --enable-managed-identity --enable-aad --dns-name-prefix $DNS_NAME_PREFIX --generate-ssh-keys
64+
```
65+
66+
## Connect to your AKS cluster
67+
68+
* Configure `kubectl` to connect to your AKS cluster using the [`az aks get-credentials`][az-aks-get-credentials] command.
69+
70+
```azurecli-interactive
71+
az aks get-credentials --resource-group $RESOURCE_GROUP --name $CLUSTER_NAME
72+
```
73+
74+
## Add the Flyte Helm repository
75+
76+
* Add the Flyte Helm repository using the `helm repo add` command.
77+
78+
```bash
79+
helm repo add flyteorg https://flyteorg.github.io/flyte
80+
```
81+
82+
## Find Flyte Helm charts
83+
84+
1. Search for Flyte Helm charts using the `helm search repo` command.
85+
86+
```bash
87+
helm search repo flyteorg
88+
```
89+
90+
The following example output shows some of the available Flyte Helm charts:
91+
92+
```output
93+
NAME CHART VERSION APP VERSION DESCRIPTION
94+
flyteorg/flyte v1.12.0 A Helm chart for Flyte Sandbox
95+
flyteorg/flyte-binary v1.12.0 1.16.0 Chart for basic single Flyte executable deployment
96+
flyteorg/flyte-core v1.12.0 A Helm chart for Flyte core
97+
flyteorg/flyte-deps v1.12.0 A Helm chart for Flyte dependencies
98+
flyteorg/flyte-sandbox 0.1.0 1.16.1 A Helm chart for the Flyte local sandbox
99+
flyteorg/flyteagent v0.1.10 A Helm chart for Flyte Agent
100+
```
101+
102+
2. Update the repository using the `helm repo update` command.
103+
104+
```bash
105+
helm repo update
106+
```
107+
108+
## Deploy a Flyte chart on AKS
109+
110+
In this section, you deploy the flyte-binary Helm chart so you can begin building and deploying data and machine learning pipelines with Flyte on AKS. The flyte-binary chart is a basic single Flyte executable deployment.
111+
112+
1. Create a namespace for your Flyte deployment using the `kubectl create namespace` command.
113+
114+
```bash
115+
kubectl create namespace <namespace-name>
116+
```
117+
118+
2. Install a Flyte Helm chart using the `helm install` command. In this example, we use the `flyte-binary` chart.
119+
120+
```bash
121+
helm install flyte-binary flyteorg/flyte-core --namespace <namespace-name>
122+
```
123+
124+
3. Verify that the Flyte deployment is running using the `kubectl get services` command.
125+
126+
```bash
127+
kubectl get services --namespace <namespace-name> --output wide
128+
```
129+
130+
The following condensed example output shows the Flyte deployment:
131+
132+
```output
133+
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
134+
flyteorg-flyte-binary-grpc ClusterIP xx.x.xx.xxx <none> 81/TCP 1m
135+
flyteorg-flyte-binary-http ClusterIP xx.x.xx.xxx <none> 80/TCP 1m
136+
flyteorg-flyte-binary-webhook ClusterIP xx.x.xx.xxx <none> 80/TCP 1m
137+
```
138+
139+
## Next steps
140+
141+
In this article, you learned how to deploy a Flyte chart on AKS. To start building and deploying data and machine learning pipelines, see the following articles:
142+
143+
* [Perform exploratory data analysis (EDA) with Flyte and Jupyter notebooks][flyte-eda]
144+
* [Orchestrate an ML pipeline with Flyte to predict housing prices across regions][flyte-pipelines]
145+
146+
<!-- LINKS -->
147+
[az-group-create]: /cli/azure/group#az-group-create
148+
[az-aks-create]: /cli/azure/aks#az-aks-create
149+
[az-aks-get-credentials]: /cli/azure/aks#az-aks-get-credentials
150+
[flyte]: https://docs.flyte.org/en/latest/introduction.html
151+
[azure-free]: https://azure.microsoft.com/free
152+
[install-azure-cli]: /cli/azure/install-azure-cli
153+
[install-helm]: https://helm.sh/docs/intro/install/
154+
[install-kubectl]: https://kubernetes.io/docs/tasks/tools/install-kubectl/
155+
[get-docker]: https://docs.docker.com/get-docker/
156+
[flyte-install]: https://flyte-next.readthedocs.io/en/latest/introduction.html#installation
157+
[flyte-eda]: https://docs.flyte.org/en/latest/flytesnacks/examples/exploratory_data_analysis/index.html
158+
[flyte-pipelines]: https://docs.flyte.org/en/latest/flytesnacks/examples/house_price_prediction/index.html

0 commit comments

Comments
 (0)