Skip to content

Commit 1ab8013

Browse files
v-dirichardsv-dirichards
authored
Merge pull request #274023 from JnHs/cn-scaling
content on replicating EDW workload in Azure
2 parents 0cb703c + 2df27fd commit 1ab8013

14 files changed

+852
-0
lines changed

‎articles/aks/TOC.yml

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -903,6 +903,22 @@
903903
href: /azure/developer/jenkins/deploy-from-github-to-aks?toc=/azure/aks/toc.json&bc=/azure/aks/breadcrumb/toc.json
904904
- name: Use Ansible to create AKS clusters
905905
href: /azure/developer/ansible/aks-configure-clusters?toc=/azure/aks/toc.json&bc=/azure/aks/breadcrumb/toc.json
906+
- name: AKS for Amazon EKS professionals
907+
items:
908+
- name: Migrate EKS event-driven workloads
909+
items:
910+
- name: Overview
911+
href: eks-edw-overview.md
912+
- name: Understand platform differences
913+
href: eks-edw-understand.md
914+
- name: Rearchitect for AKS
915+
href: eks-edw-rearchitect.md
916+
- name: Refactor app
917+
href: eks-edw-refactor.md
918+
- name: Prepare for deployment
919+
href: eks-edw-prepare.md
920+
- name: Deploy to Azure
921+
href: eks-edw-deploy.md
906922
- name: Reference
907923
items:
908924
- name: Azure CLI

‎articles/aks/eks-edw-deploy.md

Lines changed: 306 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,306 @@
1+
---
2+
title: Deploy AWS event-driven workflow (EDW) workload to Azure
3+
description: Learn how to deploy an AWS EDW workflow to Azure and how to validate your deployment.
4+
ms.topic: how-to
5+
ms.date: 06/20/2024
6+
author: JnHs
7+
ms.author: jenhayes
8+
---
9+
10+
# Deploy an AWS event-driven workflow (EDW) workload to Azure
11+
12+
In this article, you will deploy an [AWS EDW workload][eks-edw-overview] to Azure.
13+
14+
## Sign in to Azure
15+
16+
1. Sign in to Azure using the [`az login`][az-login] command.
17+
18+
```azurecli-interactive
19+
az login
20+
```
21+
22+
1. If your Azure account has multiple subscriptions, make sure to select the correct subscription. List the names and IDs of your subscriptions using the [`az account list`][az-account-list] command.
23+
24+
```azurecli-interactive
25+
az account list --query "[].{id: id, name:name }" --output table
26+
```
27+
28+
1. Select a specific subscription using the [`az account set`][az-account-set] command.
29+
30+
```azurecli-interactive
31+
az account set --subscription $subscriptionId
32+
```
33+
34+
## EDW workload deployment script
35+
36+
You use the `deploy.sh` script in the `deployment` directory of the [GitHub repository][github-repo] to deploy the application to Azure.
37+
38+
The script first checks that all of the [prerequisite tools][prerequisites] are installed. If not, the script terminates and displays an error message letting you know which prerequisites are missing. If this happens, review the prerequisites, install any missing tools, and then run the script again. You need [Node autoprovisioning (NAP) for AKS] feature flag registered on your Azure subscription. If it isn't already registered, the script executes an Azure CLI command to register the feature flag.
39+
40+
The script records the state of the deployment in a file called `deploy.state`, which is located in the `deployment` directory. You can use this file to set environment variables when deploying the app.
41+
42+
As the script executes the commands to configure the infrastructure for the workflow, it checks that each command executes successfully. If any issues occur, an error message is displayed, and the execution stops.
43+
44+
The script displays a log as it runs. You can persist the log by redirecting the log information output and saving it to the `install.log` file in the `logs` directory using the following command:
45+
46+
```bash
47+
./deployment/infra/deploy.sh | tee ./logs/install.log
48+
```
49+
50+
For more information, see the `./deployment/infra/deploy.sh` script in our [GitHub repository][github-repo].
51+
52+
### Workload resources
53+
54+
The deployment script creates the following Azure resources:
55+
56+
- **Azure resource group**: The [Azure resource group][azure-resource-group] that stores the resources created by the deployment script.
57+
- **Azure Storage account**: The Azure Storage account that contains the queue where messages are sent by the producer app and read by the consumer app, and the table where the consumer app stores the processed messages.
58+
- **Azure container registry**: The container registry provides a repository for the container that deploys the refactored consumer app code.
59+
- **Azure Kubernetes Service (AKS) cluster**: The AKS cluster provides Kubernetes orchestration for the consumer app container and has the following features enabled:
60+
61+
- **Node autoprovisioning (NAP)**: The implementation of the [Karpenter](https://karpenter.sh) node autoscaler on AKS.
62+
- **Kubernetes Event-driven Autoscaling (KEDA)**: [KEDA](https://keda.sh) enables pod scaling based on events, such as exceeding a specified queue depth threshold.
63+
- **Workload identity**: Allows you to attach role-based access policies to pod identities for enhanced security.
64+
- **Attached Azure container registry**: This feature allows the AKS cluster to pull images from repositories on the specified ACR instance.
65+
66+
- **Application and system node pool**: The script also creates an application and system node pool in the AKS cluster that has a taint to prevent application pods from being scheduled in the system node pool.
67+
- **AKS cluster managed identity**: The script assigns the `acrPull` role to this managed identity, which facilitates access to the attached Azure container registry for pulling images.
68+
- **Workload identity**: The script assigns the **Storage Queue Data Contributor** and **Storage Table Data Contributor** roles to provide role-based access control (RBAC) access to this managed identity, which is associated with the Kubernetes service account used as the identity for pods on which the consumer app containers are deployed.
69+
- **Two federated credentials**: One credential enables the managed identity to implement pod identity, and the other credential is used for the KEDA operator service account to provide access to the KEDA scaler to gather the metrics needed to control pod autoscaling.
70+
71+
## Deploy the EDW workload to Azure
72+
73+
- Make sure you're in the `deployment` directory of the project and deploy the workload using the following commands:
74+
75+
```bash
76+
cd deployment
77+
./deploy.sh
78+
```
79+
80+
## Validate deployment and run the workload
81+
82+
Once the deployment script completes, you can deploy the workload on the AKS cluster.
83+
84+
1. Set the source for gathering and updating the environment variables for `./deployment/environmentVariables.sh` using the following command:
85+
86+
```bash
87+
source ./deployment/environmentVariables.sh
88+
```
89+
90+
1. You need the information in the `./deployment/deploy.state` file to set environment variables for the names of the resources created in the deployment. Display the contents of the file using the following `cat` command:
91+
92+
```bash
93+
cat ./deployment/deploy.state
94+
```
95+
96+
Your output should show the following variables:
97+
98+
```output
99+
SUFFIX=
100+
RESOURCE_GROUP=
101+
AZURE_STORAGE_ACCOUNT_NAME=
102+
AZURE_QUEUE_NAME=
103+
AZURE_COSMOSDB_TABLE=
104+
AZURE_CONTAINER_REGISTRY_NAME=
105+
AKS_MANAGED_IDENTITY_NAME=
106+
AKS_CLUSTER_NAME=
107+
WORKLOAD_MANAGED_IDENTITY_NAME=
108+
SERVICE_ACCOUNT=
109+
FEDERATED_IDENTITY_CREDENTIAL_NAME=
110+
KEDA_SERVICE_ACCT_CRED_NAME=
111+
```
112+
113+
1. Read the file and create environment variables for the names of the Azure resources created by the deployment script using the following commands:
114+
115+
```bash
116+
while IFS= read -r; line do \
117+
echo "export $line" \
118+
export $line; \
119+
done < ./deployment/deploy.state
120+
```
121+
122+
1. Get the AKS cluster credentials using the [`az aks get-credentials`][az-aks-get-credentials] command.
123+
124+
```azurecli-interactive
125+
az aks get-credentials --resource-group $RESOURCE_GROUP --name $AKS_CLUSTER_NAME
126+
```
127+
128+
1. Verify that the KEDA operator pods are running in the `kube-system` namespace on the AKS cluster using the [`kubectl get`][kubectl-get] command.
129+
130+
```bash
131+
kubectl get pods --namespace kube-system | grep keda
132+
```
133+
134+
Your output should look similar to the following example output:
135+
136+
:::image type="content" source="media/eks-edw-deploy/sample-keda-response.png" alt-text="Screenshot showing an example response from the command to verify that KEDA operator pods are running.":::
137+
138+
## Generate simulated load
139+
140+
Now, you generate simulated load using the producer app to populate the queue with messages.
141+
142+
1. In a separate terminal window, navigate to the project directory.
143+
1. Set the environment variables using the steps in the [previous section](#validate-deployment-and-run-the-workload). 1. Run the producer app using the following command:
144+
145+
```python
146+
python3 ./app/keda/aqs-producer.py
147+
```
148+
149+
1. Once the app starts sending messages, switch back to the other terminal window.
150+
1. Deploy the consumer app container onto the AKS cluster using the following commands:
151+
152+
```bash
153+
chmod +x ./deployment/keda/deploy-keda-app-workload-id.sh
154+
./deployment/keda/deploy-keda-app-workload-id.sh
155+
```
156+
157+
The deployment script (`deploy-keda-app-workload-id.sh`) performs templating on the application manifest YAML specification to pass environment variables to the pod. Review the following excerpt from this script:
158+
159+
```bash
160+
cat <<EOF | kubectl apply -f -
161+
apiVersion: apps/v1
162+
kind: Deployment
163+
metadata:
164+
name: $AQS_TARGET_DEPLOYMENT
165+
namespace: $AQS_TARGET_NAMESPACE
166+
spec:
167+
replicas: 1
168+
selector:
169+
matchLabels:
170+
app: aqs-reader
171+
template:
172+
metadata:
173+
labels:
174+
app: aqs-reader
175+
azure.workload.identity/use: "true"
176+
spec:
177+
serviceAccountName: $SERVICE_ACCOUNT
178+
containers:
179+
- name: keda-queue-reader
180+
image: ${AZURE_CONTAINER_REGISTRY_NAME}.azurecr.io/aws2azure/aqs-consumer
181+
imagePullPolicy: Always
182+
env:
183+
- name: AZURE_QUEUE_NAME
184+
value: $AZURE_QUEUE_NAME
185+
- name: AZURE_STORAGE_ACCOUNT_NAME
186+
value: $AZURE_STORAGE_ACCOUNT_NAME
187+
- name: AZURE_TABLE_NAME
188+
value: $AZURE_TABLE_NAME
189+
resources:
190+
requests:
191+
memory: "64Mi"
192+
cpu: "250m"
193+
limits:
194+
memory: "128Mi"
195+
cpu: "500m"
196+
EOF
197+
```
198+
199+
The `azure.workload.identity/use` label in the `spec/template` section is the pod template for the deployment. Setting the label to `true` specifies that you're using workload identity. The `serviceAccountName` in the pod specification specifies the Kubernetes service account to associate with the workload identity. While the pod specification contains a reference for an image in a private repository, there's no `imagePullSecret` specified.
200+
201+
1. Verify that the script ran successfully using the [`kubectl get`][kubectl-get] command.
202+
203+
```bash
204+
kubectl get pods --namespace $AQS_TARGET_NAMESPACE
205+
```
206+
207+
You should see a single pod in the output.
208+
209+
## Monitor scale out for pods and nodes with k9s
210+
211+
You can use various tools to verify the operation of apps deployed to AKS, including the Azure portal and k9s. For more information on k9s, see the [k9s overview][k9s].
212+
213+
1. Install k9s on your AKS cluster using the appropriate guidance for your environment in the [k9s installation overview][k9s-install].
214+
1. Create two windows, one with a view of the pods and the other with a view of the nodes in the namespace you specified in the `AQS_TARGET_NAMESPACE` environment variable (default value is `aqs-demo`) and start k9s in each window.
215+
216+
You should see something similar to the following:
217+
218+
:::image type="content" source="media/eks-edw-deploy/sample-k9s-view.png" lightbox="media/eks-edw-deploy/sample-k9s-view.png" alt-text="Screenshot showing an example of the K9s view across two windows.":::
219+
220+
1. After you confirm that the consumer app container is installed and running on the AKS cluster, install the `ScaledObject` and trigger authentication used by KEDA for pod autoscaling by running the scaled object installation script (`keda-scaleobject-workload-id.sh`). using the following commands:
221+
222+
```bash
223+
chmod +x ./deployment/keda/keda-scaleobject-workload-id.sh
224+
./deployment/keda/keda-scaleobject-workload-id.sh
225+
```
226+
227+
The script also performs templating to inject environment variables where needed. Review the following excerpt from this script:
228+
229+
```bash
230+
cat <<EOF | kubectl apply -f -
231+
apiVersion: keda.sh/v1alpha1
232+
kind: ScaledObject
233+
metadata:
234+
name: aws2az-queue-scaleobj
235+
namespace: ${AQS_TARGET_NAMESPACE}
236+
spec:
237+
scaleTargetRef:
238+
name: ${AQS_TARGET_DEPLOYMENT} #K8s deployement to target
239+
minReplicaCount: 0 # We don't want pods if the queue is empty nginx-deployment
240+
maxReplicaCount: 15 # We don't want to have more than 15 replicas
241+
pollingInterval: 30 # How frequently we should go for metrics (in seconds)
242+
cooldownPeriod: 10 # How many seconds should we wait for downscale
243+
triggers:
244+
- type: azure-queue
245+
authenticationRef:
246+
name: keda-az-credentials
247+
metadata:
248+
queueName: ${AZURE_QUEUE_NAME}
249+
accountName: ${AZURE_STORAGE_ACCOUNT_NAME}
250+
queueLength: '5'
251+
activationQueueLength: '20' # threshold for when the scaler is active
252+
cloud: AzurePublicCloud
253+
---
254+
apiVersion: keda.sh/v1alpha1
255+
kind: TriggerAuthentication
256+
metadata:
257+
name: keda-az-credentials
258+
namespace: $AQS_TARGET_NAMESPACE
259+
spec:
260+
podIdentity:
261+
provider: azure-workload
262+
identityId: '${workloadManagedIdentityClientId}'
263+
EOF
264+
```
265+
266+
The manifest describes two resources: the **`TriggerAuthentication` object**, which specifies to KEDA that the scaled object is using pod identity for authentication, and the **`identityID` property**, which refers to the managed identity used as the workload identity.
267+
268+
When the scaled object is correctly installed and KEDA detects the scaling threshold is exceeded, it begins scheduling pods. If you're using k9s, you should see something like this:
269+
270+
:::image type="content" source="media/eks-edw-deploy/sample-k9s-scheduling-pods.png" lightbox="media/eks-edw-deploy/sample-k9s-scheduling-pods.png" alt-text="Screenshot showing an example of the K9s view with scheduling pods.":::
271+
272+
If you allow the producer to fill the queue with enough messages, KEDA might need to schedule more pods than there are nodes to serve. To accommodate this, Karpenter will kick in and start scheduling nodes. If you're using k9s, you should see something like this:
273+
274+
:::image type="content" source="media/eks-edw-deploy/sample-k9s-scheduling-nodes.png" lightbox="media/eks-edw-deploy/sample-k9s-scheduling-nodes.png" alt-text="Screenshot showing an example of the K9s view with scheduling nodes.":::
275+
276+
In these two images, notice how the number of nodes whose names contain `aks-default` increased from one to three nodes. If you stop the producer app from putting messages on the queue, eventually the consumers will reduce the queue depth below the threshold, and both KEDA and Karpenter will scale in. If you're using k9s, you should see something like this:
277+
278+
:::image type="content" source="media/eks-edw-deploy/sample-k9s-reduce.png" alt-text="Screenshot showing an example of the K9s view with reduced queue depth.":::
279+
280+
## Clean up resources
281+
282+
You can use the cleanup script (`/deployment/infra/cleanup.sh`) in our [GitHub repository][github-repo] to remove all the resources you created.
283+
284+
## Next steps
285+
286+
For more information on developing and running applications in AKS, see the following resources:
287+
288+
- [Install existing applications with Helm in AKS][helm-aks]
289+
- [Deploy and manage a Kubernetes application from Azure Marketplace in AKS][k8s-aks]
290+
- [Deploy an application that uses OpenAI on AKS][openai-aks]
291+
292+
<!-- LINKS -->
293+
[eks-edw-overview]: ./eks-edw-overview.md
294+
[az-login]: /cli/azure/authenticate-azure-cli-interactively#interactive-login
295+
[az-account-list]: /cli/azure/account#az_account_list
296+
[az-account-set]: /cli/azure/account#az_account_set
297+
[github-repo]: https://github.com/Azure-Samples/aks-event-driven-replicate-from-aws
298+
[prerequisites]: ./eks-edw-overview.md#prerequisites
299+
[azure-resource-group]: ../azure-resource-manager/management/overview.md
300+
[az-aks-get-credentials]: /cli/azure/aks#az_aks_get_credentials
301+
[kubectl-get]: https://kubernetes.io/docs/reference/kubectl/generated/kubectl_get/
302+
[k9s]: https://k9scli.io/
303+
[k9s-install]: https://k9scli.io/topics/install/
304+
[helm-aks]: ./kubernetes-helm.md
305+
[k8s-aks]: ./deploy-marketplace.md
306+
[openai-aks]: ./open-ai-quickstart.md

0 commit comments

Comments
 (0)