Skip to content

Commit c336baf

Browse files
v-regandownerv-regandowner
authored
Merge pull request #280315 from qpetraroia/KEDA-workload-identity
KEDA Workload Identity doc v1
2 parents 3019f58 + bae9ec3 commit c336baf

File tree

2 files changed

+379
-0
lines changed

2 files changed

+379
-0
lines changed

‎articles/aks/TOC.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -779,6 +779,8 @@
779779
href: keda-deploy-add-on-arm.md
780780
- name: Use Azure CLI
781781
href: keda-deploy-add-on-cli.md
782+
- name: Securely scale your applications using the Kubernetes Event-driven Autoscaling (KEDA) add-on and workload identity
783+
href: keda-workload-identity.md
782784
- name: Kubernetes Event-driven Autoscaler (KEDA) integrations
783785
href: keda-integrations.md
784786
- name: Troubleshoot Kubernetes Event-driven Autoscaler (KEDA)

‎articles/aks/keda-workload-identity.md

Lines changed: 377 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,377 @@
1+
---
2+
title: Securely scale your applications using the Kubernetes Event-driven Autoscaling (KEDA) add-on and workload identity
3+
description: Learn how to securely scale your applications using the KEDA add-on and workload identity on Azure Kubernetes Service (AKS).
4+
ms.service: azure-kubernetes-service
5+
author: qpetraroia
6+
ms.author: qpetraroia
7+
ms.topic: how-to
8+
ms.date: 07/08/2024
9+
ms.custom: template-how-to
10+
---
11+
12+
# Securely scale your applications using the KEDA add-on and workload identity on Azure Kubernetes Service (AKS)
13+
14+
This article shows you how to securely scale your applications with the Kubernetes Event-driven Autoscaling (KEDA) add-on and workload identity on Azure Kubernetes Service (AKS).
15+
16+
[!INCLUDE [Current version callout](./includes/keda/current-version-callout.md)]
17+
18+
## Before you begin
19+
20+
- You need an Azure subscription. If you don't have an Azure subscription, you can create a [free account](https://azure.microsoft.com/free).
21+
- You need the [Azure CLI installed](/cli/azure/install-azure-cli).
22+
- Ensure you have firewall rules configured to allow access to the Kubernetes API server. For more information, see [Outbound network and FQDN rules for Azure Kubernetes Service (AKS) clusters][aks-firewall-requirements].
23+
24+
## Create a resource group
25+
26+
* Create a resource group using the [`az group create`][az-group-create] command. Make sure you replace the placeholder values with your own values.
27+
28+
```azurecli-interactive
29+
LOCATION=<azure-region>
30+
RG_NAME=<resource-group-name>
31+
32+
az group create --name $RG_NAME --location $LOCATION
33+
```
34+
35+
## Create an AKS cluster
36+
37+
1. Create an AKS cluster with the KEDA add-on, workload identity, and OIDC issuer enabled using the [`az aks create`][az-aks-create] command with the `--enable-workload-identity`, `--enable-keda`, and `--enable-oidc-issuer` flags. Make sure you replace the placeholder value with your own value.
38+
39+
```azurecli-interactive
40+
AKS_NAME=<cluster-name>
41+
42+
az aks create \
43+
--name $AKS_NAME \
44+
--resource-group $RG_NAME \
45+
--enable-workload-identity \
46+
--enable-oidc-issuer \
47+
--enable-keda \
48+
--generate-ssh-keys
49+
```
50+
51+
1. Validate the deployment was successful and make sure the cluster has KEDA, workload identity, and OIDC issuer enabled using the [`az aks show`][az-aks-show] command with the `--query` flag set to `"[workloadAutoScalerProfile, securityProfile, oidcIssuerProfile]"`.
52+
53+
```azurecli-interactive
54+
az aks show \
55+
--name $AKS_NAME \
56+
--resource-group $RG_NAME \
57+
--query "[workloadAutoScalerProfile, securityProfile, oidcIssuerProfile]"
58+
```
59+
60+
1. Connect to the cluster using the [`az aks get-credentials`][az-aks-get-credentials] command.
61+
62+
```azurecli-interactive
63+
az aks get-credentials \
64+
--name $AKS_NAME \
65+
--resource-group $RG_NAME \
66+
--overwrite-existing
67+
```
68+
69+
## Deploy Azure Service Bus
70+
71+
1. Create an Azure Service Bus namespace using the [`az servicebus namespace create`][az-servicebus-namespace-create] command. Make sure to replace the placeholder value with your own value.
72+
73+
```azurecli-interactive
74+
SB_NAME=<service-bus-name>
75+
SB_HOSTNAME="${SB_NAME}.servicebus.windows.net"
76+
77+
az servicebus namespace create \
78+
--name $SB_NAME \
79+
--resource-group $RG_NAME \
80+
--disable-local-auth
81+
```
82+
83+
1. Create an Azure Service Bus queue using the [`az servicebus queue create`][az-servicebus-queue-create] command. Make sure to replace the placeholder value with your own value.
84+
85+
```azurecli-interactive
86+
SB_QUEUE_NAME=<service-bus-queue-name>
87+
88+
az servicebus queue create \
89+
--name $SB_QUEUE_NAME \
90+
--namespace $SB_NAME \
91+
--resource-group $RG_NAME
92+
```
93+
94+
## Create a managed identity
95+
96+
1. Create a managed identity using the [`az identity create`][az-identity-create] command. Make sure to replace the placeholder value with your own value.
97+
98+
```azurecli-interactive
99+
MI_NAME=<managed-identity-name>
100+
101+
MI_CLIENT_ID=$(az identity create \
102+
--name $MI_NAME \
103+
--resource-group $RG_NAME \
104+
--query "clientId" \
105+
--output tsv)
106+
```
107+
108+
1. Get the OIDC issuer URL using the [`az aks show`][az-aks-show] command with the `--query` flag set to `oidcIssuerProfile.issuerUrl`.
109+
110+
```azurecli-interactive
111+
AKS_OIDC_ISSUER=$(az aks show \
112+
--name $AKS_NAME \
113+
--resource-group $RG_NAME \
114+
--query oidcIssuerProfile.issuerUrl \
115+
--output tsv)
116+
```
117+
118+
1. Create a federated credential between the managed identity and the namespace and service account used by the workload using the [`az identity federated-credential create`][az-identity-federated-credential-create] command. Make sure to replace the placeholder value with your own value.
119+
120+
```azurecli-interactive
121+
FED_WORKLOAD=<federated-credential-workload-name>
122+
123+
az identity federated-credential create \
124+
--name $FED_WORKLOAD \
125+
--identity-name $MI_NAME \
126+
--resource-group $RG_NAME \
127+
--issuer $AKS_OIDC_ISSUER \
128+
--subject system:serviceaccount:default:$MI_NAME \
129+
--audience api://AzureADTokenExchange
130+
```
131+
132+
1. Create a second federated credential between the managed identity and the namespace and service account used by the keda-operator using the [`az identity federated-credential create`][az-identity-federated-credential-create] command. Make sure to replace the placeholder value with your own value.
133+
134+
```azurecli-interactive
135+
FED_KEDA=<federated-credential-keda-name>
136+
137+
az identity federated-credential create \
138+
--name $FED_KEDA \
139+
--identity-name $MI_NAME \
140+
--resource-group $RG_NAME \
141+
--issuer $AKS_OIDC_ISSUER \
142+
--subject system:serviceaccount:kube-system:keda-operator \
143+
--audience api://AzureADTokenExchange
144+
```
145+
146+
## Create role assignments
147+
148+
1. Get the object ID for the managed identity using the [`az identity show`][az-identity-show] command with the `--query` flag set to `"principalId"`.
149+
150+
```azurecli-interactive
151+
MI_OBJECT_ID=$(az identity show \
152+
--name $MI_NAME \
153+
--resource-group $RG_NAME \
154+
--query "principalId" \
155+
--output tsv)
156+
```
157+
158+
1. Get the Service Bus namespace resource ID using the [`az servicebus namespace show`][az-servicebus-namespace-show] command with the `--query` flag set to `"id"`.
159+
160+
```azurecli-interactive
161+
SB_ID=$(az servicebus namespace show \
162+
--name $SB_NAME \
163+
--resource-group $RG_NAME \
164+
--query "id" \
165+
--output tsv)
166+
```
167+
168+
1. Assign the Azure Service Bus Data Owner role to the managed identity using the [`az role assignment create`][az-role-assignment-create] command.
169+
170+
```azurecli-interactive
171+
az role assignment create \
172+
--role "Azure Service Bus Data Owner" \
173+
--assignee-object-id $MI_OBJECT_ID \
174+
--assignee-principal-type ServicePrincipal \
175+
--scope $SB_ID
176+
```
177+
178+
## Enable Workload Identity on KEDA operator
179+
180+
1. After creating the federated credential for the `keda-operator` ServiceAccount, you will need to manually restart the `keda-operator` pods to ensure Workload Identity environment variables are injected into the pod.
181+
182+
```azurecli-interactive
183+
kubectl rollout restart deploy keda-operator -n kube-system
184+
```
185+
186+
1. Confirm the keda-operator pods restart
187+
```azurecli-interactive
188+
kubectl get pod -n kube-system -lapp=keda-operator -w
189+
````
190+
191+
1. Once you've confirmed the keda-operator pods have finished rolling hit `Ctrl+c` to break the previous watch command then confirm the Workload Identity environment variables have been injected.
192+
193+
```azurecli-interactive
194+
KEDA_POD_ID=$(kubectl get po -n kube-system -l app.kubernetes.io/name=keda-operator -ojsonpath='{.items[0].metadata.name}')
195+
kubectl describe po $KEDA_POD_ID -n kube-system
196+
```
197+
198+
1. You should see output similar to the following under **Environment**.
199+
200+
```text
201+
---
202+
AZURE_CLIENT_ID:
203+
AZURE_TENANT_ID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx
204+
AZURE_FEDERATED_TOKEN_FILE: /var/run/secrets/azure/tokens/azure-identity-token
205+
AZURE_AUTHORITY_HOST: https://login.microsoftonline.com/
206+
---
207+
```
208+
209+
1. Deploy a KEDA TriggerAuthentication resource that includes the User-Assigned Managed Identity's Client ID.
210+
211+
```azurecli-interactive
212+
kubectl apply -f - <<EOF
213+
apiVersion: keda.sh/v1alpha1
214+
kind: TriggerAuthentication
215+
metadata:
216+
name: azure-servicebus-auth
217+
namespace: default # this must be same namespace as the ScaledObject/ScaledJob that will use it
218+
spec:
219+
podIdentity:
220+
provider: azure-workload
221+
identityId: $MI_CLIENT_ID
222+
EOF
223+
```
224+
225+
> [!note]
226+
> With the TriggerAuthentication in place, KEDA will be able to authenticate via workload identity. The `keda-operator` Pods use the `identityId` to authenticate against Azure resources when evaluating scaling triggers.
227+
228+
## Publish messages to Azure Service Bus
229+
230+
At this point everything is configured for scaling with KEDA and Microsoft Entra Workload Identity. We will test this by deploying producer and consumer workloads.
231+
232+
1. Create a new ServiceAccount for the workloads.
233+
234+
```azurecli-interactive
235+
kubectl apply -f - <<EOF
236+
apiVersion: v1
237+
kind: ServiceAccount
238+
metadata:
239+
annotations:
240+
azure.workload.identity/client-id: $MI_CLIENT_ID
241+
name: $MI_NAME
242+
EOF
243+
```
244+
245+
1. Deploy a Job to publish 100 messages.
246+
247+
```azurecli-interactive
248+
kubectl apply -f - <<EOF
249+
apiVersion: batch/v1
250+
kind: Job
251+
metadata:
252+
name: myproducer
253+
spec:
254+
template:
255+
metadata:
256+
labels:
257+
azure.workload.identity/use: "true"
258+
spec:
259+
serviceAccountName: $MI_NAME
260+
containers:
261+
- image: ghcr.io/azure-samples/aks-app-samples/servicebusdemo:latest
262+
name: myproducer
263+
resources: {}
264+
env:
265+
- name: OPERATION_MODE
266+
value: "producer"
267+
- name: MESSAGE_COUNT
268+
value: "100"
269+
- name: AZURE_SERVICEBUS_QUEUE_NAME
270+
value: $SB_QUEUE_NAME
271+
- name: AZURE_SERVICEBUS_HOSTNAME
272+
value: $SB_HOSTNAME
273+
restartPolicy: Never
274+
EOF
275+
````
276+
277+
1. Deploy a ScaledJob resource to consume the messages. The scale trigger will be configured to scale out every 10 messages. The KEDA scaler will create 10 jobs to consume the 100 messages.
278+
279+
```azurecli-interactive
280+
kubectl apply -f - <<EOF
281+
apiVersion: keda.sh/v1alpha1
282+
kind: ScaledJob
283+
metadata:
284+
name: myconsumer-scaledjob
285+
spec:
286+
jobTargetRef:
287+
template:
288+
metadata:
289+
labels:
290+
azure.workload.identity/use: "true"
291+
spec:
292+
serviceAccountName: $MI_NAME
293+
containers:
294+
- image: ghcr.io/azure-samples/aks-app-samples/servicebusdemo:latest
295+
name: myconsumer
296+
env:
297+
- name: OPERATION_MODE
298+
value: "consumer"
299+
- name: MESSAGE_COUNT
300+
value: "10"
301+
- name: AZURE_SERVICEBUS_QUEUE_NAME
302+
value: $SB_QUEUE_NAME
303+
- name: AZURE_SERVICEBUS_HOSTNAME
304+
value: $SB_HOSTNAME
305+
restartPolicy: Never
306+
triggers:
307+
- type: azure-servicebus
308+
metadata:
309+
queueName: $SB_QUEUE_NAME
310+
namespace: $SB_NAME
311+
messageCount: "10"
312+
authenticationRef:
313+
name: azure-servicebus-auth
314+
EOF
315+
```
316+
317+
> [!note]
318+
> ScaledJob creates a Kubernetes Job resource whenever a scaling event occurs and thus a Job template needs to be passed in when creating the resource. As new Jobs are created, Pods will be deployed with workload identity bits to consume messages.
319+
320+
1. Verify the KEDA scaler worked as intended.
321+
322+
```azurecli-interactive
323+
kubectl describe scaledjob myconsumer-scaledjob
324+
```
325+
326+
1. You should see events similar to the following.
327+
328+
```text
329+
Events:
330+
Type Reason Age From Message
331+
---- ------ ---- ---- -------
332+
Normal KEDAScalersStarted 10m scale-handler Started scalers watch
333+
Normal ScaledJobReady 10m keda-operator ScaledJob is ready for scaling
334+
Warning KEDAScalerFailed 10m scale-handler context canceled
335+
Normal KEDAJobsCreated 10m scale-handler Created 10 jobs
336+
```
337+
338+
## Next steps
339+
340+
This article showed you how to securely scale your applications using the KEDA add-on and workload identity in AKS.
341+
342+
With the KEDA add-on installed on your cluster, you can [deploy a sample application][keda-sample] to start scaling apps. For information on KEDA troubleshooting, see [Troubleshoot the Kubernetes Event-driven Autoscaling (KEDA) add-on][keda-troubleshoot].
343+
344+
To learn more about KEDA, see the [upstream KEDA docs][keda].
345+
346+
<!-- LINKS - internal -->
347+
[az-provider-register]: /cli/azure/provider#az-provider-register
348+
[az-feature-register]: /cli/azure/feature#az-feature-register
349+
[az-feature-show]: /cli/azure/feature#az-feature-show
350+
[keda-troubleshoot]: /troubleshoot/azure/azure-kubernetes/troubleshoot-kubernetes-event-driven-autoscaling-add-on?context=/azure/aks/context/aks-context
351+
[aks-firewall-requirements]: outbound-rules-control-egress.md#azure-global-required-network-rules
352+
[az-aks-update]: /cli/azure/aks#az-aks-update
353+
[az-extension-add]: /cli/azure/extension#az-extension-add
354+
[az-extension-update]: /cli/azure/extension#az-extension-update
355+
[az-group-create]: /cli/azure/group#az-group-create
356+
[az-aks-create]: /cli/azure/aks#az-aks-create
357+
[az-aks-show]: /cli/azure/aks#az-aks-show
358+
[az-aks-get-credentials]: /cli/azure/aks#az-aks-get-credentials
359+
[az-servicebus-namespace-create]: /cli/azure/servicebus/namespace#az-servicebus-namespace-create
360+
[az-servicebus-queue-create]: /cli/azure/servicebus/queue#az-servicebus-queue-create
361+
[az-identity-create]: /cli/azure/identity#az-identity-create
362+
[az-identity-federated-credential-create]: /cli/azure/identity/federated-credential#az-identity-federated-credential-create
363+
[az-role-definition-list]: /cli/azure/role/definition#az-role-definition-list
364+
[az-identity-show]: /cli/azure/identity#az-identity-show
365+
[az-servicebus-namespace-show]: /cli/azure/servicebus/namespace#az-servicebus-namespace-show
366+
[az-role-assignment-create]: /cli/azure/role/assignment#az-role-assignment-create
367+
368+
<!-- LINKS - external -->
369+
[kubectl]: https://kubernetes.io/docs/user-guide/kubectl
370+
[keda-sample]: https://github.com/kedacore/sample-dotnet-worker-servicebus-queue
371+
[keda]: https://keda.sh/docs/2.12/
372+
[kubectl-apply]: https://kubernetes.io/docs/reference/kubectl/generated/kubectl_apply/
373+
[kubectl-describe]: https://kubernetes.io/docs/reference/kubectl/generated/kubectl_describe/
374+
[kubectl-logs]: https://kubernetes.io/docs/reference/kubectl/generated/kubectl_logs/
375+
[kubectl-get]: https://kubernetes.io/docs/reference/kubectl/generated/kubectl_get/
376+
[kubectl-rollout-restart]: https://kubernetes.io/docs/reference/kubectl/generated/kubectl_rollout/kubectl_rollout_restart/
377+

0 commit comments

Comments
 (0)