Skip to content

Commit ab3d0b0

Browse files
EdB-MSFTEdB-MSFT
committed
Integrate KEDA with AKS
1 parent 1325148 commit ab3d0b0

File tree

1 file changed

+217
-6
lines changed

1 file changed

+217
-6
lines changed

‎articles/azure-monitor/essentials/integrating-keda.md

Lines changed: 217 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -15,13 +15,224 @@ Install KEDA
1515
Configure sacler
1616

1717

18-
To integrate KEDA with Azure Monitor, you need to Deploy and configure workload identity on your AKS cluster. This allows KEDA to authenticate with you Azure Monitor workspace and retrieve metrics for scaling.
18+
To integrate KEDA with Azure Monitor, you need to Deploy and configure workload identity or pod identity on your AKS cluster. This allows KEDA to authenticate with you Azure Monitor workspace and retrieve metrics for scaling.
1919

20+
This article will walk you through the steps to integrate KEDA with Azure Monitor using workload identity.
2021

21-
export RESOURCE_GROUP="rg-ed-kedatest-01"
22+
## Set up a workload identity
23+
24+
1. Start by setting up some environment variables. Change the values to suit your AKS cluster.Do not change these values for `SERVICE_ACCOUNT_NAMESPACE` and `SERVICE_ACCOUNT_NAME`. They are the namespace and name of the kubernetes service account that KEDA will use to authenticate with Azure Monitor.
25+
26+
`USER_ASSIGNED_IDENTITY_NAME` is the name of the Azure Active directory identity that will be created for KEDA.
27+
`FEDERATED_IDENTITY_CREDENTIAL_NAME` is the name of the credential that will be created for KEDA to use to authenticate with Azure.
28+
29+
```bash
30+
export RESOURCE_GROUP="rg-keda-integration"
2231
export LOCATION="eastus"
23-
export SERVICE_ACCOUNT_NAMESPACE="default"
24-
export SERVICE_ACCOUNT_NAME="workload-identity-sa"
2532
export SUBSCRIPTION="$(az account show --query id --output tsv)"
26-
export USER_ASSIGNED_IDENTITY_NAME="myIdentity"
27-
export FEDERATED_IDENTITY_CREDENTIAL_NAME="myFedIdentity"
33+
export USER_ASSIGNED_IDENTITY_NAME="keda-int-identity"
34+
export FEDERATED_IDENTITY_CREDENTIAL_NAME="kedaFedIdentity"
35+
export SERVICE_ACCOUNT_NAMESPACE="keda"
36+
export SERVICE_ACCOUNT_NAME="keda-operator"
37+
```
38+
39+
1. If your AKS cluster has not been created with workload-identity or oidc-issuer enabled, you will need to enable it. If you are not sure, you can run the following command to check if it is enabled.
40+
41+
```azurecli
42+
az aks show --resource-group $RESOURCE_GROUP --name $AKS_CLUSTER_NAME --query oidcIssuerProfile
43+
az aks show --resource-group $RESOURCE_GROUP --name $AKS_CLUSTER_NAME --query securityProfile.workloadIdentity
44+
```
45+
46+
To enable workload identity and oidc-issuer , run the following command.
47+
48+
```azurecli
49+
az aks update -g $RESOURCE_GROUP -n $AKS_CLUSTER_NAME --enable-managed-identity --enable-oidc-issuer
50+
```
51+
52+
1. Store the OIDC issuer url in an environment variable. This will be used later.
53+
54+
```bash
55+
export AKS_OIDC_ISSUER="$(az aks show -n $CLUSTER_NAME -g $RESOURCE_GROUP --query "oidcIssuerProfile.issuerUrl" -otsv)"
56+
```
57+
58+
1. Create a user assigned identity for KEDA. This identity will be used by KEDA to authenticate with Azure Monitor.
59+
60+
```azurecli
61+
az identity create --name $USER_ASSIGNED_IDENTITY_NAME --resource-group $RESOURCE_GROUP --location $LOCATION --subscription $SUBSCRIPTION
62+
```
63+
64+
The output will be similar to the following:
65+
66+
```json
67+
{
68+
"clientId": "abcd1234-abcd-abcd-abcd-9876543210ab",
69+
"id": "/subscriptions/abcdef01-2345-6789-0abc-def012345678/resourcegroups/rg-keda-integration/providers/Microsoft.ManagedIdentity/userAssignedIdentities/keda-int-identity",
70+
"location": "eastus",
71+
"name": "keda-int-identity",
72+
"principalId": "12345678-abcd-abcd-abcd-1234567890ab",
73+
"resourceGroup": "rg-keda-integration",
74+
"systemData": null,
75+
"tags": {},
76+
"tenantId": "1234abcd-9876-9876-9876-abcdef012345",
77+
"type": "Microsoft.ManagedIdentity/userAssignedIdentities"
78+
}
79+
```
80+
1. Store the `clientId` and `tenantId` in environment variables to use later.
81+
```bash
82+
export USER_ASSIGNED_CLIENT_ID="$(az identity show --resource-group $RESOURCE_GROUP --name $USER_ASSIGNED_IDENTITY_NAME --query 'clientId' -otsv)"
83+
export TENANT_ID="$(az identity show --resource-group $RESOURCE_GROUP --name $USER_ASSIGNED_IDENTITY_NAME --query 'tenantId' -otsv)"
84+
```
85+
86+
1. Assign the *Monitoring Data Reader* role user to identity for your Azure Monitor workspace. This will allow KEDA to read metrics from you workspace.
87+
88+
```azurecli
89+
az role assignment create \
90+
--assignee $USER_ASSIGNED_CLIENT_ID \
91+
--role "Monitoring Data Reader" \
92+
--scope /subscriptions/$SUBSCRIPTION/resourceGroups/<Azure Monitor Workspace resource group>/providers/microsoft.monitor/accounts/<Azure monitor workspace name>
93+
```
94+
95+
96+
1. Create the KEDA namespace, then create Kubernetes service account. This service account will be used by KEDA to authenticate with Azure.
97+
98+
```azurecli
99+
100+
az aks get-credentials -n $CLUSTER_NAME -g $RESOURCE_GROUP
101+
102+
kubectl create namespace keda
103+
104+
cat <<EOF | kubectl apply -f -
105+
apiVersion: v1
106+
kind: ServiceAccount
107+
metadata:
108+
annotations:
109+
azure.workload.identity/client-id: $USER_ASSIGNED_CLIENT_ID
110+
name: $SERVICE_ACCOUNT_NAME
111+
namespace: $SERVICE_ACCOUNT_NAMESPACE
112+
EOF
113+
```
114+
1. Check your service account by running
115+
```bash
116+
kubectl describe serviceaccount workload-identity-sa -n keda
117+
```
118+
119+
1. Establish the federated identity between the service account and the user assigned identity. This will allow the service account to use the user assigned identity to authenticate with Azure.
120+
121+
```azurecli
122+
az identity federated-credential create --name $FEDERATED_IDENTITY_CREDENTIAL_NAME --identity-name $USER_ASSIGNED_IDENTITY_NAME --resource-group $RESOURCE_GROUP --issuer $AKS_OIDC_ISSUER --subject system:serviceaccount:$SERVICE_ACCOUNT_NAMESPACE:$SERVICE_ACCOUNT_NAME --audience api://AzureADTokenExchange
123+
```
124+
125+
126+
## Deploy KEDA
127+
128+
KEDA can be deployed using YAML manifests or Helm charts. This article will use Helm charts. For more information on deploying KEDA, see [Deploying KEDA](https://keda.sh/docs/2.10/deploy/)
129+
130+
1. Deply KEDA using the following command.
131+
132+
133+
```bash
134+
helm install keda kedacore/keda --namespace keda \
135+
--set podIdentity.azureWorkload.enabled=true \
136+
--set podIdentity.azureWorkload.clientId=$USER_ASSIGNED_CLIENT_ID \
137+
--set podIdentity.azureWorkload.tenantId=$TENANT_ID
138+
```
139+
140+
1. Check your deployment by running the following command.
141+
```bash
142+
kubectl get pods -n keda
143+
```
144+
The outpout will be similar to the following:
145+
146+
```bash
147+
NAME READY STATUS RESTARTS AGE
148+
keda-admission-webhooks-ffcb8f688-kqlxp 1/1 Running 0 4m
149+
keda-operator-5d9f7d975-mgv7r 1/1 Running 1 (4m ago) 4m
150+
keda-operator-metrics-apiserver-7dc6f59678-745nz 1/1 Running 0 4m
151+
```
152+
153+
## Scalers
154+
155+
Scalers define how and when KEDA should scale a deployment. KEDA supports a variety of scalers. For more information on scalers, see [Scalers](https://keda.sh/docs/2.10/scalers/prometheus/)
156+
157+
The following yaml file defines a scaler.
158+
159+
The `serverAddress` is the Query endpoint of your Azure Monitor workspace. `metricName` is the name of the metric you want to scale on. The `query` is the query used to retrieve the metric. The `threshold` is the value at which the deployment will scale. Set thee `podIdentity.provider` according to the type of identity you are using.
160+
161+
```yml
162+
apiVersion: keda.sh/v1alpha1
163+
kind: TriggerAuthentication
164+
metadata:
165+
name: azure-managed-prometheus-trigger-auth
166+
spec:
167+
podIdentity:
168+
: azure-workload | azure # use "azure" for pod identity and "azure-workload" for workload identity
169+
identityId: <identity-id> # Optional. Default: Identity linked with the label set when installing KEDA.
170+
---
171+
apiVersion: keda.sh/v1alpha1
172+
kind: ScaledObject
173+
metadata:
174+
name: azure-managed-prometheus-scaler
175+
spec:
176+
scaleTargetRef:
177+
name: deployment-name-to-be-scaled
178+
minReplicaCount: 1
179+
maxReplicaCount: 20
180+
triggers:
181+
- type: prometheus
182+
metadata:
183+
serverAddress: https://test-azure-monitor-workspace-name-1234.eastus.prometheus.monitor.azure.com
184+
metricName: http_requests_total
185+
query: sum(rate(http_requests_total{deployment="my-deployment"}[2m])) # Note: query must return a vector/scalar single element response
186+
threshold: '100.50'
187+
activationThreshold: '5.5'
188+
authenticationRef:
189+
name: azure-managed-prometheus-trigger-auth
190+
```
191+
192+
## Troubleshooting
193+
194+
195+
### Federated credentials
196+
197+
Federated credentials can take up to 10 minutes to propagate. If you are having issues with KEDA authenticating with Azure, try the following steps.
198+
199+
The following log excerpt shows an error with the federated credentials.
200+
201+
```bash
202+
kubectl logs -n keda keda-operator-5d9f7d975-mgv7r
203+
204+
{
205+
\"error\": \"unauthorized_client\",\n \"error_description\": \"AADSTS70021: No matching federated identity record found for presented assertion.
206+
Assertion Issuer: 'https://eastus.oic.prod-aks.azure.com/abcdef01-2345-6789-0abc-def012345678/12345678-abcd-abcd-abcd-1234567890ab/'.
207+
Assertion Subject: 'system:serviceaccount:keda:keda-operator'.
208+
Assertion Audience: 'api://AzureADTokenExchange'. https://docs.microsoft.com/en-us/azure/active-directory/develop/workload-identity-federation
209+
Trace ID: 12dd9ea0-3a65-408f-a41f-5d0403a25100\\r\\nCorrelation ID: 8a2dce68-17f1-4f11-bed2-4bcf9577f2af\\r\\nTimestamp: 2023-05-30 11:11:53Z\",
210+
\"error_codes\": [\n 70021\n ],\n \"timestamp\": \"2023-05-30 11:11:53Z\",
211+
\"trace_id\": \"12345678-3a65-408f-a41f-5d0403a25100\",
212+
\"correlation_id\": \"12345678-17f1-4f11-bed2-4bcf9577f2af\",
213+
\"error_uri\": \"https://login.microsoftonline.com/error?code=70021\"\n}
214+
\n--------------------------------------------------------------------------------\n"}
215+
```
216+
217+
Check the values used to create the ServiceAccount and the credentials created with `az identity federated-credential create` and ensure the `subject` value matches the `system:serviceaccount` value.
218+
219+
### Azure Monitor workspace persmissions
220+
221+
If you are having issues with KEDA authenticating with Azure, check the permissions for the Azure Monitor workspace.
222+
The following log excerpt shows that the identity does not have read permissions for the Azure Monitor workspace.
223+
224+
```bash
225+
kubectl logs -n keda keda-operator-5d9f7d975-mgv7r
226+
227+
2023-05-30T11:15:45Z ERROR scale_handler error getting metric for scaler
228+
{"scaledObject.Namespace": "default", "scaledObject.Name": "azure-managed-prometheus-scaler", "scaler": "prometheusScaler",
229+
"error": "prometheus query api returned error. status: 403 response: {\"status\":\"error\",
230+
\"errorType\":\"Forbidden\",\"error\":\"User \\u0027abc123ab-1234-1234-abcd-abcdef123456
231+
\\u0027 does not have access to perform any of the following actions
232+
\\u0027microsoft.monitor/accounts/data/metrics/read, microsoft.monitor/accounts/data/metrics/read
233+
\\u0027 on resource \\u0027/subscriptions/abcdef01-2345-6789-0abc-def012345678/resourcegroups/rg-azmon-ws-01/providers/microsoft.monitor/accounts/azmon-ws-01\\u0027. RequestId: 123456c427f348258f3e5aeeefef834a\"}"}
234+
```
235+
236+
Ensure the identity has the `Monitoring Data Reader` role on the Azure Monitor workspace.
237+
238+

0 commit comments

Comments
 (0)