Skip to content

Commit 7a3bc92

Browse files
PMEds28PMEds28
authored
Merge pull request #239648 from EdB-MSFT/keda-for-prometheus
Keda for prometheus
2 parents 82e2f3f + 1212ba8 commit 7a3bc92

File tree

2 files changed

+258
-0
lines changed

2 files changed

+258
-0
lines changed

‎articles/azure-monitor/essentials/integrate-keda.md

Lines changed: 255 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,255 @@
1+
---
2+
title: Integrate KEDA with your Azure Kubernetes Service cluster
3+
description: How to integrate KEDA with your Azure Kubernetes Service cluster.
4+
author: EdB-MSFT
5+
ms.author: edbaynash
6+
services: azure-monitor
7+
ms.topic: how-to
8+
ms.date: 05/31/2023
9+
---
10+
11+
12+
# Integrate KEDA with your Azure Kubernetes Service cluster
13+
14+
KEDA is a Kubernetes-based Event Driven Autoscaler. KEDA lets you drive the scaling of any container in Kubernetes based on the load to be processed, by querying metrics from systems such as Prometheus. Integrate KEDA with your Azure Kubernetes Service (AKS) cluster to scale your workloads based on Prometheus metrics from your Azure Monitor workspace.
15+
16+
To integrate KEDA into your Azure Kubernetes Service, you have to deploy and configure a workload identity or pod identity on your cluster. The identity allows KEDA to authenticate with Azure and retrieve metrics for scaling from your Monitor workspace.
17+
18+
This article walks you through the steps to integrate KEDA into your AKS cluster using a workload identity.
19+
Note
20+
21+
> [!NOTE]
22+
> We recommend using Azure Active Directory workload identity. This authentication method replaces pod-managed identity (preview), which integrates with the Kubernetes native capabilities to federate with any external identity providers on behalf of the application.
23+
>
24+
> The open source Azure AD pod-managed identity (preview) in Azure Kubernetes Service has been deprecated as of 10/24/2022, and the project will be archived in Sept. 2023. For more information, see the deprecation notice. The AKS Managed add-on begins deprecation in Sept. 2023.
25+
26+
## Prerequisites
27+
28+
+ Azure Kubernetes Service (AKS) cluster
29+
+ Prometheus sending metrics to an Azure Monitor workspace. For more information, see [Azure Monitor managed service for Prometheus](./prometheus-metrics-overview.md).
30+
31+
32+
## Set up a workload identity
33+
34+
1. Start by setting up some environment variables. Change the values to suit your AKS cluster.
35+
36+
```bash
37+
export RESOURCE_GROUP="rg-keda-integration"
38+
export LOCATION="eastus"
39+
export SUBSCRIPTION="$(az account show --query id --output tsv)"
40+
export USER_ASSIGNED_IDENTITY_NAME="keda-int-identity"
41+
export FEDERATED_IDENTITY_CREDENTIAL_NAME="kedaFedIdentity"
42+
export SERVICE_ACCOUNT_NAMESPACE="keda"
43+
export SERVICE_ACCOUNT_NAME="keda-operator"
44+
```
45+
46+
+ `SERVICE_ACCOUNT_NAME` - KEDA must use the service account that was used to create federated credentials.
47+
+ `SERVICE_ACCOUNT_NAMESPACE` Both KEDA and service account must be in same namespace.
48+
+ `USER_ASSIGNED_IDENTITY_NAME` is the name of the Azure Active directory identity that's created for KEDA.
49+
+ `FEDERATED_IDENTITY_CREDENTIAL_NAME` is the name of the credential that's created for KEDA to use to authenticate with Azure.
50+
51+
1. If your AKS cluster hasn't been created with workload-identity or oidc-issuer enabled, you'll need to enable it. If you aren't sure, you can run the following command to check if it's enabled.
52+
53+
```azurecli
54+
az aks show --resource-group $RESOURCE_GROUP --name $AKS_CLUSTER_NAME --query oidcIssuerProfile
55+
az aks show --resource-group $RESOURCE_GROUP --name $AKS_CLUSTER_NAME --query securityProfile.workloadIdentity
56+
```
57+
58+
To enable workload identity and oidc-issuer, run the following command.
59+
60+
```azurecli
61+
az aks update -g $RESOURCE_GROUP -n $AKS_CLUSTER_NAME --enable-managed-identity --enable-oidc-issuer
62+
```
63+
64+
1. Store the OIDC issuer url in an environment variable to be used later.
65+
66+
```bash
67+
export AKS_OIDC_ISSUER="$(az aks show -n $CLUSTER_NAME -g $RESOURCE_GROUP --query "oidcIssuerProfile.issuerUrl" -otsv)"
68+
```
69+
70+
1. Create a user assigned identity for KEDA. This identity is used by KEDA to authenticate with Azure Monitor.
71+
72+
```azurecli
73+
az identity create --name $USER_ASSIGNED_IDENTITY_NAME --resource-group $RESOURCE_GROUP --location $LOCATION --subscription $SUBSCRIPTION
74+
```
75+
76+
The output will be similar to the following:
77+
78+
```json
79+
{
80+
"clientId": "abcd1234-abcd-abcd-abcd-9876543210ab",
81+
"id": "/subscriptions/abcdef01-2345-6789-0abc-def012345678/resourcegroups/rg-keda-integration/providers/Microsoft. ManagedIdentity/userAssignedIdentities/keda-int-identity",
82+
"location": "eastus",
83+
"name": "keda-int-identity",
84+
"principalId": "12345678-abcd-abcd-abcd-1234567890ab",
85+
"resourceGroup": "rg-keda-integration",
86+
"systemData": null,
87+
"tags": {},
88+
"tenantId": "1234abcd-9876-9876-9876-abcdef012345",
89+
"type": "Microsoft.ManagedIdentity/userAssignedIdentities"
90+
}
91+
```
92+
93+
1. Store the `clientId` and `tenantId` in environment variables to use later.
94+
```bash
95+
export USER_ASSIGNED_CLIENT_ID="$(az identity show --resource-group $RESOURCE_GROUP --name $USER_ASSIGNED_IDENTITY_NAME --query 'clientId' -otsv)"
96+
export TENANT_ID="$(az identity show --resource-group $RESOURCE_GROUP --name $USER_ASSIGNED_IDENTITY_NAME --query 'tenantId' -otsv)"
97+
```
98+
99+
1. Assign the *Monitoring Data Reader* role to the identity for your Azure Monitor workspace. This role allows the identity to read metrics from your workspace.
100+
101+
```azurecli
102+
az role assignment create \
103+
--assignee $USER_ASSIGNED_CLIENT_ID \
104+
--role "Monitoring Data Reader" \
105+
--scope /subscriptions/$SUBSCRIPTION/resourceGroups/<Azure Monitor Workspace resource group>/providers/microsoft.monitor/accounts/ <Azure monitor workspace name>
106+
```
107+
108+
109+
1. Create the KEDA namespace, then create Kubernetes service account. This service account is used by KEDA to authenticate with Azure.
110+
111+
```azurecli
112+
113+
az aks get-credentials -n $CLUSTER_NAME -g $RESOURCE_GROUP
114+
115+
kubectl create namespace keda
116+
117+
cat <<EOF | kubectl apply -f -
118+
apiVersion: v1
119+
kind: ServiceAccount
120+
metadata:
121+
annotations:
122+
azure.workload.identity/client-id: $USER_ASSIGNED_CLIENT_ID
123+
name: $SERVICE_ACCOUNT_NAME
124+
namespace: $SERVICE_ACCOUNT_NAMESPACE
125+
EOF
126+
```
127+
128+
1. Check your service account by running
129+
```bash
130+
kubectl describe serviceaccount workload-identity-sa -n keda
131+
```
132+
133+
1. Establish a federated credential between the service account and the user assigned identity. The federated credential allows the service account to use the user assigned identity to authenticate with Azure.
134+
135+
```azurecli
136+
az identity federated-credential create --name $FEDERATED_IDENTITY_CREDENTIAL_NAME --identity-name $USER_ASSIGNED_IDENTITY_NAME --resource-group $RESOURCE_GROUP --issuer $AKS_OIDC_ISSUER --subject system:serviceaccount:$SERVICE_ACCOUNT_NAMESPACE:$SERVICE_ACCOUNT_NAME --audience api://AzureADTokenExchange
137+
```
138+
139+
> [!Note]
140+
> It takes a few seconds for the federated identity credential to be propagated after being initially added. If a token request is made immediately after adding the federated identity credential, it might lead to failure for a couple of minutes as the cache is populated in the directory with old data. To avoid this issue, you can add a slight delay after adding the federated identity credential.
141+
142+
## Deploy KEDA
143+
144+
KEDA can be deployed using YAML manifests, Helm charts, or Operator Hub. This article uses Helm charts. For more information on deploying KEDA, see [Deploying KEDA](https://keda.sh/docs/2.10/deploy/)
145+
146+
Deploy KEDA using the following command.
147+
148+
```bash
149+
helm install keda kedacore/keda --namespace keda \
150+
--set podIdentity.azureWorkload.enabled=true \
151+
--set podIdentity.azureWorkload.clientId=$USER_ASSIGNED_CLIENT_ID \
152+
--set podIdentity.azureWorkload.tenantId=$TENANT_ID
153+
```
154+
155+
Check your deployment by running the following command.
156+
```bash
157+
kubectl get pods -n keda
158+
```
159+
The output will be similar to the following:
160+
161+
```bash
162+
NAME READY STATUS RESTARTS AGE
163+
keda-admission-webhooks-ffcb8f688-kqlxp 1/1 Running 0 4m
164+
keda-operator-5d9f7d975-mgv7r 1/1 Running 1 (4m ago) 4m
165+
keda-operator-metrics-apiserver-7dc6f59678-745nz 1/1 Running 0 4m
166+
```
167+
168+
## Scalers
169+
170+
Scalers define how and when KEDA should scale a deployment. KEDA supports a variety of scalers. For more information on scalers, see [Scalers](https://keda.sh/docs/2.10/scalers/prometheus/). Azure Managed Prometheus utilizes already existing Prometheus scaler to retrieve Prometheus metrics from Azure Monitor Workspace. The following yaml file is an example to use Azure Managed Prometheus.
171+
172+
```yml
173+
apiVersion: keda.sh/v1alpha1
174+
kind: TriggerAuthentication
175+
metadata:
176+
name: azure-managed-prometheus-trigger-auth
177+
spec:
178+
podIdentity:
179+
provider: azure-workload | azure # use "azure" for pod identity and "azure-workload" for workload identity
180+
identityId: <identity-id> # Optional. Default: Identity linked with the label set when installing KEDA.
181+
---
182+
apiVersion: keda.sh/v1alpha1
183+
kind: ScaledObject
184+
metadata:
185+
name: azure-managed-prometheus-scaler
186+
spec:
187+
scaleTargetRef:
188+
name: deployment-name-to-be-scaled
189+
minReplicaCount: 1
190+
maxReplicaCount: 20
191+
triggers:
192+
- type: prometheus
193+
metadata:
194+
serverAddress: https://test-azure-monitor-workspace-name-1234.eastus.prometheus.monitor.azure.com
195+
metricName: http_requests_total
196+
query: sum(rate(http_requests_total{deployment="my-deployment"}[2m])) # Note: query must return a vector/scalar single element response
197+
threshold: '100.50'
198+
activationThreshold: '5.5'
199+
authenticationRef:
200+
name: azure-managed-prometheus-trigger-auth
201+
```
202+
+ `serverAddress` is the Query endpoint of your Azure Monitor workspace. For more information, see [Query Prometheus metrics using the API and PromQL](./prometheus-api-promql.md#query-endpoint)
203+
+ `metricName` is the name of the metric you want to scale on.
204+
+ `query` is the query used to retrieve the metric.
205+
+ `threshold` is the value at which the deployment scales.
206+
+ Set the `podIdentity.provider` according to the type of identity you're using.
207+
208+
## Troubleshooting
209+
210+
The following section provides troubleshooting tips for common issues.
211+
212+
### Federated credentials
213+
214+
Federated credentials can take up to 10 minutes to propagate. If you're having issues with KEDA authenticating with Azure, try the following steps.
215+
216+
The following log excerpt shows an error with the federated credentials.
217+
218+
```
219+
kubectl logs -n keda keda-operator-5d9f7d975-mgv7r
220+
221+
{
222+
\"error\": \"unauthorized_client\",\n \"error_description\": \"AADSTS70021: No matching federated identity record found for presented assertion.
223+
Assertion Issuer: 'https://eastus.oic.prod-aks.azure.com/abcdef01-2345-6789-0abc-def012345678/12345678-abcd-abcd-abcd-1234567890ab/'.
224+
Assertion Subject: 'system:serviceaccount:keda:keda-operator'.
225+
Assertion Audience: 'api://AzureADTokenExchange'. https://docs.microsoft.com/azure/active-directory/develop/workload-identity-federation
226+
Trace ID: 12dd9ea0-3a65-408f-a41f-5d0403a25100\\r\\nCorrelation ID: 8a2dce68-17f1-4f11-bed2-4bcf9577f2af\\r\\nTimestamp: 2023-05-30 11:11:53Z\",
227+
\"error_codes\": [\n 70021\n ],\n \"timestamp\": \"2023-05-30 11:11:53Z\",
228+
\"trace_id\": \"12345678-3a65-408f-a41f-5d0403a25100\",
229+
\"correlation_id\": \"12345678-17f1-4f11-bed2-4bcf9577f2af\",
230+
\"error_uri\": \"https://login.microsoftonline.com/error?code=70021\"\n}
231+
\n--------------------------------------------------------------------------------\n"}
232+
```
233+
234+
Check the values used to create the ServiceAccount and the credentials created with `az identity federated-credential create` and ensure the `subject` value matches the `system:serviceaccount` value.
235+
236+
### Azure Monitor workspace permissions
237+
238+
If you're having issues with KEDA authenticating with Azure, check the permissions for the Azure Monitor workspace.
239+
The following log excerpt shows that the identity doesn't have read permissions for the Azure Monitor workspace.
240+
241+
```
242+
kubectl logs -n keda keda-operator-5d9f7d975-mgv7r
243+
244+
2023-05-30T11:15:45Z ERROR scale_handler error getting metric for scaler
245+
{"scaledObject.Namespace": "default", "scaledObject.Name": "azure-managed-prometheus-scaler", "scaler": "prometheusScaler",
246+
"error": "prometheus query api returned error. status: 403 response: {\"status\":\"error\",
247+
\"errorType\":\"Forbidden\",\"error\":\"User \\u0027abc123ab-1234-1234-abcd-abcdef123456
248+
\\u0027 does not have access to perform any of the following actions
249+
\\u0027microsoft.monitor/accounts/data/metrics/read, microsoft.monitor/accounts/data/metrics/read
250+
\\u0027 on resource \\u0027/subscriptions/abcdef01-2345-6789-0abc-def012345678/resourcegroups/rg-azmon-ws-01/providers/microsoft.monitor/accounts/azmon-ws-01\\u0027. RequestId: 123456c427f348258f3e5aeeefef834a\"}"}
251+
```
252+
253+
Ensure the identity has the `Monitoring Data Reader` role on the Azure Monitor workspace.
254+
255+

‎articles/azure-monitor/toc.yml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -493,6 +493,9 @@ items:
493493
- name: Azure Active Directory pod identity
494494
displayName: Prometheus,remote-write
495495
href: essentials/prometheus-remote-write-azure-ad-pod-identity.md
496+
- name: KEDA integration
497+
displayName: Prometheus
498+
href: essentials/integrate-keda.md
496499
- name: Prometheus Azure Active Directory authorization proxy
497500
displayName: Prometheus
498501
href: essentials/prometheus-authorization-proxy.md

0 commit comments

Comments
 (0)