|
| 1 | +--- |
| 2 | +title: Troubleshooting Kubernetes Event-driven Autoscaling (KEDA) add-on |
| 3 | +description: How to troubleshoot Kubernetes Event-driven Autoscaling add-on |
| 4 | +services: container-service |
| 5 | +ms.topic: article |
| 6 | +ms.date: 8/26/2021 |
| 7 | +ms.author: tomkerkhove |
| 8 | +--- |
| 9 | + |
| 10 | +# Kubernetes Event-driven Autoscaling (KEDA) AKS add-on Troubleshooting Guides |
| 11 | + |
| 12 | +When you deploy the KEDA AKS add-on, you could possibly experience problems associated with configuration of the application autoscaler. |
| 13 | + |
| 14 | +The following guide will assist you on how to troubleshoot errors and resolve common problems with the add-on, in addition to the official KEDA [FAQ][keda-faq] & [troubleshooting guide][keda-troubleshooting]. |
| 15 | + |
| 16 | +## Verifying and Troubleshooting KEDA components |
| 17 | + |
| 18 | +### Check available KEDA version |
| 19 | + |
| 20 | +You can check the available KEDA version by using the `kubectl` command: |
| 21 | + |
| 22 | +```azurecli-interactive |
| 23 | +kubectl get crd/scaledobjects.keda.sh -o custom-columns='APP:.metadata.labels.app\.kubernetes\.io/version' |
| 24 | +``` |
| 25 | + |
| 26 | +An overview will be provided with the installed KEDA version: |
| 27 | + |
| 28 | +```Output |
| 29 | +APP |
| 30 | +2.7.0 |
| 31 | +``` |
| 32 | + |
| 33 | +### Ensuring the cluster firewall is configured correctly |
| 34 | + |
| 35 | +It might happen that KEDA isn't scaling applications because it can't start up. |
| 36 | + |
| 37 | +When checking the operator logs, you might find errors similar to the following: |
| 38 | + |
| 39 | +```output |
| 40 | +1.6545953013458195e+09 ERROR Failed to get API Group-Resources {"error": "Get \"https://10.0.0.1:443/api?timeout=32s\": EOF"} |
| 41 | +sigs.k8s.io/controller-runtime/pkg/cluster.New |
| 42 | +/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/cluster/cluster.go:160 |
| 43 | +sigs.k8s.io/controller-runtime/pkg/manager.New |
| 44 | +/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/manager/manager.go:313 |
| 45 | +main.main |
| 46 | +/workspace/main.go:87 |
| 47 | +runtime.main |
| 48 | +/usr/local/go/src/runtime/proc.go:255 |
| 49 | +1.6545953013459463e+09 ERROR setup unable to start manager {"error": "Get \"https://10.0.0.1:443/api?timeout=32s\": EOF"} |
| 50 | +main.main |
| 51 | +/workspace/main.go:97 |
| 52 | +runtime.main |
| 53 | +/usr/local/go/src/runtime/proc.go:255 |
| 54 | +``` |
| 55 | + |
| 56 | +While in the metric server you might notice that it's not able to start up: |
| 57 | + |
| 58 | +```output |
| 59 | +I0607 09:53:05.297924 1 main.go:147] keda_metrics_adapter "msg"="KEDA Version: 2.7.1" |
| 60 | +I0607 09:53:05.297979 1 main.go:148] keda_metrics_adapter "msg"="KEDA Commit: " |
| 61 | +I0607 09:53:05.297996 1 main.go:149] keda_metrics_adapter "msg"="Go Version: go1.17.9" |
| 62 | +I0607 09:53:05.298006 1 main.go:150] keda_metrics_adapter "msg"="Go OS/Arch: linux/amd64" |
| 63 | +E0607 09:53:15.344324 1 logr.go:279] keda_metrics_adapter "msg"="Failed to get API Group-Resources" "error"="Get \"https://10.0.0.1:443/api?timeout=32s\": EOF" |
| 64 | +E0607 09:53:15.344360 1 main.go:104] keda_metrics_adapter "msg"="failed to setup manager" "error"="Get \"https://10.0.0.1:443/api?timeout=32s\": EOF" |
| 65 | +E0607 09:53:15.344378 1 main.go:209] keda_metrics_adapter "msg"="making provider" "error"="Get \"https://10.0.0.1:443/api?timeout=32s\": EOF" |
| 66 | +E0607 09:53:15.344399 1 main.go:168] keda_metrics_adapter "msg"="unable to run external metrics adapter" "error"="Get \"https://10.0.0.1:443/api?timeout=32s\": EOF" |
| 67 | +``` |
| 68 | + |
| 69 | +This most likely means that the KEDA add-on isn't able to start up due to a misconfigured firewall. |
| 70 | + |
| 71 | +In order to make sure it runs correctly, make sure to configure the firewall to meet [the requirements][aks-firewall-requirements]. |
| 72 | + |
| 73 | +### Enabling add-on on clusters with self-managed open-source KEDA installations |
| 74 | + |
| 75 | +While Kubernetes only allows one metric server to be installed, you can in theory install KEDA multiple times. However, it isn't recommended given only one installation will work. |
| 76 | + |
| 77 | +When the KEDA add-on is installed in an AKS cluster, the previous installation of open-source KEDA will be overridden and the add-on will take over. |
| 78 | + |
| 79 | +This means that the customization and configuration of the self-installed KEDA deployment will get lost and no longer be applied. |
| 80 | + |
| 81 | +While there's a possibility that the existing autoscaling will keep on working, it introduces a risk given it will be configured differently and won't support features such as managed identity. |
| 82 | + |
| 83 | +It's recommended to uninstall existing KEDA installations before enabling the KEDA add-on given the installation will succeed without any error. |
| 84 | + |
| 85 | +In order to determine which metrics adapter is being used by KEDA, use the `kubectl` command: |
| 86 | + |
| 87 | +```azurecli-interactive |
| 88 | +kubectl get APIService/v1beta1.external.metrics.k8s.io -o custom-columns='NAME:.spec.service.name,NAMESPACE:.spec.service.namespace' |
| 89 | +``` |
| 90 | + |
| 91 | +An overview will be provided showing the service and namespace that Kubernetes will use to get metrics: |
| 92 | + |
| 93 | +```Output |
| 94 | +NAME NAMESPACE |
| 95 | +keda-operator-metrics-apiserver kube-system |
| 96 | +``` |
| 97 | + |
| 98 | +> [!WARNING] |
| 99 | +> If the namespace is not `kube-system`, then the AKS add-on is being ignored and another metric server is being used. |
| 100 | +
|
| 101 | +[aks-firewall-requirements]: limit-egress-traffic.md#azure-global-required-network-rules |
| 102 | +[keda-troubleshooting]: https://keda.sh/docs/latest/troubleshooting/ |
| 103 | +[keda-faq]: https://keda.sh/docs/latest/faq/ |
0 commit comments