|
| 1 | +--- |
| 2 | +title: Configure Istio-based service mesh add-on for Azure Kubernetes Service (preview) |
| 3 | +description: Configure Istio-based service mesh add-on for Azure Kubernetes Service (preview) |
| 4 | +ms.topic: article |
| 5 | +ms.custom: devx-track-azurecli |
| 6 | +ms.date: 02/14/2024 |
| 7 | +ms.author: shasb |
| 8 | +--- |
| 9 | + |
| 10 | +# Configure Istio-based service mesh add-on for Azure Kubernetes Service (preview) |
| 11 | + |
| 12 | +Open-source Istio uses [MeshConfig][istio-meshconfig] to define mesh-wide settings for the Istio service mesh. Istio-based service mesh add-on for AKS builds on top of MeshConfig and classifies different properties as supported, allowed, and blocked. |
| 13 | + |
| 14 | +This article walks through how to configure Istio-based service mesh add-on for Azure Kubernetes Service and the support policy applicable for such configuration. |
| 15 | + |
| 16 | +[!INCLUDE [preview features callout](includes/preview/preview-callout.md)] |
| 17 | + |
| 18 | +## Prerequisites |
| 19 | + |
| 20 | +This guide assumes you followed the [documentation][istio-deploy-addon] to enable the Istio add-on on an AKS cluster. |
| 21 | + |
| 22 | +## Set up configuration on cluster |
| 23 | + |
| 24 | +1. Find out which revision of Istio is deployed on the cluster: |
| 25 | + |
| 26 | + ```bash |
| 27 | + az aks show -n $CLUSTER -g $RESOURCE_GROUP --query 'serviceMeshProfile' |
| 28 | + ``` |
| 29 | + |
| 30 | + Output: |
| 31 | + |
| 32 | + ``` |
| 33 | + { |
| 34 | + "istio": { |
| 35 | + "certificateAuthority": null, |
| 36 | + "components": { |
| 37 | + "egressGateways": null, |
| 38 | + "ingressGateways": null |
| 39 | + }, |
| 40 | + "revisions": [ |
| 41 | + "asm-1-18" |
| 42 | + ] |
| 43 | + }, |
| 44 | + "mode": "Istio" |
| 45 | + } |
| 46 | + ``` |
| 47 | + |
| 48 | +2. Create a ConfigMap with the name `istio-shared-configmap-<asm-revision>` in the `aks-istio-system` namespace. For example, if your cluster is running asm-1-18 revision of mesh, then the ConfigMap needs to be named as `istio-shared-configmap-asm-1-18`. Mesh configuration has to be provided within the data section under mesh. |
| 49 | + |
| 50 | + Example: |
| 51 | + |
| 52 | + ```yaml |
| 53 | + apiVersion: v1 |
| 54 | + kind: ConfigMap |
| 55 | + metadata: |
| 56 | + name: istio-shared-configmap-asm-1-18 |
| 57 | + namespace: aks-istio-system |
| 58 | + data: |
| 59 | + mesh: |- |
| 60 | + accessLogFile: /dev/stdout |
| 61 | + defaultConfig: |
| 62 | + holdApplicationUntilProxyStarts: true |
| 63 | + ``` |
| 64 | + The values under `defaultConfig` are mesh-wide settings applied for Envoy sidecar proxy. |
| 65 | + |
| 66 | +> [!CAUTION] |
| 67 | +> A default ConfigMap (for example, `istio-asm-1-18` for revision asm-1-18) is created in `aks-istio-system` namespace on the cluster when the Istio addon is enabled. However, this default ConfigMap gets reconciled by the managed Istio addon and thus users should NOT directly edit this ConfigMap. Instead users should create a revision specific Istio shared ConfigMap (for example `istio-shared-configmap-asm-1-18` for revision asm-1-18) in the aks-istio-system namespace, and then the Istio control plane will merge this with the default ConfigMap, with the default settings taking precedence. |
| 68 | + |
| 69 | +### Mesh configuration and upgrades |
| 70 | + |
| 71 | +When you're performing [canary upgrade for Istio](./istio-upgrade.md), you need create a separate ConfigMap for the new revision in the `aks-istio-system` namespace **before initiating the canary upgrade**. This way the configuration is available when the new revision's control plane is deployed on cluster. For example, if you're upgrading the mesh from asm-1-18 to asm-1-19, you need to copy changes over from `istio-shared-configmap-asm-1-18` to create a new ConfigMap called `istio-shared-configmap-asm-1-19` in the `aks-istio-system` namespace. |
| 72 | +
|
| 73 | +After the upgrade is completed or rolled back, you can delete the ConfigMap of the revision that was removed from the cluster. |
| 74 | +
|
| 75 | +## Allowed, supported, and blocked values |
| 76 | +
|
| 77 | +Fields in `MeshConfig` are classified into three categories: |
| 78 | +
|
| 79 | +- **Blocked**: Disallowed fields are blocked via addon managed admission webhooks. API server immediately publishes the error message to the user that the field is disallowed. |
| 80 | +- **Supported**: Supported fields (for example, fields related to access logging) receive support from Azure support. |
| 81 | +- **Allowed**: These fields (such as proxyListenPort or proxyInboundListenPort) are allowed but they aren't covered by Azure support. |
| 82 | + |
| 83 | +Mesh configuration and the list of allowed/supported fields are revision specific to account for fields being added/removed across revisions. The full list of allowed fields and the supported/unsupported ones within the allowed list is provided in the below table. When new mesh revision is made available, any changes to allowed and supported classification of the fields is noted in this table. |
| 84 | + |
| 85 | +### MeshConfig |
| 86 | + |
| 87 | +| **Field** | **Supported** | |
| 88 | +|-----------|---------------| |
| 89 | +| proxyListenPort | false | |
| 90 | +| proxyInboundListenPort | false | |
| 91 | +| proxyHttpPort | false | |
| 92 | +| connectTimeout | false | |
| 93 | +| tcpKeepAlive | false | |
| 94 | +| defaultConfig | true | |
| 95 | +| outboundTrafficPolicy | true | |
| 96 | +| extensionProviders | true | |
| 97 | +| defaultProvideres | true | |
| 98 | +| accessLogFile | true | |
| 99 | +| accessLogFormat | true | |
| 100 | +| accessLogEncoding | true | |
| 101 | +| enableTracing | true | |
| 102 | +| enableEnvoyAccessLogService | true | |
| 103 | +| disableEnvoyListenerLog | true | |
| 104 | +| trustDomain | false | |
| 105 | +| trustDomainAliases | false | |
| 106 | +| caCertificates | false | |
| 107 | +| defaultServiceExportTo | false | |
| 108 | +| defaultVirtualServiceExportTo | false | |
| 109 | +| defaultDestinationRuleExportTo | false | |
| 110 | +| localityLbSetting | false | |
| 111 | +| dnsRefreshRate | false | |
| 112 | +| h2UpgradePolicy | false | |
| 113 | +| enablePrometheusMerge | true | |
| 114 | +| discoverySelectors | true | |
| 115 | +| pathNormalization | false | |
| 116 | +| defaultHttpRetryPolicy | false | |
| 117 | +| serviceSettings | false | |
| 118 | +| meshMTLS | false | |
| 119 | +| tlsDefaults | false | |
| 120 | + |
| 121 | +### ProxyConfig (meshConfig.defaultConfig) |
| 122 | + |
| 123 | +| **Field** | **Supported** | |
| 124 | +|-----------|---------------| |
| 125 | +| tracingServiceName | true | |
| 126 | +| drainDuration | true | |
| 127 | +| statsUdpAddress | false | |
| 128 | +| proxyAdminPort | false | |
| 129 | +| tracing | true | |
| 130 | +| concurrency | true | |
| 131 | +| envoyAccessLogService | true | |
| 132 | +| envoyMetricsService | true | |
| 133 | +| proxyMetadata | false | |
| 134 | +| statusPort | false | |
| 135 | +| extraStatTags | false | |
| 136 | +| proxyStatsMatcher | false | |
| 137 | +| terminationDrainDuration | true | |
| 138 | +| meshId | false | |
| 139 | +| holdApplicationUntilProxyStarts | true | |
| 140 | +| caCertificatesPem | false | |
| 141 | +| privateKeyProvider | false | |
| 142 | + |
| 143 | +Fields present in [open source MeshConfig reference documentation][istio-meshconfig] but not in the above table are blocked. For example, `configSources` is blocked. |
| 144 | + |
| 145 | +> [!CAUTION] |
| 146 | +> **Support scope of configurations:** Mesh configuration allows for extension providers such as self-managed instances of Zipkin or Apache Skywalking to be configured with the Istio addon. However, these extension providers are outside the support scope of the Istio addon. Any issues associated with extension tools are outside the support boundary of the Istio addon. |
| 147 | + |
| 148 | +## Common errors and troubleshooting tips |
| 149 | + |
| 150 | +- Ensure that the MeshConfig is indented with spaces instead of tabs. |
| 151 | +- Ensure that you're only editing the revision specific shared ConfigMap (for example `istio-shared-configmap-asm-1-18`) and not trying to edit the default ConfigMap (for example `istio-asm-1-18`). |
| 152 | +- The ConfigMap must follow the name `istio-shared-configmap-<asm-revision>` and be in the `aks-istio-system` namespace. |
| 153 | +- Ensure that all MeshConfig fields are spelled correctly. If they're unrecognized or if they aren't part of the allowed list, admission control denies such configurations. |
| 154 | +- When performing canary upgrades, [check your revision specific ConfigMaps](#mesh-configuration-and-upgrades) to ensure configurations exist for the revisions deployed on your cluster. |
| 155 | +- Certain `MeshConfig` options such as accessLogging may increase Envoy's resource consumption, and disabling some of these settings may mitigate Istio data plane resource utilization. It's also advisable to use the `discoverySelectors` field in the MeshConfig to help alleviate memory consumption for Istiod and Envoy. |
| 156 | +- If the `concurrency` field in the MeshConfig is misconfigured and set to zero, it causes Envoy to use up all CPU cores. Instead if this field is unset, number of worker threads to run is automatically determined based on CPU requests/limits. |
| 157 | +- [Pod and sidecar race conditions][istio-sidecar-race-condition] in which the application starts before Envoy can be mitigated using the `holdApplicationUntilProxyStarts` field in the MeshConfig. |
| 158 | +
|
| 159 | +
|
| 160 | +[istio-meshconfig]: https://istio.io/latest/docs/reference/config/istio.mesh.v1alpha1/ |
| 161 | +[istio-sidecar-race-condition]: https://istio.io/latest/docs/ops/common-problems/injection/#pod-or-containers-start-with-network-issues-if-istio-proxy-is-not-ready |
0 commit comments