AB#8189: asm: add external TSG for Istio Add-On CNI (#10103)

* asm: add external TSG for Istio Add-On CNI

* keep consistency for prerequisites

* add references

* reword overview section

* apply PR feedback

* Apply suggestions from code review of chase

Co-authored-by: Chase Wilson <[email protected]>

* link to istio CNI addon docs

* remove Overview section

* Remove v-weizhu

* Update istio-add-on-cni-troubleshooting.md

Edit Review per CI 8189 in progress

* Update istio-add-on-cni-troubleshooting.md

Edit review per CI 8189

* Update istio-add-on-cni-troubleshooting.md

* fix link

* Fix markdown formatting in Istio CNI troubleshooting guide

Updated markdown formatting for Istio CNI troubleshooting guide.

* Update istio-add-on-cni-troubleshooting.md

* Update istio-add-on-cni-troubleshooting.md

---------

Co-authored-by: German Robayo Paz <[email protected]>
Co-authored-by: Chase Wilson <[email protected]>
Co-authored-by: Jerry Sitser <[email protected]>

Author: 4 people

support/azure/azure-kubernetes/extensions/istio-add-on-cni-troubleshooting.md

@@ -0,0 +1,140 @@
+---
+title: Istio Service Mesh Add-on CNI Troubleshooting
+description: Learn how to troubleshoot the Istio CNI add-on for Azure Kubernetes Service (AKS).
+ms.date: 10/22/2025
+ms.reviewer: gerobayopaz, kochhars
+ms.service: azure-kubernetes-service
+ms.topic: troubleshooting-general
+ms.custom: sap:Extensions, Policies and Add-Ons
+#Customer intent: As an Azure Kubernetes user, I want to troubleshoot the Istio CNI add-on so that I can use the Istio service mesh successfully.
+---
+# Istio service mesh add-on CNI troubleshooting
+
+This article discusses how to troubleshoot issues that affect the [Istio CNI][istio-cni-addon] feature for the Istio service mesh add-on for Azure Kubernetes Service (AKS).
+
+## Prerequisites
+
+- [Azure CLI](/cli/azure/install-azure-cli)
+- The `aks-preview` Azure CLI extension version 19.0.0b5 or later:
+
+    ```azurecli-interactive
+    az extension add --name aks-preview
+    ```
+
+    If you already have the `aks-preview` extension installed, update it to the latest version:
+
+    ```azurecli-interactive
+    az extension update --name aks-preview
+    ```
+- The Kubernetes [kubectl](https://kubernetes.io/docs/reference/kubectl/overview/) CLI, or a similar tool to connect to the cluster
+
+   Alternatively, you can install kubectl using Azure CLI by using the [az aks install-cli](/cli/azure/aks#az-aks-install-cli) command.
+- Make sure that your Istio service mesh uses revision `asm-1-25` or a later version. To check the current revision, run the following command:
+
+    ```azurecli-interactive
+    az aks show --resource-group <resource-group-name> --name <cluster-name> --query 'serviceMeshProfile.istio.revisions'
+    ```
+- Make sure that Istio CNI is enabled in your cluster:
+
+    ```azurecli-interactive
+    az aks show --resource-group <resource-group-name> --name <cluster-name> --query "serviceMeshProfile.istio.components.proxyRedirectioMechanism" -o table
+    ```
+
+    The command output should be `CNIChaining`. If the add-on isn't enabled, refer to [this guide to Istio CNI](/azure/aks/istio-cni).
+
+## CNI DaemonSet provisioning issues troubleshooting
+
+### Step 1: Verify that the Istio CNI DaemonSet is provisioned and ready
+
+Check that the CNI DaemonSet is deployed and that all pods are running:
+
+```bash
+kubectl get daemonset azure-service-mesh-istio-cni-addon-node -n aks-istio-system
+kubectl get pods -n aks-istio-system -l k8s-app=azure-service-mesh-istio-cni-addon-node
+```
+
+You should see a DaemonSet that has pods that run on each node in your cluster.
+
+### Step 2: Check CNI DaemonSet pod logs
+
+To identify any installation or configuration issues, examine the logs of the CNI DaemonSet pods:
+
+```bash
+kubectl logs -n aks-istio-system -l k8s-app=azure-service-mesh-istio-cni-addon-node
+```
+
+Look for error messages that are related to the following items:
+- CNI plugin installation failures
+- Network configuration errors
+- Permission or file system issues
+- Node-specific issues
+
+### Step 3: Check for node taints and tolerations
+
+Verify that the CNI DaemonSet can be scheduled on all nodes:
+
+```bash
+kubectl describe daemonset istio-cni-node -n aks-istio-system
+kubectl get nodes -o wide
+kubectl describe nodes
+```
+
+Look for node taints that might prevent CNI pod scheduling, and make sure that the DaemonSet has appropriate tolerations.
+
+## Init container injection issues troubleshooting
+### Step 1: Check whether istio-init containers are still injected
+
+For newly created pods in the mesh, verify that `istio-init` containers are no longer present:
+
+```bash
+# Check a sample pod in an injected namespace
+kubectl get pods -n $NAMESPACE -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.initContainers[*].name}{"\n"}{end}'
+```
+
+If `istio-init` containers are still being injected, CNI isn't working correctly.
+
+### Step 2: Inspect pod events for CNI-related errors
+
+Check pod events for any CNI plugin failures during pod startup:
+
+```bash
+kubectl describe pod $POD_NAME -n $NAMESPACE
+```
+
+Look for events that are related to the following items:
+- Network setup failures
+- CNI plugin errors
+- Container creation issues
+
+### Step 3: Verify istio-proxy sidecar startup
+
+Check whether the `istio-proxy` sidecar container starts successfully without the init container:
+
+```bash
+kubectl logs $POD_NAME -n $NAMESPACE -c istio-proxy
+```
+
+If CNI is working correctly, the sidecar should start normally even without the `istio-init` container.
+
+## Pod startup failure troubleshooting
+
+If pods don't start, check for `istio-validation` init container errors:
+
+```bash
+kubectl logs $POD_NAME -n $NAMESPACE -c istio-validation
+```
+
+Look for "connection refused" error messages that indicate failures in traffic redirection setup.
+
+## References
+
+[Open-source Istio's CNI troubleshooting](https://istio.io/latest/docs/ops/diagnostic-tools/cni/)
+
+[!INCLUDE [Third-party information disclaimer](../../../includes/third-party-disclaimer.md)]
+
+[!INCLUDE [Third-party contact disclaimer](../../../includes/third-party-contact-disclaimer.md)]
+
+[!INCLUDE [Azure Help Support](../../../includes/azure-help-support.md)]
+
+[istio-deploy-addon]: /azure/aks/istio-deploy-addon
+[istio-cni-addon]: /azure/aks/istio-cni

support/azure/azure-kubernetes/toc.yml

@@ -64,14 +64,14 @@ items:
   - name: Identify nodes and containers utilizing high CPU
     href: availability-performance/identify-high-cpu-consuming-containers-aks.md
   - name: Improve container image pull performance in AKS
-    href: availability-performance/container-image-pull-performance.md      
+    href: availability-performance/container-image-pull-performance.md
   - name: Troubleshoot cluster service health probe mode issues
     href: availability-performance/cluster-service-health-probe-mode-issues.md
   - name: Troubleshoot high memory consumption in disk-intensive applications
     href: availability-performance/high-memory-consumption-disk-intensive-applications.md
   - name: Troubleshoot OOMkilled in AKS clusters
-    href: availability-performance/troubleshoot-oomkilled-aks-clusters.md    
-  - name: Troubleshoot pod scheduler errors 
+    href: availability-performance/troubleshoot-oomkilled-aks-clusters.md
+  - name: Troubleshoot pod scheduler errors
     href: availability-performance/troubleshoot-pod-scheduler-errors.md
   - name: Troubleshoot node not ready
     items:
@@ -106,7 +106,7 @@ items:
     - name: Basic troubleshooting
       href: connectivity/troubleshoot-cluster-connection-issues-api-server.md
     - name: Can't access the cluster API server using authorized IP ranges
-      href: connectivity/cannot-access-cluster-api-server-using-authorized-ip-ranges.md      
+      href: connectivity/cannot-access-cluster-api-server-using-authorized-ip-ranges.md
     - name: Client IP address can't access API server
       href: connectivity/client-ip-address-cannot-access-api-server.md
     - name: Config file isn't available when connecting
@@ -130,7 +130,7 @@ items:
       - name: Can't connect to endpoints outside virtual network (public internet)
         href: connectivity/troubleshoot-connections-endpoints-outside-virtual-network.md
       - name: Can't connect to pods and services in same cluster
-        href: connectivity/troubleshoot-connection-pods-services-same-cluster.md        
+        href: connectivity/troubleshoot-connection-pods-services-same-cluster.md
       - name: Can't view resources in Kubernetes resource viewer on Azure portal
         href: connectivity/cannot-view-resources-kubernetes-resource-viewer-portal.md
       - name: TCP times out when kubectl or other third-party tools connect
@@ -169,12 +169,12 @@ items:
     href: storage/troubleshoot-common-bring-your-own-key-issues.md
   - name: Troubleshoot pods and namespaces stuck in the Terminating state
     href: storage/pods-namespaces-terminating-state.md
-  - name: Mount failures  
+  - name: Mount failures
     items:
     - name: Can't set the uid and gid mounting options on an Azure Disk
       href: storage/failure-setting-azure-disk-mount-options-uid-gid.md
     - name: Errors when mounting Blob container
-      href: storage/mounting-azure-blob-storage-container-fail.md 
+      href: storage/mounting-azure-blob-storage-container-fail.md
     - name: Errors when mounting Disk volumes
       href: storage/fail-to-mount-azure-disk-volume.md
     - name: Errors when mounting File share
@@ -233,12 +233,14 @@ items:
       href: extensions/istio-add-on-minor-revision-upgrade.md
     - name: Istio add-on plug-in CA certificate troubleshooting
       href: extensions/istio-add-on-plug-in-ca-certificate.md
+    - name: Istio add-on CNI troubleshooting
+      href: extensions/istio-add-on-cni-troubleshooting.md
   - name: Other extensions and add-ons
     items:
     - name: AKS Cost Analysis add-on issues
       href: extensions/aks-cost-analysis-add-on-issues.md
     - name: Can't pull images from container registry to cluster
-      href: extensions/cannot-pull-image-from-acr-to-aks-cluster.md  
+      href: extensions/cannot-pull-image-from-acr-to-aks-cluster.md
     - name: Troubleshoot AI toolchain operator add-on errors
       href: extensions/troubleshoot-ai-toolchain-operator-addon-issues.md
     - name: Troubleshoot Azure Key Vault Provider for Secrets Store CSI Driver
@@ -248,7 +250,7 @@ items:
     - name: Troubleshoot Dapr extension installation errors
       href: extensions/troubleshoot-dapr-extension-installation-errors.md
     - name: Troubleshoot deployment failures of Azure Marketplace offers
-      href: extensions/troubleshoot-failed-kubernetes-deployment-offer.md      
+      href: extensions/troubleshoot-failed-kubernetes-deployment-offer.md
     - name: Troubleshoot installation errors in the Azure App Configuration extension
       href: extensions/troubleshoot-app-configuration-extension-installation-errors.md
     - name: Troubleshoot managed namespaces
@@ -260,8 +262,8 @@ items:
     - name: Breaking changes in KEDA add-on 2.15 and 2.14
       href: extensions/changes-in-kubernetes-event-driven-autoscaling-add-on-214-215.md
     - name: Troubleshoot KEDA add-on
-      href: extensions/troubleshoot-kubernetes-event-driven-autoscaling-add-on.md      
-- name: Troubleshoot by error codes
+      href: extensions/troubleshoot-kubernetes-event-driven-autoscaling-add-on.md
+- name: Troubleshoot error codes
   items:
   - name: Common error codes
     href: error-codes/aks-error-code-page.md
@@ -294,9 +296,9 @@ items:
     - name: CreateOrUpdateVirtualNetworkLinkFailed error
       href: error-codes/createorupdatevirtualnetworklinkfailed-error.md
     - name: CustomPrivateDNSZoneMissingPermissionError error
-      href: error-codes/customprivatednszonemissingpermissions-error.md   
+      href: error-codes/customprivatednszonemissingpermissions-error.md
     - name: DnsServiceIpOutOfServiceCidr error
-      href: error-codes/dnsserviceipoutofservicecidr-error.md 
+      href: error-codes/dnsserviceipoutofservicecidr-error.md
     - name: ERR_VHD_FILE_NOT_FOUND error (65)
       href: create-upgrade-delete/error-code-vhdfilenotfound.md
     - name: Error "CreateOrUpdateVirtualNetworkLinkFailed"
@@ -322,7 +324,7 @@ items:
     - name: Known issues - Custom kubelet configuration on Windows
       href: create-upgrade-delete/known-issues-custom-kubelet-configuration.md
   - name: L-S
-    items:   
+    items:
     - name: LB/PvtLinkSvcWithPvtEndptConn deletion error
       href: create-upgrade-delete/cannot-delete-load-balancer-private-link-service.md
     - name: LinkedAuthorizationFailed error
@@ -332,23 +334,23 @@ items:
     - name: LoadBalancerInUseByVirtualMachineScaleSet or NetworkSecurityGroupInUseByVirtualMachineScaleSet error
       href: error-codes/networksecuritygroupinusebyvirtualmachinescaleset-error.md
     - name: Missing or invalid service principal
-      href: create-upgrade-delete/missing-or-invalid-service-principal.md        
+      href: create-upgrade-delete/missing-or-invalid-service-principal.md
     - name: MissingSubscriptionRegistration error
       href: error-codes/missingsubscriptionregistration-error.md
     - name: NetworkSecurityGroupInUseByVirtualMachineScaleSet error
       href: create-upgrade-delete/load-balancer-or-nsg-in-use-by-vm-scale-set.md
     - name: NodePoolMcVersionIncompatible error
-      href: error-codes/nodepoolmcversionincompatible-error.md 
+      href: error-codes/nodepoolmcversionincompatible-error.md
     - name: OperationIsNotAllowed errors
       href: create-upgrade-delete/operationnotallowed.md
     - name: OperationNotAllowed or PublicIPCountLimitReached error
-      href: error-codes/operationnotallowed-publicipcountlimitreached-error.md        
+      href: error-codes/operationnotallowed-publicipcountlimitreached-error.md
     - name: OrasPullNetworkTimeoutVMExtensionError
       href: error-codes/vmextensionerror-oraspullnetworktimeout.md
     - name: OrasPullUnauthorizedVMExtensionError
       href: error-codes/vmextensionerror-oraspullunauthorized.md
     - name: OutboundConnFailVMExtensionError error (50)
-      href: create-upgrade-delete/error-code-outboundconnfailvmextensionerror.md  
+      href: create-upgrade-delete/error-code-outboundconnfailvmextensionerror.md
     - name: PublicIPAddr/InUseSubnet/NetSecGrp deletion error
       href: create-upgrade-delete/cannot-delete-ip-subnet-nsg.md
     - name: PublicIPAddressCannotBeDeleted, InUseSubnetCannotBeDeleted, or InUseNetworkSecurityGroupCannotBeDeleted error
@@ -360,7 +362,7 @@ items:
     - name: QuotaExceeded or InsufficientVCPUQuota error during creation or upgrade
       href: create-upgrade-delete/quota-exceeded-during-creation-upgrade.md
     - name: RequestDisallowedByPolicy error
-      href: error-codes/requestdisallowedbypolicy-error.md 
+      href: error-codes/requestdisallowedbypolicy-error.md
     - name: ServiceCidrOverlapExistingSubnetsCidr error
       href: error-codes/servicecidroverlapexistingsubnetscidr-error.md
     - name: ServicePrincipalValidationClientError error
@@ -377,8 +379,8 @@ items:
       href: error-codes/subscriptionrequeststhrottled.md
     - name: SubscriptionRequestsThrottled error (429)
       href: create-upgrade-delete/error-code-subscriptionrequeststhrottled.md
-  - name: T-Z   
-    items:      
+  - name: T-Z
+    items:
     - name: TCP time-outs such as 10250 I/O
       href: connectivity/tcp-timeouts-dial-tcp-nodeip-10250-io-timeout.md
     - name: Throttled error
@@ -395,7 +397,7 @@ items:
       href: error-codes/unsatisfiablepdb-error.md
     - name: Upgrade issues with Gen2 VMs on Windows AKS cluster
       href: create-upgrade-delete/nodepools-not-upgraded-to-gen2-during-node-image-upgrade.md
-    - name: VirtualNetworkNotInSucceededState error 
+    - name: VirtualNetworkNotInSucceededState error
       href: error-codes/virtualnetworknotinsucceededstate-error.md
     - name: VMExtensionError_CniDownloadTimeout error
       href: error-codes/vmextensionerror-cnidownloadtimeout.md