mass-testing docs now

Author: naman-msft

scenarios/AKSClientSecretError/aks-client-secret-error.md

@@ -40,8 +40,8 @@ The issue that generates this service principal alert usually occurs for one of
 Use the following commands to retrieve the service principal profile for your AKS cluster and check the expiration date of the service principal. Make sure to set the appropriate variables for your AKS resource group and cluster name.
 
 ```azurecli
-SP_ID=$(az aks show --resource-group RESOURCE_GROUP_NAME \
-    --name AKS_CLUSTER_NAME \
+SP_ID=$(az aks show --resource-group $RESOURCE_GROUP_NAME \
+    --name $AKS_CLUSTER_NAME \
     --query servicePrincipalProfile.clientId \
     --output tsv)
 az ad app credential list --id "$SP_ID"

scenarios/AKSHealthProbeMode/aks-health-probe-mode.md

@@ -42,7 +42,7 @@ To troubleshoot these issues, follow these steps:
     ```azurecli
     export RESOURCE_GROUP="aks-rg"
     export AKS_CLUSTER_NAME="aks-cluster"
-    az aks show --resource-group $RESOURCE_GROUP --name $AKS_CLUSTER_NAME --query "loadBalancerProfile"
+    az aks show --resource-group $RESOURCE_GROUP --name $AKS_CLUSTER_NAME --query "networkProfile.loadBalancerProfile"
     ```
     Results:
 
@@ -66,24 +66,27 @@ To troubleshoot these issues, follow these steps:
     }
     ```
 
-2. Check the *overlaymgr* log to see if the cloud provider secret is updated. The keyword to look for is `cloudConfigSecretResolver`. Or check the contents of the cloud-provider-config secret in the `ccp` namespace. You can use the `kubectl get secret` command to view the secret. 
+2. Check the cloud provider configuration. In modern AKS clusters, the cloud provider configuration is managed internally and the `ccp` namespace doesn't exist. Instead, check for cloud provider related resources and verify the cloud-node-manager pods are running properly:
 
-    ```shell
-    kubectl get secret cloud-provider-config -n ccp -o yaml
+
+    ```bash
+    # Check for cloud provider related ConfigMaps in kube-system
+    kubectl get configmap -n kube-system | grep -i azure
+    
+    # Check if cloud-node-manager pods are running (indicates cloud provider integration is working)
+    kubectl get pods -n kube-system | grep cloud-node-manager
+    
+    # Check the azure-ip-masq-agent-config if it exists
+    kubectl get configmap azure-ip-masq-agent-config-reconciled -n kube-system -o yaml 2>/dev/null || echo "ConfigMap not found"
     ```
     Results:
 
     <!-- expected_similarity=0.3 -->
 
     ```output
-    apiVersion: v1
-    data:
-      cloud-config: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
-    kind: Secret
-    metadata:
-      name: cloud-provider-config
-      namespace: ccp
-      ...
+    configmap/azure-ip-masq-agent-config-reconciled   1      11h
+    
+    cloud-node-manager-rfb2w                        2/2     Running   0          16m
     ```
 
 3. Check the chart or overlay daemonset cloud-node-manager to see if the health-probe-proxy sidecar container is enabled. You can use the `kubectl get ds` command to view the daemonset.

scenarios/AKSPreviewAPILifecycle/aks-preview-api-lifecycle.md

@@ -30,7 +30,7 @@ API version as deprecation approaches.
 If you're unsure what client or tool is using this API version, check the [activity logs](/azure/azure-monitor/essentials/activity-log)
 using the following command:
 
-Set the API version you want to inspect for recent usage in the activity log.
+Set the API version you want to inspect for recent usage in the activity log. In this example, we are checking for the `2022-04-01-preview` API version.
 
 ```bash
 export API_VERSION="2022-04-01-preview"

scenarios/AzureCNIPodSubnet/azure-cni-pod-subnet.md

@@ -23,6 +23,8 @@ Azure CNI Pod Subnet assigns IP addresses to pods from a separate subnet from yo
 - Azure CLI version `2.37.0` or later and the `aks-preview` extension version `2.0.0b2` or later.
 - Register the subscription-level feature flag for your subscription: 'Microsoft.ContainerService/AzureVnetScalePreview'.
 
+## Enable Container Insights (AKS monitoring)
+
 If you have an existing cluster, you can enable Container Insights (AKS monitoring) using the following command **only if your cluster was created with monitoring enabled or is associated with a valid Log Analytics Workspace in the same region**. Otherwise, refer to Microsoft Docs for additional workspace setup requirements.
 
 ```azurecli-interactive

scenarios/CSEErrorsAKS/cse-errors-aks.md

@@ -107,7 +107,8 @@ Set up your custom Domain Name System (DNS) server so that it can do name resolu
     > **Important:** You must specify the `--name` of a valid VM in an availability set in your resource group. Here is a template for running network checks.
 
     ```azurecli
-    export DNS_IP_ADDRESS="10.0.0.10"
+    export API_FQDN=$(az aks show --resource-group $RG_NAME --name $CLUSTER_NAME --query fqdn -o tsv)
+
     az vm run-command invoke \
         --resource-group $NODE_RESOURCE_GROUP \
         --name $AVAILABILITY_SET_VM \
@@ -121,7 +122,7 @@ Set up your custom Domain Name System (DNS) server so that it can do name resolu
         --command-id RunShellScript \
         --output tsv \
         --query "value[0].message" \
-        --scripts "nslookup <api-fqdn> $DNS_IP_ADDRESS"
+        --scripts "nslookup $API_FQDN $DNS_IP_ADDRESS"
     ```
 
 For more information, see [Name resolution for resources in Azure virtual networks](/azure/virtual-network/virtual-networks-name-resolution-for-vms-and-role-instances) and [Hub and spoke with custom DNS](/azure/aks/private-clusters#hub-and-spoke-with-custom-dns).

scenarios/CniDownloadFailureAKS/cni-download-failure-aks.md

@@ -18,6 +18,7 @@ This article discusses how to identify and resolve the `CniDownloadTimeoutVMExte
 ## Prerequisites
 
 - The [Curl](https://curl.se/download.html) command-line tool
+- Network access from the same environment where AKS nodes will be deployed (same VNet, firewall rules, etc.)
 
 ## Symptoms
 
@@ -52,7 +53,7 @@ Run a Curl command to verify that your nodes can download the binaries:
 First, attempt a test download of the Azure CNI package for Linux from the official mirror endpoint.
 
 ```bash
-curl https://acs-mirror.azureedge.net/cni/azure-vnet-cni-linux-amd64-v1.0.25.tgz
+curl -I https://acs-mirror.azureedge.net/cni/azure-vnet-cni-linux-amd64-v1.0.25.tgz
 ```
 
 Results:
@@ -72,33 +73,53 @@ accept-ranges: bytes
 date: Thu, 05 Jun 2025 00:00:00 GMT
 ```
 
-This command will display binary archive data in the terminal if the download succeeds.
+This command checks if the endpoint is reachable and returns the HTTP headers. If you see a `200 OK` response, it indicates that the endpoint is accessible.
 
 Next, attempt a download with validation and save the file locally for further troubleshooting. This will help determine if SSL or outbound connectivity is correctly configured.
 
 ```bash
-curl --fail --ssl https://acs-mirror.azureedge.net/cni/azure-vnet-cni-linux-amd64-v1.0.25.tgz  --output /opt/cni/downloads/azure-vnet-cni-linux-amd64-v1.0.25.tgz
+# Create a temporary directory for testing
+mkdir -p /tmp/cni-test
+
+# Download the CNI package to the temp directory
+curl -L --fail https://acs-mirror.azureedge.net/cni/azure-vnet-cni-linux-amd64-v1.0.25.tgz --output /tmp/cni-test/azure-vnet-cni-linux-amd64-v1.0.25.tgz && echo "Download successful" || echo "Download failed"
 ```
 
 Results:
 
 <!-- expected_similarity=0.3 -->
 
 ```output
+  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
+                                 Dload  Upload   Total   Spent    Left  Speed
+100 6495k  100 6495k    0     0  8234k      0 --:--:-- --:--:-- --:--:-- 8230k
+Download successful
 ```
 
-Or as an alternative with output to /tmp and success message for clarity:
+Verify the downloaded file:
 
 ```bash
-curl -L --fail https://acs-mirror.azureedge.net/cni/azure-vnet-cni-linux-amd64-v1.0.25.tgz --output /tmp/azure-vnet-cni-test.tgz && echo "Download successful" || echo "Download failed"
+ls -la /tmp/cni-test/
+file /tmp/cni-test/azure-vnet-cni-linux-amd64-v1.0.25.tgz
 ```
 
 Results:
 
 <!-- expected_similarity=0.3 -->
 
 ```output
-Download successful
+total 6500
+drwxr-xr-x 2 user user    4096 Jun 20 10:30 .
+drwxrwxrwt 8 root root    4096 Jun 20 10:30 ..
+-rw-r--r-- 1 user user 6651392 Jun 20 10:30 azure-vnet-cni-linux-amd64-v1.0.25.tgz
+
+/tmp/cni-test/azure-vnet-cni-linux-amd64-v1.0.25.tgz: gzip compressed data, from Unix, original size modulo 2^32 20070400
+```
+
+Clean up the test files:
+
+```bash
+rm -rf /tmp/cni-test/
 ```
 
 If you can't download these files, make sure that traffic is allowed to the downloading endpoint. For more information, see [Azure Global required FQDN/application rules](/azure/aks/outbound-rules-control-egress#azure-global-required-fqdn--application-rules).

scenarios/ForbiddenErrorAKS/forbidden-error-aks.md

@@ -44,14 +44,28 @@ az aks show -g $RESOURCE_GROUP -n $CLUSTER_NAME --query aadProfile.enableAzureRb
 
 Results:
 
-<!-- expected_similarity=0.3 -->
 ```output
 false
 ```
 
+- If the result is **null** or empty, the cluster doesn't have Azure AD integration enabled. See [Solving permission issues in local Kubernetes RBAC clusters](#solving-permissions-issues-in-local-kubernetes-rbac-clusters).
 - If the result is **false**, the cluster uses Kubernetes RBAC. See [Solving permission issues in Kubernetes RBAC-based AKS clusters](#solving-permissions-issues-in-kubernetes-rbac-based-aks-clusters).
 - If the result is **true**, the cluster uses Azure RBAC. See [Solving permission issues in Azure RBAC-based AKS clusters](#solving-permissions-issues-in-azure-rbac-based-aks-clusters).
 
+### Solving permissions issues in local Kubernetes RBAC clusters
+
+If your cluster doesn't have Azure AD integration (result was null), it uses cluster admin credentials:
+
+```bash
+# Get admin credentials for full access
+az aks get-credentials --resource-group $RESOURCE_GROUP --name $CLUSTER_NAME --admin
+
+# Verify access
+kubectl get nodes
+```
+
+**Warning**: Admin credentials provide full cluster access. Use carefully and consider enabling Azure AD integration for better security.
+
 ### Solving permissions issues in Kubernetes RBAC-based AKS clusters
 
 If the cluster uses Kubernetes RBAC, permissions for the user account are configured through the creation of RoleBinding or ClusterRoleBinding Kubernetes resources. For more information, see [Kubernetes RBAC documentation](https://kubernetes.io/docs/reference/access-authn-authz/rbac/).
@@ -74,7 +88,6 @@ You can create a custom RoleBinding or ClusterRoleBinding resource to grant the
 
    Results:
 
-   <!-- expected_similarity=0.3 -->
    ```output
    [
      "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"

scenarios/KubeletIOTroubleshooting/kubelet-io-troubleshooting.md

@@ -16,6 +16,17 @@ keywords:
 
 TCP timeouts can be caused by blockages of internal traffic that runs between nodes. To investigate TCP time-outs, verify that this traffic isn't being blocked, for example, by [network security groups](/azure/aks/concepts-security#azure-network-security-groups) (NSGs) on the subnet for your cluster nodes.
 
+## Connect to the cluster
+
+First, connect to your Azure Kubernetes Service (AKS) cluster by running the following command:
+
+```bash
+export RESOURCE_GROUP=<your-resource-group>
+export CLUSTER_NAME=<your-cluster-name>
+
+az aks get-credentials --resource-group $RESOURCE_GROUP --name $CLUSTER_NAME
+```
+
 ## Symptoms
 
 Tunnel functionalities, such as `kubectl logs` and code execution, work only for pods that are hosted on nodes on which tunnel service pods are deployed. Pods on other nodes that have no tunnel service pods cannot reach to the tunnel. When viewing the logs of these pods, you receive the following error message:

scenarios/NodeNotReadyAKS/node-not-ready-aks.md

@@ -51,12 +51,31 @@ The [kubelet](https://kubernetes.io/docs/reference/command-line-tools-reference/
 Examine the output of the `kubectl describe nodes` command to find the [Conditions](https://kubernetes.io/docs/reference/node/node-status/#condition) field and the [Capacity and Allocatable](https://kubernetes.io/docs/reference/node/node-status/#capacity) blocks. Do the content of these fields appear as expected? (For example, in the **Conditions** field, does the `message` property contain the "kubelet is posting ready status" string?) In this case, if you have direct Secure Shell (SSH) access to the node, check the recent events to understand the error. Look within the */var/log/syslog* file instead of */var/log/messages* (not available on all distributions). Or, generate the kubelet and container daemon log files by running the following shell commands:
 
 ```bash
-# To check syslog file (useful on Ubuntu-based AKS nodes),
-cat /var/log/syslog
-
-# To check kubelet and containerd daemon logs,
-journalctl -u kubelet > kubelet.log
-journalctl -u containerd > containerd.log
+# First, identify the NotReady node
+export NODE_NAME=$(kubectl get nodes --no-headers | grep NotReady | awk '{print $1}' | head -1)
+
+if [ -z "$NODE_NAME" ]; then
+    echo "No NotReady nodes found"
+    kubectl get nodes
+else
+    echo "Found NotReady node: $NODE_NAME"
+    
+    # Use kubectl debug to access the node
+    kubectl debug node/$NODE_NAME -it --image=mcr.microsoft.com/dotnet/runtime-deps:6.0 -- chroot /host bash -c "
+        echo '=== Checking syslog ==='
+        if [ -f /var/log/syslog ]; then
+            tail -100 /var/log/syslog
+        else
+            echo 'syslog not found'
+        fi
+        
+        echo '=== Checking kubelet logs ==='
+        journalctl -u kubelet --no-pager | tail -100
+        
+        echo '=== Checking containerd logs ==='
+        journalctl -u containerd --no-pager | tail -100
+    "
+fi
 ```
 
 After you run these commands, examine the syslog and daemon log files for more information about the error.