Merge branch 'main' of https://github.com/MicrosoftDocs/azure-docs-pr into cni-rework

Author: asudbring

articles/aks/TOC.yml

@@ -501,24 +501,26 @@
           items:
             - name: Create a controller in AKS using Terraform
               href: create-k8s-cluster-with-aks-application-gateway-ingress.md
-            - name: Use ingress-nginx
-              items:
-                - name: Create an ingress controller
-                  href: ingress-basic.md
-                - name: Use TLS with an ingress controller
-                  href: ingress-tls.md
             - name: Use application routing add-on 
               items:
                 - name: Application routing add-on overview
                   href: app-routing.md
                 - name: Monitor using Prometheus and Grafana
                   href: app-routing-nginx-prometheus.md
+                - name: Migrate from HTTP application routing to the application routing add-on
+                  href: app-routing-migration.md
             - name: Use Application Gateway Ingress Controller add-on
               href: ../application-gateway/tutorial-ingress-controller-add-on-existing.md?toc=/azure/aks/toc.json&bc=/azure/aks/breadcrumb/toc.json
             - name: Use Istio gateway with Istio Service Mesh add-on
               href: istio-deploy-ingress.md
             - name: Use HTTP application routing add-on (retired)
               href: http-application-routing.md
+            - name: Use unmanaged ingress-nginx
+              items:
+                - name: Create an ingress controller
+                  href: ingress-basic.md
+                - name: Use TLS with an ingress controller
+                  href: ingress-tls.md
         - name: Load balancing
           items:
             - name: Create an internal load balancer

articles/aks/app-routing-migration.md

@@ -0,0 +1,130 @@
+---
+title: Migrate from HTTP application routing to the application routing add-on
+description: Learn how to migrate from the HTTP application routing feature to the application routing add-on.
+ms.topic: how-to
+ms.author: nickoman
+author: nickomang
+ms.custom: devx-track-azurecli, devx-track-linux
+ms.date: 08/18/2023
+---
+
+# Migrate from HTTP application routing to the application routing add-on
+
+In this article, you'll learn how to migrate your Azure Kubernetes Service (AKS) cluster from HTTP application routing feature to the [application routing add-on](./app-routing.md). The HTTP application routing add-on has been retired and won't work on any cluster Kubernetes version currently in support, so we recommend migrating as soon as possible to maintain a supported configuration.
+
+## Prerequisites
+
+Azure CLI version `2.49.0` or later. If you haven't yet, follow the instructions to [Install Azure CLI][install-azure-cli]. Run `az --version` to find the version, and run `az upgrade` to upgrade the version if not already on the latest.
+
+> [!NOTE]
+> These steps detail migrating from an unsupported configuration. As such, AKS cannot offer support for issues that arise during the migration process.
+
+## Update your cluster's add-ons, ingresses, and IP usage
+
+1. Enable the application routing add-on.
+
+    ```azurecli-interactive
+    az aks enable-addons -g <ResourceGroupName> -n <ClusterName> --addons web_application_routing
+    ```
+
+2. Update your ingresses, setting `ingressClassName` to `webapprouting.kubernetes.azure.com`. Remove the `kubernetes.io/ingress.class` annotation. You'll also need to update the host to one that you own, as the application routing add-on doesn't have a managed cluster DNS zone. If you don't have a DNS zone, follow instructions to [create][app-routing-dns-create] and [configure][app-routing-dns-configure] one.
+
+    Initially, your ingress configuration will look something like this:
+
+    ```yaml
+    apiVersion: networking.k8s.io/v1
+    kind: Ingress
+    metadata:
+      name: aks-helloworld
+      annotations:
+        kubernetes.io/ingress.class: addon-http-application-routing  # Remove the ingress class annotation
+    spec:
+      rules:
+      - host: aks-helloworld.<CLUSTER_SPECIFIC_DNS_ZONE>
+        http:
+          paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service: 
+                name: aks-helloworld
+                port: 
+                  number: 80
+    ```
+
+    After you've properly updated, the same configuration will look like the following:
+
+    ```yaml
+    apiVersion: networking.k8s.io/v1
+    kind: Ingress
+    metadata:
+      name: aks-helloworld
+    spec:
+      ingressClassName: webapprouting.kubernetes.azure.com # Set the ingress class property to refer to the application routing add-on ingress class
+      rules:
+      - http:
+        host: aks-helloworld.<CLUSTER_SPECIFIC_DNS_ZONE> # Replace with your own hostname
+          paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service: 
+                name: aks-helloworld
+                port: 
+                  number: 80
+    ```
+
+3. Update the ingress controller's IP (such as in DNS records) with the new IP address. You can find the new IP by using `kubectl get`. For example:
+
+    ```bash
+    kubectl get svc nginx --namespace app-routing-system -o jsonpath='{.status.loadBalancer.ingress[0].ip}'
+    ```
+
+4. Disable the HTTP application routing add-on.
+
+    ```azurecli-interactive
+    az aks disable-addons -g <ResourceGroupName> -n <ClusterName> --addons http_application_routing
+    ```
+
+## Remove and delete all HTTP application routing resources
+
+1. After the HTTP application routing add-on is disabled, some related Kubernetes resources may remain in your cluster. These resources include *configmaps* and *secrets* that are created in the *kube-system* namespace. To maintain a clean cluster, you may want to remove these resources. Look for *addon-http-application-routing* resources using the following [`kubectl get`][kubectl-get] commands:
+
+    ```bash
+    kubectl get deployments --namespace kube-system
+    kubectl get services --namespace kube-system
+    kubectl get configmaps --namespace kube-system
+    kubectl get secrets --namespace kube-system
+    ```
+
+    The following example output shows *configmaps* that should be deleted:
+
+    ```output
+    NAMESPACE     NAME                                                       DATA   AGE
+    kube-system   addon-http-application-routing-nginx-configuration         0      9m7s
+    kube-system   addon-http-application-routing-tcp-services                0      9m7s
+    kube-system   addon-http-application-routing-udp-services                0      9m7s
+    ```
+
+1. Delete remaining resources using the [`kubectl delete`][kubectl-delete] command. Make sure to specify the resource type, resource name, and namespace. The following example deletes one of the previous configmaps:
+
+    ```bash
+    kubectl delete configmaps addon-http-application-routing-nginx-configuration --namespace kube-system
+    ```
+
+1. Repeat the previous `kubectl delete` step for all *addon-http-application-routing* resources remaining in your cluster.
+
+## Next steps
+
+After migrating to the application routing add-on, learn how to [monitor ingress controller metrics with Prometheus and Grafana](./app-routing-nginx-prometheus.md).
+
+<!-- INTERNAL LINKS -->
+[install-azure-cli]: /cli/azure/install-azure-cli
+[ingress-https]: ./ingress-tls.md
+[app-routing-dns-create]: ./app-routing.md?tabs=without-osm#create-an-azure-dns-zone
+[app-routing-dns-configure]: ./app-routing.md?tabs=without-osm#configure-the-add-on-to-use-azure-dns-to-manage-dns-zones
+
+<!-- EXTERNAL LINKS -->
+[dns-pricing]: https://azure.microsoft.com/pricing/details/dns/
+[kubectl-get]: https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#get
+[kubectl-delete]: https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#delete

articles/aks/http-application-routing.md

@@ -12,7 +12,7 @@ ms.author: allensu
 # HTTP application routing add-on for Azure Kubernetes Service (AKS)
 
 > [!CAUTION]
-> The HTTP application routing add-on is in the process of being retired and isn't recommended for production use. We recommend using the [Application Routing add-on](./app-routing.md) instead.
+> The HTTP application routing add-on is in the process of being retired and isn't recommended for production use. We recommend migrating to the [Application Routing add-on](./app-routing-migration.md) instead.
 
 The HTTP application routing add-on makes it easy to access applications that are deployed to your Azure Kubernetes Service (AKS) cluster by:
 

articles/bastion/kerberos-authentication-portal.md

@@ -5,7 +5,7 @@ description: Learn how to configure Bastion to use Kerberos authentication via t
 author: cherylmc
 ms.service: bastion
 ms.topic: how-to
-ms.date: 06/12/2023
+ms.date: 09/14/2023
 ms.author: cherylmc
 
 ---
@@ -17,11 +17,11 @@ This article shows you how to configure Azure Bastion to use Kerberos authentica
 ## Considerations
 
 * The Kerberos setting for Azure Bastion can be configured in the Azure portal only and not with native client.
-* VMs migrated from on-premises to Azure are not currently supported for Kerberos. 
-* Cross-realm authentication is not currently supported for Kerberos. 
-* Changes to DNS server are not currently supported for Kerberos. After making any changes to DNS server, you will need to delete and re-create the Bastion resource.
+* VMs migrated from on-premises to Azure aren't currently supported for Kerberos. 
+* Cross-realm authentication isn't currently supported for Kerberos. 
+* Changes to DNS server aren't currently supported for Kerberos. After making any changes to DNS server, you'll need to delete and re-create the Bastion resource.
 * If additional DC (domain controllers) are added, Bastion will only recognize the first DC.
-* If additional DCs are added for different domains, the added domains cannot successfully authenticate with Kerberos.
+* If additional DCs are added for different domains, the added domains can't successfully authenticate with Kerberos.
 
 ## Prerequisites
 
@@ -56,11 +56,11 @@ In this section, the following steps help you modify your virtual network and ex
 1. [Update the DNS settings](#update-vnet-dns-servers) for your virtual network.
 1. Go to the portal page for your Bastion deployment and select **Configuration**.
 1. On the Configuration page, select **Kerberos authentication**, then select **Apply**.
-1. Bastion will update with the new configuration settings.
+1. Bastion updates with the new configuration settings.
 
 ## To verify Bastion is using Kerberos
 
-> [!NOTE] 
+> [!NOTE]
 > You must use the User Principal Name (UPN) to sign in using Kerberos.
 
 Once you have enabled Kerberos on your Bastion resource, you can verify that it's actually using Kerberos for authentication to the target domain-joined VM.
@@ -71,7 +71,10 @@ Once you have enabled Kerberos on your Bastion resource, you can verify that it'
 1. End the VM session.
 1. Connect to the target VM again using Bastion. Sign-in should succeed, indicating that Bastion used Kerberos (and not NTLM) for authentication.
 
-## Quickstart: Setup Bastion with Kerberos - Resource Manager template
+   > [!NOTE]
+   > To prevent failback to NTLM, make sure you follow the preceding steps. Enabling Kerberos (without following the procedure) won't prevent failback to NTLM.
+
+## Quickstart: Set up Bastion with Kerberos - Resource Manager template
 
 ### Review the template
 
@@ -418,21 +421,21 @@ Once you have enabled Kerberos on your Bastion resource, you can verify that it'
 The following resources have been defined in the template:
 - Deploys the following Azure resources: 
   - [**Microsoft.Network/virtualNetworks**](/azure/templates/microsoft.network/virtualnetworks): create an Azure virtual network.
-  - [**Microsoft.Network/bastionHosts**](/azure/templates/microsoft.network/bastionHosts): create a Standard SKU Bastion with a public IP and Kerberos feature enabled
-  - Create a Windows 10 ClientVM and a Windows Server 2019 ServerVM
-- Have the DNS Server of the VNET point to the private IP address of the ServerVM (domain controller).
+  - [**Microsoft.Network/bastionHosts**](/azure/templates/microsoft.network/bastionHosts): create a Standard SKU Bastion with a public IP and Kerberos feature enabled.
+  - Create a Windows 10 ClientVM and a Windows Server 2019 ServerVM.
+- Have the DNS Server of the VNet point to the private IP address of the ServerVM (domain controller).
 - Runs a Custom Script Extension on the ServerVM to promote it to a domain controller with domain name: `bastionkrb.test`.
 - Runs a Custom Script Extension on the ClientVM to have it: 
   - **Restrict NTLM: Incoming NTLM traffic** = Deny all domain accounts (this is to ensure Kerberos is used for authentication).
   - Domain-join the `bastionkrb.test` domain.
 
 ## Deploy the template
-To setup Kerberos, deploy the ARM template above by running the following PS cmd: 
+To set up Kerberos, deploy the preceding ARM template by running the following PowerShell cmd: 
 ```
 New-AzResourceGroupDeployment -ResourceGroupName <your-rg-name> -TemplateFile "<path-to-template>\KerberosDeployment.json"`
 ```
 ## Review deployed resources
-Now, login to ClientVM using Bastion with Kerberos authentication:
+Now, sign in to ClientVM using Bastion with Kerberos authentication:
 - credentials: username = `[email protected]` and password = `<password-entered-during-deployment>`.
 
 

articles/batch/best-practices.md

@@ -1,7 +1,7 @@
 ---
 title: Best practices
 description: Learn best practices and useful tips for developing your Azure Batch solutions.
-ms.date: 01/18/2023
+ms.date: 09/13/2023
 ms.topic: conceptual
 ---
 
@@ -35,13 +35,14 @@ initiates communication to the compute nodes, and compute nodes also require com
 node communication model, compute nodes initiate communication with the Batch service. Due to the reduced scope of
 inbound/outbound connections required, and not requiring Azure Storage outbound access for baseline operation, the recommendation
 is to use the simplified node communication model. Some future improvements to the Batch service will also require the simplified
-node communication model.
+node communication model. The classic node communication model will be
+[retired on March 31, 2026](batch-pools-to-simplified-compute-node-communication-model-migration-guide.md).
 
 - **Job and task run time considerations:** If you have jobs comprised primarily of short-running tasks, and the expected total task counts are small, so that the overall expected run time of the job isn't long, don't allocate a new pool for each job. The allocation time of the nodes will diminish the run time of the job.
 
 - **Multiple compute nodes:** Individual nodes aren't guaranteed to always be available. While uncommon, hardware failures, operating system updates, and a host of other issues can cause individual nodes to be offline. If your Batch workload requires deterministic, guaranteed progress, you should allocate pools with multiple nodes.
 
-- **Images with impending end-of-life (EOL) dates:** We strongly recommended avoiding images with impending Batch support
+- **Images with impending end-of-life (EOL) dates:** It's strongly recommended to avoid images with impending Batch support
 end of life (EOL) dates. These dates can be discovered via the
 [`ListSupportedImages` API](/rest/api/batchservice/account/listsupportedimages),
 [PowerShell](/powershell/module/az.batch/get-azbatchsupportedimage), or

articles/batch/security-best-practices.md

@@ -1,7 +1,7 @@
 ---
 title: Batch security and compliance best practices
 description: Learn best practices and useful tips for enhancing security with your Azure Batch solutions.
-ms.date: 11/15/2022
+ms.date: 09/13/2023
 ms.topic: conceptual
 ---
 
@@ -27,7 +27,9 @@ Pools can also be configured in one of two node communication modes, classic or
 In the classic node communication model, the Batch service initiates communication to the compute nodes, and compute nodes
 also require communicating to Azure Storage. In the simplified node communication model, compute nodes initiate communication
 with the Batch service. Due to the reduced scope of inbound/outbound connections required, and not requiring Azure Storage
-outbound access for baseline operation, the recommendation is to use the simplified node communication model.
+outbound access for baseline operation, the recommendation is to use the simplified node communication model. The classic
+node communication model will be
+[retired on March 31, 2026](batch-pools-to-simplified-compute-node-communication-model-migration-guide.md).
 
 ### Batch account authentication
 
@@ -60,10 +62,42 @@ In addition to operations specific to a Batch account, [management operations](/
 
 Batch management operations via Azure Resource Manager are encrypted using HTTPS, and each request is authenticated using Azure AD authentication.
 
-### Batch pool nodes
+### Batch pool compute nodes
 
 The Batch service communicates with a Batch node agent that runs on each node in the pool. For example, the service instructs the node agent to run a task, stop a task, or get the files for a task. Communication with the node agent is enabled by one or more load balancers, the number of which depends on the number of nodes in a pool. The load balancer forwards the communication to the desired node, with each node being addressed by a unique port number. By default, load balancers have public IP addresses associated with them. You can also remotely access pool nodes via RDP or SSH (this access is enabled by default, with communication via load balancers).
 
+#### Batch compute node OS
+
+Batch supports both Linux and Windows operating systems. Batch supports Linux with an aligned node agent for a subset of Linux OS
+distributions. It's recommended that the operating system is kept up-to-date with the latest patches provided by the OS
+publisher.
+
+Batch support for images and node agents phase out over time, typically aligned with publisher support timelines. It's
+recommended to avoid using images with impending end-of-life (EOL) dates or images that are past their EOL date.
+It's your responsibility to periodically refresh your view of the EOL dates pertinent to your pools and migrate your workloads
+before the EOL date occurs. If you're using a custom image with a specified node agent, ensure that you follow Batch support
+end-of-life dates for the image for which your custom image is derived or aligned with. An image without a specified
+`batchSupportEndOfLife` date indicates that such a date hasn't been determined yet by the Batch service. Absence of a date
+doesn't indicate that the respective image will be supported indefinitely. An EOL date may be added or updated in the future
+at any time. EOL dates can be discovered via the
+[`ListSupportedImages` API](/rest/api/batchservice/account/listsupportedimages),
+[PowerShell](/powershell/module/az.batch/get-azbatchsupportedimage), or
+[Azure CLI](/cli/azure/batch/pool/supported-images).
+
+#### Windows OS Transport Layer Security (TLS)
+
+The Batch node agent doesn't modify operating system level defaults for SSL/TLS versions or cipher suite ordering. In Windows,
+SSL/TLS versions and cipher suite order is controlled at the operating system level, and therefore the Batch node agent adopts
+the settings set by the image used by each compute node. Although the Batch node agent attempts to utilize the
+most secure settings available when possible, it can still be limited by operating system level settings.  We recommend that
+you review your OS level defaults and set them appropriately for the most secure mode that is amenable for your workflow and
+organizational requirements. For more information, please visit
+[Manage TLS](https://learn.microsoft.com/windows-server/security/tls/manage-tls) for cipher suite order enforcement and
+[TLS registry settings](https://learn.microsoft.com/windows-server/security/tls/tls-registry-settings) for SSL/TLS version
+control for Schannel SSP. Note that some setting changes require a reboot to take effect. Utilizing a newer operating system
+with modern security defaults or a [custom image](batch-sig-images.md) with modified settings is recommended instead of
+application of such settings with a Batch start task.
+
 ### Restricting access to Batch endpoints
 
 Several capabilities are available to limit access to the various Batch endpoints, especially when the solution uses a virtual network.