Merge branch 'patch-70' of https://github.com/abhilashaagarwala/azure-stack-docs-pr into aa-contrib

Author: sethmanheim

AKS-Hybrid/TOC.yml

@@ -62,7 +62,7 @@
         href: resource-manager-quickstart.md
     - name: Networking
       items:
-      - name: Create logical networks for Kubernetes clusters
+      - name: Create logical networks
         href: aks-networks.md
       - name: Use MetalLB load balancer
         items:
@@ -72,6 +72,8 @@
           href: deploy-load-balancer-portal.md
         # - name: Troubleshoot issues
         #   href: load-balancer-troubleshoot.md
+    - name: Use Arc Gateway
+      href: arc-gateway-aks-arc.md
     - name: Authentication and authorization
       items:
       - name: Enable Microsoft Entra ID authentication for Kubernetes clusters

AKS-Hybrid/aks-create-clusters-cli.md

@@ -47,7 +47,7 @@ az extension add -n connectedk8s --upgrade
 Use the `az aksarc create` command to create a Kubernetes cluster in AKS Arc. Make sure you sign in to Azure before running this command. If you have multiple Azure subscriptions, select the appropriate subscription ID using the [az account set](/cli/azure/account#az-account-set) command.
 
 ```azurecli
-az aksarc create -n $aksclustername -g $resource_group --custom-location $customlocationID --vnet-ids $logicnetId --aad-admin-group-object-ids $aadgroupID --generate-ssh-keys --load-balancer-count 0  --control-plane-ip $controlplaneIP
+az aksarc create -n $aksclustername -g $resource_group --custom-location $customlocationID --vnet-ids $logicnetId --aad-admin-group-object-ids $aadgroupID --generate-ssh-keys 
 ```
 
 After a few minutes, the command completes and returns JSON-formatted information about the cluster.

AKS-Hybrid/aks-hci-ip-address-planning.md

@@ -22,10 +22,10 @@ In the following scenario walk-through, you reserve IP addresses from a single n
 
 | IP address requirement    | Minimum number of IP addresses | How and where to make this reservation |
 |------------------|---------|---------------|
-| AKS Arc VM IPs | Reserve one IP address for every worker node in your Kubernetes cluster. For example, if you want to create 3 node pools with 3 nodes in each node pool, you need to have 9 IP addresses in your IP pool. | Reserve IP addresses for AKS Arc VMs through IP pools in Arc VM logical network. |
-| AKS Arc K8s version upgrade IPs | Because AKS Arc performs rolling upgrades, reserve one IP address for every AKS Arc cluster for Kubernetes version upgrade operations. | Reserve IP addresses for K8s version upgrade operations through IP pools in Arc VM logical networks. |
-| Control plane IP | Reserve one IP address for every Kubernetes cluster in your environment. For example, if you want to create 5 clusters in total, reserve 5 IP addresses, one for each Kubernetes cluster. | Reserve IP addresses for control plane IPs in the same subnet as the Arc VM logical network, but outside the specified IP pool. |
-| Load balancer IPs | The number of IP addresses reserved depends on your application deployment model. As a starting point, you can reserve one IP address for every Kubernetes service. | Reserve IP addresses for control plane IPs in the same subnet as the Arc VM logical network, but outside the specified IP pool. |
+| AKS Arc VM IPs | Reserve one IP address for every worker node in your Kubernetes cluster. For example, if you want to create 3 node pools with 3 nodes in each node pool, you need to have 9 IP addresses in your IP pool. | Reserve IP addresses through IP pools in Arc VM logical network. |
+| AKS Arc K8s version upgrade IPs | Because AKS Arc performs rolling upgrades, reserve one IP address for every AKS Arc cluster for Kubernetes version upgrade operations. | Reserve IP addresses through IP pools in Arc VM logical network. |
+| Control plane IP | Reserve one IP address for every Kubernetes cluster in your environment. For example, if you want to create 5 clusters in total, reserve 5 IP addresses, one for each Kubernetes cluster. | Reserve IP addresses through IP pools in Arc VM logical network. |
+| Load balancer IPs | The number of IP addresses reserved depends on your application deployment model. As a starting point, you can reserve one IP address for every Kubernetes service. | Reserve IP addresses in the same subnet as the Arc VM logical network, but outside the IP pool. |
 
 ### Example walkthrough for IP address reservation for Kubernetes clusters and applications
 
@@ -48,8 +48,7 @@ Continuing with this example, and adding it to the following table, you get:
 
 | Parameter    | Number of IP addresses | How and where to make this reservation |
 |------------------|---------|---------------|
-| AKS Arc VMs and K8s version upgrade  | Reserve 14 IP addresses | Make this reservation through IP pools in the Azure Local logical network. |
-| Control plane IP | Reserve 2 IP addresses, one for AKS Arc cluster | Use the `controlPlaneIP` parameter to pass the IP address for control plane IP. Ensure that this IP is in the same subnet as the Arc logical network, but outside the IP pool defined in the Arc logical network. |
+| AKS Arc VMs, K8s version upgrade and control plane IP  | Reserve 16 IP addresses | Make this reservation through IP pools in the Azure Local logical network. |
 | Load balancer IPs | 3 IP address for Kubernetes services, for Jane's voting application. | These IP addresses are used when you install a load balancer on cluster A. You can use the MetalLB Arc extension, or bring your own 3rd party load balancer. Ensure that this IP is in the same subnet as the Arc logical network, but outside the IP pool defined in the Arc VM logical network. |
 
 ### LNETs considerations for AKS clusters and Arc VMs

AKS-Hybrid/aks-hci-network-system-requirements.md

@@ -17,11 +17,10 @@ This article introduces core networking concepts for your VMs and applications i
 
 In this conceptual article, the following key components are introduced. These components need a static IP address in order for the AKS Arc cluster and applications to create and operate successfully:
 
-- AKS cluster VMs
-- AKS control plane IP
+- Logical network for AKS Arc VMs and control plane IP
 - Load balancer for containerized applications
 
-## Networking for AKS cluster VMs
+## Logical network for AKS Arc VMs and control plane IP
 
 Kubernetes nodes are deployed as specialized virtual machines in AKS enabled by Arc. These VMs are allocated IP addresses to enable communication between Kubernetes nodes. AKS Arc uses Azure Local logical networks to provide IP addresses and networking for the underlying VMs of the Kubernetes clusters. For more information about logical networks, see [Logical networks for Azure Local](/azure-stack/hci/manage/create-logical-networks?tabs=azurecli). You must plan to reserve one IP address per AKS cluster node VM in your Azure Local environment.
 
@@ -37,16 +36,13 @@ The following parameters are required in order to use a logical network for AKS
 | `--dns-servers`      | Space-separated list of DNS server IP addresses. Usage: `--dns-servers 10.220.32.16 10.220.32.17`. | ![Supported](media/aks-hybrid-networks/check.png) |
 | `--gateway`         | Gateway. The gateway IP address must be within the scope of the address prefix. Usage: `--gateway 10.220.32.16`. | ![Supported](media/aks-hybrid-networks/check.png) |
 | `--ip-allocation-method`         | The IP address allocation method. Supported values are "Static". Usage: `--ip-allocation-method "Static"`. | ![Supported](media/aks-hybrid-networks/check.png) |
-| `--ip-pool-start`     | The start IP address of your IP pool. The address must be in range of the address prefix. Usage: `--ip-pool-start "10.220.32.18"`.  | ![Supported](media/aks-hybrid-networks/check.png) |
-| `--ip-pool-end`       | The end IP address of your IP pool. The address must be in range of the address prefix. Usage: `--ip-pool-end "10.220.32.38"`.  | ![Supported](media/aks-hybrid-networks/check.png) |
 | `--vm-switch-name`     | The name of the VM switch. Usage: `--vm-switch-name "vm-switch-01"`. | ![Supported](media/aks-hybrid-networks/check.png) |
 
-## Control plane IP
+### Control plane IP
 
-Kubernetes uses a control plane to ensure every component in the Kubernetes cluster is kept in the desired state. The control plane also manages and maintains the worker nodes that hold the containerized applications. AKS enabled by Arc deploys the KubeVIP load balancer to ensure that the API server IP address of the Kubernetes control plane is available at all times. This KubeVIP instance requires a single immutable "control plane IP address" to function correctly.
+Kubernetes uses a control plane to ensure every component in the Kubernetes cluster is kept in the desired state. The control plane also manages and maintains the worker nodes that hold the containerized applications. AKS enabled by Arc deploys the KubeVIP load balancer to ensure that the API server IP address of the Kubernetes control plane is available at all times. This KubeVIP instance requires a single immutable "control plane IP address" to function correctly. AKS Arc automatically chooses a control plane IP for you from the logical network passed during the Kubernetes cluster create operation.
 
-> [!NOTE]
-> The control plane IP is a required parameter to create a Kubernetes cluster. You must ensure that the control plane IP address of a Kubernetes cluster does not overlap with anything else, including Arc VM logical networks, infrastructure network IPs, load balancers, etc. The control plane IP also must be within the scope of the address prefix of the logical network, but outside the IP pool. This is because the IP pool is only used for VMs, and if you choose an IP address from the IP pool for the control plane, an IP address conflict can result. Overlapping IP addresses can lead to unexpected failures for both the AKS cluster and any other place the IP address is being used. You must plan to reserve one IP address per Kubernetes cluster in your environment.
+You also have the option of passing a control plane IP. In such cases, the control plane IP must be within the scope of the address prefix of the logical network. You must ensure that the control plane IP address does not overlap with anything else, including Arc VM logical networks, infrastructure network IPs, load balancers, etc. Overlapping IP addresses can lead to unexpected failures for both the AKS cluster and any other place the IP address is being used. You must plan to reserve one IP address per Kubernetes cluster in your environment.
 
 ## Load balancer IPs for containerized applications
 
@@ -60,7 +56,7 @@ Whether you choose the Arc extension for MetalLB, or bring your own load balance
 - Provide IP addresses for your services from the same subnet as the AKS Arc VMs.
 - Use a different network and list of IP addresses if your application needs external load balancing.
 
-Regardless of the option you choose, you must ensure that the IP addresses allocated to the load balancer don't conflict with the IP addresses in the logical network or control plane IPs for your Kubernetes clusters. Conflicting IP addresses can lead to unforeseen failures in your AKS deployment and applications.
+Regardless of the option you choose, you must ensure that the IP addresses allocated to the load balancer don't conflict with the IP addresses in the logical network. Conflicting IP addresses can lead to unforeseen failures in your AKS deployment and applications.
 
 ## Proxy settings
 

AKS-Hybrid/aks-networks.md

@@ -36,9 +36,9 @@ Get-VmSwitch -SwitchType External
 ```
 
 ```output
-Name                               SwitchType       NetAdapterInterfaceDescription
-----                               ----------       ----------------------------
-ConvergedSwitch(management_compute_storage) External        Teamed-Interface
+Name                                           SwitchType      NetAdapterInterfaceDescription
+----                                           ----------      ----------------------------
+ConvergedSwitch(management_compute_storage)    External        Teamed-Interface
 ```
 
 ## Create the logical network
@@ -64,6 +64,7 @@ For static IP, the required parameters are as follows:
 | `--ip-allocation-method`   | The IP address allocation method. Supported values are "Static". Usage: `--ip-allocation-method "Static"`. |
 | `--ip-pool-start`     | The start IP address of your IP pool. The address must be in range of the address prefix. Usage: `--ip-pool-start "10.220.32.18"`.  | 
 | `--ip-pool-end`       | The end IP address of your IP pool. The address must be in range of the address prefix. Usage: `--ip-pool-end "10.220.32.38"`.  |
+
 ```azurecli
 az stack-hci-vm network lnet create --subscription $subscription --resource-group $resource_group --custom-location $customLocationID --name $lnetName --vm-switch-name $vmSwitchName --ip-allocation-method "Static" --address-prefixes $addressPrefixes --gateway $gateway --dns-servers $dnsServers --ip-pool-start $ipPoolStart --ip-pool-end $ipPoolEnd
 ```

AKS-Hybrid/arc-gateway-aks-arc.md

@@ -0,0 +1,105 @@
+---
+title: Simplify network configuration requirements with Azure Arc gateway (preview)
+description: Learn how to enable Arc gateway on AKS Arc clusters to simplify network configuration requirements
+ms.topic: how-to
+ms.date: 11/18/2024
+author: sethmanheim
+ms.author: sethm 
+ms.reviewer: abha
+ms.lastreviewed: 11/18/2024
+
+---
+
+# Simplify network configuration requirements with Azure Arc Gateway (preview)
+
+If you use enterprise proxies to manage outbound traffic, the Azure Arc gateway (preview) can help simplify the process of enabling connectivity.
+
+The Azure Arc gateway (preview) lets you:
+
+- Connect to Azure Arc by opening public network access to only seven fully qualified domain names (FQDNs).
+- View and audit all traffic that the Arc agents send to Azure via the Arc gateway.
+
+> [!IMPORTANT]
+> Azure Arc gateway is currently in preview.
+>
+> See the [Supplemental Terms of Use for Microsoft Azure Previews](https://azure.microsoft.com/support/legal/preview-supplemental-terms/) for legal terms that apply to Azure features that are in beta, preview, or otherwise not yet released into general availability.
+
+## How the Azure Arc gateway works
+
+The Arc gateway works by introducing two new components:
+
+- The **Arc gateway resource** is an Azure Resource that serves as a common front end for Azure traffic. The gateway resource is served on a specific domain/URL. You must create this resource by following the steps outlined in this article. After you successfully create the gateway resource, this domain/URL is included in the success response.
+- The **Arc Proxy** is a new component that runs as its own pod (called "Azure Arc Proxy"). This component acts as a forward proxy used by Azure Arc agents and extensions. There is no configuration required on your part for the Azure Arc Proxy. 
+
+Visit [how the Azure Arc gateway works](https://learn.microsoft.com/azure/azure-arc/kubernetes/arc-gateway-simplify-networking?tabs=azure-cli) to learn more.
+
+> [!IMPORTANT]
+> Note that Azure Local and AKS do not support TLS terminating proxies, ExpressRoute/site-to-site VPN or private endpoints.
+> In addition, there is a limit of five Arc gateway resources per Azure subscription.
+
+## Before you begin
+- Ensure you've gone through the [pre-requisites for creating AKS clusters on Azure Local](/aks-hci-network-system-requirements.md)
+- **The following Azure permissions are required** to create Arc gateway resources and manage their association with AKS Arc clusters:
+    - `Microsoft.Kubernetes/connectedClusters/settings/default/write`
+    - `Microsoft.hybridcompute/gateways/read`
+    - `Microsoft.hybridcompute/gateways/write`
+- **An Arc gateway resource** can be created using Azure CLI or Azure portal. Visit [create the Arc gateway resource in Azure](/hci/deploy/deployment-azure-arc-gateway-overview#create-the-arc-gateway-resource-in-azure) for more information on how to create an Arc gateway resource for your AKS clusters and Azure Local. Once you've created the Arc gateway resource, get the gateway resource ID by running the following command:
+
+```
+$gatewayId = "(az arcgateway show --name <gateway's name> --resource-group <resource group> --query id -o tsv)"
+```
+
+
+## Confirm access to required URLs
+
+Ensure your Arc gateway URL and all of the URLs below are allowed through your enterprise firewall: 
+
+|URL  |Purpose  |
+|---------|---------|
+|`[Your URL prefix].gw.arc.azure.com`       | Your gateway URL. This URL can be obtained by running `az arcgateway list` after you create the resource.         |
+|`management.azure.com`    |Azure Resource Manager Endpoint, required for ARM control channel.         |
+|`<region>.obo.arc.azure.com`     |Required when [Cluster connect](conceptual-cluster-connect.md) is configured.         |
+|`login.microsoftonline.com`, `<region>.login.microsoft.com`     | Microsoft Entra ID endpoint, used for acquiring identity access tokens.         |
+|`gbl.his.arc.azure.com`, `<region>.his.arc.azure.com`   |The cloud service endpoint for communicating with Arc Agents. Uses short names, for example `eus` for East US.          |
+|`mcr.microsoft.com`, `*.data.mcr.microsoft.com`     |Required to pull container images for Azure Arc agents.         |
+
+## Create AKS Arc clusters with Arc gateway enabled
+Run the following command to create AKS Arc clusters with Arc gateway enabled
+
+```azcli
+az aksarc create -n $clusterName -g $resourceGroup --custom-location $customlocationID --vnet-ids $arcVmLogNetId --aad-admin-group-object-ids $aadGroupID --gateway-id $gatewayId --generate-ssh-keys
+```
+
+## Monitor traffic
+
+To audit your gateway's traffic, view the gateway router's logs:
+
+1. Run `kubectl get pods -n azure-arc`
+2. Identify the Arc Proxy pod (its name will begin with `arc-proxy-`).
+3. Run `kubectl logs -n azure-arc <Arc Proxy pod name>`
+
+## Additional scenarios
+
+During the public preview, Arc gateway covers endpoints required for AKS Arc clusters, and a portion of endpoints required for additional Arc-enabled scenarios. Based on the scenarios you adopt, additional endpoints are still required to be allowed in your proxy.
+
+All endpoints listed for the following scenarios must be allowed in your enterprise proxy when Arc gateway is in use:
+
+- [Container insights in Azure Monitor](/azure/azure-monitor/containers/kubernetes-monitoring-firewall):
+  - `*.ods.opinsights.azure.com`
+  - `*.oms.opinsights.azure.com`
+  - `*.monitoring.azure.com`
+- [Azure Key Vault](/azure/key-vault/general/access-behind-firewall):
+  - `<vault-name>.vault.azure.net`
+- [Azure Policy](/azure/governance/policy/concepts/policy-for-kubernetes):
+  - `data.policy.core.windows.net`
+  - `store.policy.core.windows.net`
+- [Microsoft Defender for Containers](/azure/defender-for-cloud/defender-for-containers-enable?pivots=defender-for-container-arc&toc=%2Fazure%2Fazure-arc%2Fkubernetes%2Ftoc.json&bc=%2Fazure%2Fazure-arc%2Fkubernetes%2Fbreadcrumb%2Ftoc.json&tabs=aks-deploy-portal%2Ck8s-deploy-asc%2Ck8s-verify-asc%2Ck8s-remove-arc%2Caks-removeprofile-api):
+  - `*.ods.opinsights.azure.com`
+  - `*.oms.opinsights.azure.com`
+- [Azure Arc-enabled data services](/azure/azure-arc/network-requirements-consolidated?tabs=azure-cloud)
+  - `*.ods.opinsights.azure.com`
+  - `*.oms.opinsights.azure.com`
+  - `*.monitoring.azure.com`
+
+## Next steps
+- [Deploy extension for MetalLB for Azure Arc enabled Kubernetes clusters](/deploy-load-balancer-cli.md).