Merge pull request #246692 from surajmb/nexusnf-concepts

Adding Network Fabric Concepts

Author: Court72

articles/operator-nexus/TOC.yml

@@ -13,6 +13,8 @@
       href: concepts-compute.md
     - name: Storage overview
       href: concepts-storage.md
+    - name: Network Fabric overview
+      href: concepts-network-fabric.md
     - name: Nexus Kubernetes
       href: concepts-nexus-kubernetes-cluster.md
     - name: Observability

articles/operator-nexus/concepts-network-fabric.md

@@ -0,0 +1,38 @@
+---
+title: "Azure Operator Nexus: Network Fabric"
+description: Overview of Network Fabric resources for Azure Operator Nexus.
+author: surajmb
+ms.author: surmb
+ms.service: azure-operator-nexus
+ms.topic: conceptual
+ms.date: 07/10/2023
+ms.custom: template-concept
+---
+
+# Azure Operator Nexus - Network Fabric
+Azure Operator Nexus offers various capabilities to manage the lifecycle and configuration of the networking required to run the Operator's infrastructure and workloads. 
+
+Operator Nexus enables you to:
+
+* Single pane of glass to manage your on-premises networking devices and their configuration.
+* Create infrastructure and workload networks which are isolated.
+* Configure route policies to import and export specific routes to and from your existing infrastructure network.
+* Monitor and audit device performance, health, and configuration changes and take action against them via metrics, logs, and alerts.
+* Set access policies to govern who can manage the network.
+* Manage the lifecycle of the network devices.
+* Tap or mirror desired network data with Network Packet Broker.
+* Get highly available and robust control plane for your network infrastructure.
+
+:::image type="content" source="media/networking-concepts-1.png" alt-text="Screenshot of Resource Types.":::
+
+Key capabilities offered in Azure Operator Nexus Network Fabric:
+
+* **Bootstrapping and lifecycle management** - Automated bootstrapping & provisioning of network fabric resources based on network function use-cases. It provides various controls to manage network devices in operator premises via Azure APIs.
+
+* **Tenant network configuration** - Automated network configuration in Network Fabric for Container Network Functions (CNFs) and Virtual Network Functions (VNFs) that are deployed on the compute nodes. The network configuration enables east-west communication between network functions as well as north-south communication between external networks and VNFs/CNFs. 
+
+* **Observability** - Monitor the health and performance of the network fabric in real-time with metrics and logs.
+
+* **Network Policy Automation** - Automating the management of consistent network policies across the fabric to ensure security, performance, and access controls are enforced uniformly.
+
+* **Networking features built for Operators** - Support for unique features like multicast, SCTP, and jumbo frames.

articles/operator-nexus/concepts-resource-types.md

@@ -24,17 +24,67 @@ The Operator Nexus Cluster (or Instance) platform components include the infrast
 
 ### Network Fabric Controller
 
-The Network Fabric Controller (NFC) is a resource that automates the life cycle management of all network devices deployed in an Operator Nexus instance.
-NFC is hosted in a [Microsoft Azure Virtual Network](../virtual-network/virtual-networks-overview.md) in an Azure region.
-The region should be connected to your on-premises network via [Microsoft Azure ExpressRoute](../expressroute/expressroute-introduction.md).
-An NFC can manage the Network Fabric of many (subject to limits) Operator Nexus instances.
+Network Fabric Controller (NFC) is an Operator Nexus resource which runs in your subscription in your desired resource group and [Virtual Network](../virtual-network/virtual-networks-overview.md). The Network Fabric Controller acts as a bridge between the Azure control plane and your on-premises infrastructure to manage the lifecycle and configuration of the Network Devices in a Network Fabric instance.
+
+The Network Fabric Controller achieves this by establishing a private connectivity channel between your Azure environment and on-premises using [Azure ExpressRoute](../expressroute/expressroute-introduction.md) and other supporting resources which are deployed in a managed resource group. The NFC is typically the first resource which you would create to establish this connectivity to bootstrap and configure your management and workload networks.
+
+The Network Fabric Controller enables you to manage all the Network resources within your Operator Nexus instance like Network Fabric, Network Racks, Network Devices, Isolation Domains, Route Policies, etc.
+
+You can manage the lifecycle of a Network Fabric Controller via Azure using any of the supported interfaces - Azure CLI, REST API, etc. See [how to create a Network Fabric Controller](./howto-configure-network-fabric-controller.md) to learn more.
 
 ### Network Fabric
 
-The Network Fabric resource models a collection of network devices, compute servers, and storage appliances, and their interconnections. The Network Fabric resource also includes the networking required for your network functions and workloads. Each Operator Nexus instance has one Network Fabric.
+Network Fabric (NF) resource is a representation of your on-premises network topology in Azure. Every Network Fabric must be associated to and controlled by a Network Fabric Controller which is deployed in the same Azure region. You can associate multiple Network Fabric resources per Network Fabric Controller, see [Nexus Limits and Quotas](./reference-limits-and-quotas.md). A single deployment of the infrastructure is considered a Network Fabric instance.
+
+Operator Nexus allows you to create Network Fabrics based on specific SKU types, where each SKU represents the number of network racks and compute servers in each rack deployed on-premises.
+
+Each Network Fabric resource can contain a collection of network racks, network devices, isolation domains for their interconnections. Once a Network Fabric is created and you've validated that your network devices are connected, then it can be Provisioned. Provisioning a Network Fabric is the process of bootstrapping the Network Fabric instance to get the management network up.
+
+You can manage the lifecycle of a Network Fabric via Azure using any of the supported interfaces - Azure CLI, REST API, etc. See [how to create and provision a Network Fabric](./howto-configure-network-fabric.md) to learn more.
+
+### Network racks
+
+Network Rack resource is a representation of your on-premises Racks from the networking perspective. The number of network racks in an Operator Nexus instance depends on the Network Fabric SKU which was chosen while creation. In a multi-rack setup, you can create up to either 4 compute racks + 1 aggregate rack or 8 compute racks + 1 aggregate rack.
+
+Each network rack consists of Network Devices which are part of that rack. For example - Customer Edge (CE) routers, Top of Rack (ToR) Switches, Management Switches, Network Packet Brokers (NPB).
+
+The Network Rack also models the connectivity to the operator's Physical Edge switches (PEs) and the ToRs on the other Racks via Network to Network Interconnect (NNI) resource.
+
+The lifecycle of Network Rack resources is tied to the Network Fabric resource. The Network Racks are automatically created when you create the Network Fabric and the number of racks depends on the SKU which was chosen. When the Network Fabric resource is deleted, all the associated Network Racks are also deleted along with it.
+
+### Network devices
+
+Network Devices represent the Customer Edge (CE) routers, Top of Rack (ToR) Switches, Management Switches, Network Packet Brokers (NPB) which are deployed as part of the Network Fabric instance. Each Network Device resource is associated to a specific Network Rack where it is deployed.
+
+Each network device resource has a SKU, Role, Host Name, and Serial Number as properties, and can have multiple network interfaces associated. Network Interfaces contain the IPv4 and IPv6 addresses, physical identifier, interface type, and the associated connections. Network Interfaces also has the administrativeState property which indicates whether the interface is enabled or disabled.
 
-The Network Fabric Controller (NFC) performs the lifecycle management of the Network Fabric.
-It configures and bootstraps the Network Fabric resources.
+The lifecycle of the Network Interface depends on the Network Device and can exist as long as the parent network device resource exists. However, you can perform certain operations on a network interface resource like enable/disable the administrativeState via Azure using any of the supported interfaces - Azure CLI, REST API, etc.
+
+The lifecycle of the Network Device resources depends on the network rack resource and will exist as long as the parent Network Fabric resource exists. However, before provisioning the Network Fabric, you can perform certain operations on a network device like setting a custom hostname and updating the serial number of the device via Azure using any of the supported interfaces - Azure CLI, REST API, etc.
+
+### Isolation domains
+
+Isolation Domains enable east-west or north-south connectivity across Operator Nexus instance. They provide the required network connectivity between infrastructure components and also workload components. In principle, there are two types of networks which are established by isolation domains - management network and workload or tenant network. 
+
+Management network is the private connectivity that enables communication between the Network Fabric instance which is deployed on-premises and Azure Virtual Network. You can create workload or tenant networks to enable communication between the workloads which are deployed across the Operator Nexus instance.
+
+Each isolation domain is associated to a specific Network Fabric resource and has the option to be enabled/disabled. Only when an isolation domain is enabled, it's configured on the network devices and the configuration is removed once the isolation domain is removed.
+
+Primarily, there are two types of isolation domains:
+
+* Layer 2 or L2 Isolation Domains
+* Layer 3 or L3 Isolation Domains
+
+Layer 2 isolation domains enable your infrastructure and workloads communicate with each other within or across racks over a Layer 2 network. Layer 2 networks enable east-west communication within your Operator Nexus instance. You can configure an L2 isolation domain with a desired Vlan ID and MTU size, see [Nexus Limits and Quotas](./reference-limits-and-quotas.md) for MTU limits.
+
+Layer 3 isolation domains enable your infrastructure and workloads communicate with each other within or across racks over a Layer 3 network. Layer 3 networks enable east-west and north-south communication within and outside your Operator Nexus instance.
+
+There are two types of Layer 3 networks that you can create:
+
+* Internal Network
+* External Network
+
+Internal networks enable layer 3 east-west connectivity across racks within the Operator Nexus instance and external networks enable layer 3 north-south connectivity from the Operator Nexus instance to networks outside the instance. A Layer 3 isolation domain must be configured with at least one internal network and external networks are optional.
 
 ### Cluster manager
 
@@ -48,11 +98,6 @@ The Cluster (or Compute Cluster) resource models a collection of racks, bare met
 Each cluster is mapped to the on-premises Network Fabric. A cluster provides a holistic view of the deployed compute capacity.
 Cluster capacity examples include the number of vCPUs, the amount of memory, and the amount of storage space. A cluster is also the basic unit for compute and storage upgrades.
 
-### Network Rack
-
-The Network Rack consists of Consumer Edge (CE) routers, Top of Rack switches (ToRs), storage appliance, Network Packet Broker (NPB), and the Terminal Server (TS).
-The Rack also models the connectivity to the operator's Physical Edge switches (PEs) and the ToRs on the other Racks.
-
 ### Rack
 
 The Rack (or a compute rack) resource represents the compute servers (Bare Metal Machines), management servers, management switch and ToRs. The Rack is created, updated or deleted as part of the Cluster lifecycle management.
@@ -64,7 +109,7 @@ Storage Appliances represent storage arrays used for persistent data storage in
 ### Bare Metal Machine
 
 Bare Metal Machines represent the physical servers in a rack. They're lifecycle managed by the Cluster Manager.
-Bare Metal Machines are used by workloads to host Virtual Machines and Kubetnetes clusters.
+Bare Metal Machines are used by workloads to host Virtual Machines and Kubernetes clusters.
 
 ## Workload components
 
@@ -81,7 +126,7 @@ There are five Network resource types that represent a network attachment to an
 
 - **Layer 3 Network Resource**: facilitate "North-South" communication between your VMs/tenant clusters and the external network.
 
-- **Trunked Network Resource**: provides a VM or an tenant cluster access to multiple layer 3 networks and/or multiple layer 2 networks.
+- **Trunked Network Resource**: provides a VM or a tenant cluster access to multiple layer 3 networks and/or multiple layer 2 networks.
 
 ### Virtual machine
 

articles/operator-nexus/media/networking-concepts-1.png

@@ -9,7 +9,7 @@ ms.date: 06/28/2023
 ms.custom: template-reference
 ---
 
-# Nexus Limits and Quotas
+# Azure Operator Nexus limits and quotas
 
 This document provides an overview of the resource limits that apply to the components used in the Nexus solution, encompassing the resources created within Azure cloud and in on-premises instance. It outlines the specific limitations and restrictions that operators should be aware of when deploying and managing the Nexus instance across these environments.
 
@@ -41,6 +41,7 @@ The creation of the Network Fabric related resources is subject to the following
 | Layer 2 Isolation domains   | 3500 isolation domains per Nexus instance |
 | Layer 3 Isolation domains   | 200 isolation domains per Nexus instance |
 | Route policies              | 400 route policies per Nexus instance |
+| Isolation domain MTU | 1500 - 9200 |
 
 > [!NOTE]
 > * The number of Nexus instances a pair of NFC + CM can handle has been set to 20 based on some theoretical study for ExpressRoute. These numbers will be refined after more testing. 
@@ -52,7 +53,7 @@ The creation of the Network Cloud specific resources is subject to the following
 | Resource Type               | Notes |
 | --------------------------- | -------------------------|
 | Cluster Manager             |	1:1 mapping with Network Fabric Controller |
-| Cluster                     |	Up to 20 Nexus Cluster instances per Cluster Manager (within same region) [To be updated] |
+| Cluster                     |	Up to 20 Nexus Cluster instances per Cluster Manager (within same region) |
 | Racks                       |	Up to BOM-specified Compute Racks per Nexus Cluster |
 | Bare Metal Machines         |	Up to BOM-specified BareMetal machines per Rack |
 | Storage Appliances          |	Up to BOM-specified Storage appliances per Nexus Cluster instance |