Merge pull request #296427 from pjw711/pjw/second-pure

Add support for multiple storage appliances

Author: Stacyrch140

articles/operator-nexus/TOC.yml

@@ -16,6 +16,8 @@
       items:
         - name: Storage overview
           href: concepts-storage.md
+        - name: Multiple storage appliances
+          href: concepts-storage-multiple-appliances.md
         - name: Storage for Nexus Kubernetes
           href: concepts-storage-kubernetes.md
     - name: Cluster deployment and upgrades
@@ -405,6 +407,12 @@
           items:
             - name: Due To Bare Metal Machine Power Failure
               href: troubleshoot-kubernetes-cluster-stuck-workloads-due-to-power-failure.md
+    - name: Storage Appliance
+      expanded: false
+      items:
+        - name: Troubleshoot Multiple Storage appliances
+          href: troubleshoot-multiple-storage-appliances.md
+
 - name: FAQ
   href: azure-operator-nexus-faq.md
 - name: Reference

articles/operator-nexus/azure-operator-nexus-faq.md

@@ -16,9 +16,9 @@ The following sections cover some of the frequently asked questions for Azure Op
 
 ### What services does Azure Operator Nexus provide?
 Azure Operator Nexus is a managed hybrid cloud platform that supports carrier-grade network workloads. Here, the management plane lives in Azure and the control plane and user plane gets deployed on operators' premises or in Azure. It simplifies provisioning of new network services and optimizes deployment of network functions and applications on premises. The end customer can deploy containerized applications (on an on premises Nexus AKS cluster) or a virtualized workload to run these network functions. Customer gets out of the box integration with many Azure services such as Azure Monitor, Azure Container Registry, and Azure Kubernetes Services.
-          
+
 ### How do I interact with Operator Nexus instance?
-You can interact with Operator Nexus like any other Azure services using AZ CLI, API, ARM template, or portal. You can alternatively use BICEP templates.  
+You can interact with Operator Nexus like any other Azure services using AZ CLI, API, ARM template, or portal. You can alternatively use BICEP templates.
 
 ### Does customer need to deploy any resources in their subscription to deploy Azure Operator Nexus instances?
 Yes, there are some resources that customer needs to create in the respective region under their Azure subscriptions. Some of these include creation of a pair of Network Fabric Controller and Cluster Manager resource, Log Analytics Workspace, a storage account. For more details, please refer to [Azure Operator Nexus documentation](howto-azure-operator-nexus-prerequisites.md).
@@ -33,32 +33,32 @@ Yes, to ensure carrier-grade performance and high degrees of automation, you nee
 Customers should design their services with Intra-rack redundancy, Inter-rack redundancy, and globally load balancing across multiple instances. Also, for high availability, plan to spread your instances across multiple Azure regions.
 
 ### How do updates work to on-premises and to Azure components?
-Upgrades to Operator Nexus are made in two phases - Management bundle upgrades and Runtime bundle upgrades. Management bundle upgrades deals with the upgrades of Controllers in Azure, Cluster Managers in customer subscription and on-premises instances. In on-premises instances, it includes the Kubernetes controllers responsible for maintaining the state of infra resources. 
-          
+Upgrades to Operator Nexus are made in two phases - Management bundle upgrades and Runtime bundle upgrades. Management bundle upgrades deals with the upgrades of Controllers in Azure, Cluster Managers in customer subscription and on-premises instances. In on-premises instances, it includes the Kubernetes controllers responsible for maintaining the state of infra resources.
+
 Updates of Management bundle may cause interruptions to provisioning activities but it doesn't impacts the customers running workloads. Customers don't control or drive these upgrades, but these upgrades are essential to provide customers with the options to update to new runtime-based upgrades within their on-premises instances.
-          
+
 On the other hand, Runtime bundle upgrades deals with the components that require updates to the OS (Operating System) and/or workload supporting components. The update of the runtime bundle is entirely under the control of the customer and APIs can be used to perform these updates. You might observe some workload impacts during this upgrade.
 
 ### Is the storage appliance a must required device?
-For near-edge SKUs, Storage appliance is a part of the Hardware infrastructure that Operator needs to procure. 
+For near-edge SKUs, Storage appliance is a part of the Hardware infrastructure that Operator needs to procure. You can optionally choose to deploy a second storage appliance in an Azure Operator Nexus instance. For more information, see [Multiple storage appliances](./concepts-storage-multiple-appliances.md).
 
 ### Does Operator Nexus provide best practices or blueprints for deploying network functions on Operator Nexus?
-Yes, Operator Nexus comes with Nexus Ready program. With this program, Microsoft is working with industry leading Network Function partners to validate that their network functions to ensure they can run on Nexus platform. We validate these network functions on regular intervals to ensure that they stay compliant with newer versions of Nexus. Operators can now get consistent and scalable deployment of multi-vendor network functions with the Nexus Ready program. 
+Yes, Operator Nexus comes with Nexus Ready program. With this program, Microsoft is working with industry leading Network Function partners to validate that their network functions to ensure they can run on Nexus platform. We validate these network functions on regular intervals to ensure that they stay compliant with newer versions of Nexus. Operators can now get consistent and scalable deployment of multi-vendor network functions with the Nexus Ready program.
 
 ### What data stays on premises and what is available in Azure?
 From Infrastructure perspective, the data is managed via Azure APIs. The telemetry from these layers gets collected and is visible under customer subscription. Customers can use Log Analytics Workspace, storage accounts or other Analytics services in Azure to look into the telemetry from Infra layers.
-          
+
 For tenant workloads, the images get stored in ACRs (Azure Container Registry) and once deployed. Microsoft provides an option to collect the telemetry from tenant workloads into Azure but Customers can choose alternative tooling they wish to collect telemetry data or to analyze it.
 
 ### If an Azure region doesn't exist in my country/region, can I still use Operator Nexus?
-Yes, all you need is ExpressRoute connectivity to an Azure region. ExpressRoute connectivity is available at many locations. For more information, see the [Geo-locations](../expressroute/expressroute-locations.md#locations) and [connectivity providers](../expressroute/expressroute-locations.md#partners). 
+Yes, all you need is ExpressRoute connectivity to an Azure region. ExpressRoute connectivity is available at many locations. For more information, see the [Geo-locations](../expressroute/expressroute-locations.md#locations) and [connectivity providers](../expressroute/expressroute-locations.md#partners).
 
 ### Can I move my resources from one subscription to another?
 Currently, we don't support resource moves. If you need to move resources, you can consider deleting the existing controllers and using the ARM template to create another one in another location.
 
-### How many instances can be associated to a cluster manager/fabric controller pair? 
-The number of Azure Operator Nexus instances, a single pair of Network Fabric Controller and Cluster Manager can manage depends on multiple factors. It can be influenced by factors like size of Operator Nexus instances, ExpressRoute circuit bandwidth, number, and frequency of optional metrics collection, number of workloads running in Instance, destination for workload telemetry data collection and other factors. 
-          
+### How many instances can be associated to a cluster manager/fabric controller pair?
+The number of Azure Operator Nexus instances, a single pair of Network Fabric Controller and Cluster Manager can manage depends on multiple factors. It can be influenced by factors like size of Operator Nexus instances, ExpressRoute circuit bandwidth, number, and frequency of optional metrics collection, number of workloads running in Instance, destination for workload telemetry data collection and other factors.
+
 For more information, see [limits & quotas](reference-limits-and-quotas.md).
 
 ### Is it viable to redeploy a cluster that is currently running? If so, what safeguards are in place to prevent accidental deployments?
@@ -80,49 +80,49 @@ Yes, Azure Operator Nexus provides the ability to create customized VMs for host
 
 ### How do I update a bare metal server?
 To update a bare metal server within an Azure Operator Nexus instance, you can use the Azure APIs. Any update for bare metal server is part of runtime bundle upgrade. This upgrade requires the Nexus instance connectivity back to Azure.
-          
+
 ### What OS runs on the bare metal server? Can I bring my own?
 Bare metal servers are deployed with Microsoft Azure Linux (previously called CBL Mariner OS), which is thoroughly tested and is compatible with required Azure agents. There are no plans to support any other OS offering for Bare metal servers.
 
 ### How many servers does Operator Nexus support? How many racks?
-An Operator Nexus instance can have up to eight compute racks and each rack hosting upto 16 servers. These compute servers are used for running actual tenant workloads.
+An Operator Nexus instance can have up to eight compute racks and each rack hosting up to 16 servers. These compute servers are used for running actual tenant workloads.
 
 ## Networks
 
 ### Does Operator Nexus support IPv6?
-Yes, Azure Operator Nexus provides support for both IPV4 and IPV6 configuration across all layers of the stack. 
-          
+Yes, Azure Operator Nexus provides support for both IPV4 and IPV6 configuration across all layers of the stack.
+
 ### What are the networking requirements for Azure Operator Nexus?
 Here are some of the network requirements for Azure Operator Nexus:
-* Customers need to work with Microsoft partners for setting up ExpressRoute connections, 
+* Customers need to work with Microsoft partners for setting up ExpressRoute connections,
 * PE (Provider Edge) device supports 400G or 100G connections to CE (Customer Edge) device in Operator Nexus instance
 * PE must have routes to ExpressRoutes
-* IP address blocks defined for various services, VLANs for iDrac, PXE, Storage, OAM etc. 
-          
+* IP address blocks defined for various services, VLANs for iDrac, PXE, Storage, OAM etc.
+
 For more information, see [Network fabric controller](howto-configure-network-fabric-controller.md) and [Network fabric](howto-configure-network-fabric.md).
-          
+
 ### What is an isolation domain?
 Isolation domains enable Layer 2 or Layer 3 connectivity between workloads hosted across the Azure Operator Nexus instance and external networks. These constructs segment a network into authentication domains and enforces communication within required boundaries.
-          
+
 ### Does Operator Nexus support a single ToR (Top of Rack) device?
-For near-edge, the fabric is designed based on high availability model and the reason you can't have just one ToR switch.  
+For near-edge, the fabric is designed based on high availability model and the reason you can't have just one ToR switch.
 
 ### Is Network packet Broker (NPB) a hard requirement?
-For near-edge SKUs, NPBs will be part of the BOM. 
+For near-edge SKUs, NPBs will be part of the BOM.
 
 ### How do I configure the load balancing service in a Nexus Kubernetes cluster?
 The load balancer allows external services to access the services running within the cluster. You can refer to the [load balancer](./howto-kubernetes-service-load-balancer.md) article for more information.
-      
+
 ## Tenant workloads
 
-### Can I bring my own K8S cluster? 
+### Can I bring my own K8S cluster?
 Nexus platform offers customers an option to either create Nexus AKS clusters with multiple Kubernetes versions or Virtual Machines for running their workloads.
 
 ### What VNFs (Virtualized Network Functions) and CNFs (Containerized Network Functions) are certified on the platform?
 Validation of VNF and CNF functions is an ongoing activity. We'll publish the list of certified VNFs and CNFs soon.
 
 ### If a VNF or CNF isn't certified, can I still use it?
-Indeed, you can collaborate with the Nexus team to ensure there are no limitations that would prevent you from deploying these workloads. 
+Indeed, you can collaborate with the Nexus team to ensure there are no limitations that would prevent you from deploying these workloads.
 
 ### How do I deploy a VM/Container?
 Customers can use APIs to deploy VM and Nexus AKS clusters in an Operator Nexus instance. For a richer experience, Microsoft offers another service AOSM (Azure Operator Service Manager) which allows you to automate deployment of your Containerized (CNF) and Virtualized (VNF) Network Functions.
@@ -131,7 +131,7 @@ Customers can use APIs to deploy VM and Nexus AKS clusters in an Operator Nexus
 Nexus AKS clusters have a support for Read Write Many (RWX) & Read Write Once (RWO) storage classes.
 
 ### How can Operator Nexus VMs be made highly available?
-The Operators can choose to deploy VMs across multiple bare metal machines and across racks to achieve high availability. 
+The Operators can choose to deploy VMs across multiple bare metal machines and across racks to achieve high availability.
 
 ## Observability
 

articles/operator-nexus/concepts-cluster-deployment-overview.md

@@ -37,7 +37,7 @@ The ephemeral bootstrap node sequentially provisions each KCP node, and if a KCP
 
 After successful provisioning of KCP nodes, the deployment action proceeds to provision NMP nodes in parallel. If an NMP node fails to provision, the cluster deployment action fails, resulting in the cluster status being marked as failed.
 
-Upon successful provisioning of NMP nodes, a storage appliance is created before the deployment action proceeds with provisioning the compute nodes. Compute nodes are provisioned in parallel, and once the defined compute node threshold is met, the cluster status transitions from Deploying to Running. However, the remaining nodes continue undergoing the provisioning process until they too are successfully provisioned.
+Upon successful provisioning of NMP nodes, up to two storage appliances are created before the deployment action proceeds with provisioning the compute nodes. Compute nodes are provisioned in parallel, and once the defined compute node threshold is met, the cluster status transitions from Deploying to Running. However, the remaining nodes continue undergoing the provisioning process until they too are successfully provisioned.
 
 
 ## Cluster operations

articles/operator-nexus/concepts-nexus-availability.md

@@ -59,7 +59,7 @@ Go through the following steps to help plan an Operator Nexus deployment.
 
 6.  Operator Nexus supports between 1 and 8 racks per site inclusive, with each rack containing 4, 8, 12 or 16 servers. All racks must be identical in terms of number of servers. See [here](./reference-near-edge-compute.md) for specifics of the resource available for workloads. See the following diagram, and also [this article](./reference-limits-and-quotas.md) for other limits and quotas that might have an impact.
 
-7.  Operator Nexus supports one or two storage appliances. Currently, these arrays are available to workload NFs running as Kubernetes nodes. Workloads running as VMs use local storage from the server they're instantiated on.
+7.  Operator Nexus supports one or two Pure storage appliances. Storage appliances typically reach high levels of availability and data durability; precise numbers vary per vendor and per product. Azure Operator Nexus has redundant connectivity to the storage appliances on the management and data networks. Storage appliances provide storage to workload NFs running on Kubernetes. There is no support for automatic or manual migration of volumes between storage appliances, and both storage appliances are located in the aggregator rack. This means, from a storage appliance perspective, Nexus deployments with two storage appliances have the same availability characteristics as deployments with one storage appliance: a volume is only ever on one storage appliance.
 
 8.  Other factors to consider are the number of available physical sites, and any per-site limitations such as bandwidth or power.
 

articles/operator-nexus/concepts-storage-kubernetes.md

@@ -11,7 +11,7 @@ ms.custom: template-concept
 
 # Azure Operator Nexus storage for Kubernetes
 
-Each Azure Operator Nexus provides two types of persistent storage to Nexus Kubernetes cluster tenant workloads: *nexus-volume* and *nexus-shared*. Operators select the type of storage they need by creating Persistent Volume Claims (PVCs) using the *nexus-volume* or *nexus-shared* storage class. All data stored in persistent volumes is stored on a storage appliance deployed on-premises as part of the Azure Operator Nexus instance.
+Each Azure Operator Nexus provides two types of persistent storage to Nexus Kubernetes cluster tenant workloads: *nexus-volume* and *nexus-shared*. Operators select the type of storage they need by creating Persistent Volume Claims (PVCs) using the *nexus-volume* or *nexus-shared* storage class. All data stored in persistent volumes is stored on a storage appliance deployed on-premises as part of the Azure Operator Nexus instance. Azure Operator Nexus requires one storage appliance and supports up to two storage appliances per Azure Operator Nexus instance.
 
 > [!IMPORTANT]
 > Azure Operator Nexus doesn't support ephemeral volumes. Nexus recommends that the persistent volume storage mechanisms described in this document are used for all Nexus Kubernetes cluster workload volumes as these mechanisms provide the highest levels of performance and availability. All storage in Azure Operator Nexus is provided by the storage appliance. There's no support for storage provided by baremetal machine disks.
@@ -45,6 +45,8 @@ status:
   phase: Bound
 ```
 
+Some Azure Operator Nexus deployments may have two storage appliances installed. Persistent volume claims (PVCs) using the *nexus-volume* storage class can place the associated persistent volumes onto a specific storage appliance by using the *storageApplianceName* annotation. More information is available in [this document](./concepts-storage-multiple-appliances.md).
+
 ### StorageClass: nexus-shared
 
 In situations where a shared file system is required, the *nexus-shared* storage class is available. This storage class provides a highly available shared storage solution by enabling multiple pods in the same Nexus Kubernetes cluster to concurrently access and share the same volume. The *nexus-shared* storage class is backed by a highly available NFS storage service. This NFS storage service (storage pool currently limited to a maximum size of 1 TiB) is available per Cloud Service Network (CSN). The NFS storage service is deployed automatically on creation of a CSN resource. Any Nexus Kubernetes cluster attached to the CSN can provision persistent volumes from this shared storage pool. Nexus-shared supports both Read Write Once (RWO) and Read Write Many (RWX) access modes. What that means is that the workload applications can make use of either of these access modes to access the shared storage.