You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/azure-vmware/azure-vmware-solution-nsx-scale-and-performance-recommendations-for-vmware-hcx.md
+58-60Lines changed: 58 additions & 60 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -3,7 +3,7 @@ title: NSX Scale and Performance Recommendations for VMware HCX
3
3
description: Learn about the default NSX Topology in Azure VMware Solution and recommended practices to mitigate performance issues around HCX migration use cases.
4
4
ms.topic: how-to
5
5
ms.service: azure-vmware
6
-
ms.date: 12/18/2024
6
+
ms.date: 12/19/2024
7
7
ms.custom: engagement-fy25
8
8
---
9
9
@@ -15,99 +15,97 @@ In this article, learn about the default NSX topology in Azure VMware Solution,
15
15
16
16
The Azure VMware Solution NSX default topology has the following configuration:
17
17
18
-
*Three node NSX Manager cluster.
18
+
*Three node NSX Manager cluster.
19
19
20
-
*NSX Edge and Gateway for North-bound traffic:
20
+
*NSX Edge and Gateway for North-bound traffic:
21
21
22
-
*Two Large Form Factor NSX Edges, deployed in an NSX Edge cluster.
22
+
*Two Large Form Factor NSX Edges, deployed in an NSX Edge cluster.
23
23
24
-
*A Default NSX Tier-0 Gateway in Active/Active mode.
24
+
*A Default NSX Tier-0 Gateway in Active/Active mode.
25
25
26
-
* A Default NSX Tier-1 Gateway in Active/Standby mode.
26
+
* A Default NSX Tier-1 Gateway in Active/Standby mode.
27
+
28
+
* A Default HCX-UPLINK segment connected to default Tier-1 Gateway.
27
29
28
-
* A Default HCX-UPLINK segment connected to default Tier-1 Gateway.
29
-
30
30
Customers typically host their application workloads by creating new NSX segments and attaching them to the default Tier-1 Gateway. Additionally, customers with an HCX migration use case use the default HCX-uplink segment, which is also connected to the default Tier-1 Gateway.
31
31
32
32
The default NSX topology for Azure VMware Solution, where all traffic exits through the default Tier-1 Gateway, may not be optimal based on customer traffic flows and throughput requirements.
33
33
34
-
### Potential Challenge:
34
+
### Potential Challenge
35
35
36
36
Here are some potential challenges and the recommended configurations to optimize the NSX Edge data path resource.
37
37
38
-
* All the north-bound network traffic (Migrations, L2 Extensions, VM traffic outbound of Azure VMware Solution) uses the default Tier-1 Gateway which is in Active/Standby mode.
38
+
* All the north-bound network traffic (Migrations, L2 Extensions, VM traffic outbound of Azure VMware Solution) uses the default Tier-1 Gateway which is in Active/Standby mode.
39
+
40
+
* In the default Active/Standby mode, the Tier-1 Gateway only uses the Active Edge VM for all north-bound traffic.
39
41
40
-
* In the default Active/Standby mode, the Tier-1 Gateway only uses the Active Edge VM for all north-bound traffic.
42
+
* The second Edge VM, which is standby, is not used for north-bound traffic.
41
43
42
-
* The second Edge VM, which is standby, is not used for north-bound traffic.
44
+
* Depending on the throughput requirements, and flows this could potentially create a bottleneck on the Active Edge VM.
43
45
44
-
* Depending on the throughput requirements, and flows this could potentially create a bottleneck on the Active Edge VM.
46
+
### Recommended Practices
45
47
46
-
### Recommended Practices:
47
48
It is possible to change the NSX North-bound network connectivity to distribute the traffic evenly to both Edge VMs. Creating an additional Tier-1 Gateways and distributing the NSX segments across multiple Tier-1 Gateways evenly distributes traffic across the Edge VMs. For an HCX migration use case, the recommendation would be to move HCX Layer 2 (L2) Extension and migration traffic to a newly created Tier-1 Gateway, so it uses the NSX Edge resource optimally.
48
49
49
50
To make an Active Edge for a given Tier-1 Gateway predictable, it is recommended to create an additional Tier-1 Gateway with the High Availability (HA) Mode set to Active/Standby with the Failover mode set to preemptive. This configuration allows you to select a different active Edge VM than the one in use by the default Tier-1 Gateway. This naturally splits north-bound traffic across multiple Tier-1 Gateways, so both NSX Edges are optimally utilized, thus avoiding a potential bottleneck with the default NSX topology.
50
51
51
-
:::image type="content" source="media/nsxt/default-nsx-topology.png" alt-text="Diagram showing the default nsx topology in Azure VMware Solution." border="false" lightbox="media/nsxt/default-nsx-topology.png":::
52
+
:::image type="content" source="media/nsxt/default-nsx-topology.png" alt-text="Diagram showing the default NSX topology in Azure VMware Solution." border="false" lightbox="media/nsxt/default-nsx-topology.png":::
52
53
53
54
### NSX Edge performance characteristics
54
55
55
56
Each of the NSX Edge Virtual machine (EVM) can support up to approximately ~20 Gbps based on the number of flows, packet size, and services enabled on the NSX gateways. Each of the Edge VMs (Large form factors) has four Data Plane Development Kit (DPDK) enabled CPU cores, essentially each of the DPDK core could process up to ~5 Gbps traffic, based on flow hashing, packet size, and services enabled on NSX gateway. For more information on NSX Edge performance, see the VMware NSX-T Reference Design Guide section 8.6.2.*
56
57
57
58
## Monitor, Identify, and Fix potential Edge data path Performance Bottlenecks
58
59
59
-
Using the built-in NSX alarm framework is recommended to monitor and identify key NSX edge performance metrics.
60
+
Using the built-in NSX alarm framework is recommended to monitor and identify key NSX Edge performance metrics.
60
61
61
-
### How to Monitor and Identify NSX Edge Data Path Resource Constraints:
62
+
### How to Monitor and Identify NSX Edge Data Path Resource Constraints
62
63
63
64
NSX Edge performance can be monitored and identified by using the built-in NSX alarm framework. The following critical NSX Edge alarms identify the NSX Edge data path resource constraints:
To mitigate the issue, here are a few options to consider.
84
81
85
82
Mitigation options:
86
-
1. Edge Scale-UP: NSX Edge Scale-UP from Large (four DPDK CPU) to X-Large (eight DPDK CPU) form factor could resolve part of the issue.
83
+
84
+
1. Edge Scale-UP: NSX Edge Scale-UP from Large (four DPDK CPU) to X-Large (eight DPDK CPU) form factor could resolve part of the issue.
87
85
88
86
* Edge Scale up provides additional CPU and memory for data path packet processing.
89
87
90
88
* Edge Scale up may not help if you have one or more heavy flows, for example, HCX Network Extension (NE) to Network Extension (NE) traffic, as this traffic could potentially pin to one of the DPDK CPU cores.
91
89
92
-
2.Tier-1 Gateway Topology Change: Change the Azure VMware Solution NSX default Tier-1 Gateway topology with multiple Tier-1 Gateways to split the traffic across multiple Edge VMs
90
+
2.Tier-1 Gateway Topology Change: Change the Azure VMware Solution NSX default Tier-1 Gateway topology with multiple Tier-1 Gateways to split the traffic across multiple Edge VMs
93
91
94
92
* More details in the next section with an example of HCX migration use case.
95
93
96
-
3.Edge Scale-OUT: If customer has large number of Hosts in the SDDC and workloads, NSX Edge Scale-OUT (from two Edges to four Edges) could be an option to add additional NSX Edge data path resources.
94
+
3.Edge Scale-OUT: If customer has large number of Hosts in the SDDC and workloads, NSX Edge Scale-OUT (from two Edges to four Edges) could be an option to add additional NSX Edge data path resources.
97
95
98
96
* However, NSX Edge Scale-OUT is effective only with a change in the NSX default Tier-1 Gateway topology to distribute the traffic optimally across all four Edge VMs. More details in the next section with an example of HCX migration use case.
99
97
100
-
### Default and configuration recommendations to the NSX Edge data path performance.
98
+
### Default and configuration recommendations to the NSX Edge data path performance
101
99
102
100
Here are a few configuration recommendations to mitigate an NSX Edge VMs performance challenges.
103
101
104
-
1.By default, Edge VMs are part of Azure VMware Solution management resource pool on vCenter. All appliances in the management resource pool have dedicated computing resources assigned.
102
+
1.By default, Edge VMs are part of Azure VMware Solution management resource pool on vCenter. All appliances in the management resource pool have dedicated computing resources assigned.
105
103
106
-
2.By default, Edge VMs are hosted on different Hosts with anti-affinity rules applied, to avoid multiple heavy packet processing workloads on same hosts.
104
+
2.By default, Edge VMs are hosted on different Hosts with anti-affinity rules applied, to avoid multiple heavy packet processing workloads on same hosts.
107
105
108
-
3.Disable the Tier-1 Gateway Firewall if it is not required to get better packet processing power. (By default, the Tier-1 Gateway Firewall is enabled).
106
+
3.Disable the Tier-1 Gateway Firewall if it is not required to get better packet processing power. (By default, the Tier-1 Gateway Firewall is enabled).
109
107
110
-
4.Verify that NSX Edge VMs and HCX Network Extension (NE) appliances are on separate hosts, to avoid multiple heavy packet processing workloads on same hosts.
108
+
4.Verify that NSX Edge VMs and HCX Network Extension (NE) appliances are on separate hosts, to avoid multiple heavy packet processing workloads on same hosts.
111
109
112
110
5. Verify for HCX migration use case, that the HCX Network Extension (NE) and HCX Interconnect (IX) appliances have the CPU reserved. Reserving the CPU allows HCX to optimally process the HCX migration traffic. (By default, these appliances have no CPU reservations).
113
111
@@ -123,7 +121,7 @@ Given the nature of HCX use case traffic pattern and default Azure VMware Soluti
123
121
124
122
In general, creating additional Tier-1 Gateways and distributing segments across multiple Tier-1 Gateways helps to mitigate potential NSX Edge data path bottleneck. The steps outlined show how to create and move an HCX uplink segment to the new Tier-1 Gateway. This allows you to separate out HCX traffic from workload VM traffic.
125
123
126
-
:::image type="content" source="media/nsxt/nsx-traffic-flow-additional-tier-1-gateway.png" alt-text="Diagram showing nsx traffic flow in Azure VMware Solution with an additional Tier-1 gateway." border="false" lightbox="media/nsxt/nsx-traffic-flow-additional-tier-1-gateway.png":::
124
+
:::image type="content" source="media/nsxt/nsx-traffic-flow-additional-tier-1-gateway.png" alt-text="Diagram showing NSX traffic flow in Azure VMware Solution with an additional Tier-1 gateway." border="false" lightbox="media/nsxt/nsx-traffic-flow-additional-tier-1-gateway.png":::
127
125
128
126
### Detailed Steps (Mitigate Edge VM bottleneck)
129
127
@@ -133,34 +131,34 @@ The creation of an additional Tier-1 Gateway can help mitigate potential Edge VM
133
131
134
132
Distributed Only Option:
135
133
136
-
1.No Edge Cluster can be selected.
134
+
1.No Edge Cluster can be selected.
137
135
138
-
2.All connected Segments and Service Ports must be advertised.
136
+
2.All connected Segments and Service Ports must be advertised.
139
137
140
-
3.No stateful services are available in the Distributed Only option.
138
+
3.No stateful services are available in the Distributed Only option.
>In a Distributed Only High Availability (HA) Mode, traffic is distributed across all Edge VMs. Workload traffic and Migration traffic may traverse the Active Edge at the same time.
146
144
147
145
Active/Standby Option:
148
146
149
-
1.Select the **Edge Cluster**.
147
+
1.Select the **Edge Cluster**.
150
148
151
-
2.For Auto Allocate Edges- Select **No** on the radio button.
149
+
2.For Auto Allocate Edges- Select **No** on the radio button.
152
150
153
-
3.Select the **Edge VM** that is not currently active as the preferred option.
151
+
3.Select the **Edge VM** that is not currently active as the preferred option.
154
152
155
-
4.For the **Fail Over** setting, select **Preemptive**, this ensures that traffic will always failback to the preferred Edge VM selected in Step 3.
153
+
4.For the **Fail Over** setting, select **Preemptive**, this ensures that traffic will always failback to the preferred Edge VM selected in Step 3.
156
154
157
-
5.Select **All Connected Segments and Service Ports** to be advertised.
155
+
5.Select **All Connected Segments and Service Ports** to be advertised.
158
156
159
157
6. Select **Save**.
160
158
161
159
An Active/Standby configuration with the preferred Edge VM defined allows you to force traffic the Edge VM that is not the Active Edge on the Default Tier-1 Gateway. If the Edge cluster is scaled-out to four Edges, creating the new Tier-1 Gateway and selecting Edge VM 03 and Edge VM 04 may be a better option to isolate HCX traffic completely.
>Microsoft Recommends the Active/Standby HA Mode when additional Tier-1 Gateways are created. This allows customers to seperate Workload and migration traffic across different Edge VMs.
@@ -174,42 +172,42 @@ Select the newly created Tier-1 Gateway when creating your new NSX Segment.
174
172
>[!NOTE]
175
173
>When creating a new NSX Segment, customers can utilize the Azure VMware Solution reserved IP space. For example, a new segment can be created with an IP range of 10.18.75.129/26, assuming the following IP space 10.18.72.0/22 was used to create the Azure VMware Solution Private Cloud.
176
174
177
-
:::image type="content" source="media/nsxt/nsx-segment-creation.png" alt-text="Diagram showing the creation of a nsx segment." border="false" lightbox="media/nsxt/nsx-segment-creation.png":::
175
+
:::image type="content" source="media/nsxt/nsx-segment-creation.png" alt-text="Diagram showing the creation of an NSX segment." border="false" lightbox="media/nsxt/nsx-segment-creation.png":::
178
176
179
177
## Create an HCX Network Profile
180
178
181
179
For detailed steps on how to Create an HCX Network Profile. [HCX Network Profile](configure-vmware-hcx.md#create-network-profiles)
182
180
183
-
1.Navigate to the HCX Portal select **Interconnect**, and then select **Network Profile**.
181
+
1.Navigate to the HCX Portal select **Interconnect**, and then select **Network Profile**.
184
182
185
-
2.Select **Create Network Profile**.
183
+
2.Select **Create Network Profile**.
186
184
187
-
3.Select **NSX Network**, and choose the newly created **HCX Uplink segment**.
185
+
3.Select **NSX Network**, and choose the newly created **HCX Uplink segment**.
188
186
189
-
4.Add the desired **IP Pool range**.
187
+
4.Add the desired **IP Pool range**.
190
188
191
-
5.(Optional) Select **HCX Uplink** as the HCX Traffic Type.
189
+
5.(Optional) Select **HCX Uplink** as the HCX Traffic Type.
192
190
193
-
6.Select **Create**.
191
+
6.Select **Create**.
194
192
195
-
:::image type="content" source="media/hcx/hcx-uplink-network-profile.png" alt-text="Diagram showing the creation of a hcx network profile." border="false" lightbox="media/nsxt/hcx-uplink-network-profile.png":::
193
+
:::image type="content" source="media/hcx/hcx-uplink-network-profile.png" alt-text="Diagram showing the creation of a HCX network profile." border="false" lightbox="media/nsxt/hcx-uplink-network-profile.png":::
196
194
197
195
Once the new HCX Uplink Network Profile is created, update the existing Service Mesh and edit the default uplink profile with the newly created Network Profile.
198
196
199
197
:::image type="content" source="media/hcx/hcx-service-mesh-edit.png" alt-text="Diagram showing how to edit an existing HCX service mesh." border="false" lightbox="media/nsxt/hcx-service-mesh-edit.png":::
200
198
201
-
7.Select the existing **Service Mesh** and select **Edit**.
199
+
7.Select the existing **Service Mesh** and select **Edit**.
202
200
203
-
8.Edit the default Uplink with the newly created Network Profile.
201
+
8.Edit the default Uplink with the newly created Network Profile.
204
202
205
-
9.Select **Service Mesh Change**.
203
+
9.Select **Service Mesh Change**.
206
204
207
-
:::image type="content" source="media/hcx/hcx-in-service-mode.png" alt-text="Diagram showing how to edit an in service mode on a hcx Network extension appliance." border="false" lightbox="media/nsxt/hcx-in-service-mode.png":::
205
+
:::image type="content" source="media/hcx/hcx-in-service-mode.png" alt-text="Diagram showing how to edit an in service mode on a HCX Network extension appliance." border="false" lightbox="media/nsxt/hcx-in-service-mode.png":::
208
206
209
207
>[!Note]
210
208
>In-Service Mode of the HCX Network Extension appliances should be considered to reduce downtime during this Service Mesh edit.
211
209
212
-
10.Select **Finish**.
210
+
10.Select **Finish**.
213
211
214
212
[!IMPORTANT]Downtime varies depending on the Service Mesh change created. It is recommended to allocate 5 minutes of downtime for these changes to take effect.
0 commit comments