You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/azure-vmware/azure-vmware-solution-nsx-scale-and-performance-recommendations-for-vmware-hcx.md
+25-47Lines changed: 25 additions & 47 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -7,61 +7,57 @@ ms.date: 12/18/2024
7
7
ms.custom: engagement-fy25
8
8
---
9
9
10
-
11
10
# NSX Scale and performance recommendations for VMware HCX
12
11
13
-
14
-
15
12
In this article, learn about the default NSX topology in Azure VMware Solution, NSX data path performance characteristics, how to identify NSX data path resource constraints and recommended configurations to help mitigate resource constraints and optimize over all data path performance for HCX migrations.
16
13
17
14
## Azure VMware Solution NSX Default topology
18
15
19
16
The Azure VMware Solution NSX default topology has the following configuration:
20
17
21
-
• Three node NSX Manager cluster.
22
-
23
-
• NSX Edge and Gateway for North-bound traffic:
18
+
* Three node NSX Manager cluster.
24
19
25
-
• Two Large Form Factor NSX Edges, deployed in an NSX Edge cluster.
20
+
*NSX Edge and Gateway for North-bound traffic:
26
21
27
-
• A Default NSX Tier-0 Gateway in Active/Active mode.
28
-
29
-
• A Default NSX Tier-1 Gateway in Active/Standby mode.
30
-
31
-
• A Default HCX-UPLINK segment connected to default Tier-1 Gateway.
22
+
* Two Large Form Factor NSX Edges, deployed in an NSX Edge cluster.
32
23
24
+
* A Default NSX Tier-0 Gateway in Active/Active mode.
33
25
26
+
* A Default NSX Tier-1 Gateway in Active/Standby mode.
34
27
28
+
* A Default HCX-UPLINK segment connected to default Tier-1 Gateway.
29
+
35
30
Customers typically host their application workloads by creating new NSX segments and attaching them to the default Tier-1 Gateway. Additionally, customers with an HCX migration use case use the default HCX-uplink segment, which is also connected to the default Tier-1 Gateway.
36
31
37
-
The default NSX topology for Azure VMware Solution, where all traffic exits through the default Tier-1 Gateway, may not be optimal based on customer traffic flows and throughput requirements. Here are some potential challenges and the recommended configurations to optimize the NSX Edge data path resource.
32
+
The default NSX topology for Azure VMware Solution, where all traffic exits through the default Tier-1 Gateway, may not be optimal based on customer traffic flows and throughput requirements.
38
33
39
34
### Potential Challenge:
40
35
41
-
• All the north-bound network traffic (Migrations, L2 Extensions, VM traffic outbound of Azure VMware Solution) uses the default Tier-1 Gateway which is in Active/Standby mode.
36
+
Here are some potential challenges and the recommended configurations to optimize the NSX Edge data path resource.
42
37
43
-
• In the default Active/Standby mode, the Tier-1 Gateway only uses the Active Edge VM for all north-bound traffic.
38
+
* All the north-bound network traffic (Migrations, L2 Extensions, VM traffic outbound of Azure VMware Solution) uses the default Tier-1 Gateway which is in Active/Standby mode.
44
39
45
-
• The second Edge VM, which is standby, is not used for north-bound traffic.
40
+
* In the default Active/Standby mode, the Tier-1 Gateway only uses the Active Edge VM for all north-bound traffic.
46
41
47
-
• Depending on the throughput requirements, and flows this could potentially create a bottleneck on the Active Edge VM.
42
+
* The second Edge VM, which is standby, is not used for north-bound traffic.
43
+
44
+
* Depending on the throughput requirements, and flows this could potentially create a bottleneck on the Active Edge VM.
48
45
49
46
### Recommended Practices:
50
47
It is possible to change the NSX North-bound network connectivity to distribute the traffic evenly to both Edge VMs. Creating an additional Tier-1 Gateways and distributing the NSX segments across multiple Tier-1 Gateways evenly distributes traffic across the Edge VMs. For an HCX migration use case, the recommendation would be to move HCX Layer 2 (L2) Extension and migration traffic to a newly created Tier-1 Gateway, so it uses the NSX Edge resource optimally.
51
48
52
49
To make an Active Edge for a given Tier-1 Gateway predictable, it is recommended to create an additional Tier-1 Gateway with the High Availability (HA) Mode set to Active/Standby with the Failover mode set to preemptive. This configuration allows you to select a different active Edge VM than the one in use by the default Tier-1 Gateway. This naturally splits north-bound traffic across multiple Tier-1 Gateways, so both NSX Edges are optimally utilized, thus avoiding a potential bottleneck with the default NSX topology.
53
-
54
-
:::image type="content" source="media/nsxt/default-nsx-topology.png" alt-text="Diagram showing the default nsx topology in Azure VMware Solution." border="false" lightbox="media/nsxt/default-nsx-topology.png":::
55
-
56
50
57
-
Figure 1: Depicts the default NSX topology in Azure VMware Solution
51
+
:::image type="content" source="media/nsxt/default-nsx-topology.png" alt-text="Diagram showing the default nsx topology in Azure VMware Solution." border="false" lightbox="media/nsxt/default-nsx-topology.png":::
58
52
59
53
### NSX Edge performance characteristics
60
54
61
55
Each of the NSX Edge Virtual machine (EVM) can support up to approximately ~20 Gbps based on the number of flows, packet size, and services enabled on the NSX gateways. Each of the Edge VMs (Large form factors) has four Data Plane Development Kit (DPDK) enabled CPU cores, essentially each of the DPDK core could process up to ~5 Gbps traffic, based on flow hashing, packet size, and services enabled on NSX gateway. For more information on NSX Edge performance, see the VMware NSX-T Reference Design Guide section 8.6.2.*
62
56
63
57
## Monitor, Identify, and Fix potential Edge data path Performance Bottlenecks
64
58
59
+
Using the built-in NSX alarm framework is recommended to monitor and identify key NSX edge performance metrics.
60
+
65
61
### How to Monitor and Identify NSX Edge Data Path Resource Constraints:
66
62
67
63
NSX Edge performance can be monitored and identified by using the built-in NSX alarm framework. The following critical NSX Edge alarms identify the NSX Edge data path resource constraints:
@@ -73,37 +69,33 @@ NSX Edge performance can be monitored and identified by using the built-in NSX a
To mitigate the issue, here are a few options to consider.
92
84
93
85
Mitigation options:
94
86
1. Edge Scale-UP: NSX Edge Scale-UP from Large (four DPDK CPU) to X-Large (eight DPDK CPU) form factor could resolve part of the issue.
95
87
96
-
• Edge Scale up provides additional CPU and memory for data path packet processing.
88
+
* Edge Scale up provides additional CPU and memory for data path packet processing.
97
89
98
-
• Edge Scale up may not help if you have one or more heavy flows, for example, HCX Network Extension (NE) to Network Extension (NE) traffic, as this traffic could potentially pin to one of the DPDK CPU cores.
90
+
* Edge Scale up may not help if you have one or more heavy flows, for example, HCX Network Extension (NE) to Network Extension (NE) traffic, as this traffic could potentially pin to one of the DPDK CPU cores.
99
91
100
92
2. Tier-1 Gateway Topology Change: Change the Azure VMware Solution NSX default Tier-1 Gateway topology with multiple Tier-1 Gateways to split the traffic across multiple Edge VMs
101
93
102
-
• More details in the next section with an example of HCX migration use case.
94
+
* More details in the next section with an example of HCX migration use case.
103
95
104
96
3. Edge Scale-OUT: If customer has large number of Hosts in the SDDC and workloads, NSX Edge Scale-OUT (from two Edges to four Edges) could be an option to add additional NSX Edge data path resources.
105
97
106
-
• However, NSX Edge Scale-OUT is effective only with a change in the NSX default Tier-1 Gateway topology to distribute the traffic optimally across all four Edge VMs. More details in the next section with an example of HCX migration use case.
98
+
* However, NSX Edge Scale-OUT is effective only with a change in the NSX default Tier-1 Gateway topology to distribute the traffic optimally across all four Edge VMs. More details in the next section with an example of HCX migration use case.
107
99
108
100
### Default and configuration recommendations to the NSX Edge data path performance.
109
101
@@ -133,14 +125,14 @@ In general, creating additional Tier-1 Gateways and distributing segments across
133
125
134
126
:::image type="content" source="media/nsxt/nsx-traffic-flow-additional-tier-1-gateway.png" alt-text="Diagram showing nsx traffic flow in Azure VMware Solution with an additional Tier-1 gateway." border="false" lightbox="media/nsxt/nsx-traffic-flow-additional-tier-1-gateway.png":::
135
127
136
-
Figure 4: NSX traffic Flow with additional Tier-1 Gateways created.
137
-
138
-
139
128
### Detailed Steps (Mitigate Edge VM bottleneck)
140
129
130
+
The creation of an additional Tier-1 Gateway can help mitigate potential Edge VM bottlenecks.
131
+
141
132
[Create an NSX Tier-1 Gateway](tutorial-nsx-tier-1-gateway.md).
142
133
143
134
Distributed Only Option:
135
+
144
136
1. No Edge Cluster can be selected.
145
137
146
138
2. All connected Segments and Service Ports must be advertised.
>In a Distributed Only High Availability (HA) Mode, traffic is distributed across all Edge VMs. Workload traffic and Migration traffic may traverse the Active Edge at the same time.
156
146
157
-
158
147
Active/Standby Option:
159
148
160
149
1. Select the **Edge Cluster**.
@@ -173,12 +162,9 @@ An Active/Standby configuration with the preferred Edge VM defined allows you to
>Microsoft Recommends the Active/Standby HA Mode when additional Tier-1 Gateways are created. This allows customers to seperate Workload and migration traffic across different Edge VMs.
180
167
181
-
182
168
## Create a new Segment for HCX Uplink and attach to the new Tier-1 Gateway
183
169
184
170
For detailed instructions on NSX Segment creation. [NSX Segment Creation](tutorial-nsx-t-network-segment.md)
@@ -190,8 +176,6 @@ Select the newly created Tier-1 Gateway when creating your new NSX Segment.
190
176
191
177
:::image type="content" source="media/nsxt/nsx-segment-creation.png" alt-text="Diagram showing the creation of a nsx segment." border="false" lightbox="media/nsxt/nsx-segment-creation.png":::
192
178
193
-
Figure 7: NSX Segment creation for new HCX Uplink network.
194
-
195
179
## Create an HCX Network Profile
196
180
197
181
For detailed steps on how to Create an HCX Network Profile. [HCX Network Profile](configure-vmware-hcx.md#create-network-profiles)
@@ -210,23 +194,17 @@ For detailed steps on how to Create an HCX Network Profile. [HCX Network Profile
210
194
211
195
:::image type="content" source="media/hcx/hcx-uplink-network-profile.png" alt-text="Diagram showing the creation of a hcx network profile." border="false" lightbox="media/nsxt/hcx-uplink-network-profile.png":::
212
196
213
-
Figure 8: HCX Network profile creation.
214
-
215
197
Once the new HCX Uplink Network Profile is created, update the existing Service Mesh and edit the default uplink profile with the newly created Network Profile.
216
198
217
199
:::image type="content" source="media/hcx/hcx-service-mesh-edit.png" alt-text="Diagram showing how to edit an existing HCX service mesh." border="false" lightbox="media/nsxt/hcx-service-mesh-edit.png":::
218
200
219
-
Figure 9: HCX Service Mesh edit.
220
-
221
201
7. Select the existing **Service Mesh** and select **Edit**.
222
202
223
203
8. Edit the default Uplink with the newly created Network Profile.
224
204
225
205
9. Select **Service Mesh Change**.
226
206
227
207
:::image type="content" source="media/hcx/hcx-in-service-mode.png" alt-text="Diagram showing how to edit an in service mode on a hcx Network extension appliance." border="false" lightbox="media/nsxt/hcx-in-service-mode.png":::
228
-
229
-
Figure 10: HCX In-Service Mode
230
208
231
209
>[!Note]
232
210
>In-Service Mode of the HCX Network Extension appliances should be considered to reduce downtime during this Service Mesh edit.
0 commit comments