You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/azure-vmware/azure-vmware-solution-nsx-scale-and-performance-recommendations-for-vmware-hcx.md
+27-27Lines changed: 27 additions & 27 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -32,24 +32,24 @@ In this article, you will learn about the default NSX topology in Azure VMware S
32
32
33
33
34
34
35
-
Customers typically host their application workloads by creating new NSX segments and attaching them to the default Tier-1 Gateway. Additionally, customers with an HCX migration use case will use the default HCX-uplink segment, which is also connected to the default Tier-1 Gateway.
35
+
Customers typically host their application workloads by creating new NSX segments and attaching them to the default Tier-1 Gateway. Additionally, customers with an HCX migration use case uses the default HCX-uplink segment, which is also connected to the default Tier-1 Gateway.
36
36
37
37
The default NSX topology for Azure VMware Solution, where all traffic exits through the default Tier-1 Gateway, may not be optimal based on customer traffic flows and throughput requirements. Here are some potential challenges and the recommended configurations to optimize the NSX Edge data path resource.
38
38
39
39
### Potential Challenge:
40
40
41
41
• All the north-bound network traffic (Migrations, L2 Extensions, VM traffic outbound of Azure VMware Solution) uses the default Tier-1 Gateway which is in Active/Standby mode.
42
42
43
-
• In the default Active/Standby mode, the Tier-1 Gateway will only use the Active Edge VM for all north-bound traffic.
43
+
• In the default Active/Standby mode, the Tier-1 Gateway only uses the Active Edge VM for all north-bound traffic.
44
44
45
45
• The second Edge VM, which is standby, is not used for north-bound traffic.
46
46
47
47
• Depending on the throughput requirement and flows this could potentially create a bottleneck on the Active Edge VM.
48
48
49
49
### Recommended Practices:
50
-
It is possible to change the NSX North-bound network connectivity to distribute the traffic evenly to both Edge VM’s. This can be done by creating additional Tier-1 Gateways and distributing the NSX segments across multiple Tier-1 Gateways. For an HCX migration use case, the recommendation would be to move HCX Layer 2 (L2) Extension and migration traffic to a newly created Tier-1 Gateway, so it uses the NSX Edge resource optimally.
50
+
It is possible to change the NSX North-bound network connectivity to distribute the traffic evenly to both Edge VMs. This can be done by creating additional Tier-1 Gateways and distributing the NSX segments across multiple Tier-1 Gateways. For an HCX migration use case, the recommendation would be to move HCX Layer 2 (L2) Extension and migration traffic to a newly created Tier-1 Gateway, so it uses the NSX Edge resource optimally.
51
51
52
-
To make an Active Edge for a given Tier-1 Gateway predictable it is recommended to create an additional Tier-1 Gateway with the High Availability (HA) Mode set to Active/Standby with the Failover mode set to preemptive. This configuration would allow you to select a different active Edge VM then the one in use by the default Tier-1 Gateway. This naturally splits north-bound traffic across multiple Tier-1 Gateways, so both NSX Edges are optimally utilized, thus avoiding a potential bottleneck with the default NSX topology.
52
+
To make an Active Edge for a given Tier-1 Gateway predictable, it is recommended to create an additional Tier-1 Gateway with the High Availability (HA) Mode set to Active/Standby with the Failover mode set to preemptive. This configuration allows you to select a different active Edge VM then the one in use by the default Tier-1 Gateway. This naturally splits north-bound traffic across multiple Tier-1 Gateways, so both NSX Edges are optimally utilized, thus avoiding a potential bottleneck with the default NSX topology.
53
53
54
54
:::image type="content" source="media/nsxt/default-nsx-topology.png" alt-text="Diagram showing the default nsx topology in Azure VMware Solution." border="false" lightbox="media/nsxt/default-nsx-topology.png":::
55
55
@@ -58,9 +58,9 @@ Figure 1: Depicts the default NSX topology in Azure VMware Solution
58
58
59
59
### NSX Edge performance characteristics
60
60
61
-
Each of the NSX Edge Virtual machine (EVM) can support up to approximately ~20 Gbps based on the number of flows, packet size and services enabled on the NSX gateways. Each of the Edge VMs (Large form factors) has 4 Data Plane Development Kit (DPDK) enabled CPU cores, essentially each of the DPDK core could process up to ~5 Gbps traffic, based on flow hashing, packet size, and services enabled on NSX gateway. For more information on NSX Edge performance, refer to the VMware NSX-T Reference Design Guide section 8.6.2.*
61
+
Each of the NSX Edge Virtual machine (EVM) can support up to approximately ~20 Gbps based on the number of flows, packet size, and services enabled on the NSX gateways. Each of the Edge VMs (Large form factors) has four Data Plane Development Kit (DPDK) enabled CPU cores, essentially each of the DPDK core could process up to ~5 Gbps traffic, based on flow hashing, packet size, and services enabled on NSX gateway. For more information on NSX Edge performance, see the VMware NSX-T Reference Design Guide section 8.6.2.*
62
62
63
-
## Monitor, Identify and Fix potential Edge data path Performance Bottlenecks
63
+
## Monitor, Identify, and Fix potential Edge data path Performance Bottlenecks
64
64
65
65
### How to Monitor and Identify NSX Edge Data Path Resource Constraints:
To mitigate the issue, here are a few options to consider.
92
92
93
93
Mitigation options:
94
-
1. Edge Scale-UP: NSX Edge Scale-UP from Large (4 DPDK CPU) to X-Large (8 DPDK CPU) form factor could resolve part of the issue.
94
+
1. Edge Scale-UP: NSX Edge Scale-UP from Large (four DPDK CPU) to X-Large (eight DPDK CPU) form factor could resolve part of the issue.
95
95
96
96
• Edge Scale up provides additional CPU and memory for data path packet processing.
97
97
98
98
• Edge Scale up may not help if you have one or more heavy flows, for example, HCX Network Extension (NE) to Network Extension (NE) traffic, as this traffic could potentially pin to one of the DPDK CPU cores.
99
99
100
-
2. Tier-1 Gateway Topology Change: Change the Azure VMware Solution NSX default Tier-1 Gateway topology with multiple Tier-1 Gateways to split the traffic across multiple Edge VM’s
100
+
2. Tier-1 Gateway Topology Change: Change the Azure VMware Solution NSX default Tier-1 Gateway topology with multiple Tier-1 Gateways to split the traffic across multiple Edge VMs
101
101
102
102
• More details in the next section with an example of HCX migration use case.
103
103
104
-
3. Edge Scale-OUT: If customer has large number of Hosts in the SDDC and workloads, NSX Edge Scale-OUT (from 2 Edges to 4 Edges) could be an option to add additional NSX Edge data path resources.
104
+
3. Edge Scale-OUT: If customer has large number of Hosts in the SDDC and workloads, NSX Edge Scale-OUT (from two Edges to four Edges) could be an option to add additional NSX Edge data path resources.
105
105
106
-
• However, NSX Edge Scale-OUT is effective only with a change in the NSX default Tier-1 Gateway topology to distribute the traffic optimally across all 4 Edge VM’s. More details in the next section with an example of HCX migration use case.
106
+
• However, NSX Edge Scale-OUT is effective only with a change in the NSX default Tier-1 Gateway topology to distribute the traffic optimally across all four Edge VMs. More details in the next section with an example of HCX migration use case.
107
107
108
108
### Default and configuration recommendations to the NSX Edge data path performance.
109
109
110
-
Here are a few configuration recommendations to mitigate an NSX Edge VM’s performance challenges.
110
+
Here are a few configuration recommendations to mitigate an NSX Edge VMs performance challenges.
111
111
112
-
1. By default, Edge VM’s are part of Azure VMware Solution management resource pool on vCenter, all appliances in the management resource pool have dedicated computing resources assigned.
112
+
1. By default, Edge VMs are part of Azure VMware Solution management resource pool on vCenter, all appliances in the management resource pool have dedicated computing resources assigned.
113
113
114
-
2. By default, Edge VM’s are hosted on different Hosts with anti-affinity rules applied, to avoid multiple heavy packet processing workloads on same hosts.
114
+
2. By default, Edge VMs are hosted on different Hosts with anti-affinity rules applied, to avoid multiple heavy packet processing workloads on same hosts.
115
115
116
-
3. Disable Tier-1 Gateway Firewall if it is not required to get better packet processing power. (By default, the Tier-1 Gateway Firewall is enabled).
116
+
3. Disable the Tier-1 Gateway Firewall if it is not required to get better packet processing power. (By default, the Tier-1 Gateway Firewall is enabled).
117
117
118
-
4. Make sure NSX Edge VM’s and HCX Network Extension (NE) appliances are on separate hosts, to avoid multiple heavy packet processing workloads on same hosts.
118
+
4. Make sure NSX Edge VMs and HCX Network Extension (NE) appliances are on separate hosts, to avoid multiple heavy packet processing workloads on same hosts.
119
119
120
-
5. For HCX migration use case, be sure that the HCX Network Extension (NE) and HCX Interconnect (IX) appliances have the CPU reserved. Reserving the CPU will allow HCX to optimally process the HCX migration traffic. (By default, these appliances have no CPU reservations).
120
+
5. Make sure for HCX migration use case, be sure that the HCX Network Extension (NE) and HCX Interconnect (IX) appliances have the CPU reserved. Reserving the CPU will allow HCX to optimally process the HCX migration traffic. (By default, these appliances have no CPU reservations).
121
121
122
122
## How to optimize Azure VMware Solution NSX Data Path Performance - HCX Use Case
123
123
124
-
One of the most frequent scenarios that may potentially reach the NSX Edges data path limit is the HCX migration and network extension use case. The reason being HCX migration and network extension creates heavy flows (single flow between Network Extenders) this will be hashed to single edge and single DPDK core within the Edge VM. This could potentially limit HCX migration and Network extension traffic up to 5Gbps, based on the flow hashing.
124
+
One of the most frequent scenarios that may potentially reach the NSX Edges data path limit is the HCX migration and network extension use case. The reason being HCX migration and network extension creates heavy flows (single flow between Network Extenders) this will be hashed to single edge and single DPDK core within the Edge VM. This could potentially limit HCX migration and Network extension traffic up to 5 Gbps, based on the flow hashing.
125
125
126
126
HCX Network extenders have a throughput limit of 4-6 Gbps per appliance. A recommended practice is to deploy multiple HCX NE appliances to distribute the load across them, ensuring reliable performance. This also allows multiple network flows, improving network hashing across different NSX Edges and cores within an NSX Edge VM.
127
127
128
128
Given the nature of HCX use case traffic pattern and default Azure VMware Solution topology, here are few recommended practices to mitigate NSX Edge VM bottlenecks.
In general, creating additional Tier-1 Gateways and distributing segments across multiple Tier-1 Gateways helps to mitigate potential NSX Edge data path bottleneck. The steps outlined below show how to create and move an HCX uplink segment to the new Tier-1 Gateway. This allows you to separate out HCX traffic from workload VM traffic.
132
+
In general, creating additional Tier-1 Gateways and distributing segments across multiple Tier-1 Gateways helps to mitigate potential NSX Edge data path bottleneck. The steps outlined below show how to create and move an HCX uplink segment to the new Tier-1 Gateway. This allows you to separate out HCX traffic from workload VM traffic.
133
133
134
134
:::image type="content" source="media/nsxt/nsx-traffic-flow-additional-tier-1-gateway.png" alt-text="Diagram showing nsx traffic flow in Azure VMware Solution with an additional Tier-1 gateway." border="false" lightbox="media/nsxt/nsx-traffic-flow-additional-tier-1-gateway.png":::
135
135
@@ -152,7 +152,7 @@ Distributed Only Option:
152
152
Figure 5: Tier-1 Gateway Distributed Only Option.
153
153
154
154
>[!IMPORTANT]
155
-
>In a Distributed Only High Availability (HA) Mode, traffic is distributed across all Edge VM's. Workload traffic and Migration traffic may traverse the Active Edge at the same time.
155
+
>In a Distributed Only High Availability (HA) Mode, traffic is distributed across all Edge VMs. Workload traffic and Migration traffic may traverse the Active Edge at the same time.
156
156
157
157
158
158
Active/Standby Option:
@@ -163,20 +163,20 @@ Active/Standby Option:
163
163
164
164
3. Select the **Edge VM** that is not currently active as the preferred option.
165
165
166
-
4. For the **Fail Over** setting, select **Preemptive**, this will ensure that traffic will always failback to the preferred Edge VM selected in Step 3.
166
+
4. For the **Fail Over** setting, select **Preemptive**, this ensures that traffic will always failback to the preferred Edge VM selected in Step 3.
167
167
168
168
5. Select **All Connected Segments and Service Ports** to be advertised.
169
169
170
170
6. Select **Save**.
171
171
172
-
An Active/Standby configuration with the preferred Edge VM defined allows you to force traffic the Edge VM that is not the Active Edge on the Default Tier-1 Gateway. If the Edge cluster is scaled-out to 4 Edges, creating the new Tier-1 Gateway and selecting Edge VM 03 and Edge VM 04 may be a better option to isolate HCX traffic completely.
172
+
An Active/Standby configuration with the preferred Edge VM defined allows you to force traffic the Edge VM that is not the Active Edge on the Default Tier-1 Gateway. If the Edge cluster is scaled-out to four Edges, creating the new Tier-1 Gateway and selecting Edge VM 03 and Edge VM 04 may be a better option to isolate HCX traffic completely.
>Microsoft Recommends the Active/Standby HA Mode when additional Tier-1 Gateways are created. This allows customers to seperate Workload and migration traffic across different Edge VM's.
179
+
>Microsoft Recommends the Active/Standby HA Mode when additional Tier-1 Gateways are created. This allows customers to seperate Workload and migration traffic across different Edge VMs.
180
180
181
181
182
182
## Create a new Segment for HCX Uplink and attach to the new Tier-1 Gateway
@@ -188,15 +188,15 @@ Select the newly created Tier-1 Gateway when creating your new NSX Segment.
188
188
>[!NOTE]
189
189
>When creating a new NSX Segment, customers can utilize the Azure VMware Solution reserved IP space. For example, a new segment can be created with an IP range of 10.18.75.129/26, assuming the following IP space 10.18.72.0/22 was used to create the Azure VMware Solution Private Cloud.
190
190
191
-
:::image type="content" source="media/nsxt/nsx-segment-creation.png" alt-text="Diagram showing the creation of an nsx segment." border="false" lightbox="media/nsxt/nsx-segment-creation.png":::
191
+
:::image type="content" source="media/nsxt/nsx-segment-creation.png" alt-text="Diagram showing the creation of a nsx segment." border="false" lightbox="media/nsxt/nsx-segment-creation.png":::
192
192
193
193
Figure 7: NSX Segment creation for new HCX Uplink network.
194
194
195
195
## Create an HCX Network Profile
196
196
197
197
For detailed steps on how to Create an HCX Network Profile. [HCX Network Profile](configure-vmware-hcx.md#create-network-profiles)
198
198
199
-
1. Navigate to the HCX Portal, select **Interconnect** and then select **Network Profile**.
199
+
1. Navigate to the HCX Portal select **Interconnect**, and then select **Network Profile**.
200
200
201
201
2. Select **Create Network Profile**.
202
202
@@ -208,7 +208,7 @@ For detailed steps on how to Create an HCX Network Profile. [HCX Network Profile
208
208
209
209
6. Select **Create**.
210
210
211
-
:::image type="content" source="media/hcx/hcx-uplink-network-profile.png" alt-text="Diagram showing the creation of an hcx network profile." border="false" lightbox="media/nsxt/hcx-uplink-network-profile.png":::
211
+
:::image type="content" source="media/hcx/hcx-uplink-network-profile.png" alt-text="Diagram showing the creation of a hcx network profile." border="false" lightbox="media/nsxt/hcx-uplink-network-profile.png":::
212
212
213
213
Figure 8: HCX Network profile creation.
214
214
@@ -224,7 +224,7 @@ Figure 9: HCX Service Mesh edit.
224
224
225
225
9. Select **Service Mesh Change**.
226
226
227
-
:::image type="content" source="media/hcx/hcx-in-service-mode.png" alt-text="Diagram showing how to edit an in service mode on an HCX Network extension appliance." border="false" lightbox="media/nsxt/hcx-in-service-mode.png":::
227
+
:::image type="content" source="media/hcx/hcx-in-service-mode.png" alt-text="Diagram showing how to edit an in service mode on a hcx Network extension appliance." border="false" lightbox="media/nsxt/hcx-in-service-mode.png":::
[!IMPORTANT]Downtime will vary depending on the Service Mesh change created. It is recommended to allocate 5 minutes of downtime for these changes to take effect.
236
+
[!IMPORTANT]Downtime varies depending on the Service Mesh change created. It is recommended to allocate 5 minutes of downtime for these changes to take effect.
0 commit comments