Skip to content

Commit 1219ebf

Browse files
v-regandownerv-regandowner
authored
Merge pull request #220992 from v-ksreedevan/09Dec-StuckReplication
Troubleshoot Stuck replications
2 parents 3e044b8 + 8bf221b commit 1219ebf

File tree

2 files changed

+126
-0
lines changed

2 files changed

+126
-0
lines changed

‎articles/migrate/toc.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -299,6 +299,8 @@
299299
items:
300300
- name: Troubleshoot replication issues in agentless VMware migration
301301
href: troubleshoot-changed-block-tracking-replication.md
302+
- name: Troubleshoot stuck replication and slow migration issues in agentless VMware
303+
href: troubleshoot-replication-vmware.md
302304
- name: Troubleshoot web app migration issues
303305
href: troubleshoot-webapps-migration.md
304306
- name: Reference

‎articles/migrate/troubleshoot-replication-vmware.md

Lines changed: 124 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,124 @@
1+
---
2+
title: Troubleshoot slow replication or stuck migration in agentless VMware migration
3+
description: Learn how to troubleshoot issues with slow replication or stuck migration for VMware.
4+
author: piyushdhore-microsoft
5+
ms.author: piyushdhore
6+
ms.manager: vijain
7+
ms.topic: troubleshooting
8+
ms.date: 12/08/2022
9+
ms.custom: mvc, engagement-fy23
10+
---
11+
12+
# Troubleshoot slow replication or stuck migration issues in agentless VMware migration
13+
14+
This article helps you troubleshoot slow replication or stuck migration issues that you may encounter when you replicate on-premises VMware VMs using the Azure Migrate: Server Migration agentless method.
15+
16+
17+
## Replication is slow or stuck for VM
18+
19+
While performing replications, you might observe that replication for a particular VM isn't progressing at the expected pace. Generally, the underlying reason for this issue is an unavailability or scarcity of some resources required for replication. The resources might be consumed by other VMs that are replicating or some other process running on the appliance in the datacenter.
20+
21+
Following are some reasons that generally cause this issue and remediations.
22+
23+
### NFC buffer size low
24+
25+
The Azure Migrate appliance operates under the constraint of using 32 MB of NFC buffer to concurrently replicate 8 disks on the ESXi host. An NFC buffer size of less than 32 MB might cause slow replication.
26+
You may also get the following exception:
27+
28+
Exception: GatewayErrorHandling.GatewayServiceException: The operation failed with the error 'Memory allocation failed. Out of memory.'
29+
30+
#### Remediation
31+
32+
You can increase the NFC buffer size beyond 32 MB to increase concurrency. The setting needs to be done on both the ESXi host and on the appliance. If not, the replication may perform even worse.
33+
34+
> [!Caution]
35+
> Increasing the size to more than 32 MB might cause resource constraints in the environment. Before proceeding, consult with the System Administrator to understand the implications.
36+
37+
**Changes in ESXi host**
38+
39+
1. SSH to the ESXi host as root.
40+
2. Use the vi editor to open “/etc/vmware/hostd/config.xml”.
41+
3. Find the section that looks like the one below:
42+
43+
```
44+
<nfcsvc>
45+
<enabled>true</enabled>
46+
<maxMemory>134217728</maxMemory>
47+
<maxStreamMemory>10485760</maxStreamMemory>
48+
<path>libnfcsvc.so</path>
49+
</nfcsvc>
50+
```
51+
4. Edit the value of `maxMemory` to the value (in Bytes) that you’d like to configure for the NFC buffer. In this example, it's set to 128 MB (128 * 1024 * 1024).
52+
5. Save and exit.
53+
6. Restart the management agents from the shell using the following commands:
54+
- /etc/init.d/hostd restart
55+
- /etc/init.d/vpxa restart
56+
57+
**Changes in Appliance**
58+
59+
1. Sign in to the Azure Migrate appliance as an administrator using the Remote Desktop.
60+
2. Open the GatewayDataWorker.json file in the "%programdata%\Microsoft Azure\Config" folder.
61+
3. Create an empty json file if it doesn’t exist and paste the following text in the new file created.
62+
```
63+
{
64+
"HostBufferSizeInMB": "32",
65+
}
66+
```
67+
4. Change the value of `HostBufferSizeInMB` to the value that you set in the ESXi host.
68+
5. Save and exit.
69+
6. Restart the Azure Migrate gateway service that is running on the appliance. Open PowerShell and execute the following:
70+
- net stop asrgwy (wait for the service to stop)
71+
- net start asrgwy 
72+
73+
74+
### ESXi host available RAM low
75+
76+
When the ESXi host on which the replicating VM is present is too busy, the replication process will slow down due to unavailability of RAM.
77+
78+
#### Remediation
79+
80+
Use VMotion to move the VM with slow replication to an ESXi host, which isn't too busy.
81+
82+
### Network bandwidth
83+
84+
Replications might be slow because of low network bandwidth available to the Azure Migrate appliance. Low bandwidth might be due to other applications using up the bandwidth or presence of bandwidth throttling applications or a proxy setting restricting the bandwidth use of replication appliance.
85+
86+
#### Remediation
87+
88+
In case of low bandwidth, you can first reduce the number of applications using network bandwidth. Check with your network administrator if any throttling application or proxy setting is present.
89+
90+
### Disk I/O
91+
92+
Replications can be slow because the server that is being replicated has too much load on it and this is causing high I/O operations on disks attached to it. It's advised to reduce the load on the server to increase the replication speed. You may also encounter the following error:
93+
94+
The last replication cycle for the virtual machine ‘VM Name’ failed. Encountered timeout event.
95+
96+
If no action is taken, the replication will proceed and be completed with a delay.
97+
98+
### Disk write rates
99+
100+
Replications can be slower than expected if the data upload speed is higher than the write speed of the disk that you selected while enabling replication. To get better speeds at same upload speeds, you would need to restart the replication and select **Premium** while selecting the disk type for replication.
101+
102+
> [!Caution]
103+
> The disk type recommended during Assessment might not be **Premium** for a particular VM. In this case, switching to Premium disk to improve replication speeds isn't advisable since it might not be required post migration to have a Premium disk attached to this VM.
104+
105+
106+
## Migration operation on VM is stuck
107+
108+
While triggering migration for a particular VM, you might observe that the migration is stuck at some stage (queued or delta sync) longer than expected. Generally, the underlying reason for this issue is an unavailability or scarcity of some resources required for migration. The resources might be consumed by other VMs that are replicating or some other process running on appliance on in the datacenter. Following are some reasons that generally cause this issue and the remedies.
109+
110+
### NFC buffer size low
111+
112+
If an IR cycle for a server with large disks is ongoing while migration is triggered for second VM, the second VM’s migration job can get stuck. Even though migration jobs are given high priority, the NFC buffer might not be available for migration. In this case, it's recommended to stop or pause the initial replication of servers with large disks and complete the migration of the second VM.
113+
114+
### Ongoing delta sync cycle isn't complete
115+
116+
If migration is triggered during an ongoing delta replication cycle, it would be queued. The delta replication cycle on the VM will be completed first after which migration will start. The time required to trigger migration depends on the time taken to complete one delta sync cycle.
117+
118+
### Shutdown of on-premises VM taking longer than usual
119+
120+
Try to migrate without shutting down the VM or turn off the VM manually and then migrate it.
121+
122+
## Next steps
123+
[Learn more](tutorial-migrate-vmware.md) about migrating VMware VMs.
124+

0 commit comments

Comments
 (0)