Failed waiting for data. Error 195887250. Migration determined a failure by the VMX

[Moudy_87]

Hi Guys, another vMotion storage migration and another 2 failures I am doing a storage vMotion from VMFS6 datastore to NFS datastore. Hot migration fails but if I shutdown the VMs and do the storage vMotion it works fine. I have also tried to migrate the VM using the compute and the storage to my other host but the same thing happens again. I had similar issue where vMotion was failing but was a different error and it was due to a switch I added and forgot to set the jumbo frames, so the vmkping and MTU has been covered already in this post.

This is the new error I get on the first machine;

Relocate virtual machine

Status:

Failed waiting for data. Error 195887250. Migration determined a failure by the VMX.

Error stack:Failed to copy one or more disks.A fatal internal error occurred. See the virtual machine's log for more details.Failed waiting for data. Error 195887250. Migration determined a failure by the VMX.

This is the error for the second machine;

Status:

Failed waiting for data. Error 195887107. Not found.

Error stack:Failed to copy one or more disks.A fatal internal error occurred. See the virtual machine's log for more details.Failed waiting for data. Error 195887107. Not found.

I don't have any conenction issues and the files are located on the datastore no problems the VMs are operating fine and

If I power off these machine and do the same migration it works fine no problems!! Any help will be really appreciated.

[Moudy_87]

I have reconfigured the vMotion on a new vmKernal instead of having it using the management vmKernal, again vmKernal works fine but I get the some sort of a similar issue and can't migrate still.

Description:

Cannot migrate DC from esxi02.home.local, Datastore02_VMFS to esxi01.home.local, Datastore01_NFS in HomeLab Datacenter

Event Type Description:Failed to migrate the virtual machine for reasons described in the event message

Possible Causes:

• The virtual machine did not migrate. This condition can occur if vMotion IPs are not configured, the source and destination hosts are not accessible, and so on. Action: Check the reason in the event message to find the cause of the failure. Ensure that the vMotion IPs are configured on source and destination hosts, the hosts are accessible, and so on.

This my vmKernel config, pretty simple no vlans or complicated networking.

host 1

Port properties
Network label	VMkernel-vMotion
VLAN ID	None (0)
TCP/IP stack	vMotion
Enabled services	vMotion
IPv4 settings
DHCP	Disabled
IPv4 address	192.168.1.200 (static)
Subnet mask	255.255.255.0
Default gateway	192.168.1.254

host 2

Port properties
Network label	VMkernel-vMotion
VLAN ID	None (0)
TCP/IP stack	vMotion
Enabled services	vMotion
IPv4 settings
DHCP	Disabled
IPv4 address	192.168.1.201 (static)
Subnet mask	255.255.255.0
Default gateway	192.168.1.254

and they both have the same exact settings in the properties tab.

Port properties
Network label	VMkernel-vMotion
VLAN ID	None (0)
TCP/IP stack	vMotion
Enabled services	vMotion

[daphnissov]

A migration of a powered-off VM is not considered a vMotion. You most likely have a networking issue between the two vmkernel ports on both hosts tagged for vMotion. Verify by conducting a vmkping between those vmkernel ports. If you cannot ping between them, vMotion will not work. https://kb.vmware.com/s/article/1003728

[Moudy_87]

Seems you didn't notice that I have mentioned already the vmkping works fine and I can't see any issues with the network, however the NFS datastore is sitting on a windows server machine in the same subnet and it's mounted fine on the ESXi hosts fine not sure what would cause the problem.

[daphnissov]

Just for the record, would you please conduct a vmkping between those vMotion vmkernel ports and post the output along with the command you used?

[Moudy_87]

Here is the results

[root@ESXi01:~] vmkping -s 8972 -d 192.168.1.201

PING 192.168.1.201 (192.168.1.201): 8972 data bytes

8980 bytes from 192.168.1.201: icmp_seq=0 ttl=64 time=0.693 ms

8980 bytes from 192.168.1.201: icmp_seq=1 ttl=64 time=0.698 ms

8980 bytes from 192.168.1.201: icmp_seq=2 ttl=64 time=0.680 ms

--- 192.168.1.201 ping statistics ---

3 packets transmitted, 3 packets received, 0% packet loss

round-trip min/avg/max = 0.680/0.690/0.698 ms

[root@ESXi01:~] vmkping -I vmk2 -S vmotion 192.168.1.201

PING 192.168.1.201 (192.168.1.201): 56 data bytes

64 bytes from 192.168.1.201: icmp_seq=0 ttl=64 time=0.166 ms

64 bytes from 192.168.1.201: icmp_seq=1 ttl=64 time=0.186 ms

64 bytes from 192.168.1.201: icmp_seq=2 ttl=64 time=0.178 ms

--- 192.168.1.201 ping statistics ---

3 packets transmitted, 3 packets received, 0% packet loss

round-trip min/avg/max = 0.166/0.177/0.186 ms

---------------------------------------------

[root@ESXi02:~] vmkping -s 8972 -d 192.168.1.200

PING 192.168.1.200 (192.168.1.200): 8972 data bytes

8980 bytes from 192.168.1.200: icmp_seq=0 ttl=64 time=0.644 ms

8980 bytes from 192.168.1.200: icmp_seq=1 ttl=64 time=0.674 ms

8980 bytes from 192.168.1.200: icmp_seq=2 ttl=64 time=0.682 ms

--- 192.168.1.200 ping statistics ---

3 packets transmitted, 3 packets received, 0% packet loss

round-trip min/avg/max = 0.644/0.667/0.682 ms

[root@ESXi02:~] vmkping -I vmk2 -S vmotion 192.168.1.200

PING 192.168.1.200 (192.168.1.200): 56 data bytes

64 bytes from 192.168.1.200: icmp_seq=0 ttl=64 time=0.182 ms

64 bytes from 192.168.1.200: icmp_seq=1 ttl=64 time=0.185 ms

64 bytes from 192.168.1.200: icmp_seq=2 ttl=64 time=0.185 ms

--- 192.168.1.200 ping statistics ---

3 packets transmitted, 3 packets received, 0% packet loss

round-trip min/avg/max = 0.182/0.184/0.185 ms

--------------------------------------------------

I have tried to vMotion from the NFS datastore to the 2 hosts on VMFS6 datastore works perfectly fine no problems, tried to vMotion the same machine back to the NFS datastore failed :S

[daphnissov]

More than likely, this is due to Microsoft's garbage implementation of the NFS protocol, which no one should ever use. Throw that junk away and use another implementation. Even something like FreeNAS works better.

[Moudy_87]

Thanks for your input, will look for alternatives.

[aaronatbaidu]

I'm using Linux kernel 5.4.0.52 an have this problem with a similar setup. vmotion, nfs server, fails immediately with storage migration.

[udaykumar-blr]

similar error noticed with RHEL 7.x  in vSphere 6.5 env with unformatted RAW disks (vmdk, not RDM) attached to the vm.

All other vms were able to migrate except the RHEL vm.

Failed waiting for data. Error 195887250. Migration determined a failure by the VMX.

[vmkfix-SSA]

I followed the below article to resolve the issue. May be it will helpful you.

https://vmkfix.blogspot.com/2023/05/storage-migration-failed.html