- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Failed waiting for data. Error 195887250. Migration determined a failure by the VMX
Hi Guys, another vMotion storage migration and another 2 failures I am doing a storage vMotion from VMFS6 datastore to NFS datastore. Hot migration fails but if I shutdown the VMs and do the storage vMotion it works fine. I have also tried to migrate the VM using the compute and the storage to my other host but the same thing happens again. I had similar issue where vMotion was failing but was a different error and it was due to a switch I added and forgot to set the jumbo frames, so the vmkping and MTU has been covered already in this post.
This is the new error I get on the first machine;
Relocate virtual machine
Status:
Failed waiting for data. Error 195887250. Migration determined a failure by the VMX.
This is the error for the second machine;
Status:
Failed waiting for data. Error 195887107. Not found.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I have reconfigured the vMotion on a new vmKernal instead of having it using the management vmKernal, again vmKernal works fine but I get the some sort of a similar issue and can't migrate still.
Description:
Cannot migrate DC from esxi02.home.local, Datastore02_VMFS to esxi01.home.local, Datastore01_NFS in HomeLab Datacenter
Event Type Description:Failed to migrate the virtual machine for reasons described in the event message
Possible Causes:
- The virtual machine did not migrate. This condition can occur if vMotion IPs are not configured, the source and destination hosts are not accessible, and so on. Action: Check the reason in the event message to find the cause of the failure. Ensure that the vMotion IPs are configured on source and destination hosts, the hosts are accessible, and so on.
This my vmKernel config, pretty simple no vlans or complicated networking.
host 1
| Port properties | |
| Network label | VMkernel-vMotion |
| VLAN ID | None (0) |
| TCP/IP stack | vMotion |
| Enabled services | vMotion |
| IPv4 settings | |
| DHCP | Disabled |
| IPv4 address | 192.168.1.200 (static) |
| Subnet mask | 255.255.255.0 |
| Default gateway | 192.168.1.254 |
host 2
| Port properties | |
| Network label | VMkernel-vMotion |
| VLAN ID | None (0) |
| TCP/IP stack | vMotion |
| Enabled services | vMotion |
| IPv4 settings | |
| DHCP | Disabled |
| IPv4 address | 192.168.1.201 (static) |
| Subnet mask | 255.255.255.0 |
| Default gateway | 192.168.1.254 |
and they both have the same exact settings in the properties tab.
| Port properties | |
| Network label | VMkernel-vMotion |
| VLAN ID | None (0) |
| TCP/IP stack | vMotion |
| Enabled services | vMotion |
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
A migration of a powered-off VM is not considered a vMotion. You most likely have a networking issue between the two vmkernel ports on both hosts tagged for vMotion. Verify by conducting a vmkping between those vmkernel ports. If you cannot ping between them, vMotion will not work. https://kb.vmware.com/s/article/1003728
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Seems you didn't notice that I have mentioned already the vmkping works fine and I can't see any issues with the network, however the NFS datastore is sitting on a windows server machine in the same subnet and it's mounted fine on the ESXi hosts fine not sure what would cause the problem.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Just for the record, would you please conduct a vmkping between those vMotion vmkernel ports and post the output along with the command you used?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Here is the results
[root@ESXi01:~] vmkping -s 8972 -d 192.168.1.201
PING 192.168.1.201 (192.168.1.201): 8972 data bytes
8980 bytes from 192.168.1.201: icmp_seq=0 ttl=64 time=0.693 ms
8980 bytes from 192.168.1.201: icmp_seq=1 ttl=64 time=0.698 ms
8980 bytes from 192.168.1.201: icmp_seq=2 ttl=64 time=0.680 ms
--- 192.168.1.201 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.680/0.690/0.698 ms
[root@ESXi01:~] vmkping -I vmk2 -S vmotion 192.168.1.201
PING 192.168.1.201 (192.168.1.201): 56 data bytes
64 bytes from 192.168.1.201: icmp_seq=0 ttl=64 time=0.166 ms
64 bytes from 192.168.1.201: icmp_seq=1 ttl=64 time=0.186 ms
64 bytes from 192.168.1.201: icmp_seq=2 ttl=64 time=0.178 ms
--- 192.168.1.201 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.166/0.177/0.186 ms
---------------------------------------------
[root@ESXi02:~] vmkping -s 8972 -d 192.168.1.200
PING 192.168.1.200 (192.168.1.200): 8972 data bytes
8980 bytes from 192.168.1.200: icmp_seq=0 ttl=64 time=0.644 ms
8980 bytes from 192.168.1.200: icmp_seq=1 ttl=64 time=0.674 ms
8980 bytes from 192.168.1.200: icmp_seq=2 ttl=64 time=0.682 ms
--- 192.168.1.200 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.644/0.667/0.682 ms
[root@ESXi02:~] vmkping -I vmk2 -S vmotion 192.168.1.200
PING 192.168.1.200 (192.168.1.200): 56 data bytes
64 bytes from 192.168.1.200: icmp_seq=0 ttl=64 time=0.182 ms
64 bytes from 192.168.1.200: icmp_seq=1 ttl=64 time=0.185 ms
64 bytes from 192.168.1.200: icmp_seq=2 ttl=64 time=0.185 ms
--- 192.168.1.200 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.182/0.184/0.185 ms
--------------------------------------------------
I have tried to vMotion from the NFS datastore to the 2 hosts on VMFS6 datastore works perfectly fine no problems, tried to vMotion the same machine back to the NFS datastore failed :S
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
More than likely, this is due to Microsoft's garbage implementation of the NFS protocol, which no one should ever use. Throw that junk away and use another implementation. Even something like FreeNAS works better.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thanks for your input, will look for alternatives.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I'm using Linux kernel 5.4.0.52 an have this problem with a similar setup. vmotion, nfs server, fails immediately with storage migration.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
similar error noticed with RHEL 7.x in vSphere 6.5 env with unformatted RAW disks (vmdk, not RDM) attached to the vm.
All other vms were able to migrate except the RHEL vm.
Failed waiting for data. Error 195887250. Migration determined a failure by the VMX.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I followed the below article to resolve the issue. May be it will helpful you.
https://vmkfix.blogspot.com/2023/05/storage-migration-failed.html