vMotion error

[jooji]

Hi,

Im getting the below error message when attempting to migrate a live VM approx 5TB, this has happened on a few other VMs but after a few attempts they eventually migrate. The problem is the large VMs, ive had to migrate 2 over the weekend offline. The destination datastore has 9TB free (this was also erroring when there was 12TB free) so i cant see it being disk space. Its all 10G networking source and dest with a 1Gb uplink in the middle between both old and new clusters. VMware support havent been much help, they just ask to do it offline but i find it to be slower and obviously causes disruption. Just wondering if anyone has seen the same message before? The error codes are just VMk error codes that state the obvious, no disk space and timeout... but i cant see why it would say that when i have TB's of free space!

Failed waiting for data. Error 195887319. No space left on device. vMotion migration [171704324:5711899582906550958] failed to read stream keepalive: Connection closed by remote host, possibly due to timeout Failed to copy source (/vmfs/volumes/5de534ab-6a9b9d19-1f82-5cb9019c650c/VMNAME/VMDISK.vmdk) to destination (/vmfs/volumes/616edfcc-37d17a56-d2a3-e43d1a4fe1b0/VMNAME/VMDISK.vmdk): No space left on device. Migration to host <IPADDRESS> failed with error Connection closed by remote host, possibly due to timeout (195887167). 2021-11-15T10:27:45.305639Z File system specific implementation of Ioctl[file] failed.

[ABreitbach]

Hello,

did you find a solution or the reason for it?

I got exactly the same error today for a bigger VM > 3TB.

Migration of all other VMs with 40 to 300GB went fine.

We only have 1Gbit/s connection on same switch.

The failure occurs at different states.

Regards

[jooji]

Hello,

I didn't no, i tried a cold migration but that didn't work either. VMware support weren't much help, they never usually are with vMotion issues.

In the end i used Veeam replication to copy the VM to the new destination datastore.

[ABreitbach]

hmm,

some researches pointed to network timeouts due to load balancing...

I have now reconfigured to use a dedicated vMotion NIC on the neighbor port on the switch.

Already transferred 500GB. 2.5 TB remaining. Hoping the best.

Did you use replication or quick-migration features inside Veeam?

Regards

[jooji]

Yeah i had a dedicated vMotion VLAN using single 10GbE NICs either side but the link between the infrastructures in the middle was 1GbE. Obviously expected a bottleneck but not the issues i was seeing. I was expecting VMware support to come back and tell me to change some form of timeout setting somewhere. I've never had any issues like this with vMotion, but this customer is one of those where every rock you turn over some nasty is starting you in the face! ha

I created a full replication job. Quick-migration just uses vMotion anyway and the other forms of migration tools in Veeam are usually cold migrations or require the VM to be paused.

[ABreitbach]

It failed again at 1.2 TB 😞

I have now started the Veeam quick migration and marked the checkbox to route the traffic over the Veeam server.

It's only 30% of speed but maybe runs fine without the errors...