Vmotion fails after 10% with "operation timed out"

[AtleJensen]

I have a problem with one of my VM's.

When I try to migrate it with Vmotion it fails after about 1 min with the message "operation timed out".

If I try to migrate other VM's from the same source host to the same destination host it does so without any problems.

I run ESX server 3.0.1 and I can ping both hosts from each other.

I think the problem started after we upgraded the Virtual Center server to patch 2.

We had problems with HA agents on both ESX servers, but fixed it with a restart of the agent deamon, patience and reconnect.

Best regards

Atle

Message was edited by:

AtleJensen

[warrenwalker]

Have you checked any logs for clues?

[VMKR9]

If you are Vmotioning from Server1 to Server2 ssh to server 1 and run

tail -f /var/log/vmkernel

start your Vmotion as it is running the logs are going to start generating in the vmkernel file, when it fails see if you can see the reason...

[AtleJensen]

This is what I get from the logfile on the destination host:

Jul 4 11:56:24 esxserver1 vmkernel: 33:21:56:31.518 cpu0:1969)WARNING: Swap: vm 1969: 1480: Failed to create swap file '/volumes/45bee416-a7f2f03a-10fe-0019bbc904c2/Exchange/Exchange-b7e9b68b.vswp.15132': Host doesn't have a journal

Jul 4 11:56:24 esxserver1 vmkernel: 33:21:56:31.518 cpu0:1969)WARNING: World: vm 1970: 699: init fn swap failed with: No swap file!

Jul 4 11:56:24 esxserver1 vmkernel: 33:21:56:31.518 cpu0:1969)WARNING: World: vm 1970: 1530: WorldInit failed: trying to cleanup.

On the source host there is no changes in the logfile.

[VMKR9]

Is there anything in the vmware.log for the vm at the same time as the vmotion failing? not seen that error before. Is the vm quite busy? does it have a lot of memory assigned to it?

[AtleJensen]

It is our mailserver, so it has some load.

Host memory usage: 2,28 GB

Guest memory usage: 325 MB

If I try to view the vmware.log file I only get "Device or resource busy" (I tried to use "tail" and "more").

Message was edited by:

AtleJensen

[Morpheus]

We had the problem, that we could migrate VMs to some ESX hosts and to some ESX hosts it did not work. In our case, the swapfile could not be created and the migration failed.

The reason was, that we configured the storage (NAS) on the ESX hosts in different ways. We used the DNS Name, Full qualified DNS Name or IP Adress of the NAS.

After we reconfigured the NAS on the ESX hosts in exactly the same way, the problem was fixed.

This link describes a similar problem.

http://www.vmware.com/community/thread.jspa?threadID=53797&start=0&tstart=0

Every resource has to be configured exactly the same way on each ESX host you would like to use for vmotion!

Hope this helps.

Best Regards,

Frank

[AtleJensen]

We are using a SAN storage, so I don't think IP/DNS issue is the case here. Unfortunately.

I can ping all the vmotion ports from all hosts, and the ip range on these are on a seperate vlan.

Atm I think I will try to shut the VM down and do a cold migrate to see if that works.

Just curios about how much space is needed on the storage that the boot Harddisk is located to do Vmotion?

[Nautilus]

Hi AtleJensen,

i have get this Problem more than one time.

1. ping and vmping from any service console Ip any other IPs of your VI3 Servers

2. check the ressources on your server. When you try to vmotion a vm to a Vi3 hosts with not enought reousrces, the will fail back after 10%

All the messages are logged in /var/log vmkern...... or in the log files in any VM.

I hope this helsp you

[LasseHT]

try and delete the Vmotion network (on the host server) then recreate it, we had a problem where that helped.

[bggb29]

Verify the subnet mask and ip address on the hosts vmotion network.

I had the same issue it was the vmotion ip address

[AtleJensen]

Now I have done an offline migration of the VM and it work out all ok.

After I start the VM again and do a vMotion migration it comes up with the same error message as before.

I have also done what some people suggested, to delete the vMotion network and create it again. It did not help.

The ip addresses on the VLAN for the vMotion is checked again and again, even the subnet mask. Nothing I can find wrong.

The destination host has enough resources, so that shouldn't be the problem.

The vMotion port is on the same vSwitch as the vmkernel and the service console. Can that have something to do with it?

Message was edited by:

AtleJensen

[DougBaer]

While that is generally not recommended (sharing SC with VMotion ports), I don't think it is causing the problem here. However, if your VMotion and SC IPs are on the same subnet, I suppose it could create funniness.

Can you do a vmkping from the source and destination hosts to each other? Use the VMotion IPs to verify that they can see each other on that network.

vmkping 10.0.0.1[/code]

... give it a few seconds to get started since it sometimes takes a bit to get going.

When I have seen VMotions fail at 10%, it has normally been a network issue (bad cabling, routing issue, goofy VLAN trunking, etc.).

[RobBuxton]

Have a look at what Swap files are around in the VM guests directory.

I've had an issue where I had more than one and even where the one I had seemed to be corrupt.

My "fix" was to shut down the guest and then mv the swap file away. You should be able to start a guest with no vswap file.

Once done, try the vmotion.

[AtleJensen]

Thank you RobBuxton, I will try what you suggested when it is possible.

To DoughBaer: The vMotion network is on a separate VLAN, with nothing else than vMotion. I can ping both vMotion ports from each host. Some of the guests can be migrated with vMotion, but not all.

I would like to thank everyone that have answered this post here. I will post the result here after I have tried the vswap file.

Message was edited by:

AtleJensen

[MayurPatel]

Check the VMotion NIC on your hosts are correctly configured for negotiation. If you are running 100Mb then make sure it is set at Full Duplex and if your switches are Gb then set the NIC to 1000Mb instead of Autonegotiate.

[NorbK]

I had the exact issue where a cold migration worked but timed out at 10% and it turns out someone else brought up a machine with the IP address I had for VMotion. I brought down the ESX server, pinged my VMotion IP and hunted down the guy who grabbed it!

[DougBaer]

I had the exact issue where a cold migration worked

but timed out at 10% and it turns out someone else

brought up a machine with the IP address I had for

VMotion. I brought down the ESX server, pinged my

VMotion IP and hunted down the guy who grabbed it!

...and if this is what happened, be sure to throttle that person soundly!

[alb]

Try the following:

Edit the virtual machine settings, select the Options Tab, then click on Advanced in the left hand box. Now select "Hide NX flag from guest"

I'm sure I had something like this and the root of the problem was that the servers were at different patch levels.

Al.

[lawries]

This may sound dumb, but have you looked at the obvious which is a CD Device is attached to the VM that you are trying to VMotion.