AtleJensen
Enthusiast
Enthusiast

Vmotion fails after 10% with "operation timed out"

I have a problem with one of my VM's.

When I try to migrate it with Vmotion it fails after about 1 min with the message "operation timed out".

If I try to migrate other VM's from the same source host to the same destination host it does so without any problems.

I run ESX server 3.0.1 and I can ping both hosts from each other.

I think the problem started after we upgraded the Virtual Center server to patch 2.

We had problems with HA agents on both ESX servers, but fixed it with a restart of the agent deamon, patience and reconnect.

Best regards

Atle

Message was edited by:

AtleJensen

0 Kudos
20 Replies
warrenwalker
Enthusiast
Enthusiast

Have you checked any logs for clues?

0 Kudos
VMKR9
Expert
Expert

If you are Vmotioning from Server1 to Server2 ssh to server 1 and run

tail -f /var/log/vmkernel

start your Vmotion as it is running the logs are going to start generating in the vmkernel file, when it fails see if you can see the reason...

AtleJensen
Enthusiast
Enthusiast

This is what I get from the logfile on the destination host:

Jul 4 11:56:24 esxserver1 vmkernel: 33:21:56:31.518 cpu0:1969)WARNING: Swap: vm 1969: 1480: Failed to create swap file '/volumes/45bee416-a7f2f03a-10fe-0019bbc904c2/Exchange/Exchange-b7e9b68b.vswp.15132': Host doesn't have a journal

Jul 4 11:56:24 esxserver1 vmkernel: 33:21:56:31.518 cpu0:1969)WARNING: World: vm 1970: 699: init fn swap failed with: No swap file!

Jul 4 11:56:24 esxserver1 vmkernel: 33:21:56:31.518 cpu0:1969)WARNING: World: vm 1970: 1530: WorldInit failed: trying to cleanup.

On the source host there is no changes in the logfile.

0 Kudos
VMKR9
Expert
Expert

Is there anything in the vmware.log for the vm at the same time as the vmotion failing? not seen that error before. Is the vm quite busy? does it have a lot of memory assigned to it?

0 Kudos
AtleJensen
Enthusiast
Enthusiast

It is our mailserver, so it has some load.

Host memory usage: 2,28 GB

Guest memory usage: 325 MB

If I try to view the vmware.log file I only get "Device or resource busy" (I tried to use "tail" and "more").

Message was edited by:

AtleJensen

0 Kudos
Morpheus
Contributor
Contributor

We had the problem, that we could migrate VMs to some ESX hosts and to some ESX hosts it did not work. In our case, the swapfile could not be created and the migration failed.

The reason was, that we configured the storage (NAS) on the ESX hosts in different ways. We used the DNS Name, Full qualified DNS Name or IP Adress of the NAS.

After we reconfigured the NAS on the ESX hosts in exactly the same way, the problem was fixed.

This link describes a similar problem.

http://www.vmware.com/community/thread.jspa?threadID=53797&start=0&tstart=0

Every resource has to be configured exactly the same way on each ESX host you would like to use for vmotion!

Hope this helps.

Best Regards,

Frank

0 Kudos
AtleJensen
Enthusiast
Enthusiast

We are using a SAN storage, so I don't think IP/DNS issue is the case here. Unfortunately.

I can ping all the vmotion ports from all hosts, and the ip range on these are on a seperate vlan.

Atm I think I will try to shut the VM down and do a cold migrate to see if that works.

Just curios about how much space is needed on the storage that the boot Harddisk is located to do Vmotion?

0 Kudos
Nautilus
Enthusiast
Enthusiast

Hi AtleJensen,

i have get this Problem more than one time.

1. ping and vmping from any service console Ip any other IPs of your VI3 Servers

2. check the ressources on your server. When you try to vmotion a vm to a Vi3 hosts with not enought reousrces, the will fail back after 10%

All the messages are logged in /var/log vmkern...... or in the log files in any VM.

I hope this helsp you

Kind regards Nautilus ________________________________________________________________________________________________________________________________________________ If you found this or any other post helpful please consider the use of the Helpfull/Correct buttons to award points
0 Kudos
LasseHT
Enthusiast
Enthusiast

try and delete the Vmotion network (on the host server) then recreate it, we had a problem where that helped.

0 Kudos
bggb29
Expert
Expert

Verify the subnet mask and ip address on the hosts vmotion network.

I had the same issue it was the vmotion ip address

AtleJensen
Enthusiast
Enthusiast

Now I have done an offline migration of the VM and it work out all ok.

After I start the VM again and do a vMotion migration it comes up with the same error message as before.

I have also done what some people suggested, to delete the vMotion network and create it again. It did not help.

The ip addresses on the VLAN for the vMotion is checked again and again, even the subnet mask. Nothing I can find wrong.

The destination host has enough resources, so that shouldn't be the problem.

The vMotion port is on the same vSwitch as the vmkernel and the service console. Can that have something to do with it?

Message was edited by:

AtleJensen

0 Kudos
DougBaer
VMware Employee
VMware Employee

While that is generally not recommended (sharing SC with VMotion ports), I don't think it is causing the problem here. However, if your VMotion and SC IPs are on the same subnet, I suppose it could create funniness.

Can you do a vmkping from the source and destination hosts to each other? Use the VMotion IPs to verify that they can see each other on that network.

vmkping 10.0.0.1[/code]

... give it a few seconds to get started since it sometimes takes a bit to get going.

When I have seen VMotions fail at 10%, it has normally been a network issue (bad cabling, routing issue, goofy VLAN trunking, etc.).

Doug Baer, Staff Architect, Sr. Manager of vPod Architecture team for the VMware Hands-on Labs | VCDX #019, vExpert 2012-20 | @dobaer
0 Kudos
RobBuxton
Enthusiast
Enthusiast

Have a look at what Swap files are around in the VM guests directory.

I've had an issue where I had more than one and even where the one I had seemed to be corrupt.

My "fix" was to shut down the guest and then mv the swap file away. You should be able to start a guest with no vswap file.

Once done, try the vmotion.

0 Kudos
AtleJensen
Enthusiast
Enthusiast

Thank you RobBuxton, I will try what you suggested when it is possible.

To DoughBaer: The vMotion network is on a separate VLAN, with nothing else than vMotion. I can ping both vMotion ports from each host. Some of the guests can be migrated with vMotion, but not all.

I would like to thank everyone that have answered this post here. I will post the result here after I have tried the vswap file.

Message was edited by:

AtleJensen

0 Kudos
MayurPatel
Expert
Expert

Check the VMotion NIC on your hosts are correctly configured for negotiation. If you are running 100Mb then make sure it is set at Full Duplex and if your switches are Gb then set the NIC to 1000Mb instead of Autonegotiate.

0 Kudos
NorbK
Enthusiast
Enthusiast

I had the exact issue where a cold migration worked but timed out at 10% and it turns out someone else brought up a machine with the IP address I had for VMotion. I brought down the ESX server, pinged my VMotion IP and hunted down the guy who grabbed it!

0 Kudos
DougBaer
VMware Employee
VMware Employee

I had the exact issue where a cold migration worked

but timed out at 10% and it turns out someone else

brought up a machine with the IP address I had for

VMotion. I brought down the ESX server, pinged my

VMotion IP and hunted down the guy who grabbed it!

...and if this is what happened, be sure to throttle that person soundly! Smiley Wink

Doug Baer, Staff Architect, Sr. Manager of vPod Architecture team for the VMware Hands-on Labs | VCDX #019, vExpert 2012-20 | @dobaer
0 Kudos
alb
Contributor
Contributor

Try the following:

Edit the virtual machine settings, select the Options Tab, then click on Advanced in the left hand box. Now select "Hide NX flag from guest"

I'm sure I had something like this and the root of the problem was that the servers were at different patch levels.

Al.

0 Kudos
lawries
Contributor
Contributor

This may sound dumb, but have you looked at the obvious which is a CD Device is attached to the VM that you are trying to VMotion.

0 Kudos