Machines wont migrate off host

[paradoxxaz]

So this is going to seem pretty broad, but I have a 3 node cluster. One of the nodes was throwing HA errors all over the place last week. I was able to finesse it enough (restarting services etc) to get it running. There are about 12 VM's on it.

Now, whenever I try to migrate a machine off, it just hangs at 10%, never moves.. ulitimately i think it errors with a time out...

Any ideas? One node is on update 4, the others, including this one, arent even on update 4, thats where I am trying to get. This Friday is maintenance, so I need to get all off so I can update. How the *&% am I going to get these machines off without just killing the hosts and letting HA take over? if it even will.

Thanks guys..

[Troy_Clavell]

if I shut down the VM it migrates

remember, I cold migration does not use the vmkernel port, it uses the service console. So, that is not a good test.

[paradoxxaz]

i did the vmkping to the hosts themselves, posted above... is this what you wanted?

[paradoxxaz]

3 packets transmitted, 3 packets received, 0% packet loss

round-trip min/avg/max = 0.286/0.439/0.672 ms

[Troy_Clavell]

sorry, I missed that....long thread

[paradoxxaz]

so the vmkping <hostname> pings the vmkernal?

[Troy_Clavell]

as suggested earlier you may want to restart hostd on at least one of the ESX Hosts in question?

>service mgmt-vmware restart

remember the host will go into a not responding state and the vm's will go disconnected, but don't panic, it's just vCenter losing connection. When the Host(s) comes back into vCenter in a connnected state, try to issue the vmotion again.

[paradoxxaz]

have doen this too many times to list.

[Troy_Clavell]

so the vmkping <hostname> pings the vmkernal?

you have an IP address for your vmkernel port(vmotion). When you issue the vmkping command from one ESX Host, you want the destination ping to be the vmkernel (vmotion) IP address.

make sense?

[paradoxxaz]

now when I try to migratre anything off the host I get the error:

a general system error: failed to initiate vmotion dest (vim.fault.AlreadyExists)

uhg

[Troy_Clavell]

you don't have any "orphaned" VM's do you? Can you post a screen shot of your vCenter instance?

[paradoxxaz]

8 is the issue, 9 is @ update 4, 10 is fine, but just as behind on updates as 8 (attached)

[Troy_Clavell]

is it possible for you to shutdown the vm that is having the problem, remove it from inventory, then browse the datastore and add the VM back to inventory power it on and try the migration again?

[paradoxxaz]

they all are!!!! nothing will migrate off 8!

Takes hours to migrate a single vm off 8

[max2479]

paradoxxaz

we have established that the cold migration works which uses sc as Troy has mentioned.

disable and enable vmotion. it might do the trick.

go to configiration of the esx host. networking -> find the vmkernel port and go to properties. you will see check box there enabled. disable it and enable it back

Also I am seeing you are running Update manager, correct? you might want to stop any update when migrating...

if you found this or any other answer useful please consider allocating points for helpful or correct answers <

[Troy_Clavell]

I don't want to keep pushing you down the road, that may or may not work. You have also tried restarting your vCenter Server Service?

Maybe open an SR? This will give VMware a webex session into your environment and may be able to find something we can't find by just going through thread comments.

[paradoxxaz]

already tried that

[paradoxxaz]

I may have to open a SR

[Troy_Clavell]

>a general system error: failed to initiate vmotion dest (vim.fault.AlreadyExists)

That error bothers me... To me it seems as though your VM's won't migrate because somehow it appears the process of vmotion is failing because it thinks there is another VM registered with the same name.

Please keep me posted as to what VMware Support says.

[paradoxxaz]

will do.

[beagle_jbl]

I have had to face HA problems a number of times.

To state the obvious, have you tried going into VI Client, right clicking the Host, and selecting "reconfigure for HA" and ensure it doesn't error out.

Also DNS can be a nasty thing and the cause of almost ALL of my HA woes in the past. For example, case matters in DNS names with ESX, so I always ensure my host names and DNS entries are lower case. You may want to verify your DNS names match EXACTLY (seriously - double and triple check) and that they resolve properly on all the hosts in your cluster. I also like to ensure that all my reverse DNS zones are accurate as well.

If neither of those work, you may want to schedule an outage to get the VM's off that bad host, remove the host from the cluster and then re-add it.

If that fails, I would leave the VM's off and reinstall ESX... because any other issues would probably take more time to resolve than the half a day it takes you to rebuild.