Error moving guest between ESXi 4 servers

[mmarseglia]

Hi,

I have two free ESXi servers vm1 and vm2. They are connected to the same NFS datastore.

I have a guest, guest2, that I originally built on vm2. I registered guest2 on my second esxi4 server, vm1, by browsing the NFS datastore.

I powered off guest2 on vm2. I attempted to power on guest2 on vm1 but I got an error

I checked the ntp settings on both hosts and found I had not configured ntp on vm1. I configured vm1 with the same ntp servers as vm2, got the time sync'd and they are now both at the same time.

I removed guest2 from vm1, re-registered it, and I got the same error.

The vSphere client gave this error message: "A general system error occurred: The virtual machine could not start"

I got more information from /var/log/vmware/host-d.log. guest2 is called "Backups". Not sure that it matters, but it is an Ubuntu JeOS guest.

2009-06-11 19:27:59.262 57D25B90 info 'TaskManager' Task Created : haTask-240-vim.VirtualMachine.powerOn-1852795575

2009-06-11 19:27:59.262 57D25B90 verbose 'vm:/vmfs/volumes/47dea0bb-b98659eb/Backups/Backups.vmx' Power on request recieved

2009-06-11 19:27:59.262 57D25B90 verbose 'vm:/vmfs/volumes/47dea0bb-b98659eb/Backups/Backups.vmx' Reconfigure ethernet backing if required

2009-06-11 19:27:59.263 57D25B90 info 'ha-eventmgr' Event 74 : Backups on host vm1 in ha-datacenter is starting

2009-06-11 19:27:59.263 57D25B90 info 'vm:/vmfs/volumes/47dea0bb-b98659eb/Backups/Backups.vmx' State Transition (VM_STATE_OFF -> VM_STATE_POWERING_ON)

2009-06-11 19:27:59.263 57D25B90 verbose 'ha-host' ModeMgr::Begin: op = normal, current = normal, count = 4

2009-06-11 19:27:59.263 57D25B90 verbose 'ha-license-manager' Load: Loading existing file: /etc/vmware/license.cfg

2009-06-11 19:27:59.281 57D25B90 verbose 'vm:/vmfs/volumes/47dea0bb-b98659eb/Backups/Backups.vmx' PowerOn request queued

2009-06-11 19:27:59.301 57CE4B90 verbose 'vm:/vmfs/volumes/47dea0bb-b98659eb/Backups/Backups.vmx' Retrieved current VM state from foundry 7, 6

2009-06-11 19:27:59.301 120DAB90 info 'Libs' /vm/#600948508ace93ca/: VMHSVMCbPower: Setting state of VM to powerOn with option soft

2009-06-11 19:27:59.303 120DAB90 info 'vm:/vmfs/volumes/47dea0bb-b98659eb/Backups/Backups.vmx' VMHS: Exec()'ing /bin/vmx

2009-06-11 19:27:59.303 120DAB90 info 'Libs' VMHS: VMKernel_ForkExec(/bin/vmx, detached=1): rc=0 pid=919913

2009-06-11 19:27:59.354 120DAB90 warning 'Libs' VMHS: Looks like pid 919913 really died.

2009-06-11 19:27:59.355 120DAB90 info 'Libs' FoundryVMDBPowerOpCallback: VMDB reports power op failed for VM /vmfs/volumes/47dea0bb-b98659eb/Backups/Backups.vmx with error msg = "Failed to launch peer process" and error code -46.

2009-06-11 19:27:59.355 57CA3B90 verbose 'vm:/vmfs/volumes/47dea0bb-b98659eb/Backups/Backups.vmx' Retrieved current VM state from foundry 5, 2

2009-06-11 19:27:59.355 57D66B90 info 'Vmsvc' Failed to do Power Op: Error: (3041) The virtual machine could not start

2009-06-11 19:27:59.355 57D66B90 warning 'vm:/vmfs/volumes/47dea0bb-b98659eb/Backups/Backups.vmx' Failed operation

2009-06-11 19:27:59.356 57D66B90 info 'ha-eventmgr' Event 75 : Cannot power On Backups on vm1 in ha-datacenter. A general system error occurred:

2009-06-11 19:27:59.356 57D66B90 info 'vm:/vmfs/volumes/47dea0bb-b98659eb/Backups/Backups.vmx' State Transition (VM_STATE_POWERING_ON -> VM_STATE_OFF)

2009-06-11 19:27:59.356 57D66B90 verbose 'ha-host' ModeMgr::End: op = normal, current = normal, count = 5

2009-06-11 19:27:59.356 57D66B90 info 'TaskManager' Task Completed : haTask-240-vim.VirtualMachine.powerOn-1852795575 Status error

No clue what to do with the information in the log. I'm stuck.

No, I can't go from vm1 to vm2 either, I get the same error. I'm a bit concered because if either server blows up I'm afraid I won't be able to register my vm's with another ESXi server.

I'm running the free version of esxi4, VMkernel vm1 4.0.0 #1 SMP Release build-164009 Apr 30 2009 19:02:35 x86_64 unknown

[DSTAVERT]

Are the hosts similar hardware. AMD vs Intel. One support VT and the other not. Hardware on the compatibility list. I have had a server able to install 4 but not run 64bit VM's.

[mmarseglia]

You raise a good point. They should be identical HP servers but I am unsure if VT is enabled on both, one, or none. I will check on this.

--Update

I checked and they are the same hardware and VT is enabled on both servers. Good suggestion, thank you.

[admin]

Did you check the VM's log file? It should be in the same directory where the .vmx and the vmdk file are located, and should be called vmware.log.

Could you post the contents of the vmware.log file as well?

[norregaard]

maybe the guest haven't been shut properly down on the ESX even though it appears to be in VI client and you have a hung process or a lock on one the ESX servers..

-

---

If you have a VM that is stuck and you can't power it off via Virtual Center, then here's some usable commands for troubleshooting and killing the VM from the Service Console:

Show running VM's:

vm-support -x

vmware-cmd -l

See the current state of a VM:

vmware-cmd /

-

---

http://www.vi-tips.com/2009/04/howto-power-off-or-kill-vm-that-is.html

if this is not the case, you could create a new empty VM with the same specs as the existing one, and then attach the .vmdk disk file to the new VM (and delete the old VM)

[mmarseglia]

Thank you, I tried these commands and did not see the VM running on the other host.

I tried doing a V to V conversion and then run the new VM on the other host. That did not work. Is doing the V-to-V the same as you describe here?

if this is not the case, you could create a new empty VM with the same specs as the existing one, and then attach the .vmdk disk file to the new VM (and delete the old VM)

[DSTAVERT]

I tried doing a V to V conversion and then run the new VM on the other host. That did not work. Is doing the V-to-V the same as you describe here?

What happened when it "did not work"? You need to make sure you don't do any harm to the existing files. What files are listed in the vm directory? What are the messages when you try to start the VM? Post the contents of the vmware.log file.

[DSTAVERT]

Check to see what the file permissions are from the NFS server.

[mmarseglia]

I tried everything folks suggested. I had a few other variables in my situation and I think one of them fixed the issue. Not really sure what the real culprit was.

I do know that when I tried a V-to-V conversion it did not work.

I upgraded from ESXi3.5 to 4 and upgraded each guest. I may have upgraded this guest on esx2 while keeping it registered on esx1. esx1 thought it was still a v4 Guest, not v7

I installed vmtools on the guest.

When I originally moved the guest it was suspended. It started up OK on esx2 but I had problems moving it back esx1. While running it on esx2 I've powered it off/on on a couple times while I was working on the OS.

I un-registered and re-registered the guest on esx1.

It seems to work OK now and I can power the guest up on either host.

[mmarseglia]

Thought of that and checked those too. I thought maybe there was a lock or something on the files.