vCenter 5.0 (Appliance) Issues (Stack Error & Migration Issues)

[BCS-Satori]

Greetings,

I am having numerous vCenter 5.0 (Appliance) issues and I am having strong considerations on returning to vCenter 4.2 even though it will be painful as most machines are running VM Verison 8 and VMFS 5.

There are two issues currently plaguing my new environment:

1. I am unable to migrate a live virtual machine from one datastore or host to another.  The process is able to reach 21% upon which I received an "Operation timed out." error.  From migrate to error message takes approximately one minute.  If the virtual machine is powered off I am able to migrate the machine without issue.
2. I am unable to power on virtual machines on random a random chassis  because of a unexpected stack error:  I patched ESXi to 504890  surpassing the Sept. patch for this issue however the issue is still  present.  The only solution is to reboot the physical chassis upon which  I am able to power on machines for a limited time before the issue  reoccurs.

/var/log/vmkernel.log (shows)

2011-11-14T18:05:26.394Z cpu9:9018)Config: 346: "SIOControlFlag2" = 1, Old Value: 0, (Status: 0x0)
2011-11-14T18:05:29.435Z cpu12:9289)VmkAccess: 637: hostd-vix-worke: access denied:: dom:appDom(2), obj:forkExecSys(88), mode:syscall_allow(2)
2011-11-14T18:05:29.435Z cpu12:9289)VmkAccess: 320: VMkernel syscall num 1
2011-11-14T18:05:29.436Z cpu6:9018)Config: 346: "SIOControlFlag2" = 0, Old Value: 1, (Status: 0x0)

/var/log/hostd.log (shows)

2011-11-14T18:09:02.568Z [51F99B90 info 'vm:/vmfs/volumes/4ebd2cac-0318e7bb-390c-984be104fb1e/BCSRAD01/BCSRAD01.vmx'] VMHS: Exec()'ing /bin/vmx
2011-11-14T18:09:02.568Z [51F99B90 info 'Libs'] VMHS: VMKernel_ForkExec(/bin/vmx, detached=1): rc=195887383 pid=-1
2011-11-14T18:09:02.568Z [51F99B90 warning 'Libs'] VMHSHostExecVMX: unable to spawn /bin/vmx: Operation not permitted
2011-11-14T18:09:02.568Z [51F99B90 warning 'Libs'] VMHS_LaunchVMCb failed: Failed to launch peer process
2011-11-14T18:09:02.568Z [51F99B90 info 'Libs'] FoundryVMDBPowerOpCallback: VMDB reports power op failed for VM /vmfs/volumes/4ebd2cac-0318e7bb-390c-984be104fb1e/BCSRAD01/BCSRAD01.vmx with error msg = "The virtual machine could not start. Make sure VMware ESX is installed correctly and you have rights to run the software and to access all directories it uses, including directories in which the software is installed, directories containing the virtual disk and configuration files, and directories for temporary files." and error code -46.

Any assistance would be great.  Thanks.

[aoctav]

Hi BCS-Satori,

did you find a solution to your situation? I am experiencing the same behaviour at the moment. 😞

fyi: I am using vCenter Server 5.0 on a Windows 2008 R2 Server, so it does not seem to be a problem specific to the Appliance.

edit: I can power on a machine after i removed it from the inventory, added it again and migrated it once. WTF??

Best regards,

Andreas

[BCS-Satori]

I do have an open case with VMware but my technician does not seem know what is happening at this time.

I did also notice that I am unable to modify and running virtual machine (i.e.: adding addtional network card, hard drive, etc.)  when the issue is occuring.

WORKAROUND:

I have found a temporary workaround to the problem by restarting the  management agents on each ESXi 5.0 host having issues either migrating  to or powering on.  After the agents are restarted I can migrate three machines upon which the errors come back always on the fourth machine.

Once I have a response from VMware I will post their solution (if there is one, I have a strong suspition we are waiting for the next patch)...

[RParker]

Ok first, we have the appliance, and we don't have these issues.  The appliance is the same for every person, it's not even a self install so I seriously doubt it's a vCenter Appliance issue, since everyone would have the same problem, and we don't.

Secondly, try to power on the VM from host (skip the vCenter) and see if you get the same error.

Since you mention that you patched the server and this problem was related to a patch, you can simply migrate VM's off that host, reinstall the host (FRESH Install) and migrate the VM's back.  This seems like a ESX host problem (or possible hardware firmware) NOT vCenter problem.

Before you consider that vCenter is the issue, you might try to isolate the problem first.

[Xaneth]

Any word on a fix for this yet?  Just got my HA cluster up and running and having these very same issues.  It's been a year now, hopefully this thread has just been dead?  I of course opened my own ticket, so I guess we should see one way or another.                 

[Jason_Knight]

FYI, I resolved my issue by reinstalling the VMWare hypervisor on the new host.  Apparently there was some sort of corruption in my install.  This is what VMWare support had me do.

I'm replying to Xaneth, as this is my other VMWare profile.   

[adprom]

I'm getting this on 5.1 with same sort of issues. Done a fresh esx install etc. Wasn' getting this on 5.0

2012-09-17T14:50:36.683Z [77B43B90 info 'Libs'] VMHS: VMKernel_ForkExec(/bin/vmx, detached=1): rc=195887383 pid=2017633088

2012-09-17T14:50:36.683Z [77B43B90 warning 'Libs'] VMHSSpawnVMXUW: unable to spawn /bin/vmx: Operation not permitted

2012-09-17T14:50:36.683Z [77B43B90 info 'Libs'] VigorClient_StartAndConnect Failed: Failed to launch peer process

General behaviour is that you do a VM hardware upgrade or vmotion between data stores and the first operation succeeds. However from there, any subsequent action such a datastore migration fails with a timeout. Trying to power a VM fails with general system error. The way to get it working again is to reboot, or login via the cli and run services.sh restart. Quite frustrating and no real reason known for it failing yet. Hardware appears to be fine.