VMware Cloud Community
ejm381
Contributor
Contributor

Unable to start vcenter service

Greetings.

Unable to connect to Vcenter 4.1 server using Vsphere client. Vcenter server has been in production for a year, no recent changes, running Win2003R2 with local SQL Express 2005 DB connecting to three Vsphere 4 hosts.

Server running at 100% CPU. Reboot, same. Unable to start "VirtualCenter Server" service..it will go to a "started" status, the server will jump to 100% CPU for 15 seconds and then the service fails.

Checked DB. Less than 10% free space, set to autogrow, should be fine but I expanded anyway. Same issue.

IIS is not installed on the vcenter server, never has been.

Looking at vxpd logs, these are the warn/error entries:

[2011-01-16 15:14:25.607 05724 warning 'Libs'] [VpxdOsLayer] Couldn't read registry entry managedIP

[2011-01-16 15:14:25.623 05724 error 'App'] [OptionMgr] Ignoring unknown entry from DB: VirtualCenter.VMotionEncryption

[2011-01-16 15:14:42.185 05724 warning 'Libs'] [VpxdOsLayer] Couldn't read registry entry VCVmId

[2011-01-16 15:14:44.091 01044 warning 'VpxdMoLock'] ***WARNING*** Lock host-10 mode SHARE held for 1126 ms
[2011-01-16 15:14:45.248 01044 warning 'VpxdMoLock'] ***WARNING*** Lock host-84404 mode SHARE held for 1088 ms

[2011-01-16 15:14:55.091 00728 warning 'VpxdMoLock'] ***WARNING*** Lock domain-c83577 mode EXCLUSIVE held for 1254 ms
[2011-01-16 15:14:55.107 00728 warning 'VpxdMoLock'] ***WARNING*** Lock host-10 mode EXCLUSIVEALL held for 1254 ms

[2011-01-16 15:14:56.279 00728 error 'App'] Win32 exception: Stack overflow (0xc00000fd)
[2011-01-16 15:14:56.279 00728 error 'App']    eip: 0x197e1fb esp: 0x49f2fcc ebp: 0x49f3040
[2011-01-16 15:14:56.279 00728 error 'App']    eax: 0x49f5064 ebx: 00000000 ecx: 0x1302d48 edx: 0xfffffffe edi: 0x3ddd038 esi: 0x000007

Not sure what brought all this about.

Any guidance is most apprciated.

Thanks.

-Ed

Reply
0 Kudos
7 Replies
EVW
Enthusiast
Enthusiast

User Account that is responsible for starting this service might be locked out. Check this service account in Active Directory to see if it is locked out.

Also check to following KB: http://kb.vmware.com/kb/1003346

Reply
0 Kudos
ejm381
Contributor
Contributor

Thanks.

"VMware VirtualCenter Server" service is using "Local System" account so AD lockout not a possibility for my situation.

I don't believe the KB would apply in this case since this is a fresh Vcenter install, never upgraded (4.1 is the first version we've used). Prior to this Vcenter install we were managing hosts individually.

Reply
0 Kudos
AureusStone
Expert
Expert

Hey

Have a look at this KB.

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=102983...

That seems to be your problem.  I haven't encountered this issue, so I can't do more then point you to the KB. Smiley Happy

Reply
0 Kudos
idle-jam
Immortal
Immortal

is the mssql service started? you will need it to be started before the vcenter can run,

Reply
0 Kudos
ejm381
Contributor
Contributor

Thanks.

Not running DRS, just HA. KB doesn't seem to apply to my issue.

Reply
0 Kudos
ejm381
Contributor
Contributor

Thanks,

SQL service is started and DB seems fine. I connected and expanded DB initially as I expected low DB space. SQL DB doesn't appear to be the problem...at least it's not obviously the problem.

Reply
0 Kudos
ejm381
Contributor
Contributor

I believe my issue was ultimately caused by having too many snapshots (65) on one particular VM. Now, it's not normal for us to have anywhere near that many snaps...ever. I believe this was caused by VDR (Vmware Data Recovery), Double-Take For Vmware, or a combination of both. To make matters worse, we were low on disk space on the LUN containing the troubled VM, so when we deleted the snaps using snapshot manager, the snaps were orphaned (not listed anymore in Vsphere, but still there if viewed via SSH).

We had been using Double-Take to do some daily backups for a few key production VM's. We installed/configured VDR 1.2 and started to use that to do similair things. I think these two tools may have been competing to handle snaps for the same VM....and ultimately caused the snaps to not be removed after backup. I know VDR had issues with not cleaning up snaps, but I thought the issue was resolved with 1.2, hence my suspicion that it might have been caused by a comination of things.

So, to recover we cloned the troubled VM from the command line as a quick way to bring together all the snaps to a new VM with a single (up to date) VMDK..."vmkfstools -i <source vmdk> <destination vmdk>".

Once we spun the new VM up and confirmed it was fine we deleted the old VM entry (all files) via WINSCP. Rebooted Vcenter and were able to start all services and connect to it from Vsphere.

Double-Take has been retired and VDR is back up and running for now...but I'm seriously considering taking a look at the other Vm Backup products on the market.

Reply
0 Kudos