VirtualCenter Server keeps 'dying'

[zaanpenguin]

Hello everyone,

We've been running 3 VMware hosts, 1 iSCSI SAN-box and 1 VC for management, all on ESX 3.5. The VC machine becomes unresponsive within a short period of time of 'being online', with the following symptoms:

- Webserver webaccess works, but after entering login/password and clicking login it hangs

- Logging in with VI client takes a while (mouse cursor changes a few times) and then still gives a timeout error

- The VMware VirtualCenter Service is running on the machine, stopping and starting it brings it back to life but it dies again after a short while

- The machine is not out of disk space (18+ GB free)

- SQLEXPRESS service is running

- Network connectivity is fine, I can Remote Desktop in and do the above step of Stopping/Starting the service to fix it, but it's unworkable to do this every hour

- Sometimes the VC will become unresponsive when being logged in and doing work, creating sync issues between what's on the hosts etc.

I've checked the Tomcat directory's logfiles, but most files are empty (0 bytes) or don't contain errors.

Can anyone give me some pointers on how to go about fixing this? Without the VC administration is rather 'time-intensive'

TIA, A. Eijkhoudt

[runclear]

Whats the Resources look like on the VC server itself (ie: memory and cpu) , not that it matters but are we talking about a VM vc or a Physical VC box?

-

---

[zaanpenguin]

It's an older HP ML server, dual Xeon (P3), 2.3GB RAM, 2x36GB HW RAID1 boot, 8x36GB HW RAID5 storage, dual NIC. A physical machine therefore, with no other tasks than running the VC environment.

[runclear]

I've personally never ran VC on a box that old (P3), but im curious if that could be the performance bottleneck... obviously after troubleshooting everything else... id spin that box into a VM and see if it performs any better

Unless your hosts are equally configured like your VC hardware.... ?

-

---

[zaanpenguin]

The 3 hosts and storage server are modern HP boxes (we run 100-150 VMs problem-free). The management box should be more than capable of running VC, it's barely breaking a sweat either CPU- or memory-wise: 2-20% CPU, more than 1.5GB of RAM available. In the past it was running fine, something has gone 'wrong' with the VC box and I have no idea where to look (which reminds me: the Event Viewer doesn't seem to contain any VC-related errors either).

The VirtualCenter Service needs restarts to keep it working.

[DSTAVERT]

How big is the database. SQLExpress has a limit make sure you haven't exceeded it. 4GB I think.

[zaanpenguin]

From what I can tell with SQLEXPRESS Management Studio (sorry I'm not a frequent user of MS SQL Express 2005) , it's 486MB, 5MB free, unrestricted growth is enabled for both the VMDB and VMDB_log.

[DSTAVERT]

OS updates??

VMware updates??

HP firmware updates??

Test hardware??

This is just going to be the elimination of things that aren't causing the problem. Luckily you get to do the eliminating.

[zaanpenguin]

I did some more research, and it seems that the problem is/was related to VPXD.EXE taking up 100%. It apparently does this because VirtualCenter overloads SQLExpress with statistics gathering. We don't use this, so I disabled the setting in VC last night and the CPU usage immediately dropped. Also, the VC environment was still responsive this morning...

So, crossing my fingers this solved the problem.

[DSTAVERT]

Have a look at http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=100625...

[zaanpenguin]

Thank you! I added that option as well and will see if the VC will stay up ...

[DSTAVERT]

Whatever you don't uncross your fingers. If you are already at VirtualCenter 2.5 Update 2 you might want to register an SR

[Chamon]

I think your issue is SQL Express. It is meant for SMB with up to 50 VMs you have 100-150. Take a look here for some more information.

http://vmetc.com/2009/01/02/why-not-virtualcenter-with-sql-2005-express/