Virtual Center 2.5 VirtualCenter Server service stopping problem

[cls55]

Hi,

On my VC2.5 server (Windows XP using sql2005 express from VC 2.5 install), I am having a problem with the VMware VirtualCenter Server Service stopping. When I see I have been kicked off (from the client error message) I go to the server and see that the service is stopped and I manually start it without a problem. This is causing problems with automated testing we are running.

I get the following, in order:

Event Type: Error
Event Source: VMware VirtualCenter Server
Event Category: None
Event ID: 1000
Date: 7/19/2008
Time: 3:48:24 AM
Description:
The description for Event ID ( 1000 ) in Source ( VMware VirtualCenter Server ) cannot be found. The local computer may not have the necessary registry information or message DLL files to display messages from a remote computer. You may be able to use the /AUXSOURCE= flag to retrieve this description; see Help and Support for details. The following information is part of the event: An unrecoverable problem has occurred, stopping the VMware VirtualCenter service. Check database connectivity before restarting. Error: Error[VdbODBCError] (-1) "ODBC error: (40001) - [SQL Native Client][SQL Server]Transaction (Process ID 54) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction." is returned when executing SQL statement "SELECT vpx_vm.id FROM vpx_vm LEFT JOIN vpx_ip_address ON vpx_vm.id = vpx_ip_address.entity_id WHERE (vpx_vm.ip_address = ? OR vpx_ip_address.ip_address = ?) AND vpx_vm.is_template = 0".

Then I get:

Event Type: Information
Event Source: VMware VirtualCenter Server
Event Category: None
Event ID: 1000
Date: 7/19/2008
Time: 3:48:24 AM
Description:
The description for Event ID ( 1000 ) in Source ( VMware VirtualCenter Server ) cannot be found. The local computer may not have the necessary registry information or message DLL files to display messages from a remote computer. You may be able to use the /AUXSOURCE= flag to retrieve this description; see Help and Support for details. The following information is part of the event: Forcing shutdown of VMware VirtualCenter now.

Then there are a few MSSQL messages:

Event Type: Information
Event Source: MSSQL$SQLEXP_VIM
Event Category: (2)
Event ID: 2803
Date: 7/19/2008
Time: 3:48:25 AM
Description:
SQL Server has encountered 1 occurrence(s) of cachestore flush for the 'Object Plans' cachestore (part of plan cache) due to some database maintenance or reconfigure operations.
Event Type: Information
Event Source: MSSQL$SQLEXP_VIM
Event Category: (2)
Event ID: 2803
Date: 7/19/2008
Time: 3:48:25 AM
Description:
SQL Server has encountered 1 occurrence(s) of cachestore flush for the 'SQL Plans' cachestore (part of plan cache) due to some database maintenance or reconfigure operations.
Event Type: Information
Event Source: MSSQL$SQLEXP_VIM
Event Category: (2)
Event ID: 2803
Date: 7/19/2008
Time: 3:48:25 AM
Description:
SQL Server has encountered 1 occurrence(s) of cachestore flush for the 'Bound Trees' cachestore (part of plan cache) due to some database maintenance or reconfigure operations.

Any ideas?

[Rubeck]

I would start with trying to install and run VC on a supported host OS instead of WinXP..... Really.., no need to troubleshoot if the host OS platform isn't supported, IMO.

/Rubeck

[Rubeck]

Ohhh.. It does run on XP.. Sorry about that, mate.

/Rubeck

[jasonca36]

I just encountered the same exact issue on a machine running Windows 2003 Server. The DB is also on that server as it is our lab environment.

[aldikan]

Hi,

Was it always like this or started to happen recently?

Trying to link it to the lates updates done to VC OS, or possibly antivirus...

When you restart the service, how long will it take for problem to appear?

Also, please specify if your VC is virtual or physical box.

Alex

[jasonca36]

It started to happen recently. Since it is a lab, I do have users that also connect to their ESX hosts via the client as root, but that only seemed to cause issues in old versions of vmware. This is also a machine that was upgraded from 2.0. During the upgrade I switched to the built in SQL Express client and removed SQL 2005. After that, it had run well for quite some time.

When I restart the service, things are great for a day or so. Then I will start to see performance issues and eventually the connection to the console will time out saying it took too long for the machine to respond. There is also an application message in event viewer that my database restarted when I get the SQL Flush Error. Weekends, or when no one is on, it never happens.

[jasonca36]

Forgot...It's a physical box 2.5 GIGS, dual cpu & I did try stopping AntiVirus when it first started happening. Thanks.

[aldikan]

Hi Jason,

In your case there might be an issue with SQL express, not keeping up with the load.

How many guests/hosts you have in the lab?

[jasonca36]

I keep between 5-6 hosts in the lab and each one has anywhere from 3-15 vm's running. No VMotion, very basic.

[espi3030]

This has happened to me from time to time. What I was told is the transaction log is full, so then our DBA's did some cleaning up on the database and this cleared up the problem for me. So I would definitely have someone looak at your database settings.

Hope this helps!

[Rockapot]

The ESX Server 3 and VirtualCenter Installation Guide states that SQL Express can be used for environments up to 5 Hosts and 50 VM's so you are probably reaching the limit with SQL express usage irrespective of the issue you are facing now in the environment.

Carl

[mike_laspina]

Hi,

Do you have any maintenance tasks running?

Is someone accessing the DB outside of the VC process?

Dead locks can only occur when two independant querys are waiting on each other and niether can continue.

So look for active connections performing queries using SQL Studios management console.