VMware Cloud Community
ccastellow
Enthusiast
Enthusiast

VirtualCenter service stops during SQL2005 cluster failover

I've noticed this on a few occasions, where the VirtualCenter server shuts down during a failover of the SQL2005 cluster that hosts the VC database. The following events are logged:

Event Type: Error

Event Source: VMware VirtualCenter Server

Event Category: None

Event ID: 1000

Date: 8/7/2009

Time: 1:15:30 PM

User: N/A

Computer:

Description:

The description for Event ID ( 1000 ) in Source ( VMware VirtualCenter Server ) cannot be found. The local computer may not have the necessary registry information or message DLL files to display messages from a remote computer. You may be able to use the /AUXSOURCE= flag to retrieve this description; see Help and Support for details. The following information is part of the event: An unrecoverable problem has occurred, stopping the VMware VirtualCenter service. Check database connectivity before restarting. Error: Error[VdbODBCError] (-1) "ODBC error: (42000) - [SQL Native Client][SQL Server]SHUTDOWN is in progress." is returned when executing SQL statement "UPDATE VPX_ENTITY SET NAME = ? , TYPE_ID = ? , PARENT_ID = ? WHERE ID = ?".

...and then:

Event Type: Information

Event Source: VMware VirtualCenter Server

Event Category: None

Event ID: 1000

Date: 8/7/2009

Time: 1:15:30 PM

User: N/A

Computer:

Description:

The description for Event ID ( 1000 ) in Source ( VMware VirtualCenter Server ) cannot be found. The local computer may not have the necessary registry information or message DLL files to display messages from a remote computer. You may be able to use the /AUXSOURCE= flag to retrieve this description; see Help and Support for details. The following information is part of the event: Forcing shutdown of VMware VirtualCenter now.

The service stays down until it is manually restarted. The service is configured to restart upon a failure, but if the service is being shutdown (technically not a failure, right?) then this does little to help. Ideas or suggestions would be greatly appreciated.

Thanks

Tags (2)
Reply
0 Kudos
9 Replies
azn2kew
Champion
Champion

You may want to try these troubleshooting steps to see if it helps

If you found this information useful, please consider awarding points for "Correct" or "Helpful". Thanks!!!

Regards,

Stefan Nguyen

VMware vExpert 2009

iGeek Systems Inc.

VMware, Citrix, Microsoft Consultant

If you found this information useful, please consider awarding points for "Correct" or "Helpful". Thanks!!! Regards, Stefan Nguyen VMware vExpert 2009 iGeek Systems Inc. VMware vExpert, VCP 3 & 4, VSP, VTSP, CCA, CCEA, CCNA, MCSA, EMCSE, EMCISA
Reply
0 Kudos
AndreTheGiant
Immortal
Immortal

With old version of VC 2.x a minimal timeout in SQL service will stop the entire VC service.

Are you using VC 2.5U4 or U5?

How does it take a SQL failover?

Andre

Andrew | http://about.me/amauro | http://vinfrastructure.it/ | @Andrea_Mauro
Reply
0 Kudos
ccastellow
Enthusiast
Enthusiast

I am using VC 2.5U4. I believe the SQL failover takes appoximately 30 seconds for the database to become available again on the new active node. Is there a way to increase the timeout value?

Reply
0 Kudos
AndreTheGiant
Immortal
Immortal

Is there a way to increase the timeout value?

Sorry, no idea.

Maybe there is some ODBC configuration, but in VC I do not know how to change the DB timeout value.

Andre

Andrew | http://about.me/amauro | http://vinfrastructure.it/ | @Andrea_Mauro
cnetbuild1
Contributor
Contributor

I have the exact same problem. Any direction to solving this would be appreciated.

Reply
0 Kudos
ccastellow
Enthusiast
Enthusiast

I finally contacted VMware Technical Support regarding this issue. I was told that the VirtualCenter application is shutting down the VPXD service after the connection to the database is lost. There is no way to modify the timeout value from VirtualCenter. Because the service is stopped, and is technically not a service failure or crash, the recovery actions that are configured for the service are not executed. In other words, if you have the service set to automatically restart after a failure or crash, a restart will not be attempted as the service is being shutdown by the application itself.

Right now, I am exploring having SCOM 2007 monitor this service, and have a recovery action setup to attempt to restart it. Additionally, I will explore using a script to monitor the service or set up an event trigger.

Reply
0 Kudos
embo500
Contributor
Contributor

I am testing a way around this by triggering a powershell script based on EventID 1000 from source VMWare VirtualCenter Server in the Event Log. When an event from vCenter with ID 1000 is logged, I fire off this powershell script that looks for the "ODBC error" and "SHUTDOWN is in progress" phrases, and restarts vpxd after 30 seconds if it finds them (this way Info event 1000's won't do anything).

Here's the code if it's useful to anyone:

-


$logentry = Get-EventLog -LogName Application | Where {$_.EventId -eq "1000"} | Where {$_.Source -eq "VMWare VirtualCenter Server"} | Select -First 1

if ($logentry.Message -match "ODBC error")

{

if ($logentry.Message -match "SHUTDOWN is in progress")

{

Start-Sleep -s 30

Start-Service vpxd

}

}

-


Reply
0 Kudos
IB_IT
Expert
Expert

Exact same problem here.  I have searched high and low for a solution to this in VI3....since we are moving to vsphere 4.1, I am wondering if anyone knows if there are any DB timeout values configurable in that infrastructure?

Reply
0 Kudos
mark_chuman
Hot Shot
Hot Shot

Just wanted to add that we see the same type behavior (VC service initiates shutdown during remote SQL cluster fail-over) and have verified with VMware that there is no way to adjust the timeout value on VC (as of 4.0, U1).  Similar to other solutions mentioned in the post we have addressed the problem with an automated restart process integrated into VC monitoring/alert steps.

Reply
0 Kudos