vcenter 5.5 service fails

[Pascalloz]

Hi all,

since a couple of days our vcenter is reporting the following event:

The description for Event ID 1000 from source VMware VirtualCenter Server cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

If the event originated on another computer, the display information had to be saved with the event.

The following information was included with the event:

An unrecoverable problem has occurred, stopping the VMware VirtualCenter service. Error: Error[VdbODBCError] (-1) "ODBC error: (40001) - [Microsoft][SQL Server Native Client 10.0][SQL Server]Transaction (Process ID 88) was deadlocked on lock | communication buffer resources with another process and has been chosen as the deadlock victim. Rerun the transaction." is returned when executing SQL statement "SELECT MO_ID, MO_TYPE, TYPE_ID, ARRAY_ID, VALUE FROM VPX_TEXT_ARRAY WHERE MO_TYPE = ? AND MO_ID = ?"

the message resource is present but the message is not found in the string/message table

If i reboot the vcenter server all the services are starting normally also the vcenter server service, and after a x period of time, the service crashes and give us above error message. And we are unable to connect with the client

[Alim786]

Have you got latest ESXi updates ?

Are you on ESXi 5.5 Update 1 ?

[Pascalloz]

yes i am, i installed it yesterday. we had the problems before the update, and after the update

[Alim786]

Was this a new install of 5.5 or an upgrade ?

[Pascalloz]

Everything is build up from scratch 3 months ago. So yes it a new install, on a new system.

And everything was running perfectly till a couple of weeks ago.

[Alim786]

It obviously points to a SQL Server issue. I am not a SQL DB person. I would just check that the ODBC connection is ok. I would look at the SQL server DB and check to see if any obvious errors. As it is 5.5 and not come across this before, logging a case with VMware might be the best course of action unless someone else can contribute.

[Pascalloz]

true, but i saw on the internet that there where people who had simmalir problems only with other versions of vcenter.

but i have look, and like you i am not a sql admin, so i have no idea how to solve the problem. Because from my point of view i also suspect the issue is in the database and not in the application of vcenter. We are running on SQL 2008 R2

And we also have another error message in the event viewer:

Faulting application name: vpxd.exe, version: 5.5.0.40799, time stamp: 0x5307e9e2

Faulting module name: KERNELBASE.dll, version: 6.1.7601.18229, time stamp: 0x51fb1677

Exception code: 0xe06d7363

Fault offset: 0x000000000000940d

Faulting process id: 0x105c

Faulting application start time: 0x01cf4a753a553975

Faulting application path: D:\VMware\Infrastructure\VirtualCenter Server\vpxd.exe

Faulting module path: C:\Windows\system32\KERNELBASE.dll

Report Id: 473f66fa-b669-11e3-a3d8-000c2946f5be

[rh5592]

check the vCenter log file location and see if there are vpxd crash dump files. If you are seeing them everytime your vCenter service crashes then you have to open a ticket with Vmware support to have it investigated. I have a few open SR's with support also for vCenter 5.5 crash but the symptoms are not the same as the one you are experiencing.

[Pascalloz]

Well i have the same problem again. After the first time we had the problem i decided that to re-install vcenter, since i had documented the whole process. I deleted everything from the vcenter vm and re-installed everything with a new database. (i only kept the vum database. )

and for the first two weeks everything was running fine. And then it happens again, in the beginning it happend once, and then a couple of days nothing, and then it started to happen more frequently.at the point that now it happens 10mins after i restarted the vcenter server.

I created a call with vmware, but they only asked me some basic questions (how many hosts and vms, database size, log retention.) and that i should check with my db admin... which we dont have...

And yes there are vpxd cash dump files.

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

The description for Event ID 1000 from source VMware VirtualCenter Server cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

If the event originated on another computer, the display information had to be saved with the event.

The following information was included with the event:

An unrecoverable problem has occurred, stopping the VMware VirtualCenter service. Error: Error[VdbODBCError] (-1) "ODBC error: (40001) - [Microsoft][SQL Server Native Client 10.0][SQL Server]Transaction (Process ID 81) was deadlocked on lock | communication buffer resources with another process and has been chosen as the deadlock victim. Rerun the transaction." is returned when executing SQL statement "SELECT MO_ID, MO_TYPE, TYPE_ID, ARRAY_ID, VALUE FROM VPX_TEXT_ARRAY WHERE MO_TYPE = ? AND MO_ID = ?"

the message resource is present but the message is not found in the string/message table

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Faulting application name: vpxd.exe, version: 5.5.0.40799, time stamp: 0x5307e9e2

Faulting module name: KERNELBASE.dll, version: 6.1.7601.18229, time stamp: 0x51fb1677

Exception code: 0xe06d7363

Fault offset: 0x000000000000940d

Faulting process id: 0x1538

Faulting application start time: 0x01cf581553472e9b

Faulting application path: D:\VMware\Infrastructure\VirtualCenter Server\vpxd.exe

Faulting module path: C:\Windows\system32\KERNELBASE.dll

Report Id: a838f82b-c408-11e3-b60d-000c2946f5be

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

The description for Event ID 1000 from source VMware VirtualCenter Server cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

If the event originated on another computer, the display information had to be saved with the event.

The following information was included with the event:

Starting VMware VirtualCenter 5.5.0 build-1623101

the message resource is present but the message is not found in the string/message table

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

[kashifkarar01]

Hello Pascalloz ,

Please check if you have IIS service started in your vCenter installed VM. If yes, please stop that service and start the vCenter server service.

Also, please download SQL Management Studio from Microsoft:

http://social.msdn.microsoft.com/Forums/sqlserver/en-US/4e5d5374-9422-4699-b6c8-fe72e96bb533/microso...  

Then follow the instructions as mentioned in the following KB:

kb.vmware.com/kb/1000455

If the answer is helpful or correct please mark it !!!!!

[Spencer_Faulkne]

Check the resource utilization on your SQL server just before the incident occurs. If this looks like a memory leak it is likely a 3rd party software connecting to the vCenter Server.  Check the sessions inside vCenter to try and find out what account the tool is connecting. Check plugins on all the VI clients.