vcenter server stopped working after reboot

[bdabhs]

Our vcenter server seems to have stopped working after we rebooted it this morning.  It has been up and running for over a year now and we've rebooted it several times since then without any problems.  When we try to open the vsphere client which is on the vcenter server itself we get an error that says "The vsphere client could not connect to. The server took too long to respond."

I tested the ODBC and it tested fine.  The SQL database is on another server and not hosted locally to this server.  I would reboot the SQL server but I can't right now because I have other DB's running.

I had a look at the vpxd logs and noticed the following errors:

[2012-06-21 10:57:31.222 03200 error 'App'] [Vdb::IsRecoverableErrorCode] Unable to recover from HY010:0

[2012-06-21 10:57:31.222 03200 error 'App'] [VdbStatement] SQL execution failed: INSERT INTO VPX_BINARY_DATA WITH (ROWLOCK) (ID, DATA_TYPE, BIN_DATA, CREATED_TIME, CHANGE_ID) VALUES (?, ?, ?, ?, ?)
[2012-06-21 10:57:31.222 03200 error 'App'] [VdbStatement] Execution elapsed time: 16 ms
[2012-06-21 10:57:31.222 03200 error 'App'] [VdbStatement] Diagnostic data from driver is HY010:1:0:[Microsoft][SQL Native Client]Function sequence error
[2012-06-21 10:57:31.222 03200 error 'App'] [VdbStatement] Bind parameters:
[2012-06-21 10:57:31.222 03200 error 'App'] [VdbStatement] datatype: 1, size: 4, arraySize: 0
[2012-06-21 10:57:31.222 03200 error 'App'] [VdbStatement] value = 16
[2012-06-21 10:57:31.222 03200 error 'App'] [VdbStatement] datatype: 11, size: 30, arraySize: 0
[2012-06-21 10:57:31.222 03200 error 'App'] [VdbStatement] value = "Ldap.BackupData"
[2012-06-21 10:57:31.222 03200 error 'App'] [VdbStatement] datatype: 12, size: -50372847, arraySize: 0
[2012-06-21 10:57:31.222 03200 error 'App'] [VdbStatement] datatype: 10, size: 23, arraySize: 0
[2012-06-21 10:57:31.222 03200 error 'App'] [VdbStatement] datatype: 11, size: 10, arraySize: 0
[2012-06-21 10:57:31.222 03200 error 'App'] [VdbStatement] value = "45067"

---------------------------------------------------------------------------------------------------------------------

[2012-06-21 10:57:31.222 03200 error 'App'] [Vdb::IsRecoverableErrorCode] Unable to recover from HY010:0

----------------------------------------------------------------------------------------------------------------------

[2012-06-21 11:02:03.329 03868 error 'App'] [LDAP Client] [LDAP Client] Failed to delete LDAP entryCN=cim-ui,CN=53CF3AE6-748D-48B5-A661-4735DEDD47B4,OU=ComponentSpecs,OU=Health,dc=virtualcenter,dc=vmware,dc=int: 0x20 (Object does not exist.)
[2012-06-21 11:02:03.334 03868 error 'App'] [VpxdHealth::UnregisterHealthUrl] LDAP problem. Unable to unregister health spec for cim-ui
[2012-06-21 11:02:03.421 03868 error 'App'] [LDAP Client] [LDAP Client] Failed to delete LDAP entryCN=com.vmware.vim.stats.report,CN=53CF3AE6-748D-48B5-A661-4735DEDD47B4,OU=ComponentSpecs,OU=Health,dc=virtualcenter,dc=vmware,dc=int: 0x20 (Object does not exist.)
[2012-06-21 11:02:03.421 03868 error 'App'] [VpxdHealth::UnregisterHealthUrl] LDAP problem. Unable to unregister health spec for com.vmware.vim.stats.report
[2012-06-21 11:02:03.422 03868 error 'App'] [LDAP Client] [LDAP Client] Failed to delete LDAP entryCN=health-ui,CN=53CF3AE6-748D-48B5-A661-4735DEDD47B4,OU=ComponentSpecs,OU=Health,dc=virtualcenter,dc=vmware,dc=int: 0x20 (Object does not exist.)
[2012-06-21 11:02:03.422 03868 error 'App'] [VpxdHealth::UnregisterHealthUrl] LDAP problem. Unable to unregister health spec for health-ui
[2012-06-21 11:02:03.422 03868 error 'App'] [LDAP Client] [LDAP Client] Failed to delete LDAP entryCN=hostdiag,CN=53CF3AE6-748D-48B5-A661-4735DEDD47B4,OU=ComponentSpecs,OU=Health,dc=virtualcenter,dc=vmware,dc=int: 0x20 (Object does not exist.)
[2012-06-21 11:02:03.423 03868 error 'App'] [VpxdHealth::UnregisterHealthUrl] LDAP problem. Unable to unregister health spec for hostdiag
[2012-06-21 11:02:03.423 03868 error 'App'] [LDAP Client] [LDAP Client] Failed to delete LDAP entryCN=license-ui,CN=53CF3AE6-748D-48B5-A661-4735DEDD47B4,OU=ComponentSpecs,OU=Health,dc=virtualcenter,dc=vmware,dc=int: 0x20 (Object does not exist.)
[2012-06-21 11:02:03.423 03868 error 'App'] [VpxdHealth::UnregisterHealthUrl] LDAP problem. Unable to unregister health spec for license-ui
[2012-06-21 11:02:03.424 03868 error 'App'] [LDAP Client] [LDAP Client] Failed to delete LDAP entryCN=VirtualCenter,CN=53CF3AE6-748D-48B5-A661-4735DEDD47B4,OU=ComponentSpecs,OU=Health,dc=virtualcenter,dc=vmware,dc=int: 0x20 (Object does not exist.)
[2012-06-21 11:02:03.424 03868 error 'App'] [VpxdHealth::UnregisterHealthUrl] LDAP problem. Unable to unregister health spec for VirtualCenter

[Troy_Clavell]

do you have any event log errors on the Windows Host where you have vCenter installed?  Could it be a disk space issue on the SQL Server (transaction logs)

[bdabhs]

I'm getting the event ID 1000 for source VMware VirtualCenter Server with the following details "[Ldap] Failed to save LDAP backup data to database: "ODBC error: (HY010) - [Microsoft][SQL Native Client]Function sequence error" is returned when executing SQL statement "INSERT INTO VPX_BINARY_DATA WITH (ROWLOCK) (ID, DATA_TYPE, BIN_DATA, CREATED_TIME, CHANGE_ID) VALUES (?, ?, ?, ?, ?)"

As far as disk space issues I'm thinking that it could be that.  I have 2 vmware databases called VCDB & VCUPDATE:

the VCDB has 14% free for data and 98% free for logs.

the VCUPDATE has 0% free for data and 81% free for logs.

[GreatWhiteTec]

That would do it. Can you increase the size of the DBs? What version of SQL are you running? What recovery model are you running? Simple or Full?

[bdabhs]

We are running SQL 2005 and the recovery model is FULL for both DB's.  How do I go about increasing the size?

NOTE: the vcupdate data is set for autogrowth by 1 mb, unrestricted growth

          the vcupdate_log is set for autogrowth by 10 percent, restricted growth by 2,097,152(MB)

[Troy_Clavell]

I would start cleaning up your VCDB.  shrinking the transaction logs and use simple mode as your recovery model

http://kb.vmware.com/kb/1003980

[bdabhs]

I'm not sure that is the solution as we use BE2012 and make full backups followed by transaction backups, thus we have it set to Full.

[Troy_Clavell]

you should still shrink the database, or better yet do some DB maintenance.  (which would require downtime for vCenter)

http://kb.vmware.com/kb/1025914

[GreatWhiteTec]

You can increase the DB size by going to Mangement Studio>Right-click on the DB>Properties> Files> increase size for both DB and Log. You can also adjust the autogrowth values. in addition you can follow Troy's recommendation.

[bdabhs]

Any idea what the VCUPDATE database is?  I've increased the size so I now have 5% free disk space, but still can't get vcenter to work:smileyconfused:

[Troy_Clavell]

don't know... what is your ODBC connection pointing to?

[GreatWhiteTec]

I'm guessing that is your Update Manager if you made a separate DB for it during installation. You want to focus on the other DB.

[bdabhs]

It is pointing to VCDB.  I should also mention that the reason I rebooted in the first place was because I put 1 of our hosts in maintenance mode so I could reboot that host.  Once the host came back up, I went to take it out of maintenance mode but vcenter locked up on when trying to do so.  Ever since then, I've been unable to get back in.  The host is back up and running though, but the VM's that are on that host are not up.

[bdabhs]

what's also strange is that if I try to stop any of the services related to vcenter I get an error "could not stop.......error 1503 ther service didn't respond in a timely fashion"

[Troy_Clavell]

just our of curiosity you didn't install IIS or any other application that is using port 80?

With that said, focusing on your VCDB's cleanup is key, in my opinion.

[bdabhs]

nope...haven't done anything really to the server, not even windows udpates in the past 60 days.

[bdabhs]

Isn't there a way to check if the port is actually open and listening?

[Troy_Clavell]

http://kb.vmware.com/kb/1003971

[GreatWhiteTec]

The host is back up and running though, but the VM's that are on that host are not up.

The vCenter being down should not affect other VMs aside from the fact that vDS is down if you use it. Those VMs should be able to power on on the host itself. How many hosts do you have? Is the vCenter VM running on the Host that was worked on or a different host? Can  you provide more info on your set up?

[bdabhs]

just noticed something real strang in that when if i ping my server name from itself i get "Reply from ::1: time<1ms" Not sure why that is?  If I ping it from another server I get the usual IP address.