VMware Cloud Community
bjblackmore1982
Contributor
Contributor
Jump to solution

VMware vCenter service stops with SQL DB size errors

Hi,

We run VMware vCenter 4.1 with 3 ESX hosts, and 50 VMs. vCenter is running on Windows 2008 R2, with SQL 2005 Express SP2.

Yesterday our virtual backup failed, saying it couldn't connect to the vCenter agent. When I looged into the issue, it appeared that the vCenter Server service had stopped. Up on further investigation it appears that the SQL Express database had reached 4GB in size, which is the limited, and this caused the vCenter service to fall over.

I've checked some forum posts, and have performed the following:

  • Set a database retention policy for tasks & events, this was originally un-checked, but I have checked it, and set it to 90 days initially, and now 30 days.
  • Found a script on the VMware forum called VCDB_table_cleanup_MSSQL_V4.X.sql which can cleanup the database. I set it to delete data older (changing SET @DELETE_DATA = 1) than 90 days, and it removed 130,000+ rows.
  • Performed a database shrink & file shrink on the data file - this only got me back 9mb?
  • Performed a database shrink & file shrink on the log file - this got back a little more, about 100mb.
  • Changed the log file size from 460mb to 1024mb, in the database properties > files settings.

However, even after all of the above, when I restart the server, or stop & start the vCenter service, the following errors are logged in the application log:
-------------------------------------------
Log Name:      Application
Source:        MSSQL$SQLEXP_VIM
Date:          16/08/2011 08:51:44
Event ID:      1827
Task Category: (2)
Level:         Error
Keywords:      Classic
User:          SYSTEM
Description:
CREATE DATABASE or ALTER DATABASE failed because the resulting cumulative database size would exceed your licensed limit of 4096 MB per database.
-------------------------------------------
Log Name:      Application
Source:        MSSQL$SQLEXP_VIM
Date:          16/08/2011 08:51:44
Event ID:      1105
Task Category: (2)
Level:         Error
Keywords:      Classic
User:          SYSTEM
Description:
Could not allocate space for object 'dbo.VPX_BINARY_DATA'.'PK_VPX_BIN_DATA' in database 'VIM_VCDB' because the 'PRIMARY' filegroup is full. Create disk space by deleting unneeded files, dropping objects in the filegroup, adding additional files to the filegroup, or setting autogrowth on for existing files in the filegroup.
-------------------------------------------
Log Name:      Application
Source:        VMware VirtualCenter Server
Date:          16/08/2011 08:51:44
Event ID:      1000
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Description:
The description for Event ID 1000 from source VMware VirtualCenter Server cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

If the event originated on another computer, the display information had to be saved with the event.

The following information was included with the event:

[Ldap] Failed to save LDAP backup data to database: "ODBC error: (42000) - [Microsoft][SQL Native Client][SQL Server]Could not allocate space for object 'dbo.VPX_BINARY_DATA'.'PK_VPX_BIN_DATA' in database 'VIM_VCDB' because the 'PRIMARY' filegroup is full. Create disk space by deleting unneeded files, dropping objects in the filegroup, adding additional files to the filegroup, or setting autogrowth on for existing files in the filegroup." is returned when executing SQL statement "INSERT INTO VPX_BINARY_DATA WITH (ROWLOCK) (ID, DATA_TYPE, BIN_DATA, CREATED_TIME, CHANGE_ID) VALUES (?, ?, ?, ?, ?)"
the message resource is present but the message is not found in the string/message table
-------------------------------------------

Can anyone provide another solution to this issue? We need to be able to perform a successful backup!

Any help or suggestions appreciated!

Ben

Reply
0 Kudos
1 Solution

Accepted Solutions
a_p_
Leadership
Leadership
Jump to solution

Due to the size of the database the query may take some time. I'd suggest you let it run and just be patient.

Other option are to migrate to a "full" SQL Server 2005/2008 version (see http://kb.vmware.com/kb/7960893), or - in case this is an option for you - to migrate to the currently experimentally supported (starting with vCenter Server 4.1 Update 1) SQL Express 2008 version which supports up to 10 GB per database. (see http://www.vmware.com/pdf/vsphere4/r40/vsp_compatibility_matrix.pdf) Although it states 32 and 64 bit in the compatibility matrix, it looks like there are currently issues with the 64-bit edition (see http://kb.vmware.com/kb/1036085)

André

View solution in original post

Reply
0 Kudos
4 Replies
a_p_
Leadership
Leadership
Jump to solution

Just to make sure, did you follow exactly the steps in http://kb.vmware.com/kb/1025914

From what you mentioned, I assume you did, however just want to make sure before digging deeper.

André

Reply
0 Kudos
bjblackmore1982
Contributor
Contributor
Jump to solution

Hi,

Thanks for the reply.

I did everything mentioned in the KB article. The max settings in dbo.VPX_PARAMETER were already set, but executing the dbo.cleanup_events_tasks_proc just seemed to run forever, 25 minutes and it was still showing 'Executing query...', the Results pane didn't show anything, and the file size hadn't decreased at all. So not sure if it actually did anything!

Ben

Reply
0 Kudos
a_p_
Leadership
Leadership
Jump to solution

Due to the size of the database the query may take some time. I'd suggest you let it run and just be patient.

Other option are to migrate to a "full" SQL Server 2005/2008 version (see http://kb.vmware.com/kb/7960893), or - in case this is an option for you - to migrate to the currently experimentally supported (starting with vCenter Server 4.1 Update 1) SQL Express 2008 version which supports up to 10 GB per database. (see http://www.vmware.com/pdf/vsphere4/r40/vsp_compatibility_matrix.pdf) Although it states 32 and 64 bit in the compatibility matrix, it looks like there are currently issues with the 64-bit edition (see http://kb.vmware.com/kb/1036085)

André

Reply
0 Kudos
bjblackmore1982
Contributor
Contributor
Jump to solution

Hi,

I let dbo.cleanup_events_tasks_proc run again, took a total of 28 minutes before returning 1 row of '0'. After that I performed a database shrink again, and the size went from 4GB to 1.5GB, so this appears to have been successful.

I'm just going to perform a reboot of the server, and try running a backup of the virtual infrastructure before marking the thread as resolved, but it's looking hopeful!

Many thanks for your help!

Ben

Reply
0 Kudos