Hi,
We run VMware vCenter 4.1 with 3 ESX hosts, and 50 VMs. vCenter is running on Windows 2008 R2, with SQL 2005 Express SP2.
Yesterday our virtual backup failed, saying it couldn't connect to the vCenter agent. When I looged into the issue, it appeared that the vCenter Server service had stopped. Up on further investigation it appears that the SQL Express database had reached 4GB in size, which is the limited, and this caused the vCenter service to fall over.
I've checked some forum posts, and have performed the following:
However, even after all of the above, when I restart the server, or stop & start the vCenter service, the following errors are logged in the application log:
-------------------------------------------
Log Name: Application
Source: MSSQL$SQLEXP_VIM
Date: 16/08/2011 08:51:44
Event ID: 1827
Task Category: (2)
Level: Error
Keywords: Classic
User: SYSTEM
Description:
CREATE DATABASE or ALTER DATABASE failed because the resulting cumulative database size would exceed your licensed limit of 4096 MB per database.
-------------------------------------------
Log Name: Application
Source: MSSQL$SQLEXP_VIM
Date: 16/08/2011 08:51:44
Event ID: 1105
Task Category: (2)
Level: Error
Keywords: Classic
User: SYSTEM
Description:
Could not allocate space for object 'dbo.VPX_BINARY_DATA'.'PK_VPX_BIN_DATA' in database 'VIM_VCDB' because the 'PRIMARY' filegroup is full. Create disk space by deleting unneeded files, dropping objects in the filegroup, adding additional files to the filegroup, or setting autogrowth on for existing files in the filegroup.
-------------------------------------------
Log Name: Application
Source: VMware VirtualCenter Server
Date: 16/08/2011 08:51:44
Event ID: 1000
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Description:
The description for Event ID 1000 from source VMware VirtualCenter Server cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.
If the event originated on another computer, the display information had to be saved with the event.
The following information was included with the event:
[Ldap] Failed to save LDAP backup data to database: "ODBC error: (42000) - [Microsoft][SQL Native Client][SQL Server]Could not allocate space for object 'dbo.VPX_BINARY_DATA'.'PK_VPX_BIN_DATA' in database 'VIM_VCDB' because the 'PRIMARY' filegroup is full. Create disk space by deleting unneeded files, dropping objects in the filegroup, adding additional files to the filegroup, or setting autogrowth on for existing files in the filegroup." is returned when executing SQL statement "INSERT INTO VPX_BINARY_DATA WITH (ROWLOCK) (ID, DATA_TYPE, BIN_DATA, CREATED_TIME, CHANGE_ID) VALUES (?, ?, ?, ?, ?)"
the message resource is present but the message is not found in the string/message table
-------------------------------------------
Can anyone provide another solution to this issue? We need to be able to perform a successful backup!
Any help or suggestions appreciated!
Ben
Due to the size of the database the query may take some time. I'd suggest you let it run and just be patient.
Other option are to migrate to a "full" SQL Server 2005/2008 version (see http://kb.vmware.com/kb/7960893), or - in case this is an option for you - to migrate to the currently experimentally supported (starting with vCenter Server 4.1 Update 1) SQL Express 2008 version which supports up to 10 GB per database. (see http://www.vmware.com/pdf/vsphere4/r40/vsp_compatibility_matrix.pdf) Although it states 32 and 64 bit in the compatibility matrix, it looks like there are currently issues with the 64-bit edition (see http://kb.vmware.com/kb/1036085)
André
Just to make sure, did you follow exactly the steps in http://kb.vmware.com/kb/1025914
From what you mentioned, I assume you did, however just want to make sure before digging deeper.
André
Hi,
Thanks for the reply.
I did everything mentioned in the KB article. The max settings in dbo.VPX_PARAMETER were already set, but executing the dbo.cleanup_events_tasks_proc just seemed to run forever, 25 minutes and it was still showing 'Executing query...', the Results pane didn't show anything, and the file size hadn't decreased at all. So not sure if it actually did anything!
Ben
Due to the size of the database the query may take some time. I'd suggest you let it run and just be patient.
Other option are to migrate to a "full" SQL Server 2005/2008 version (see http://kb.vmware.com/kb/7960893), or - in case this is an option for you - to migrate to the currently experimentally supported (starting with vCenter Server 4.1 Update 1) SQL Express 2008 version which supports up to 10 GB per database. (see http://www.vmware.com/pdf/vsphere4/r40/vsp_compatibility_matrix.pdf) Although it states 32 and 64 bit in the compatibility matrix, it looks like there are currently issues with the 64-bit edition (see http://kb.vmware.com/kb/1036085)
André
Hi,
I let dbo.cleanup_events_tasks_proc run again, took a total of 28 minutes before returning 1 row of '0'. After that I performed a database shrink again, and the size went from 4GB to 1.5GB, so this appears to have been successful.
I'm just going to perform a reboot of the server, and try running a backup of the virtual infrastructure before marking the thread as resolved, but it's looking hopeful!
Many thanks for your help!
Ben