VCSA DB full 100% disk use VPXD not running

[muetstem]

Our vSphere 5 uses the VCSA with de embedded database. After a while the mountpoint /storage/db on partition /dev/sdb3 gets 100% full (20GB partition).

vmware-vpxd is stoped and we cannot connect with the vSphere client to the vcenter server (that makes sense of course)
When we look into the DB2 database, we see a big file on /storage/db/db2/home/db2inst1/db2inst1/NODE0000/VCDB/T0000002/C0000000.LRG

We replaced this file with a empty one, but I cannot start the database after that action. We restored the file from a backup.

We have 4 ESXI5 servers and 60 virtual machines. When we use vSphere 5 for the first time we increased the log parameters and used the calulator for predicting the size of the database (VMware gives 16GB) now our volume is full.

Is there a way to increase the database, or does vmware-vpxd not a clean-up from the database?

[muetstem]

A few things that I have discovered: I made a call with technical support VMware:

Our statiscis level was to high, level 4, we brought it back to level 1 or 2.

The calculations for the size of de DB was not correct producing losts of MB's in the database.

The problem is still not solved. To solve this I restored a backup of our VCSA, the size of which was /dev/sdb3 about 50% of the volume.

After a while the volume size increased to 75% around 14GB which I think is still to much, because of the statistics level that was set to 0 (turned off).

Then I followed the advice in the KB 2007388

I run truncate table vc.vpx_hist_stat1 truncate table vc.vpx_sample_time1 (and repeated for stat2,3 and 4 / time2,3 and 4)

After truncating these tables the disk space was not given back as free space.

Looking in the tables I see the folowing:

TSIZE Table size in bytes.

Table: VC.VPXV_DEVICE_COUNTER tsize  2351583

Table: VC.VPX_EVENT tsize tsize 13439815

Table: VC.VPX_EVENT_ARG tsize 15995147

Table: VC.VPX_STAT_COUNTER tsize 1839357

Table: VC.VPX_TOPN_PAST_DAY tsize 9819056

Total volume is 20GB....where does the space go??

Can you please give me some advice as how to solve this issue and release the extra disk space.

[rverchere]

Hello,

I get the same issue last week.

I made a quick and dirty workaround to continue using VCSA, before having a clean solution :

- I created a 3rd virtual disk with 50 GB size

- I copied the /storage/db directory to that new disk

It works, but I will have the same problem in few weeks when the 50GB drive will be full. Meanwhile I can continue using VCSA and try to find a proper solution.

If you need more details or help, feel free to ask.

I also updated the vcenter server parameters to store only 30 days of logs in the database, but no differences yet.