7.0U1: every 2 mins: lots of Events "vCenter xxx File System Resource status changed from Yellow to Green"

[UThomas]

Hello,

after updating my VCSA from 6.7U3 to 7.0U1 I get every two minutes the same events (vim.event.ResourceExhaustionStatusChangedEvent)

vCenter xxx File System Resource status changed from Yellow to Green on ...

I can't see any change from green to yellow.

Are there any hints?

regards,

Ulrich

[Lalegre]

Hey UThomas​,

This could be an issue regarding the Linux partitions to be higher in used space than the Threshold defined by the product. If this is happening for all the products and changing constantly between Green and Yellow I assume this could be because of logs increasing and then the rotation cleaning them continuously.

Could you please log inside your VCSA using SSH and list the Filesystems used space by running df -h

[UThomas]

Hi,

the appliance type before upgrade was small/default and I changed that to small/large.

Filesystem                                Size  Used Avail Use% Mounted on
devtmpfs                                  9.3G     0  9.3G   0% /dev
tmpfs                                     9.4G  968K  9.4G   1% /dev/shm
tmpfs                                     9.4G  1.2M  9.4G   1% /run
tmpfs                                     9.4G     0  9.4G   0% /sys/fs/cgroup
/dev/sda3                                  46G  5.2G   39G  12% /
/dev/sda2                                 120M   27M   85M  24% /boot
/dev/mapper/vtsdblog_vg-vtsdblog           15G   57M   14G   1% /storage/vtsdblog
/dev/mapper/lifecycle_vg-lifecycle         98G  3.6G   90G   4% /storage/lifecycle
/dev/mapper/log_vg-log                    9.8G  743M  8.6G   8% /storage/log
/dev/mapper/seat_vg-seat                  541G  735M  513G   1% /storage/seat
/dev/mapper/dblog_vg-dblog                 15G  441M   14G   4% /storage/dblog
/dev/mapper/vtsdb_vg-vtsdb                541G  103M  513G   1% /storage/vtsdb
/dev/mapper/netdump_vg-netdump            985M  2.5M  915M   1% /storage/netdump
/dev/mapper/archive_vg-archive             49G  275M   47G   1% /storage/archive
/dev/mapper/updatemgr_vg-updatemgr         98G  374M   93G   1% /storage/updatemgr
/dev/mapper/imagebuilder_vg-imagebuilder  9.8G   37M  9.3G   1% /storage/imagebuilder
/dev/mapper/db_vg-db                      9.8G  362M  9.0G   4% /storage/db
/dev/mapper/autodeploy_vg-autodeploy      9.8G   48M  9.3G   1% /storage/autodeploy
tmpfs                                     9.4G  4.8M  9.3G   1% /tmp
/dev/mapper/core_vg-core                   49G   53M   47G   1% /storage/core
tmpfs                                     1.0M     0  1.0M   0% /var/spool/snmp

[Lalegre]

Mmm I can see that you have plenty of free space. After increasing the size did you restart the vCenter?

[UThomas]

Yes, of course. But this didn't help.

[Lalegre]

And one more question, were you facing this issues before changing the size? Also if you connect to the VAMI on port 5480 which is the status of the services?

[UThomas]

All services are operational in VAMI.

For clarification : I choose during the vCSA upgrade the new deployment type "small" with "large" data size. I didn't change anything manually.

[SRT65]

Hi,

Did you get a resolution for this. I have exact same issue on two brand new installs of VC 7.0.0d (no upgrade involved)

Cheers,

SRT

[UThomas]

@SRT65  Did you get it fixed?

I ended up with a complete new install an restore a backup

[SRT65]

@UThomas 

Only worked out a workaround.

I did inadvertantly notice that the messages only continued for 24 hours after reboot of vCenter then it stopped. I then went in and changed an XML file I found that I changed the frequency of the messages from every 2 mins to every 4 hours. After that I only get the messages six times after each reboot now which I can live with.

[SRT65]

Just in case anyone is interested. You need to SSH in to the vCenter and edit the file

/etc/vmware/statsmonitor/statsMonitor.xml

Change the line:

<storageSizeSamplingPeriodSec>120</storageSizeSamplingPeriodSec>

To:

<storageSizeSamplingPeriodSec>14400</storageSizeSamplingPeriodSec>

This sets to every 14400 sec (4 hours) but could set to what ever you like I guess. I felt 4 hours was a good compromise so not to miss a real issue if it arises.

[UThomas]

Thanks for sharing your solution with us.

I opened a service ticket, but after about 3 days the event messages are gone. Strange thing, I didn't change anything during that time.