Hi
Our VCSA 6.7 is complaining: Log Disk Exhaustion on vmw-vcenter
We have a rather small scale environment (5 ESXi using this vmw-vcenter) and set it up with the recommended requirements back then when it comes to scale.
Seems like /storage/log is the culprit.
Rather than simply extending the disk, I'd like to resolve this properly.
root@vmw-vcenter [ / ]# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 4.9G 0 4.9G 0% /dev
tmpfs 4.9G 816K 4.9G 1% /dev/shm
tmpfs 4.9G 712K 4.9G 1% /run
tmpfs 4.9G 0 4.9G 0% /sys/fs/cgroup
/dev/sda3 11G 6.7G 3.4G 67% /
tmpfs 4.9G 1.4M 4.9G 1% /tmp
/dev/sda1 120M 28M 84M 25% /boot
/dev/mapper/core_vg-core 25G 1.7G 22G 8% /storage/core
/dev/mapper/log_vg-log 9.8G 9.2G 67M 100% /storage/log
/dev/mapper/db_vg-db 9.8G 1.4G 7.9G 15% /storage/db
/dev/mapper/dblog_vg-dblog 15G 326M 14G 3% /storage/dblog
/dev/mapper/seat_vg-seat 9.8G 357M 8.9G 4% /storage/seat
/dev/mapper/netdump_vg-netdump 985M 1.3M 916M 1% /storage/netdump
/dev/mapper/autodeploy_vg-autodeploy 9.8G 34M 9.2G 1% /storage/autodeploy
/dev/mapper/imagebuilder_vg-imagebuilder 9.8G 23M 9.2G 1% /storage/imagebuilder
/dev/mapper/updatemgr_vg-updatemgr 99G 1.4G 92G 2% /storage/updatemgr
/dev/mapper/archive_vg-archive 50G 27G 21G 57% /storage/archive
root@vmw-vcenter [ / ]#
One of the main culprits seem to be these two files:
root@vmw-vcenter [ /storage/log/vmware ]# du -a | sort -n -r | head -n 5
9556268 .
3123924 ./rbd
2964000 ./rbd/rbd-syslog.log.1
2547020 ./vmdird
2547016 ./vmdird/vmdird-syslog.log.1
What's the proper way to truncate/rotate them?
Log Disk Exhaustion on vmw-vcenter Log Disk Exhaustion on vmw-vcenter
Usually those should be rotated by default. You can delete the .1 log files to check this just keeping in mind you'd obviously lose them for historical analysis purposes.
Seems the issue was resolved in a later update; but I had to delete the .1 log files first.
Feel free to mark your own answer as correct in that case.