/storage/updatemgr/jetty-temp/ keep filling disk to 100%

[ahbosch]

We recently updated to vCenter 6.7.0.54000 build 20540798, and now the folder "/storage/updatemgr/jetty-temp/" keeps on getting filled to 100%; even when increasing the disk space from 99G -> 246G, the utilization keeps on growing.

I have already reset the vmware-updatemgr database, and this had no effect.

We are getting the same issue with "/storage/core" keeps on getting filled with, for example, "core.worker.65048" files. I read that you can delete these core.worker.* files, which I have a cronjob setup to delete every 15 mins as it fills up 100GB very quickly.

We never had this issue before the update form a previous version of 6.7.

Does any one have any ideas ?

root@vcenter [ /storage/updatemgr/jetty-temp ]# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 12G 0 12G 0% /dev
tmpfs 12G 732K 12G 1% /dev/shm
tmpfs 12G 692K 12G 1% /run
tmpfs 12G 0 12G 0% /sys/fs/cgroup
/dev/sda3 11G 6.0G 4.1G 60% /
tmpfs 12G 1.5M 12G 1% /tmp
/dev/sda1 120M 34M 78M 31% /boot
/dev/mapper/autodeploy_vg-autodeploy 25G 54M 24G 1% /storage/autodeploy
/dev/mapper/imagebuilder_vg-imagebuilder 25G 44M 24G 1% /storage/imagebuilder
/dev/mapper/seat_vg-seat 50G 3.9G 43G 9% /storage/seat
/dev/mapper/log_vg-log 25G 1.6G 22G 7% /storage/log
/dev/mapper/archive_vg-archive 197G 1.3G 187G 1% /storage/archive
/dev/mapper/core_vg-core 50G 52M 47G 1% /storage/core
/dev/mapper/netdump_vg-netdump 9.8G 23M 9.2G 1% /storage/netdump
/dev/mapper/db_vg-db 25G 938M 23G 4% /storage/db
/dev/mapper/dblog_vg-dblog 25G 1.2G 23G 5% /storage/dblog
/dev/mapper/updatemgr_vg-updatemgr 246G 113G 123G 48% /storage/updatemgr

UPDATE:
The increased volume is also now 100% in less than 12 hours.
/dev/mapper/updatemgr_vg-updatemgr 246G 235G 0 100% /storage/updatemgr

FIX UPDATE:
Think I found the issue, also think it's a bug that needs to be addressed as this came from a fresh VC install and on another upgraded VC to the same version mentioned above.

Looked in the VUM Server logs "/var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server-log4cpp.log"
ans saw that the Jetty service was crashing "Jetty server keeps crashing! Giving up."
Googled the error and came across this KB, https://kb.vmware.com/s/article/83778
Followed the steps and I had the "$Server" entry missing.
The Jetty server seems to be happy and now more creation of the "core.worker.*" files too.

[maksym007]

vCenter has a direct connection to internet?  Maybe its downloading something. 

[maksym007]

Check also that article:

https://www.provirtualzone.com/how-to-add-extra-space-to-vcenter-for-the-upgrade/ 

[ahbosch]

Yes it does have access, but 100G and growing, that's seems to be alot.
But I did go ahead and look at the FIrewall sessions and not seeing any active "downloadings" session from vCenter

Since my initial post it has already grown to 142G
/dev/mapper/updatemgr_vg-updatemgr 246G 142G 93G 61% /storage/updatemgr

[ahbosch]

thanks, I followed a similar article to upgrade the existing "/storage/updatemgr" from 99g -> 246G

[scott28tt]

Your duplicate post in one of the SDK areas has been reported to the moderators, since it is not needed and in the wrong area.

[ahbosch]

Thanks @scott28tt 

If you can do me a solid and remove it, that would be great.

Also since you're an employee of VMware, any insight into my issue?

[scott28tt]

I'm not a moderator, I can't remove posts, that's why I reported it so the moderators are informed.

I happen to be a VMware employee, I don't know anything about your issue.

[ahbosch]

Issue fixed, look at original post for update