vCenter 7U3 - Log exhaustion - WCP Service

[rom2311]

Hey everyone,

I have an environment with a vCenter version 7U3 build - 21477706.
We have an error in the environment regarding Log exhaustion in the /storage/log disk.

We tried inspecting further following the KBs: 
1. Troubleshooting vCenter Appliance /storage/log directory is 80% or more full (83070) (vmware.com)
2. vCenter Server Appliance disk space is full (76563) (vmware.com)

The only thing I've noticed is with the /storage/log/vmware/wcp folder having a huge log file that takes up almost all of the disk space.

It can be seen in the screenshot that the problematic file is - stdstream.log.stderr
I've tried finding more information regarding the file and the WCP service but I couldn't really understand the purpose and use of the service and if deleting the file and restarting the service is the solution.

Any help would be appreciated, 
Thanks, Rom.

[battybishop]

Workload Control Plane (WCP), is a service that runs within the vCenter appliance and is responsible for managing the integration of Kubernetes with vSphere.

Have you checked the /var/log/vmware/wcp/wcpsvc.log to identify issues?

[markey165]

You could try running cat /storage/log/vmware/wcp/stdstream.log.stderr to examine the contents and see what's causing it to get so large.   If its easier, you could also SCP the file off and examine it with notepad++

I've had something similar with a different service which was a known issue, and i recall the fix there was to simply zero size the file, but obviously it would useful to know what's causing it to get so large in the first place. Its obviously had (or is still having) some kind of issue that's causing that log to fill with errors. If in doubt, and you can't determine the cause, I'd recommend raising a support call, they should be able to advise you of best course of action!

HTH

[Ajay1988]

This could be a bug. I recall similar was fixed way back in early release of 7.0 . This could be something else but similar . Better to open SR with VMware Support and share VC bundle. 

Can you go share output of below commands.

du /storage/log/vmware/ -ahx . | sort -rh | head -n 20
du -hs * | sort -rh | head -10

[rom2311]

Hey, 

Sorry for the late response, been sick the last 2 weeks..
@battybishop , I did check that log and the only thing repeating multiple times - 

@markey165 I've tried looking into the /storage/log/vmware/wcp/stdstream.log.sderr log itself, but it seems unreadable. I'm now trying to download it and watch it on notepad.

@Ajay1988, It does make sense, as I recall a similar problem in earlier releases as well, thought maybe I just couldn't find the KBs that talk about it..
Here is the output of du /storage/log/vmware/ -ahx . | sort -rh | head -n 20 -

Anyways, I will try to upgrade the vCenter to 7u3n and see if it solves the problem and in the meantime I will open a SR as well.

[Ajay1988]

Upgrade to 7.0 U3n . Should fix the issue. Zero out or truncate /storage/log/vmware/wcp/stdstream.log.stderr . You might have to stop wcp service. Then upgrade and check.

I don't have issue in my lab.

root@is-dhcp42-38 [ ~ ]# rpm -qa | grep -i wcp-
VMware-wcp-0.0.23-21882390.x86_64
root@is-dhcp42-38 [ ~ ]# vpxd -v
VMware VirtualCenter 7.0.3 build-21958406

Can you also provide below output :
tail -n 20 /storage/log/vmware/wcp/stdstream.log.stderr

[rom2311]

Hey @Ajay1988, 

This is the output for tail -n 20 /storage/log/vmware/wcp/stdstream.log.stderr - 

Still, no real error I know how to solve.. We have decided to upgrade next week so I'll keep you posted 🙂

[battybishop]

I think upgrading to the latest version is your best bet