VMware Cloud Community
rom2311
Contributor
Contributor

vCenter 7U3 - Log exhaustion - WCP Service

Hey everyone,

I have an environment with a vCenter version 7U3 build - 21477706.
We have an error in the environment regarding Log exhaustion in the /storage/log disk.

We tried inspecting further following the KBs: 
1. Troubleshooting vCenter Appliance /storage/log directory is 80% or more full (83070) (vmware.com)
2. vCenter Server Appliance disk space is full (76563) (vmware.com)

The only thing I've noticed is with the /storage/log/vmware/wcp folder having a huge log file that takes up almost all of the disk space.

rom2311_0-1694342092456.png

It can be seen in the screenshot that the problematic file is - stdstream.log.stderr
I've tried finding more information regarding the file and the WCP service but I couldn't really understand the purpose and use of the service and if deleting the file and restarting the service is the solution.

Any help would be appreciated, 
Thanks, Rom.

Tags (3)
Reply
0 Kudos
13 Replies
battybishop
Hot Shot
Hot Shot

Workload Control Plane (WCP), is a service that runs within the vCenter appliance and is responsible for managing the integration of Kubernetes with vSphere.

Have you checked the /var/log/vmware/wcp/wcpsvc.log to identify issues?

Reply
0 Kudos
markey165
Hot Shot
Hot Shot

You could try running cat /storage/log/vmware/wcp/stdstream.log.stderr to examine the contents and see what's causing it to get so large.   If its easier, you could also SCP the file off and examine it with notepad++

 

I've had something similar with a different service which was a known issue, and i recall the fix there was to simply zero size the file, but obviously it would useful to know what's causing it to get so large in the first place. Its obviously had (or is still having) some kind of issue that's causing that log to fill with errors. If in doubt, and you can't determine the cause, I'd recommend raising a support call, they should be able to advise you of best course of action!

 

HTH

_____________________________________________
If this post helps you, please leave Kudo | or mark this reply as an answer
Reply
0 Kudos
Ajay1988
Expert
Expert

This could be a bug. I recall similar was fixed way back in early release of 7.0 . This could be something else but similar . Better to open SR with VMware Support and share VC bundle. 

 

Can you go share output of below commands.

du /storage/log/vmware/ -ahx . | sort -rh | head -n 20
du -hs * | sort -rh | head -10

If you think your queries have been answered
Mark this response as "Correct" or "Helpful".

Regards,
AJ
Reply
0 Kudos
rom2311
Contributor
Contributor

Hey, 

Sorry for the late response, been sick the last 2 weeks..
@battybishop , I did check that log and the only thing repeating multiple times - 

rom2311_0-1695716860218.png

@markey165 I've tried looking into the /storage/log/vmware/wcp/stdstream.log.sderr log itself, but it seems unreadable. I'm now trying to download it and watch it on notepad.

@Ajay1988, It does make sense, as I recall a similar problem in earlier releases as well, thought maybe I just couldn't find the KBs that talk about it..
Here is the output of du /storage/log/vmware/ -ahx . | sort -rh | head -n 20 -

rom2311_0-1695717519564.png

Anyways, I will try to upgrade the vCenter to 7u3n and see if it solves the problem and in the meantime I will open a SR as well.

 

Reply
0 Kudos
Ajay1988
Expert
Expert

Upgrade to 7.0 U3n . Should fix the issue. Zero out or truncate /storage/log/vmware/wcp/stdstream.log.stderr . You might have to stop wcp service. Then upgrade and check.

I don't have issue in my lab.

root@is-dhcp42-38 [ ~ ]# rpm -qa | grep -i wcp-
VMware-wcp-0.0.23-21882390.x86_64
root@is-dhcp42-38 [ ~ ]# vpxd -v
VMware VirtualCenter 7.0.3 build-21958406

Can you also provide below output :
tail -n 20 /storage/log/vmware/wcp/stdstream.log.stderr

If you think your queries have been answered
Mark this response as "Correct" or "Helpful".

Regards,
AJ
Reply
0 Kudos
rom2311
Contributor
Contributor

Hey @Ajay1988

This is the output for tail -n 20 /storage/log/vmware/wcp/stdstream.log.stderr - 

rom2311_0-1695726003472.png

 

Still, no real error I know how to solve.. We have decided to upgrade next week so I'll keep you posted 🙂

battybishop
Hot Shot
Hot Shot

I think upgrading to the latest version is your best bet

Reply
0 Kudos
liambourke
Contributor
Contributor

I am having a similar issue.

What does the stdstream.log file do?

Reply
0 Kudos
navina
Enthusiast
Enthusiast

Hope this helps
https://kb.vmware.com/s/article/94709?lang=en_US

Regards,
Navin A
Reply
0 Kudos
jrgalleg
Enthusiast
Enthusiast

I have the same issue.

What I can see is 2 possible solutions:

https://kb.vmware.com/s/article/94709

https://kb.vmware.com/s/article/95479

Can you please suggest me the best option?

My VCenter version is: vCenter Server 7.0 U3o

Thx in advance.

Reply
0 Kudos
markey165
Hot Shot
Hot Shot

@jrgalleg - I'd refer you to the 2nd link you posted (95749) and follow the instructions for the workaround. Specifically :

To mitigate this problem, please take a fresh backup of the vCenter Server Appliance, then remove all stdstream.log* files from the folder and restart the wcp service:

# service-control --stop wcp; rm /var/log/vmware/wcp/stdstream.log*; service-control --start wcp

 

In other words, delete (using the rm command) the files that end with a number (as shown in Black below), but don't delete the one highlighted in red, as that one is the live log file. The others are backups. If you wish to take a copy of the files to be on the safe side (although its unlikely you would need them as they are just log files) you can use WinSCP to connect over Port 22 (the SSH port) and copy them off

1.4M  -rw-------  1 wcp  lwisRegReader 1.4M <DATE> <TIME> stdstream.log-1.stderr
34M  -rw-------  1 wcp  lwisRegReader  34M <DATE> <TIME>stdstream.log-4.stderr
111M  -rw-------  1 wcp  lwisRegReader 111M <DATE> <TIME> stdstream.log.stderr1.1G  -rw-------  1 wcp  lwisRegReader 1.1G <DATE> <TIME> stdstream.log-3.stderr
1.6G  -rw-------  1 wcp  lwisRegReader 1.6G <DATE> <TIME> stdstream.log-0.stderr 

 
Once you have restarted the service you can then monitor the log to see if the problem returns. If it does, I would then raise a call with VMware support to see what is causing the logs to grow so big.

HTH

_____________________________________________
If this post helps you, please leave Kudo | or mark this reply as an answer
Scott_BHF
Contributor
Contributor

Thanks

 

This worked a treat for me

 

 

markey165
Hot Shot
Hot Shot

@Scott_BHF No worries, glad it helped you 😊

_____________________________________________
If this post helps you, please leave Kudo | or mark this reply as an answer
Reply
0 Kudos