VMware Cloud Community
mqzd23
Enthusiast
Enthusiast

vCenter not working after log4 mitigation

Version: vCenter 6.7 on Windows server

I mitigated vCenter log4 vulnerability as described here: https://kb.vmware.com/s/article/87096 This means I have altered vsphere-ui.json, vsphere-client.json, log4j-core.jar and JndiLookup.class. But now my vCenter doesn't seem to be working anymore. Whenever I go the vCenter in my browser, it just times out. I have backups of the files but I'm wondering what's going wrong since I believe I followed the exact instructions from the KB. I've already tried a server reboot but no difference.

Anyone idea what's going on?

 

 

 

 

C:\Program Files\VMware\vCenter Server\bin>.\service-control --start --all
Operation not cancellable. Please wait for it to finish...
Performing start operation on service vmware-cis-config...
Successfully started service vmware-cis-config
Performing start operation on service VMWareAfdService...
Successfully started service VMWareAfdService
Performing start operation on service VMWareDirectoryService...
Successfully started service VMWareDirectoryService
Performing start operation on service VMWareCertificateService...
Successfully started service VMWareCertificateService
Performing start operation on service VMwareIdentityMgmtService...
Successfully started service VMwareIdentityMgmtService
Performing start operation on service VMwareSTS...
Successfully started service VMwareSTS
Performing start operation on service VMwareDNSService...
Successfully started service VMwareDNSService
Performing start operation on profile: ALL...
Service-control failed. Error: Failed to start vmon. Error: (1056, 'StartService
', 'An instance of the service is already running.')

C:\Program Files\VMware\vCenter Server\bin>.\service-control --status
Stopped:
 EsxAgentManager VMWareCAMService VMwareComponentManager VServiceManager content
-library mbcs rhttpproxy vPostgres vapiEndpoint vimPBSM vmonapi vmsyslogcollecto
r vmware-autodeploy-waiter vmware-imagebuilder vmware-license vmware-network-cor
edump vmware-perfcharts vmwareServiceControlAgent vpxd vpxd-svcs vsan-health vsp
here-ui vspherewebclientsvc
StartPending:
 vmon
Running:
 VMWareAfdService VMWareCertificateService VMWareDirectoryService VMwareDNSServi
ce VMwareIdentityMgmtService VMwareSTS vmware-cis-config

C:\Program Files\VMware\vCenter Server\bin>

C:\Program Files\VMware\vCenter Server\bin>.\service-control --stop
Operation not cancellable. Please wait for it to finish...
Performing stop operation on profile: ALL...
Successfully stopped profile: ALL.

C:\Program Files\VMware\vCenter Server\bin>.\service-control --start --all
Operation not cancellable. Please wait for it to finish...
Performing start operation on service vmware-cis-config...
Successfully started service vmware-cis-config
Performing start operation on service VMWareAfdService...
Successfully started service VMWareAfdService
Performing start operation on service VMWareDirectoryService...
Successfully started service VMWareDirectoryService
Performing start operation on service VMWareCertificateService...
Successfully started service VMWareCertificateService
Performing start operation on service VMwareIdentityMgmtService...
Successfully started service VMwareIdentityMgmtService
Performing start operation on service VMwareSTS...
Successfully started service VMwareSTS
Performing start operation on service VMwareDNSService...
Successfully started service VMwareDNSService
Performing start operation on profile: ALL...
Service-control failed. Error: Failed to start vmon. Error: (1056, 'StartService
', 'An instance of the service is already running.')

 

edit: I had to have the server running again so I reverted log4j-core.jar and JndiLookup.class, same problem. After that I reverted vsphere-ui.json, vsphere-client.json, still same problem. The last step to completely revert everything is to undo the actions from vMON.py script. I'm currently trying to figure out what the script did.

 

Reply
0 Kudos
14 Replies
cwc-kavaa
Contributor
Contributor

We have the same issue...

Reply
0 Kudos
ronancosta
Contributor
Contributor

Same problem

Reply
0 Kudos
cwc-kavaa
Contributor
Contributor

Did anyone of you already found a solution? 

Still waiting on a Call from vmware...

Reply
0 Kudos
mqzd23
Enthusiast
Enthusiast

No solution yet for me. If you got a call back with some more info, I'm eager to hear.

I reverted log4j-core.jar and JndiLookup.class, same problem. After that I reverted vsphere-ui.json, vsphere-client.json, still same problem. The last step to completely revert everything is to undo the actions from the vMON.py script. I'm currently trying to figure out what the script did.

edit: nvm, I didn't have to undo the script it seems. A reboot made my server available again but without mitigations so I'm still curious on how to solve that.

Reply
0 Kudos
cwc-kavaa
Contributor
Contributor

We have put back all the files as well and rebooted, but did not work.

Reply
0 Kudos
Putuque
Contributor
Contributor

Exactly the same case but with vcenter 7.0.2 Version: 7.0.2.00400
Only changed vi /usr/lib/vmware-vmon/java-wrapper-vmon file as a workaround from:

Original
exec $java_start_bin $jvm_dynargs $security_dynargs $original_args
 
Updated
log4j_arg="-Dlog4j2.formatMsgNoLookups=true"
exec $java_start_bin $jvm_dynargs $log4j_arg $security_dynargs $original_args


and rebooted services.. does not come up anymore

Sent a ticket

Putuque
Contributor
Contributor

resolved my case

went to compare the original:
/usr/lib/vmware-vmon/java-wrapper-vmon file
and backuped /usr/lib/vmware-vmon/java-wrapper-vmon.bak file

and even if I did not change that myself then there was a empty line in the beginning of the file.
After removing that and restarting the vcenter. All is fine again.. 

Seniore
Enthusiast
Enthusiast

Hi,

we've had the same issue (VMON not starting).
Since then VMware has updated their KB with the following sentence:
"Make a backup of the below two .json files in a different location than the originals"
In the first try we had the backups in the same folder, after re-doing everything and moving them to a different location the service started successfully.

cwc-kavaa
Contributor
Contributor

If we only knew that! 😄

We made the same mistake i think, at least we had the files moved, but also left .bak files in original location.

We ended up migrating to vCenter 7 in the end. We got the old vCenter server working somehow, don't ask me how... But it did eventually.

 

Reply
0 Kudos
ScottO88
Contributor
Contributor

Having the same issues with the vCenter running on the Virtual Appliance.

Reply
0 Kudos
ronancosta
Contributor
Contributor

KB87096 was fixed, after retry following the KB it worked correctly.

https://kb.vmware.com/s/article/87096

Note: Take a snapshot before the changes.

Reply
0 Kudos
kylebegle
Contributor
Contributor

Just throwing my hat in the ring, also on 6.7 on Windows. We are having the same issue, vmon is "Start Pending" and will not come back up. I followed the KB article AFTER the changes clarifying that the backup files should be moved to another directory.

Support ticket is open, will update if any fix is found.

Reply
0 Kudos
kylebegle
Contributor
Contributor

Update: TAC was able to get our server back up and running by just repeating the steps in KB 87096, then rebooting the server.

Reply
0 Kudos
esxnoobofnoobs
Contributor
Contributor

i have a similar issue, but vimPBSM wont start ( i think its part of sps )

but i only got to step1 and stopped. firstly those 2 json files it askes to remove lines, well those lines of code dont exist...

the vcenter is acessible and working,besides this not running.

 

i have backups of the json files in the different location

i noticed all the files in C:\ProgramData\VMware\vCenterServer\cfg\vmware-vmon\svcCfgfiles have been touched my the vmon py script. the touched files seem to have increased in size a little and some extra code added.

i have all the files from another vcenter healthy server, should i stop all services

copy all from healthy to to bad vcenter, and start services.

PS on 6.5.0 BUILD 9451637

my first psc seem to go ok.

Reply
0 Kudos