vCenter 6.7 Server down

[elbruce]

I'm running vCenter Server (on Windows Server 2016 standard) in my lab environment.  Several days ago - the C: drive ran out of space.  I expanded the drive, and it now has 15GB of free space.

However - eve after multiple reboots I am not able to access vCenter.  I am getting the following error:

503 Service Unavailable (Failed to connect to endpoint: [class Vmacore::Http::NamedPipeServiceSpec:0x00000133860f0350] _serverNamespace = / action = Allow _pipeName =\\.\pipe\vmware-vpxd-webserver-pipe)

Running service-control --status returns the following:

Running:
EsxAgentManager VMWareAfdService VMWareCertificateService VMWareDirectoryService VMwareComponentManager VMwareDNSService VMwareIdentityMgmtService VMwareSTS rhttpproxy vPostgres vapiEndpoint vmon vmonapi vmware-cis-config vmware-license vmwareServiceControlAgent vpxd-svcs vsphere-ui vspherewebclientsvc
Stopped:
VMWareCAMService VServiceManager content-library mbcs vimPBSM vmsyslogcollector vmware-autodeploy-waiter vmware-imagebuilder vmware-network-coredump vmware-perfcharts vpxd vsan-health

I am pretty sure the issue is that the vpxd service is not running, so I attempted to start it with service-control --start vpxd and got the following:

Operation not cancellable. Please wait for it to finish...
Performing start operation on service vpxd...
Error executing start on service vpxd. Details {
"problemId": null,
"resolution": null,
"detail": [
{
"localized": "An error occurred while starting service 'vpxd'",
"translatable": "An error occurred while starting service '%(0)s'",
"id": "install.ciscommon.service.failstart",
"args": [
"vpxd"
]
}
],
"componentKey": null
}
Service-control failed. Error: {
"problemId": null,
"resolution": null,
"detail": [
{
"localized": "An error occurred while starting service 'vpxd'",
"translatable": "An error occurred while starting service '%(0)s'",
"id": "install.ciscommon.service.failstart",
"args": [
"vpxd"
]
}
],
"componentKey": null
}

I checked in c:\programdata\vmware\vCenterServer\logs\vmware-vpx - but there is no vlxd.log.  There IS a vpxd-271.log file, but the last entry in that file is from 11/23 (when the disk space issue happened).  The last few lines are:

2020-11-23T12:27:18.718-05:00 info vpxd[09992] [Originator@6876 sub=vpxLro opID=sps-Main-262111-509-workitem389021-92] [VpxLRO] -- FINISH lro-2850471
2020-11-23T12:27:24.642-05:00 error vpxd[10128] [Originator@6876 sub=Health] Failed to write vpxd health XML to file C:\ProgramData\VMware\vCenterServer\cfg\sca\health\vmware-vpxd-health-status.xml. Error: There is not enough space on the disk.
2020-11-23T12:27:28.599-05:00 info vpxd[09976] [Originator@6876 sub=vpxLro opID=sps-Main-262111-509-79] [VpxLRO] -- BEGIN lro-2850473 -- SessionManager -- vim.SessionManager.sessionIsActive -- 52e9c951-fc0c-24db-b58b-bca266d6743b(52550011-38e8-e374-dc15-dd7ccaef55a5)

Can anyone provide some insight into where to start troubleshooting this issue?

Thanks in advance.

[elbruce]

After some further troubleshooting - I found the following I the vMon.log:

2020-11-27T12:35:28.982-05:00| host-3516| I125: <vpxd-prestart> Constructed command : "C:\Program Files\VMware\vCenter Server\python\python.exe" "C:\Program Files\VMware\vCenter Server/vpxd/py/vpxd-prestart.py"
2020-11-27T12:35:30.064-05:00| host-3516| W115: <vpxd> Service pre-start command's stderr: E
2020-11-27T12:35:30.064-05:00| host-3516| W115: <vpxd> Service pre-start command's stderr: n
2020-11-27T12:35:30.064-05:00| host-3516| W115: <vpxd> Service pre-start command's stderr: t
2020-11-27T12:35:30.064-05:00| host-3516| W115: <vpxd> Service pre-start command's stderr: i
2020-11-27T12:35:30.064-05:00| host-3516| W115: <vpxd> Service pre-start command's stderr: t
2020-11-27T12:35:30.064-05:00| host-3516| W115: <vpxd> Service pre-start command's stderr: y
2020-11-27T12:35:30.064-05:00| host-3516| W115: <vpxd> Service pre-start command's stderr: :
2020-11-27T12:35:30.065-05:00| host-3516| W115: <vpxd> Service pre-start command's stderr:
2020-11-27T12:35:30.065-05:00| host-3516| W115: <vpxd> Service pre-start command's stderr: l
2020-11-27T12:35:30.065-05:00| host-3516| W115: <vpxd> Service pre-start command's stderr: i
2020-11-27T12:35:30.065-05:00| host-3516| W115: <vpxd> Service pre-start command's stderr: n
2020-11-27T12:35:30.065-05:00| host-3516| W115: <vpxd> Service pre-start command's stderr: e
2020-11-27T12:35:30.065-05:00| host-3516| W115: <vpxd> Service pre-start command's stderr:
2020-11-27T12:35:30.065-05:00| host-3516| W115: <vpxd> Service pre-start command's stderr: 1
2020-11-27T12:35:30.065-05:00| host-3516| W115: <vpxd> Service pre-start command's stderr: : parser error : Document is empty
2020-11-27T12:35:30.065-05:00| host-3516| W115+
2020-11-27T12:35:30.065-05:00| host-3516| W115+ ^
2020-11-27T12:35:30.065-05:00| host-3516| W115:
2020-11-27T12:35:30.065-05:00| host-3516| W115: <vpxd> Service pre-start command's stderr: Failed to update memStopInMB in vpxd.cfg :xmlReadFd() failed
2020-11-27T12:35:30.082-05:00| host-3516| E110: <vpxd> Service pre-start command failed with exit code 1.

So - I tried running the vlxd-prestart by hand:

"C:\Program Files\VMware\vCenter Server\python\python.exe" "C:\Program Files\VMware\vCenter Server/vpxd/py/vpxd-prestart.py"

And got the following:

Entity: line 1: parser error : Document is empty

^
Failed to update memStopInMB in vpxd.cfg :xmlReadFd() failed

This lead me to look at the vlxd.cfg file - and it appears to be BLANK:

C:\ProgramData\VMware\vCenterServer\cfg\vmware-vpx\vpxd.cfg  shows - bytes, and when opened with Notepad is blank.

Is there any way to rebuild this vlxd.cfg file?  Can I just copy it over from a working vCenter server from another environment?

[bryanvaneeden]

The reason the log file isn't newer is probably because the vpxd service hasn't started anymore since that day. This could be a tricky thing to solve. What are some of the other log files showing? Is this the only service that cannot start?

[elbruce]

After some further troubleshooting - I found the following the von.log:

2020-11-27T12:35:28.982-05:00| host-3516| I125: Received start request for vpxd
2020-11-27T12:35:28.982-05:00| host-3516| I125: <vpxd-prestart> Constructed command : "C:\Program Files\VMware\vCenter Server\python\python.exe" "C:\Program Files\VMware\vCenter Server/vpxd/py/vpxd-prestart.py"
2020-11-27T12:35:30.064-05:00| host-3516| W115: <vpxd> Service pre-start command's stderr: E
2020-11-27T12:35:30.064-05:00| host-3516| W115: <vpxd> Service pre-start command's stderr: n
2020-11-27T12:35:30.064-05:00| host-3516| W115: <vpxd> Service pre-start command's stderr: t
2020-11-27T12:35:30.064-05:00| host-3516| W115: <vpxd> Service pre-start command's stderr: i
2020-11-27T12:35:30.064-05:00| host-3516| W115: <vpxd> Service pre-start command's stderr: t
2020-11-27T12:35:30.064-05:00| host-3516| W115: <vpxd> Service pre-start command's stderr: y
2020-11-27T12:35:30.064-05:00| host-3516| W115: <vpxd> Service pre-start command's stderr: :
2020-11-27T12:35:30.065-05:00| host-3516| W115: <vpxd> Service pre-start command's stderr:
2020-11-27T12:35:30.065-05:00| host-3516| W115: <vpxd> Service pre-start command's stderr: l
2020-11-27T12:35:30.065-05:00| host-3516| W115: <vpxd> Service pre-start command's stderr: i
2020-11-27T12:35:30.065-05:00| host-3516| W115: <vpxd> Service pre-start command's stderr: n
2020-11-27T12:35:30.065-05:00| host-3516| W115: <vpxd> Service pre-start command's stderr: e
2020-11-27T12:35:30.065-05:00| host-3516| W115: <vpxd> Service pre-start command's stderr:
2020-11-27T12:35:30.065-05:00| host-3516| W115: <vpxd> Service pre-start command's stderr: 1
2020-11-27T12:35:30.065-05:00| host-3516| W115: <vpxd> Service pre-start command's stderr: : parser error : Document is empty
2020-11-27T12:35:30.065-05:00| host-3516| W115+
2020-11-27T12:35:30.065-05:00| host-3516| W115+ ^
2020-11-27T12:35:30.065-05:00| host-3516| W115:
2020-11-27T12:35:30.065-05:00| host-3516| W115: <vpxd> Service pre-start command's stderr: Failed to update memStopInMB in vpxd.cfg :xmlReadFd() failed
2020-11-27T12:35:30.082-05:00| host-3516| E110: <vpxd> Service pre-start command failed with exit code 1.

I tried to run the restart command: "C:\Program Files\VMware\vCenter Server\python\python.exe" "C:\Program Files\VMware\vCenter Server/vpxd/py/vpxd-prestart.py" and got the following:

Entity: line 1: parser error : Document is empty

^
Failed to update memStopInMB in vpxd.cfg :xmlReadFd() failed

This lead me to have a look at the vpxd config file located here: C:\ProgramData\VMware\vCenterServer\cfg\vmware-vpx\vpxd.cfg

The config file is showing 0 bytes and appears to be empty.

So - how do I go about rebuilding the vpxd.cfg file?  Can I copy one over from another vCenter server?

Thanks!!

[bryanvaneeden]

Well yes that makes sense, if the VPXD configuration file is missing it's not able to start. You could use a VPXD from another vCenter, but honestly I don't know if that will work unless you change everything in the exact way it was before. But you can definitely try.

[sjesse]

if this is a lab get off of the windows vcenter server 6.7 is the last version it's viable.

[SrVMwarer]

memStopInMB in vpxd.cfg

If I remember correctly, but I was on VCSA : )) 

Stop vMon, and start it again in the logs for vMon it will tell which vpxd.cfg is missing using another vCenter I was able to edit the corrupted file 

[Lalegre]

Hey @elbruce,

So the vpxd.cfg values between vCenters is mostly the same just a few changes need to be done regarding some inputs of course. Do you have any backup from where to get this configuration file?

I presume it got corrupted because of the lack of space.