503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x00007f290400b950] _serverNamespace = / action = Allow _pipeName =/var/run/vmware/vpxd-webserver-pipe)

paulsamkpz · ‎02-11-2020

My vCenter is using a shared Datastore and that datastore got full recently and my vcenter started throwing this error in UI.

503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x00007f290400b950] _serverNamespace = / action = Allow _pipeName =/var/run/vmware/vpxd-webserver-pipe)

Once i tried to cleanup some space and start the service's back, i am getting this error while starting vpxd:

Operation not cancellable. Please wait for it to finish...

Performing start operation on service vpxd...

Successfully started service vmware-vmon

Error executing start on service vpxd. Details {

"resolution": null,

"componentKey": null,

"detail": [

{

"args": [

"vpxd"

],

"translatable": "An error occurred while starting service '%(0)s'",

"id": "install.ciscommon.service.failstart",

"localized": "An error occurred while starting service 'vpxd'"

}

],

"problemId": null

}

Service-control failed. Error: {

"resolution": null,

"componentKey": null,

"detail": [

{

"args": [

"vpxd"

],

"translatable": "An error occurred while starting service '%(0)s'",

"id": "install.ciscommon.service.failstart",

"localized": "An error occurred while starting service 'vpxd'"

}

],

"problemId": null

}

Any help would be appreciated!!

Thanks

NathanosBlightc · ‎02-11-2020

Are you sure about the enough free space inside the VCSA guest OS?

Can you check it again please? with running following command in the VCSA shell:

cd /

du -ah --max-depth=1

Please mark my comment as the Correct Answer if this solution resolved your problem

paulsamkpz · ‎02-11-2020

I ran the command and here is the result:

48K ./root

36K ./tmp

1.3G ./var

0 ./srv

4.0K ./ssh

0 ./lib

0 ./sys

179M ./opt

464K ./dev

4.0K ./vapi-endpoint

2.5M ./tftpboot

306M ./etc

33M ./boot

0 ./sbin

3.9G ./usr

0 ./media

200K ./home

1.1M ./vasecurity

680K ./run

12G ./storage

12K ./mnt

0 ./bin

16K ./lost+found

12K ./.cache

0 ./lib64

du: cannot access './proc/65003/task/65003/fd/4': No such file or directory

du: cannot access './proc/65003/task/65003/fdinfo/4': No such file or directory

du: cannot access './proc/65003/fd/3': No such file or directory

du: cannot access './proc/65003/fdinfo/3': No such file or directory

0 ./proc

18G .

NathanosBlightc · ‎02-11-2020

Please retry this command for /storage directory:

cd /storage

du -ah --max-depth=2

Check contents of every directory that is occupied with large space.

I think there is too much log files (or historical logs) or maybe cache content in your VCSA, so it cause the volume fills more than normal and then prevent some critical services to start and interrupted them

Please mark my comment as the Correct Answer if this solution resolved your problem

paulsamkpz · ‎02-11-2020

Here is the output from /storage

4.0K ./mbcs/mbcsspec.properties

8.0K ./mbcs

4.0K ./updatemgr/software-update

4.0K ./updatemgr/jetty-temp

16K ./updatemgr/lost+found

4.0K ./updatemgr/patch-store-temp

8.0K ./updatemgr/patch-store

40K ./updatemgr

4.0K ./imagebuilder/vibs

4.0K ./imagebuilder/depot

16K ./imagebuilder/lost+found

4.0K ./imagebuilder/exports

32K ./imagebuilder

20K ./sps/tomcat

24K ./sps

16K ./dblog/lost+found

161M ./dblog/vpostgres

161M ./dblog

4.0K ./sqlite-state/vmca

4.0K ./sqlite-state/vmafd

12K ./sqlite-state

16K ./db/lost+found

125M ./db/vpostgres

4.0K ./db/journal

41M ./db/vmware-vmdir

68K ./db/vmware-imagebuilder

132K ./db/vmware-vmafd

20K ./db/vpostgres_ssl

165M ./db

4.0K ./vmware-vmon/vapi-endpoint.start.cmd

132K ./vmware-vmon/.svcStats

4.0K ./vmware-vmon/cm.start.cmd

4.0K ./vmware-vmon/vmonapi.start.cmd

4.0K ./vmware-vmon/defaultStartProfile

4.0K ./vmware-vmon/cis-license.start.cmd

4.0K ./vmware-vmon/vsm.start.cmd

4.0K ./vmware-vmon/signingcert.crt

4.0K ./vmware-vmon/vmonapi-health.xml

4.0K ./vmware-vmon/sps.start.cmd

4.0K ./vmware-vmon/vsphere-client.start.cmd

4.0K ./vmware-vmon/sca.start.cmd

4.0K ./vmware-vmon/eam.start.cmd

4.0K ./vmware-vmon/content-library.start.cmd

4.0K ./vmware-vmon/perfcharts.start.cmd

4.0K ./vmware-vmon/vpxd-svcs.start.cmd

0 ./vmware-vmon/vmonserver

4.0K ./vmware-vmon/vsphere-ui.start.cmd

200K ./vmware-vmon

4.0K ./service-state/updatemgr

4.0K ./service-state/imagebuilder

4.0K ./service-state/vmware-vsan-health

4.0K ./service-state/vmidentity

4.0K ./service-state/vsphere-client

4.0K ./service-state/vmafd

4.0K ./service-state/rbd

4.0K ./service-state/vsphere-ui

4.0K ./service-state/likewise

4.0K ./service-state/vmdir

44K ./service-state

16K ./autodeploy/lost+found

4.0K ./autodeploy/backup

4.0K ./autodeploy/cache

28K ./autodeploy

8.0K ./core/software-update

469M ./core/core.postgres.831

16K ./core/lost+found

4.0K ./core/netdumps

469M ./core

4.0K ./pre-failover/vmdir_request_restore.sh

8.0K ./pre-failover

4.0K ./applmgmt/backup_restore

8.0K ./applmgmt

4.0K ./vsphere-client/ceip

240K ./vsphere-client/SerenityDB

248K ./vsphere-client

2.0G ./log/vmware

16K ./log/lost+found

4.0K ./log/remote

2.0G ./log

4.0K ./vmware-pod

4.0K ./vsan-health

4.0K ./vsphere-ui/ceip

40K ./vsphere-ui/SerenityDB

48K ./vsphere-ui

20K ./vpxd-svcs/tomcat

24K ./vpxd-svcs

16K ./netdump/lost+found

20K ./netdump

4.0K ./seat/software-update

180M ./seat/vpostgres

16K ./seat/lost+found

180M ./seat

4.0K ./vsan-dps

9.0G ./archive/vpostgres

16K ./archive/lost+found

4.0K ./archive/software-update

9.0G ./archive

12G .

NathanosBlightc · ‎02-12-2020

So check the following directory (based on your attached result)

2.0G	./log/vmware
2.0G	./log
9.0G	./archive/vpostgres
9.0G	./archive

and remove some old and unused (archive) files. But Strongly recommend to provide a backup from the VCSA before removing any old file/folder. Then try again to start the interrupted services or reboot the VM and give me the final result

Please mark my comment as the Correct Answer if this solution resolved your problem

All

VPXD service not starting.

503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x00007f290400b950] _serverNamespace = / action = Allow _pipeName =/var/run/vmware/vpxd-webserver-pipe)