Unable to login to VCSA

[sebkang]

I've recently encountered an issue whereby I was unable to login to the VCSA 6.0 web client. When accessing the web client, the error below is shown.

503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x7f0db405f700] _serverNamespace = / _isRedirect = false _pipeName =/var/run/vmware/vpxd-webserver-pipe)

I then proceed to do a restart on the server and found out that some of the services are failing to start due to a "no space left on device" error. Next I tried to ssh into the server using the root account to determine if the disks were indeed full but I was denied access. Attempting to login to the dcui also gets a "Invalid login or password" error. Thinking that I may have forgotten my password for my root account, I then proceed to change my password but was then shown another error as shown below.

Error while closing temporary shadow file: No space left on device

Error: Password NOT changed

So basically I am completely locked out of my VCSA as I am unable to login through my web client or through ssh. Is there any way that I can resolve this situation or a fresh install is going to be necessary?

[krobertsIAA]

Maybe give this a shot?

VMware Knowledge Base

Mound the drives but before resetting the password, look for where all the space has gone? Delete old logs, then reset the password.

[msripada]

It looks like the VCSA is out of space.

If you can login thru shell or ssh via vcsa, you can try the KB VMware Knowledge Base

If you are unable to login with the ssh or shell of VCSA

You can login to the VCSA from the VMware Knowledge Base  till Step 6. Once you are in, you can go ahead and type the command df -h to check which partition is full.

or remove the audit.log (if root partition is full) and reboot the VCSA after resetting the password.

Once you are in, you can check the services whether they are up and running.

Thanks,

MS

[admin]

see below KB for this issue,

https://kb.vmware.com/s/article/1007638

[sebkang]

Thanks for the numerous advise. I was able to login via console and ssh after clearing the logs which were causing the disk to be 100% full. However I am still encountering this error when trying to access the web client.

503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x7f0db405f700] _serverNamespace = / _isRedirect = false _pipeName =/var/run/vmware/vpxd-webserver-pipe)

I then proceed to run "service-control --status --all" which gave me this result.

Stopped:

vmware-cm (VMware Component Manager) vmware-eam (VMware ESX Agent Manager) vmware-invsvc (VMware Inventory Service) vmware-mbcs (VMware Message Bus Configuration Service) vmware-netdumper (VMware vSphere ESXi Dump Collector) vmware-rbd-watchdog (VMware vSphere Auto Deploy Waiter) vmware-sca (VMware Service Control Agent) vmware-vapi-endpoint (VMware vAPI Endpoint) vmware-vdcs (VMware Content Library Service) vmware-vpx-workflow (VMware vCenter Workflow Manager) vmware-vpxd (VMware vCenter Server) vsphere-client ()

As expected, vmware-vpxd has stopped running. I then proceed to start the service manually but was unable to. A quick look at /var/log/vmware/vpxd/vpxd.log gave me the below result at the end of the page.

2017-11-14T05:05:43.027Z info vpxd[7F4B3F78C7A0] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] [CreateServiceContent] Try to connect to SSO VMOMI endpoint.

2017-11-14T05:05:43.032Z warning vpxd[7F4B3F78C7A0] [Originator@6876 sub=Default] Closing Response processing in unexpected state: 3

2017-11-14T05:05:43.032Z warning vpxd[7F4B3F78C7A0] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] [RetryOnConnectionFailure] Vmomi::Fault::SystemError while trying to connect to SSO Admin server: N7Vmacore4Soap24InvalidResponseExceptionE(Invalid response code: 404 Not Found)

2017-11-14T05:05:43.032Z info vpxd[7F4B3F78C7A0] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] [CreateServiceContent] Will attempt to connect again in 10 seconds.

2017-11-14T05:05:53.033Z info vpxd[7F4B3F78C7A0] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] [CreateServiceContent] Try to connect to SSO VMOMI endpoint.

2017-11-14T05:05:53.038Z warning vpxd[7F4B3F78C7A0] [Originator@6876 sub=Default] Closing Response processing in unexpected state: 3

2017-11-14T05:05:53.038Z warning vpxd[7F4B3F78C7A0] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] [RetryOnConnectionFailure] Vmomi::Fault::SystemError while trying to connect to SSO Admin server: N7Vmacore4Soap24InvalidResponseExceptionE(Invalid response code: 404 Not Found)

2017-11-14T05:05:53.038Z info vpxd[7F4B3F78C7A0] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] [CreateServiceContent] Will attempt to connect again in 10 seconds.

2017-11-14T05:06:03.039Z info vpxd[7F4B3F78C7A0] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] [CreateServiceContent] Try to connect to SSO VMOMI endpoint.

2017-11-14T05:06:03.045Z warning vpxd[7F4B3F78C7A0] [Originator@6876 sub=Default] Closing Response processing in unexpected state: 3

2017-11-14T05:06:03.045Z warning vpxd[7F4B3F78C7A0] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] [RetryOnConnectionFailure] Vmomi::Fault::SystemError while trying to connect to SSO Admin server: N7Vmacore4Soap24InvalidResponseExceptionE(Invalid response code: 404 Not Found)

2017-11-14T05:06:03.045Z error vpxd[7F4B3F78C7A0] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] [RetryOnConnectionFailure] Max attempts (10) reached. Giving up ...

2017-11-14T05:06:03.045Z error vpxd[7F4B3F78C7A0] [Originator@6876 sub=[SSO][SsoFactory_CreateFacade]] Unable to create SSO facade: N7Vmacore4Soap24InvalidResponseExceptionE(Invalid response code: 404 Not Found).

2017-11-14T05:06:03.045Z warning vpxd[7F4B3F78C7A0] [Originator@6876 sub=VpxProfiler] Init [Vpx::Common::Sso::SsoFactory_CreateFacade(sslContext, ssoFacadeConstPtr)] took 90063 ms

2017-11-14T05:06:03.045Z error vpxd[7F4B3F78C7A0] [Originator@6876 sub=Main] [Init] Init failed: Vpx::Common::Sso::SsoFactory_CreateFacade(sslContext, ssoFacadeConstPtr)

--> Backtrace:

-->

--> [backtrace begin] product: VMware VirtualCenter, version: 6.0.0, build: build-2776510, tag: vpxd

--> backtrace[00] libvmacore.so[0x003C00C4]: Vmacore::System::Stacktrace::CaptureWork(unsigned int)

--> backtrace[01] libvmacore.so[0x001EDBD3]: Vmacore::System::SystemFactoryImpl::CreateQuickBacktrace(Vmacore::Ref<Vmacore::System::Backtrace>&)

--> backtrace[02] libvmacore.so[0x001988DD]: Vmacore::Throwable::Throwable(std::string const&)

--> backtrace[03] vpxd[0x00BD075E]: Vmomi::Fault::SystemError::Exception::Exception(std::string const&)

--> backtrace[04] vpxd[0x00BC9791]

--> backtrace[05] vpxd[0x00BBA0D0]

--> backtrace[06] vpxd[0x00AF8789]

--> backtrace[07] libc.so.6[0x0001EC36]

--> backtrace[08] vpxd[0x00AF81DD]

--> [backtrace end]

-->

2017-11-14T05:06:03.048Z warning vpxd[7F4B3F78C7A0] [Originator@6876 sub=VpxProfiler] ServerApp::Init [TotalTime] took 182491 ms

2017-11-14T05:06:03.048Z error vpxd[7F4B3F78C7A0] [Originator@6876 sub=Default] Failed to intialize VMware VirtualCenter. Shutting down...

2017-11-14T05:06:03.048Z info vpxd[7F4B3F78C7A0] [Originator@6876 sub=SupportMgr] Wrote uptime information

2017-11-14T05:08:03.049Z info vpxd[7F4B3F78C7A0] [Originator@6876 sub=Default] Forcing shutdown of VMware VirtualCenter now

Would anyone be able to advise what I should do next in order to resolve this issue? Thanks

[daphnissov]

Reboot the appliance if you haven't already.

[sebkang]

Yes. Appliance has already been rebooted but still the same error persists