Issue with vCenter 5.5

[pengwin]

On Tuesday morning one of our physical hosts crashed. I managed to reboot the host OK, got all the VMs up and running again, but I've not been able to connect to vCenter through vSphere since.

I've tried rebooting the host, tried restoring both a recent backup and one a couple of months old to both the host it was running on and to another one, and have ensured that it has enough disk space (only using 41%) - none of which have worked.

I can ping the vCenter server, and connect to the web GUI, but every time I try to connect via vSphere I get the following error:

vSphere could not connect to <IP>.

An unknown connection error occurred. (The request failed because of a connection failure. (Unable to connect to the remote sever.))

Looking at the console on boot, the issues appears to be with the vpxd service as it stalls for a while on "Waiting for vpxd to initialize" before failing.

The vpxd log:

Section for VMware VirtualCenter, pid=5817, version=5.5.0, build=2646482, option =Release

------ In-memory logs start -------- mem> 2018-10-24T21:50:52.403Z [7FEF2201B740 verbose 'vpxdvpxdSignal'] Signal han dler for signal 1 is installed mem> 2018-10-24T21:50:52.403Z [7FEF2201B740 verbose 'vpxdvpxdSignal'] Signal han dler for signal 2 is installed mem> 2018-10-24T21:50:52.403Z [7FEF2201B740 verbose 'vpxdvpxdSignal'] Signal han dler for signal 15 is installed mem> 2018-10-24T21:50:52.403Z [7FEF2201B740 verbose 'vpxdvpxdSignal'] Signal han dler for signal 10 is installed mem> 2018-10-24T21:50:52.403Z [7FEF2201B740 info 'Hooks'] Hooks Initialized

------ In-memory logs end -------- 2018-10-24T21:50:52.421Z [7FEF2201B740 info 'Default'] Logging uses fast path: t rue 2018-10-24T21:50:52.421Z [7FEF2201B740 info 'Default'] Handling bora/lib logs wi th VmaCore facilities 2018-10-24T21:50:52.421Z [7FEF2201B740 info 'Default'] Initialized channel manag er 2018-10-24T21:50:52.425Z [7FEF2201B740 info 'Default'] Current working directory : /storage/log/vmware/vpx 2018-10-24T21:50:52.425Z [7FEF2201B740 info 'ThreadPool'] Thread pool on asio: Min Io, Max Io, Min Task, Max Task, Max Concurency: 2, 200, 2, 90, 2147483647 2018-10-24T21:50:52.425Z [7FEF2201B740 info 'ThreadPool'] Thread enlisted 2018-10-24T21:50:52.425Z [7FEF181EA700 info 'ThreadPool'] Thread enlisted 2018-10-24T21:50:52.425Z [7FEF182EC700 info 'ThreadPool'] Thread enlisted 2018-10-24T21:50:52.425Z [7FEF1836D700 info 'ThreadPool'] Thread enlisted 2018-10-24T21:50:52.425Z [7FEF1826B700 info 'ThreadPool'] Thread enlisted 2018-10-24T21:50:52.425Z [7FEF2201B740 info 'Default'] Log path: /var/log/vmware/vpx 2018-10-24T21:50:52.425Z [7FEF2201B740 info 'Default'] Initializing SSL 2018-10-24T21:50:52.427Z [7FEF2201B740 info 'Default'] Vmacore::InitSSL: handshakeTimeoutUs = 120000000 2018-10-24T21:50:52.427Z [7FEF18169700 info 'ThreadPool'] Thread enlisted 2018-10-24T21:50:52.428Z [7FEF2201B740 info 'vpxdvpxdDaemon'] Changed working directory to /var/log/vmware/vpx 2018-10-24T21:50:52.436Z [7FEF2201B740 info 'Default'] Starting VMware VirtualCenter 5.5.0 build-2646482 2018-10-24T21:50:52.436Z [7FEF2201B740 info 'Default'] Log directory: /var/log/vmware/vpx. 2018-10-24T21:50:52.436Z [7FEF2201B740 info 'vpxdvpxdMain'] Account name: root 2018-10-24T21:50:52.436Z [7FEF2201B740 info 'commonvpxLro'] [VpxLRO] 640 max LROs 2018-10-24T21:50:52.436Z [7FEF2201B740 info 'commonvpxLro'] [VpxLRO] 12 reserved internal LROs 2018-10-24T21:50:52.436Z [7FEF2201B740 info 'commonvpxLro'] [VpxLRO] 12 reserved blocker LROs 2018-10-24T21:50:52.436Z [7FEF2201B740 info 'commonvpxLro'] [VpxLRO] 12 reserved short LROs 2018-10-24T21:50:52.436Z [7FEF2201B740 info 'commonvpxLro'] [VpxLRO] 8 reserved long LROs 2018-10-24T21:50:52.436Z [7FEF2201B740 info 'commonvpxLro'] [VpxLRO] 600-second completed task lifetime 2018-10-24T21:50:52.436Z [7FEF2201B740 info 'commonvpxLro'] [VpxLRO] 60-second minimum completed task lifetime 2018-10-24T21:50:52.436Z [7FEF2201B740 info 'commonvpxLro'] [VpxLRO] 200 maximum completed tasks 2018-10-24T21:50:52.436Z [7FEF2201B740 info 'commonvpxLro'] [VpxLRO] 600-second unregistered task lifetime 2018-10-24T21:50:52.437Z [7FEF2201B740 info 'utilvpxdVdb'] Registry Item DB 5 value is '' 2018-10-24T21:50:52.437Z [7FEF2201B740 info 'utilvpxdVdb'] [VpxdVdb::SetDBType] Logging in to DSN: VMware VirtualCenter with username vc 2018-10-24T21:50:52.575Z [7FEF2201B740 info 'Default'] [VdbSchemaLoader::LookupSchemaFromDB] Loaded schema and index information from DB 2018-10-24T21:50:52.594Z [7FEF2201B740 info 'utilvpxdDbSchemaHash'] [VpxdDbSchemaHash::GetSchemaFilename]: sqlDir: /usr/lib/vmware-vpx/sql/ 2018-10-24T21:50:52.594Z [7FEF2201B740 info 'utilvpxdDbSchemaHash'] [VpxdDbSchemaHash::CheckSchema]: schema file: /usr/lib/vmware-vpx/sql/VCDBPostgreSQL.sql 2018-10-24T21:50:52.608Z [7FEF2201B740 info 'linuxvpxLdap_linux'] [LdapBackup] Making sure LDAP instance VMwareVCMSDS is running 2018-10-24T21:50:52.608Z [7FEF2201B740 info 'linuxvpxLdap_linux'] [LdapBackup] Checking service ldap for running 2018-10-24T21:50:52.635Z [7FEF2201B740 info 'Default'] Looking for vpx result filter at /usr/lib/vmware-vpx/vpxResultFilter.xml 2018-10-24T21:50:52.640Z [7FEF2201B740 info 'MoLock'] LRO tracing: enabled: false 2018-10-24T21:50:52.641Z [7FEF2201B740 info 'MoLock'] LRO tracing: LRO name: 2018-10-24T21:50:52.641Z [7FEF1836D700 info 'vpxdvpxdMain' opID=SWI-6058ed8] Pre-loading symbols... 2018-10-24T21:50:52.643Z [7FEF1836D700 info 'vpxdvpxdMain' opID=SWI-6058ed8] Done pre-loading symbols. 2018-10-24T21:50:52.657Z [7FEF2201B740 info 'Default'] Looking for vpx result filter at /usr/lib/vmware-vpx/vpxResultFilter.xml 2018-10-24T21:50:52.662Z [7FEF2201B740 info 'QueryServiceProvider'] Initializing QueryServiceProviderHelperImpl 2018-10-24T21:50:52.933Z [7FEF2201B740 info 'QueryServiceProvider'] Starting with generation number: 3615389 2018-10-24T21:50:52.933Z [7FEF2201B740 info 'vpxservicesMoServiceDirectory'] [ServiceDirectory] Local instance UUID: 14E1E7CF-D145-48EE-B85B-76BF190E817E 2018-10-24T21:50:52.936Z [7FEF2201B740 error 'linuxvpxLdap_linux'] [LDAP Client] Failed to modify LDAP entry cn=VIMAPI,cn=14E1E7CF-D145-48EE-B85B-76BF190E817E,ou=Instances,dc=virtualcenter,dc=vmware,dc=int: 0x1 (Operations error) 2018-10-24T21:50:52.936Z [7FEF2201B740 error 'linuxvpxLdap_linux'] [LDAP Client] LdapApi::modify_s: key (vmw-vc-SSLThumbprint) 2018-10-24T21:50:52.937Z [7FEF2201B740 error 'vpxservicesMoServiceDirectory'] [ServiceDirectory::UpdateServiceInfo] Error while modifying service-endpoint SSL thumbprint cn=VIMAPI,cn=14E1E7CF-D145-48EE-B85B-76BF190E817E,ou=Instanc es,dc=virtualcenter,dc=vmware,dc=int 2018-10-24T21:50:52.937Z [7FEF2201B740 error 'vpxdvpxdMain'] [Init] Init failed: Ldap error --> Backtrace: --> backtrace[00] rip 00007fef1bf80014 Vmacore::System::Stacktrace::CaptureWork(unsigned int) --> backtrace[01] rip 00007fef1be6a132 Vmacore::System::SystemFactoryImpl::CreateQuickBacktrace(Vmacore::Ref<Vmacore::System::Backtrace>&) --> backtrace[02] rip 00007fef1bdcb8a5 Vmacore::Throwable::Throwable(std::string const&) --> backtrace[03] rip 00007fef2304c49f Vmomi::Fault::SystemError::Exception::Exception(std::string const&) --> backtrace[04] rip 00007fef244f7e7b /usr/lib/vmware-vpx/vpxd(+0x235be7b) [0x7fef244f7e7b] --> backtrace[05] rip 00007fef244fa96d /usr/lib/vmware-vpx/vpxd(+0x235e96d) [0x7fef244fa96d] --> backtrace[06] rip 00007fef244faeb5 /usr/lib/vmware-vpx/vpxd(+0x235eeb5) [0x7fef244faeb5] --> backtrace[07] rip 00007fef23036c2f /usr/lib/vmware-vpx/vpxd(+0xe9ac2f) [0x7fef23036c2f] --> backtrace[08] rip 00007fef2304aec3 /usr/lib/vmware-vpx/vpxd(+0xeaeec3) [0x7fef2304aec3] --> backtrace[09] rip 00007fef2302ddae /usr/lib/vmware-vpx/vpxd(+0xe91dae) [0x7fef2302ddae] --> backtrace[10] rip 00007fef19eefc16 /lib64/libc.so.6(_libc_start_main+0xe6) [0x7fef19eefc16] --> backtrace[11] rip 00007fef2302d009 /usr/lib/vmware-vpx/vpxd(+0xe91009) [0x7fef2302d009] --> 2018-10-24T21:50:52.939Z [7FEF2201B740 error 'Default'] Failed to intialize VMware VirtualCenter. Shutting down... 2018-10-24T21:50:52.939Z [7FEF2201B740 info 'vpxdvpxdSupportManager'] Wrote uptime information

Despite all my Googling, I've not been able to resolve this. Can anyone help please?

Cheers.

[VMBoy79]

Is it a stateless ESXI host.?