VCSA 6.5 won't boot start vpxd after thunderstorm unclean shutdown

[cgetut1]

I have a previously working vcenter appliance 6.5 that was in a bad state after an unclean shutdown from a thunderstorm. Initially it was not fully booting, but I ran fsck on the volumes that needed it, and it gets all the way to the running info screen. After that vpxd fails to fully start. I can't find any more volumes that are in a bad state, but it appears that I might have some SQL corruption. Note the SQL error below.

Error message when trying to open web client is: 503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x00007faab404a230] _serverNamespace = / action = Allow _pipeName =/var/run/vmware/vpxd-webserver-pipe)

Output of service-control --status --all

Running:

applmgmt lwsmd pschealth vmafdd vmcad vmdird vmdnsd vmonapi vmware-analytics vmware-cis-license vmware-cm vmware-content-library vmware-eam vmware-perfcharts vmware-pod vmware-postgres-archiver vmware-rbd-watchdog vmware-rhttpproxy vmware-sca vmware-sps vmware-statsmonitor vmware-sts-idmd vmware-stsd vmware-updatemgr vmware-vapi-endpoint vmware-vmon vmware-vpostgres vmware-vpxd-svcs vmware-vsan-health vmware-vsm vsphere-client vsphere-ui

Stopped:

vmcam vmware-imagebuilder vmware-mbcs vmware-netdumper vmware-vcha vmware-vpxd vsan-dps

Grep of all lines with error in them from vpxd log file:

2020-05-23T15:13:12.693Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Unable to read from '/etc/motd':N7Vmacore23FileIONotFoundExceptionE(Could not find file  : /etc/motd)

2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: AgentUpgrade.autoUpgradeAgents

2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: AgentUpgrade.checkPeriodSeconds

2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: alarms.upgraded

2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: DBProc.Log.Level.Stats.Purge1

2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: DBProc.Log.Level.Stats.Purge2

2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: DBProc.Log.Level.Stats.Purge3

2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: DBProc.Log.Level.Stats.Rollup1

2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: DBProc.Log.Level.Stats.Rollup2

2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: DBProc.Log.Level.Stats.Rollup3

2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: DBProc.Log.Level.Topn.Calc1

2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: DBProc.Log.Level.Topn.Calc2

2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: DBProc.Log.Level.Topn.Calc3

2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: DBProc.Log.Level.Topn.Calc4

2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: DBProc.Log.Level.Topn.Purge1

2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: DBProc.Log.Level.Topn.Purge2

2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: DBProc.Log.Level.Topn.Purge3

2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: DBProc.Log.Level.Topn.Purge4

2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: TOPN_LOG_BUFFER

2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: TOPN_LOGING_MODE

2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: TOPN_STATS_DELAY_MINS_1

2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: TOPN_STATS_DELAY_MINS_2

2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: TOPN_STATS_DELAY_MINS_3

2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: TOPN_STATS_DELAY_MINS_4

2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: VirtualCenter.CacheSize

2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: VirtualCenter.LDAPAdminPrincipal

2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: VirtualCenter.VimWebServicesUrl

2020-05-23T15:13:12.704Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Skipping bad entry config.vpxd.enableDebugBrowse from DB. Resetting to default.Exception: Fault cause: vmodl.fault.InvalidArgument

2020-05-23T15:13:15.388Z error vpxd[04354] [Originator@6876 sub=Vsan] Data type vim.cluster.VsanClusterHclInfo already exists.

2020-05-23T15:13:15.388Z error vpxd[04354] [Originator@6876 sub=Vsan] Managed type vim.cluster.VsanClusterHealthSystem already exists.

2020-05-23T15:13:15.389Z error vpxd[04354] [Originator@6876 sub=Vsan] Data type vim.host.VsanHclControllerInfo already exists.

2020-05-23T15:13:15.389Z error vpxd[04354] [Originator@6876 sub=Vsan] Managed type vim.host.VsanHealthSystem already exists.

2020-05-23T15:13:15.389Z error vpxd[04354] [Originator@6876 sub=Vsan] Data type vim.host.VsanHostHclInfo already exists.

2020-05-23T15:13:15.389Z error vpxd[04354] [Originator@6876 sub=Vsan] Managed type vim.host.VsanDiskManagementSystem already exists.

2020-05-23T15:13:15.389Z error vpxd[04354] [Originator@6876 sub=Vsan] Data type vim.vsan.host.DiskMapInfoEx already exists.

2020-05-23T15:13:15.389Z error vpxd[04354] [Originator@6876 sub=Vsan] Data type vim.vsan.host.VsanDiskManagementSystemCapability already exists.

2020-05-23T15:13:15.389Z error vpxd[04354] [Originator@6876 sub=Vsan] Data type vim.host.VSANCmmdsFaultDomainInfo already exists.

2020-05-23T15:13:15.389Z error vpxd[04354] [Originator@6876 sub=Vsan] Data type vim.host.VSANCmmdsNodeInfo already exists.

2020-05-23T15:13:15.389Z error vpxd[04354] [Originator@6876 sub=Vsan] Data type vim.host.VSANCmmdsPreferredFaultDomainInfo already exists.

2020-05-23T15:13:15.389Z error vpxd[04354] [Originator@6876 sub=Vsan] Data type vim.host.VSANStretchedClusterHostCapability already exists.

2020-05-23T15:13:15.389Z error vpxd[04354] [Originator@6876 sub=Vsan] Data type vim.host.VSANStretchedClusterHostInfo already exists.

2020-05-23T15:13:15.389Z error vpxd[04354] [Originator@6876 sub=Vsan] Managed type vim.host.VsanStretchedClusterSystem already exists.

2020-05-23T15:13:19.242Z error vpxd[04354] [Originator@6876 sub=profileUtil] [DeserializeFromFile] reading failed: Could not find file  : /var/lib/vmware/hpMetadataCache.xml

2020-05-23T15:13:22.233Z error vpxd[04569] [Originator@6876 sub=Main opID=CheckCertificateExpiry-35702e2f] Unable to get certificate count for APPLMGMT_PASSWORD from VECS localhost, error: 0

2020-05-23T15:13:23.413Z error vpxd[04435] [Originator@6876 sub=vmomi.soapStub[12]] initial service state request failed, disabling pings. error=HTTP Status:400 'Bad Request'

2020-05-23T15:13:23.729Z error vpxd[04354] [Originator@6876 sub=AuthorizeManager] [ACL] Adding unresolved permission for user "VSPHERE.LOCAL\Administrator"

2020-05-23T15:13:23.729Z error vpxd[04354] [Originator@6876 sub=AuthorizeManager] [ACL] Adding unresolved permission for group "CAROLINA\Domain Admins"

2020-05-23T15:13:24.280Z error vpxd[04354] [Originator@6876 sub=AuthorizeManager] [ACL] Adding unresolved permission for user "VSPHERE.LOCAL\Administrator"

2020-05-23T15:13:24.280Z error vpxd[04354] [Originator@6876 sub=AuthorizeManager] [ACL] Adding unresolved permission for group "VSPHERE.LOCAL\Administrators"

2020-05-23T15:13:24.280Z error vpxd[04354] [Originator@6876 sub=AuthorizeManager] [ACL] Adding unresolved permission for user "VSPHERE.LOCAL\vsphere-webclient-80cfe84e-9df1-47e4-93e3-d2dfc3e72d72"

2020-05-23T15:13:24.280Z error vpxd[04354] [Originator@6876 sub=AuthorizeManager] [ACL] Adding unresolved permission for user "VSPHERE.LOCAL\vpxd-80cfe84e-9df1-47e4-93e3-d2dfc3e72d72"

2020-05-23T15:13:24.280Z error vpxd[04354] [Originator@6876 sub=AuthorizeManager] [ACL] Adding unresolved permission for user "VSPHERE.LOCAL\vpxd-extension-80cfe84e-9df1-47e4-93e3-d2dfc3e72d72"

2020-05-23T15:13:24.346Z error vpxd[04668] [Originator@6876 sub=vmomi.soapStub[13]] initial service state request failed, disabling pings. error=HTTP Status:400 'Bad Request'

2020-05-23T15:13:25.152Z error vpxd[04354] [Originator@6876 sub=OsLayer_linux] [VpxOsLayer] Failed to write to config: Permission denied for file  : /etc/vmware-vpx/vpxd.cfg.tmp

2020-05-23T15:14:05.925Z error vpxd[04454] [Originator@6876 sub=[SSO] opID=26e74229] [UserDirectorySso] GetUserInfo exception: N7Vmacore9Authorize25AuthUserNotFoundExceptionE(User localos\com.vmware.vim.eam)

2020-05-23T15:14:05.927Z error vpxd[04454] [Originator@6876 sub=[SSO] opID=26e74229] [UserDirectorySso] NormalizeUserName(com.vmware.vim.eam, false) exception: N7Vmacore9Authorize25AuthUserNotFoundExceptionE(User localos\com.vmware.vim.eam)

2020-05-23T15:14:10.713Z error vpxd[04429] [Originator@6876 sub=HttpSvc.HTTPService] Failed to read request; stream: <io_obj p:0x00007f4f5039b388, h:-1, <UNIX '/var/run/vmware/vpxd-webserver-pipe'>, <UNIX ''> FD Closed>, error: N7Vmacore16TimeoutExceptionE(Operation timed out: Stream: <io_obj p:0x00007f4f5039b388, h:-1, <UNIX '/var/run/vmware/vpxd-webserver-pipe'>, <UNIX ''> FD Closed>, duration: 00:00:45.250489 (hh:mm:ss.us))

2020-05-23T15:16:17.602Z error vpxd[04458] [Originator@6876 sub=MoEnvBrowser opID=75353f56] Can not find option descriptor for key vmx-14

2020-05-23T15:20:00.010Z error vpxd[04467] [Originator@6876 sub=Default] [VdbStatement] Execute result code: -1

2020-05-23T15:20:00.010Z error vpxd[04467] [Originator@6876 sub=Default] [VdbStatement] SQL execution failed:  select rule_topn1_proc()

2020-05-23T15:20:00.010Z error vpxd[04467] [Originator@6876 sub=Default] [VdbStatement] Execution elapsed time: 11 ms

2020-05-23T15:20:00.010Z error vpxd[04467] [Originator@6876 sub=Default] [VdbStatement] Statement diagnostic data from driver is XX001:0:1:ERROR: could not read block 2389 in file "base/16395/32425": read only 0 of 8192 bytes;

2020-05-23T15:20:00.010Z error vpxd[04467] [Originator@6876 sub=Default] [VdbStatement] Connection diagnostic data from driver is XX001:0:110:ERROR: could not read block 2389 in file "base/16395/32425": read only 0 of 8192 bytes

2020-05-23T15:20:00.010Z error vpxd[04467] [Originator@6876 sub=Default] [VdbStatement] Bind parameters:

2020-05-23T15:20:00.010Z error vpxd[04467] [Originator@6876 sub=Default] [Vdb::IsRecoverableErrorCode] Unable to recover from XX001:1

2020-05-23T15:20:00.010Z error vpxd[04467] [Originator@6876 sub=Default] [Vdb::IsRecoverableErrorCode] Unable to recover from XX001:110

2020-05-23T15:20:00.010Z error vpxd[04467] [Originator@6876 sub=Default] [VdbStatement] SQLError was thrown: "ODBC error: (XX001) - ERROR: could not read block 2389 in file "base/16395/32425": read only 0 of 8192 bytes;

[RajeevVCP4]

It seems your are using vcenter server appliances if yes SQL will not come in picture.

If you have external PSC

Do one thing

Power of Vcneter server then PSC

Power on PSC once it up check all service status

then power on VC after 10 minutes check service status

[cgetut1]

No.. only 3 systems in my infrastructure. Vcenter VCSA guest (virtual guest), and 3 ESXi hosts.

Nothing else is separate.

[a_p_]

Just a quick though.

In case that the  vCenter configuration isn't really complex, it may be worth considering to simply deploy a new vCSA, or restore it from a previous backup you have one.

André