3 Replies Latest reply on May 23, 2020 11:15 AM by a.p.

    VCSA 6.5 won't boot start vpxd after thunderstorm unclean shutdown

    cgetut1 Lurker

      I have a previously working vcenter appliance 6.5 that was in a bad state after an unclean shutdown from a thunderstorm. Initially it was not fully booting, but I ran fsck on the volumes that needed it, and it gets all the way to the running info screen. After that vpxd fails to fully start. I can't find any more volumes that are in a bad state, but it appears that I might have some SQL corruption. Note the SQL error below.

       

      Error message when trying to open web client is: 503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x00007faab404a230] _serverNamespace = / action = Allow _pipeName =/var/run/vmware/vpxd-webserver-pipe)

       

      Output of service-control --status --all

      Running:

      applmgmt lwsmd pschealth vmafdd vmcad vmdird vmdnsd vmonapi vmware-analytics vmware-cis-license vmware-cm vmware-content-library vmware-eam vmware-perfcharts vmware-pod vmware-postgres-archiver vmware-rbd-watchdog vmware-rhttpproxy vmware-sca vmware-sps vmware-statsmonitor vmware-sts-idmd vmware-stsd vmware-updatemgr vmware-vapi-endpoint vmware-vmon vmware-vpostgres vmware-vpxd-svcs vmware-vsan-health vmware-vsm vsphere-client vsphere-ui

      Stopped:

      vmcam vmware-imagebuilder vmware-mbcs vmware-netdumper vmware-vcha vmware-vpxd vsan-dps

       

       

      Grep of all lines with error in them from vpxd log file:

      2020-05-23T15:13:12.693Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Unable to read from '/etc/motd':N7Vmacore23FileIONotFoundExceptionE(Could not find file  : /etc/motd)

      2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: AgentUpgrade.autoUpgradeAgents

      2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: AgentUpgrade.checkPeriodSeconds

      2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: alarms.upgraded

      2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: DBProc.Log.Level.Stats.Purge1

      2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: DBProc.Log.Level.Stats.Purge2

      2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: DBProc.Log.Level.Stats.Purge3

      2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: DBProc.Log.Level.Stats.Rollup1

      2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: DBProc.Log.Level.Stats.Rollup2

      2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: DBProc.Log.Level.Stats.Rollup3

      2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: DBProc.Log.Level.Topn.Calc1

      2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: DBProc.Log.Level.Topn.Calc2

      2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: DBProc.Log.Level.Topn.Calc3

      2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: DBProc.Log.Level.Topn.Calc4

      2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: DBProc.Log.Level.Topn.Purge1

      2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: DBProc.Log.Level.Topn.Purge2

      2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: DBProc.Log.Level.Topn.Purge3

      2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: DBProc.Log.Level.Topn.Purge4

      2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: TOPN_LOG_BUFFER

      2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: TOPN_LOGING_MODE

      2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: TOPN_STATS_DELAY_MINS_1

      2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: TOPN_STATS_DELAY_MINS_2

      2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: TOPN_STATS_DELAY_MINS_3

      2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: TOPN_STATS_DELAY_MINS_4

      2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: VirtualCenter.CacheSize

      2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: VirtualCenter.LDAPAdminPrincipal

      2020-05-23T15:13:12.700Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Ignoring unknown entry from DB: VirtualCenter.VimWebServicesUrl

      2020-05-23T15:13:12.704Z error vpxd[04354] [Originator@6876 sub=MoOptionMgr] Skipping bad entry config.vpxd.enableDebugBrowse from DB. Resetting to default.Exception: Fault cause: vmodl.fault.InvalidArgument

      2020-05-23T15:13:15.388Z error vpxd[04354] [Originator@6876 sub=Vsan] Data type vim.cluster.VsanClusterHclInfo already exists.

      2020-05-23T15:13:15.388Z error vpxd[04354] [Originator@6876 sub=Vsan] Managed type vim.cluster.VsanClusterHealthSystem already exists.

      2020-05-23T15:13:15.389Z error vpxd[04354] [Originator@6876 sub=Vsan] Data type vim.host.VsanHclControllerInfo already exists.

      2020-05-23T15:13:15.389Z error vpxd[04354] [Originator@6876 sub=Vsan] Managed type vim.host.VsanHealthSystem already exists.

      2020-05-23T15:13:15.389Z error vpxd[04354] [Originator@6876 sub=Vsan] Data type vim.host.VsanHostHclInfo already exists.

      2020-05-23T15:13:15.389Z error vpxd[04354] [Originator@6876 sub=Vsan] Managed type vim.host.VsanDiskManagementSystem already exists.

      2020-05-23T15:13:15.389Z error vpxd[04354] [Originator@6876 sub=Vsan] Data type vim.vsan.host.DiskMapInfoEx already exists.

      2020-05-23T15:13:15.389Z error vpxd[04354] [Originator@6876 sub=Vsan] Data type vim.vsan.host.VsanDiskManagementSystemCapability already exists.

      2020-05-23T15:13:15.389Z error vpxd[04354] [Originator@6876 sub=Vsan] Data type vim.host.VSANCmmdsFaultDomainInfo already exists.

      2020-05-23T15:13:15.389Z error vpxd[04354] [Originator@6876 sub=Vsan] Data type vim.host.VSANCmmdsNodeInfo already exists.

      2020-05-23T15:13:15.389Z error vpxd[04354] [Originator@6876 sub=Vsan] Data type vim.host.VSANCmmdsPreferredFaultDomainInfo already exists.

      2020-05-23T15:13:15.389Z error vpxd[04354] [Originator@6876 sub=Vsan] Data type vim.host.VSANStretchedClusterHostCapability already exists.

      2020-05-23T15:13:15.389Z error vpxd[04354] [Originator@6876 sub=Vsan] Data type vim.host.VSANStretchedClusterHostInfo already exists.

      2020-05-23T15:13:15.389Z error vpxd[04354] [Originator@6876 sub=Vsan] Managed type vim.host.VsanStretchedClusterSystem already exists.

      2020-05-23T15:13:19.242Z error vpxd[04354] [Originator@6876 sub=profileUtil] [DeserializeFromFile] reading failed: Could not find file  : /var/lib/vmware/hpMetadataCache.xml

      2020-05-23T15:13:22.233Z error vpxd[04569] [Originator@6876 sub=Main opID=CheckCertificateExpiry-35702e2f] Unable to get certificate count for APPLMGMT_PASSWORD from VECS localhost, error: 0

      2020-05-23T15:13:23.413Z error vpxd[04435] [Originator@6876 sub=vmomi.soapStub[12]] initial service state request failed, disabling pings. error=HTTP Status:400 'Bad Request'

      2020-05-23T15:13:23.729Z error vpxd[04354] [Originator@6876 sub=AuthorizeManager] [ACL] Adding unresolved permission for user "VSPHERE.LOCAL\Administrator"

      2020-05-23T15:13:23.729Z error vpxd[04354] [Originator@6876 sub=AuthorizeManager] [ACL] Adding unresolved permission for group "CAROLINA\Domain Admins"

      2020-05-23T15:13:24.280Z error vpxd[04354] [Originator@6876 sub=AuthorizeManager] [ACL] Adding unresolved permission for user "VSPHERE.LOCAL\Administrator"

      2020-05-23T15:13:24.280Z error vpxd[04354] [Originator@6876 sub=AuthorizeManager] [ACL] Adding unresolved permission for group "VSPHERE.LOCAL\Administrators"

      2020-05-23T15:13:24.280Z error vpxd[04354] [Originator@6876 sub=AuthorizeManager] [ACL] Adding unresolved permission for user "VSPHERE.LOCAL\vsphere-webclient-80cfe84e-9df1-47e4-93e3-d2dfc3e72d72"

      2020-05-23T15:13:24.280Z error vpxd[04354] [Originator@6876 sub=AuthorizeManager] [ACL] Adding unresolved permission for user "VSPHERE.LOCAL\vpxd-80cfe84e-9df1-47e4-93e3-d2dfc3e72d72"

      2020-05-23T15:13:24.280Z error vpxd[04354] [Originator@6876 sub=AuthorizeManager] [ACL] Adding unresolved permission for user "VSPHERE.LOCAL\vpxd-extension-80cfe84e-9df1-47e4-93e3-d2dfc3e72d72"

      2020-05-23T15:13:24.346Z error vpxd[04668] [Originator@6876 sub=vmomi.soapStub[13]] initial service state request failed, disabling pings. error=HTTP Status:400 'Bad Request'

      2020-05-23T15:13:25.152Z error vpxd[04354] [Originator@6876 sub=OsLayer_linux] [VpxOsLayer] Failed to write to config: Permission denied for file  : /etc/vmware-vpx/vpxd.cfg.tmp

      2020-05-23T15:14:05.925Z error vpxd[04454] [Originator@6876 sub=[SSO] opID=26e74229] [UserDirectorySso] GetUserInfo exception: N7Vmacore9Authorize25AuthUserNotFoundExceptionE(User localos\com.vmware.vim.eam)

      2020-05-23T15:14:05.927Z error vpxd[04454] [Originator@6876 sub=[SSO] opID=26e74229] [UserDirectorySso] NormalizeUserName(com.vmware.vim.eam, false) exception: N7Vmacore9Authorize25AuthUserNotFoundExceptionE(User localos\com.vmware.vim.eam)

      2020-05-23T15:14:10.713Z error vpxd[04429] [Originator@6876 sub=HttpSvc.HTTPService] Failed to read request; stream: <io_obj p:0x00007f4f5039b388, h:-1, <UNIX '/var/run/vmware/vpxd-webserver-pipe'>, <UNIX ''> FD Closed>, error: N7Vmacore16TimeoutExceptionE(Operation timed out: Stream: <io_obj p:0x00007f4f5039b388, h:-1, <UNIX '/var/run/vmware/vpxd-webserver-pipe'>, <UNIX ''> FD Closed>, duration: 00:00:45.250489 (hh:mm:ss.us))

      2020-05-23T15:16:17.602Z error vpxd[04458] [Originator@6876 sub=MoEnvBrowser opID=75353f56] Can not find option descriptor for key vmx-14

      2020-05-23T15:20:00.010Z error vpxd[04467] [Originator@6876 sub=Default] [VdbStatement] Execute result code: -1

      2020-05-23T15:20:00.010Z error vpxd[04467] [Originator@6876 sub=Default] [VdbStatement] SQL execution failed:  select rule_topn1_proc()

      2020-05-23T15:20:00.010Z error vpxd[04467] [Originator@6876 sub=Default] [VdbStatement] Execution elapsed time: 11 ms

      2020-05-23T15:20:00.010Z error vpxd[04467] [Originator@6876 sub=Default] [VdbStatement] Statement diagnostic data from driver is XX001:0:1:ERROR: could not read block 2389 in file "base/16395/32425": read only 0 of 8192 bytes;

      2020-05-23T15:20:00.010Z error vpxd[04467] [Originator@6876 sub=Default] [VdbStatement] Connection diagnostic data from driver is XX001:0:110:ERROR: could not read block 2389 in file "base/16395/32425": read only 0 of 8192 bytes

      2020-05-23T15:20:00.010Z error vpxd[04467] [Originator@6876 sub=Default] [VdbStatement] Bind parameters:

      2020-05-23T15:20:00.010Z error vpxd[04467] [Originator@6876 sub=Default] [Vdb::IsRecoverableErrorCode] Unable to recover from XX001:1

      2020-05-23T15:20:00.010Z error vpxd[04467] [Originator@6876 sub=Default] [Vdb::IsRecoverableErrorCode] Unable to recover from XX001:110

      2020-05-23T15:20:00.010Z error vpxd[04467] [Originator@6876 sub=Default] [VdbStatement] SQLError was thrown: "ODBC error: (XX001) - ERROR: could not read block 2389 in file "base/16395/32425": read only 0 of 8192 bytes;