ESXi host suddenly failed

[rebelfalls]

I had a situation recently where 4 of my hosts disconnected on the same day. It happened mostly out of office hours in the early morning. 

I'm trying to find out the reason so I checked my hostd.log and found errors like those below, but cannot find any real explanation for these errors online. Any suggestions?

2020-11-18T12:35:16.152Z info hostd[2832459] [Originator@6876 sub=Libs opID=c1d9c9a2] NetstackInstanceImpl: congestion control algorithm: newreno
2020-11-18T12:35:31.254Z error hostd[2839930] [Originator@6876 sub=Default] [LikewiseGetDomainJoinInfo:354] QueryInformation(): ERROR_FILE_NOT_FOUND (2/0):
-->
2020-11-18T12:35:35.666Z info hostd[2832352] [Originator@6876 sub=Solo.VmwareCLI opID=c1d9c9ef user=dcui:vsanmgmtd] Dispatch list
2020-11-18T12:35:35.667Z info hostd[2832352] [Originator@6876 sub=Libs opID=c1d9c9ef user=dcui:vsanmgmtd] VsanInfoImpl: Refresh config generation
2020-11-18T12:35:35.669Z info hostd[2832442] [Originator@6876 sub=Solo.VmwareCLI opID=c1d9c9f0 user=dcui:vsanmgmtd] Dispatch list
2020-11-18T12:35:35.671Z info hostd[2832442] [Originator@6876 sub=Solo.VmwareCLI opID=c1d9c9f0 user=dcui:vsanmgmtd] Dispatch list done
2020-11-18T12:35:35.681Z info hostd[2832317] [Originator@6876 sub=Solo.VmwareCLI opID=c1d9c9f4 user=dcui:vsanmgmtd] Dispatch list
2020-11-18T12:35:35.682Z info hostd[2832317] [Originator@6876 sub=Solo.VmwareCLI opID=c1d9c9f4 user=dcui:vsanmgmtd] Dispatch list done
2020-11-18T12:35:35.682Z info hostd[2832317] [Originator@6876 sub=Solo.VmwareCLI opID=c1d9c9f4 user=dcui:vsanmgmtd] Dispatch list failed
2020-11-18T12:35:35.682Z info hostd[2832317] [Originator@6876 sub=Default opID=c1d9c9f4 user=dcui:vsanmgmtd] AdapterServer caught exception: N5Vmomi18DynamicMethodFault9ExceptionE(Fault cause: vim.EsxCLI.CLIFault
--> )
2020-11-18T12:35:35.683Z info hostd[2832317] [Originator@6876 sub=Solo.Vmomi opID=c1d9c9f4 user=dcui:vsanmgmtd] Activation [N5Vmomi10ActivationE:0x00000028181aa9a0] : Invoke done [list] on [vim.EsxCLI.network.ip.route.ipv6:ha-cli-handler-network-ip-route-ipv6]
2020-11-18T12:35:35.683Z verbose hostd[2832317] [Originator@6876 sub=Solo.Vmomi opID=c1d9c9f4 user=dcui:vsanmgmtd] Arg netstack:
--> (null)
2020-11-18T12:35:35.683Z info hostd[2832317] [Originator@6876 sub=Solo.Vmomi opID=c1d9c9f4 user=dcui:vsanmgmtd] Throw vim.EsxCLI.CLIFault
2020-11-18T12:35:35.683Z info hostd[2832317] [Originator@6876 sub=Solo.Vmomi opID=c1d9c9f4 user=dcui:vsanmgmtd] Result:
--> (vim.EsxCLI.CLIFault) {
--> faultCause = (vmodl.MethodFault) null,
--> faultMessage = <unset>,
--> errMsg = (string) [
--> "Ipv6 not Enabled"
--> ]
--> msg = ""
--> }
2020-11-18T12:36:31.261Z error hostd[2840069] [Originator@6876 sub=Default] [LikewiseGetDomainJoinInfo:354] QueryInformation(): ERROR_FILE_NOT_FOUND (2/0):
-->
2020-11-18T12:36:53.090Z info hostd[2832444] [Originator@6876 sub=Default] IPMI SEL sync took 0 seconds 0 sel records, last 1
2020-11-18T12:36:53.091Z info hostd[2832351] [Originator@6876 sub=Libs opID=c1d9caf2] NetstackInstanceImpl: congestion control algorithm: newreno
2020-11-18T12:36:53.091Z info hostd[2832351] [Originator@6876 sub=Libs opID=c1d9caf2] NetstackInstanceImpl: congestion control algorithm: newreno
2020-11-18T12:36:53.091Z info hostd[2832351] [Originator@6876 sub=Libs opID=c1d9caf2] NetstackInstanceImpl: congestion control algorithm: newreno
2020-11-18T12:37:31.268Z error hostd[2840090] [Originator@6876 sub=Default] [LikewiseGetDomainJoinInfo:354] QueryInformation(): ERROR_FILE_NOT_FOUND (2/0):
-->
2020-11-18T12:37:33.050Z warning hostd[2832317] [Originator@6876 sub=Hostsvc.NetworkProvider] Skip saving dvport xxx to : filePath empty
2020-11-18T12:37:33.050Z warning hostd[2832317] [Originator@6876 sub=Hostsvc.NetworkProvider] Skip saving dvport xxx to : filePath empty
2020-11-18T12:37:33.050Z warning hostd[2832317] [Originator@6876 sub=Hostsvc.NetworkProvider] Skip saving dvport xxxto : filePath empty
2020-11-18T12:37:33.050Z warning hostd[2832317] [Originator@6876 sub=Hostsvc.NetworkProvider] Skip saving dvport xxx to : filePath empty
2020-11-18T12:37:33.051Z warning hostd[2832317] [Originator@6876 sub=Hostsvc.NetworkProvider] Skip saving dvport vxxxto : filePath empty
2020-11-18T12:37:33.051Z warning hostd[2832317] [Originator@6876 sub=Hostsvc.NetworkProvider] Skip saving dvport xxx to : filePath empty
2020-11-18T12:37:33.051Z warning hostd[2832317] [Originator@6876 sub=Hostsvc.NetworkProvider] Skip saving dvport xxx to : filePath empty
2020-11-18T12:38:30.231Z info hostd[2832450] [Originator@6876 sub=Default] IPMI SEL sync took 0 seconds 0 sel records, last 1
2020-11-18T12:38:30.232Z info hostd[2832459] [Originator@6876 sub=Libs opID=c1d9cc56] NetstackInstanceImpl: congestion control algorithm: newreno
2020-11-18T12:38:30.232Z info hostd[2832459] [Originator@6876 sub=Libs opID=c1d9cc56] NetstackInstanceImpl: congestion control algorithm: newreno
2020-11-18T12:38:30.232Z info hostd[2832459] [Originator@6876 sub=Libs opID=c1d9cc56] NetstackInstanceImpl: congestion control algorithm: newreno
2020-11-18T12:38:31.276Z error hostd[2840304] [Originator@6876 sub=Default] [LikewiseGetDomainJoinInfo:354] QueryInformation(): ERROR_FILE_NOT_FOUND (2/0):
-->
2020-11-18T12:39:27.741Z info hostd[2832440] [Originator@6876 sub=Vimsvc.TaskManager opID=sps-Main-404103-685-71-24-cd2a user=vpxuser:VSPHERE.LOCAL\vpxd-extension-98991ed4-2a80-4062-a465-b6fe60dc215b] Task Created : haTask--vim.vslm.host.CatalogSyncManager.queryCatalogChange-3613934940
2020-11-18T12:39:27.742Z info hostd[2832442] [Originator@6876 sub=Default opID=sps-Main-404103-685-71-24-cd2a user=vpxuser:VSPHERE.LOCAL\vpxd-extension-98991ed4-2a80-4062-a465-b6fe60dc215b] Transfer to exception eraro code: 403, message:
2020-11-18T12:39:27.743Z info hostd[2832442] [Originator@6876 sub=Default opID=sps-Main-404103-685-71-24-cd2a user=vpxuser:VSPHERE.LOCAL\vpxd-extension-98991ed4-2a80-4062-a465-b6fe60dc215b] AdapterServer caught exception: N3Vim5Fault8NotFound9ExceptionE(Fault cause: vim.fault.NotFound
--> )
2020-11-18T12:39:27.746Z info hostd[2832442] [Originator@6876 sub=Vimsvc.TaskManager opID=sps-Main-404103-685-71-24-cd2a user=vpxuser:VSPHERE.LOCAL\vpxd-extension-] Task Completed : haTask--vim.vslm.host.CatalogSyncManager.queryCatalogChange-3613934940 Status error
2020-11-18T12:39:31.285Z error hostd[2840311] [Originator@6876 sub=Default] [LikewiseGetDomainJoinInfo:354] QueryInformation(): ERROR_FILE_NOT_FOUND (2/0):

[scott28tt]

@rebelfalls 

Moderator: Please do not create multiple threads on the same topic - I see 2 other threads which have now been archived.

[madkid82]

Can you share the other threads>? I cannot find them

[Virt-aid]

Rebelfalls

what do you mean by failed? Did the host rebooted on its own?
Are these hostd logs are the time of the issue? please share vmkernel and vobd logs prior to reboot.

If it’s auto rebooted, the usual suspect will be a BIOS configuration or power management. 

[vmwareusermatth]

Seeing the same issue. Did you ever find the root cause?

2023-12-09T07:23:53.055Z info hostd[1256262] [Originator@6876 sub=Default] IPMI SEL sync took 0 seconds 0 sel records, last 39

2023-12-09T07:23:53.075Z warning hostd[1256262] [Originator@6876 sub=Default] load_v6_addresses: Ignore BMC V6 IP addr 56, not active, state=1

2023-12-09T07:23:53.096Z warning hostd[1256262] [Originator@6876 sub=Default] load_v6_addresses: Ignore BMC V6 IP addr 59, not active, state=1

2023-12-09T07:23:53.122Z info hostd[1256181] [Originator@6876 sub=Libs opID=ea87f552] NetstackInstanceImpl: congestion control algorithm: newreno

2023-12-09T08:07:12.048Z - time the service was last started 2023-12-09T08:07:12.043Z, Section for VMware ESX, pid=1051004, version=7.0.3, build=22348816, option=Release