vSphere 503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http16Local...

[basco2]

This is the 3rd time it happened since I started using ESXi 7.0. I'm currently running ver.7.0.2 kernel 17867351 on Dell PowerEdge R730

Host became unavaiable showing an error:

503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http16LocalServiceSpecE:0x000000d9f218bd00] _serverNamespace = / action = Allow _port = 8309)

vCenter was running but lost connection with host so I was unable to check anything, I couldn't open host in webrowser (error above), but I was able to login to the host from console and ssh.

I have tried restarting Restarting Management Agents, it didn't help.. 

Regarding solutions there's very little on the Internet 

Finally I ended up restarting ESXi, but this shouldn't be a solution. 

What's even worst my logs from /var/log after reboot are gone, I have only current ones is there any other place I should look for esxi logs?

[basco2]

Host became unavaiable showing an error:

503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http16LocalServiceSpecE:0x000000d9f218bd00] _serverNamespace = / action = Allow _port = 8309)

vCenter was running but lost connection with host so I was unable to check anything, I couldn't open host in webrowser (error above), but I was able to login to the host from console and ssh.

I have tried restarting Restarting Management Agents, it didn't help.. 

Regarding solutions there's very little on the Internet 

Finally I ended up restarting ESXi, but this shouldn't be a solution. 

What's even worst my logs from /var/log after reboot are gone, I have only current ones is there any other place I should look for esxi logs?

[Srijithk]

Hey Basco,

is the logs stored in anon persistent location? if yes then please persist the logging via https://kb.vmware.com/s/article/1033696 or https://kb.vmware.com/s/article/2003322

Now coming to the issue there could be multiple reasons but few of the primary are listed below

>> look at vmkernel log (prior reboot if present) and see if there are any scsi aborts be at device level or the host HBA itself, you can decode the sense codes using https://www.virten.net/vmware/vmware-esxi-scsi-sense-code-decoder-v2/

>> sometimes it can be a vpxa or sfcb crash as well or at times also a known issue with the build, if you have coredump configured look at the var/core directory and see if there are any z-dumps generated

>> this issue can also be with certain memory leaks in the process (you shall see the snippets in the form of admission failure in the kernel logs)

few articles on memory leaks

https://kb.vmware.com/s/article/76043
https://kb.vmware.com/s/article/82091
https://www.dell.com/support/kbdoc/en-in/000167824/dell-emc-vxrail-esxi-host-shows-not-responding-or...

>> Also the build you mentioned 17867351 if you are using uSB/SD cards as a boot device then look out for bootbank not found errors in vmkwarning.log if they are then patch the host to 7.0 u2 c as per the below articles

https://kb.vmware.com/s/article/83963

https://kb.vmware.com/s/article/83376

Thanks,

Srijith