503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http16LocalServiceSpecE:0x000000a63051bb30] _serverNamespace = / action = Allow _port = 8309)

[cattlerustler]

Help    

I have ESXi 6.7 running, after a funny issue with the webportal not working correctly, stuck just before the login screen, I reset the management agents, now all I presented with is

503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http16LocalServiceSpecE:0x000000a63051bb30] _serverNamespace = / action = Allow _port = 8309)

Have tried resetting the management agents again and the network agent both via ssh and on the console, I'm now at a loss please help me.

Thanks

Adrian

[ronnieferrell]

---

@Mhtd wrote:

I used "services.sh restart"  command with SSH to restart all management agents on the host,

It worked for me.

---

This was the ticket for me. thanks you!

[TD99]

Having same issue (Host not responding in vCenter).

• VMware ESXi, 6.7.0, 18828794
  
• Model: HPE ProLiant DL360 Gen9

Here are the troubleshooting steps I performed:

1. Restarted Management agents in ESXi Using Direct Console User Interface (DCUI).
   
   2. Tried changing management IP and DNS name to something else.
   
   3. Restarted the ESXi host daemon and vCenter Agent services using these commands:
   /etc/init.d/hostd restart
   /etc/init.d/vpxa restart
   
   4. Confirmed the services were running:
   [root@esx-kp-02:~] /etc/init.d/vpxa status
   vpxa is running
   
   [root@esx-kp-02:~] /etc/init.d/hostd status
   hostd is running.
   
   5. Restarted all management agents on the host, using the command:
   services.sh restart
   
   6. Tried to reset the management network on VMkernel interface "vmk0", using the command (which failed):
   [root@esx-kp-02:~] esxcli network ip interface set -e false -i vmk0; esxcli network ip interface set -e true -i vmk0
   Connection failed
   Connection failed
   
   7. Attempts to access ESXi web interface at "https://esx-kp-02.hs.local/" results in error:
   503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http16LocalServiceSpecE:0x00000093edeaac30] _serverNamespace = / action = Allow _port = 8309)

Tried everything above (except rebooting the host which fixes the issue 95% of the time). No luck.

Contacted HPE Support who says the internal microSD card (NAND flash) where ESXi is installed, is failing. SCSI write commands sent to it are not responding.

In the ESXi DCUI, I pressed Alt-F12 and saw highlighted rows referring to "mpx.vmhba32" which is the USB Storage Controller on the host. See attached screenshots.

Only solution is to reboot host and place into Maintenance Mode, then set-up for microSD card replacement.

[ssboy]

Hi

I'll tell you about my "adventure"

Homelab Version: 7.0 Update 3  without vCenter. A very simple network config: two homelab connected to a lan and a internet connection via router.

A few days ago, suddenly, I can't connect to the host and the error in question appears. I connected the host via SSH, restarted the services (services.sh restart), restarted the server ... I tried everything! I was able to contact the host but after few seconds the error in question appeared.

In the same network I have another homelab 6.0.0 (Build 3620759) which instead worked correctly. and was always reachable.

Everything in the network seemed to work correctly ... except one thing: due to a failure of the provider, the internet connection was absent! When the internet connection was restored, after restarting the host services, it was possible to connect to the host and the error did not reoccur.
I did some tests and the result is always the same: if I disconnect the internet for more than 5-10 minutes while I am connected to the host, the host becomes unreachable with error. By restoring the internet connection, after restarting the services, the host becomes reachable again and remains reachable as long as the internet connection remains stable.

.Does the host need to contact some external source? What causes the error?

Simon

[achandler85]

Guys,
I just fixed if for my scenario:
- ESXi was controlled by Vcenter
- I removed veeam backup of one VM on the ESXi
- Then ESXi required disk consolidation for this VM
- Then ESXi became unavailable
- Then I restarted this service via CLI
- Then I restarted this service via KVM (It wasn't available until I reset it via cli)
- Vcenter also wasn't able to communicate with this ESXi
- Then cat /etc/vmware/vpxa/vpxa.cfg
You should see ESXi and VCenter IP addresses.
I removed Vcenter IP address using vi /etc/vmware/vpxa/vpxa.cfg
and restarted vpxa service
- Then ESXi Web interface became available and I reconnected it in Vcenter
- Then cat /etc/vmware/vpxa/vpxa.cfg again - the Vcenter IP was added back

[sanchezmwhs]

Hey achandler85

I am running into this issue and I also have Veeam as a backup service on many vms I am wondering if veeam is causing this because a restart works but then it just breaks again.

[rlucky1]

Hi, I've got the same issue and after checking status, I do see that vsphere-ui is stopped. When trying to start it I get this error:

C:\Program Files\VMware\vCenter Server\bin>service-control --start vsphere-ui
Operation not cancellable. Please wait for it to finish...
Performing start operation on service vsphere-ui...
Error executing start on service vsphere-ui. Details {
"componentKey": null,
"resolution": null,
"problemId": null,
"detail": [
{
"translatable": "An error occurred while starting service '%(0)s'",
"id": "install.ciscommon.service.failstart",
"localized": "An error occurred while starting service 'vsphere-ui'"
,
"args": [
"vsphere-ui"
]
}
]
}
Service-control failed. Error: {
"componentKey": null,
"resolution": null,
"problemId": null,
"detail": [
{
"translatable": "An error occurred while starting service '%(0)s'",
"id": "install.ciscommon.service.failstart",
"localized": "An error occurred while starting service 'vsphere-ui'"
,
"args": [
"vsphere-ui"
]
}
]
}

Do you have any ideas?

-Lucky

[EBMgmbH]

You have to restart the Management Agents.

from the Terminal: F2 - connect as Root/Troubleshooting Options/Restart Management Agents/
Enter F11 Enter Escape
via Putty:
service mgmt-vmware restart

Usually you also have the message HA Master Agent was not found as an error message in Vsphere.

[ilsava]

it's because the ssl certificate is invalid, try to check these two files rui.key and rui.crt in /etc/vmware/ssl, use the two file certificate that are either valid or invalid from vmware installer or you can get it from the installer esxi from vmware workstation installed esxi 6.7 then copy the two files, then restart using /etc/init.d/hostd restart then check the browser ui again, the crux of your problem is only in those two files.

[adamabel1]

I ran into this last week while swapping out a dedicated storage NIC for another that was using the iSCSI software adapter.  I rebooted my host because the datastores were not coming up as "not consumed"  when the system came back up after a very long boot up process I got the 503 unavailable error.  I tried all the fixes listed here to date and none of them work.  What I found out after I had reinstalled the OS and it came back up is that it was related to the iSCSI connection to my Netapp array.  After I reinstalled I setup the software iSCSI adapter again and when I went to reboot to fix another issue I got the same 503 unavailable error.  Once I took the iqn's out of the initiator group on the Netapp and rebooted everything worked again.  I'm not sure why this is since the OS is stored on a local SSD mirrored set of drive on the host.  Hopefully this help someone else running into this problem. 

[meaguilarg]

Spoiler

It work!

[gaseker]

Saved my day...