Foxdemon
Contributor
Contributor

Cant login to vsphere webgui and 1 of esxi hosts after reboot of vcenter server.

Jump to solution

First thing first, I have to admit we are a very inexperienced team, following after a lot of senior engineers left our company. We are rebuilding our staging servers, which havent been touched for about 250 days. So after clogging and rebuilding our veeam server, we wanted to start on the vcenter infrastructure.

After rebooting our Vcenter server VM, we can access the Webgui of vsphere, but it doesnt show anything. The webgui of Vcenter gave the following error: "Could not connect to one or more vCenter Server Systems:vcenter1.local:443/sdk error in the vSphere Web Client. It shows nothing but a white inventory etc.

Also the first ESXi host cant be accessed anymore from the webgui. We have 3 hosts of which 2 of them can be accessed, (also the one with the vcenter server). The host that cant be accessed does response to Ping request.

Cause of this all the whole setup doesnt work anymore. Veeam cant access the VM's to backup etc. Does someone know what to do?

Thanks in advance,

Henri

Tags (3)
1 Solution

Accepted Solutions
AhmedIbrahimVMw
Enthusiast
Enthusiast

The iDRAC is a feature that allows you to connect to the server using an out-of-band management network.

It requires a separate IP address, which needs to be configured from the server’s BIOS, and of course, requires a physical network connectivity from the network switch to a specific port on the server.

If you don’t have a physical or remote access to the server, then I don’t think there’s much you can do right now.

You might want to wait till you or somebody else have a way to access to this server to restart the management agents to see if this fixes the issue or not.

If the issue persists, then you might need to safely shut down the VMs using SSH if they’re Linux or RDP if they’re Windows, then reboot the server.

View solution in original post

8 Replies
Foxdemon
Contributor
Contributor

Forgot to mention: when typing vcenter server ip address it goes to the empty Webgui. But when typing its hostname its gives the following error:

503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x00007f7f1007b7d0] _serverNamespace = / action = Allow _pipeName =/var/run/vmware/vpxd-webserver-pipe)

0 Kudos
AhmedIbrahimVMw
Enthusiast
Enthusiast

Hi

Regarding the host that doesn’t respond to ping requests, you’ll need to open its console (using iLO, iDRAC, KVM, etc. ) to confirm that it is up and running, and is configured with the correct TCP/IP settings.

Regarding the vCenter Server, the error message might indicate that one of the services is not running correctly, specifically the VPXD service.

Hot check the service status, you’ll need to log into the vCenter Server machine using SSH if it is an appliance, or RDP if it is Windows then run the following command:

service-control —status vpxd

if it is stopped, then run the following command:

service-control —start —all

Then run the following command to confirm that the VPXD is running :

service-control —status —all

You can find more info  about this command in the following KB article:

https://kb.vmware.com/s/article/2109887

Foxdemon
Contributor
Contributor

Thanks Ahmed,

This steps have resolve the issue with the vcenter server as the vmware-vpxd was indeed not running. Still left with one of the isxhosts not connected with the error:

Cannot contact the specified host (172.16.91.51). The host may not be available on the network, a network configuration problem may exist, or the management services on this host may not be responding.

Ping request to the host does work, but thats it. No ssh, and no webgui. When nmap the host it gives the following output:

PORT     STATE  SERVICE

22/tcp   closed ssh

80/tcp   open   http

427/tcp  open   svrloc

443/tcp  open   https

902/tcp  open   iss-realsecure

2222/tcp closed EtherNetIP-1

5988/tcp closed wbem-http

5989/tcp open   wbem-https

8000/tcp open   http-alt

8080/tcp closed http-proxy

8300/tcp open   tmi

9080/tcp open   glrpc

(edit) Something to add. I can ping the VM's that are running on the host.

0 Kudos
AhmedIbrahimVMw
Enthusiast
Enthusiast

Log in the ESXi shell, either physically or remotely using a console like iLO, iDRAC, etc., then run the following command:

services.sh restart

Read the following KB for cautions that should be taken into consideration before you run this command:

https://kb.vmware.com/s/article/1003490

Once the command finishes, try to type the IP address of the ESXi into a web browser, to see how it goes.

If you successfully connect to the ESXi web interface, then check the connectivity from vCenter Server client

Foxdemon
Contributor
Contributor

Thanks again! Maybe this are some basic questions, but as i said we dont have much experience with this issues, so this is helping me a lot.

The hosts are located off site, so i cant access them physically right away. This will be the last resort i guess. But i found out that there are iDRAC on them. (it are Dell PowerEdge R710's)

I have installed the srvadmin-all package on my linux laptop, start the service and try to connect to the host? with https://<ipadress>:1311 This doesn't seem to work. Is the iDRAC IP the same as the host IP?

The only other IP-adresses that i can find in my network that are unknown to us (which are exactly 6, as there are 6 hosts) have no open ports what so ever. Can this be that the iDRAC's aren't configured or something

Thanks in advance

0 Kudos
AhmedIbrahimVMw
Enthusiast
Enthusiast

The iDRAC is a feature that allows you to connect to the server using an out-of-band management network.

It requires a separate IP address, which needs to be configured from the server’s BIOS, and of course, requires a physical network connectivity from the network switch to a specific port on the server.

If you don’t have a physical or remote access to the server, then I don’t think there’s much you can do right now.

You might want to wait till you or somebody else have a way to access to this server to restart the management agents to see if this fixes the issue or not.

If the issue persists, then you might need to safely shut down the VMs using SSH if they’re Linux or RDP if they’re Windows, then reboot the server.

Foxdemon
Contributor
Contributor

Thanks again, yeah the problem is, this is our staging and testing environment and there is zero documentation on it whatsoever. The person who managed this has left the company and we are left with it without much knowledge how the infrastructure is build up etc. I will try to find out if there is someone who knows about the physical network connection of the iDRAC.

If that doesn't yield anything i guess the only option is to shut down the VM's that are on it (I know i can reach them through SSH), and then go to the server location to see what we can do there.

With a last resort in shutting it down and power it on manually.

Thanks for all your answers

0 Kudos
Foxdemon
Contributor
Contributor

So wanted to let you know, and maybe someone who has the same problem later on.

I have been to the datacenter and rebooted the ESXi-host. This solved the problems, as it restarted all the services.

I also found the iDRAC Ip adresses of all the hosts in the cluster on the little screen in front of the racks. (And documented it..)

So in the future we will be able to resolve this kind of issues on a remote location.

Henri

0 Kudos