ESXi not responding, what to do?

[WillL]

Hi,

Both ESXi hosts are not responding, show disconnected in vCenter how hosted VM are still running. Tried to telnet to ports like 22, 80, 443, 902, they all were closed right away by ESXi. It's in a remote location, I'm trying to get someone to take a look at console, these are our proof of concept servers for vCloud.

What options do we have? hard reset the power? What logs should I look at once online again?

Thanks,

William

[RaZaKKaZaR]

Hello William,

Have you considered downloading the vMA for free from VMware's Virtual Appliance site to remotely administer the servers? As long as you can ping the servers, vMA should be able to connect to them to execute commands. Alternatively, you could build a Linux VM and install the vCLI component to perform the same remote administration commands.

Telnet will not work on the ESXi servers.

Regards,

Trevor

=====================================================================================

If any of my responses have been helpful in any way, please rate accordinly. Thank you and Happy VM'ing! :smileygrin:

[WillL]

Telnet is just to check if ports are open, not the actual telnet protocol. As I mentioned, unable to ssh or browse the web page or using vSphere client. ESXi seem to close the any port right away for some reason, however it's still pingable on vmk0, the hosted VM I can still log into but not ESXi hosts.

I don't have vMA installed, too bad.

[WillL]

I'm also wondering if ESXi host is in a special mode which only allow local console access but hosted VMs are still accessible as normal.

[RaZaKKaZaR]

Hello William,

Yes, there is a local console mode that has a limited command set, but I believe you have to be at the physical box to access it. Usually, the VM Tech Support Team steps you through how to access it if you need to.

The vMA is a Virtual Appliance that can be run either on an ESXi Server that you can still manage or on VMware Workstation 7. You can download it for free from VMware's Virtual Appliance site. As long as you have network connectivity to the boxes, it should be able to connect for you to run commands remotely on those servers.

Regards,

Trevor

=====================================================================================

If any of my responses have been helpful in any way, please rate accordinly. Thank you and Happy VM'ing! :smileygrin:

[athlon_crazy · Accepted solution]

You may need someone to go physically to the ESXi console and restart management agents KB

[WillL]

Thanks Trevor.

I had enabled Tech Support mode, so I was able to ssh on top of browser and vSphere client. They all stopped working for some reason, I guess only the local console or logs can reveal what happened.

The following shows ESXi host closes network connection right away:

telnet esxi01 22

Trying 10.192.106.205...

Connected to esxi01 (10.192.106.205).

Escape character is '^]'.

Connection closed by foreign host.

[WillL]

Thanks, we will try the KB and report back.

[WillL]

Restarting management agents resolved our issue, thank you!

Now I'm looking at the logs:

Issue on vCloud console was reported on 9pm UTC (5pm EDT) 19th, vCenter logged both ESXi disconnect around 10:30pm UTC.

On ESXi server 1, messages are gone for 19th, vpx log is filled with this:

Section for VMware VirtualCenter Agent, pid=5169612, version=4.1.0, build=build-258902, option=Release

did not find a VM with ID 44 in the vmList

VM with vmid = 44 not found

did not find a VM with ID 44 in the vmList

VM with vmid = 44 not found

Failed to fetch current stats. Exception: No buffer space available

On ESXi server 2, messages are gone for 19th, however this one is interesting:

Oct 20 23:38:09 root: sfcbd-watchdog:Restarting SFCB! Log a bug!!!

...

Oct 20 23:38:11 sfcb-sfcb[11045055]: --- Using /etc/sfcb/sfcb.cfg

Oct 20 23:38:11 vmkernel: 71:07:04:03.499 cpu2:4381358)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 27000000030000000 type CHAR: Would block

Oct 20 23:38:11 vmkernel: 71:07:04:03.503 cpu3:11045058)WARNING: Tcpip: 1313: socreate(type=1, proto=6) failed with error 55

Oct 20 23:38:11 sfcb-sfcb[11045058]: --- Cannot listen on ipv4 port 5989 (Bad file descriptor)

Oct 20 23:38:11 vmkernel: 71:07:04:03.504 cpu6:11061443)WARNING: Tcpip: 1313: socreate(type=1, proto=6) failed with error 55

Oct 20 23:38:11 sfcb-sfcb[11061443]: --- Cannot listen on ipv4 port 5988 (Bad file descriptor)

Oct 20 23:38:11 cimslp: --- Using /etc/sfcb/sfcb.cfg

Oct 20 23:38:12 sfcb-HTTP-Daemon[11061443]: --- accept error Bad file descriptor

Oct 20 23:38:12 vmkernel: 71:07:04:04.505 cpu2:11061443)User: 2428: wantCoreDump : sfcb-HTTP-Daemo -enabled : 0

Oct 20 23:38:12 sfcb-ProviderManager[11045055]: getProcess fork failed: Function not implemented

Oct 20 23:38:12 sfcb-ProviderManager[11045055]: sfcbd is stopping, ignoring fork failure

Oct 20 23:38:12 sfcb-ProviderManager[11045055]: Semctl failed rc -1, error Invalid argument

Oct 20 23:38:12 vmkernel: 71:07:04:04.508 cpu2:11049159)User: 2428: wantCoreDump : sfcb-ProviderMa -enabled : 0

Oct 20 23:38:12 cimslp: --- SLP Agent got error code 1 while doing enumInstances. Trying again (attempt 1, sleeping 15) ---

Oct 20 23:38:12 vmkernel: 71:07:04:04.569 cpu4:11061440)WARNING: Tcpip: 1313: socreate(type=1, proto=0) failed with error 55

vpx log is attached. once a while this message:

UUID: Unable to open /dev/mem: No such file or directory

Not sure what exactly happened.

Thanks,

William

[athlon_crazy]

Have you log the case with VMware Tech ?

[WillL]

We are now evaluating vCloud Director under trial license, don't think we can engage tech support.