Vsphere client connection error on HP server w/ ESXi 5.0 U1

[Saltshaker]

I have a new HP Proliant Gen8 server with HP customized 5.0 U1 on it that I was able to connect to via vsphere client without a problem a couple of weeks ago. I do not have vcenter server, but just the other day I now get:

"An unknown connection error occurred. (The request failed because of a connection failure. (Unable to connect to the remote server))."

while trying to connect.

All VMs are still running fine. I am able to connect/RDP to them and ping them.
I am able to ping the ESXi host IP as well. No firewall issues or changes, I am able to connect to another host fine. SSH is not enabled so I cannot access that way, if it is then I can't putty into it anyway. There is also an ILO port that I can access fine, no issues seen. At the physical console, I hit F2, got the login, enter creds and nothing. Now it does not respond at the console.

So my questions are, has anyone encountered this at all or have any idea what could be causing this connection issue and/or the console from not responding after login?
What would be the best course of action? Should I just log into each VM, shut them down and hard reboot the server? Hopefully that may correct the problem? Any advice would be appreciated. Thanks in advance.

Chris.

[clarkwayne]

You can try restarting the management agents through the local console: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=100349... (video of how to do that). As it says it's under management options.

[Saltshaker]

I appreciate the reply and that is what I was hoping to do but the local console stopped responding after I tried to login with my creds. Now nothing happens despite what I hit on the keyboard.

[sparrowangelste]

it cant be lockdown mode, because you dont have vcenter

ive seen something liek this, i have waited up to 2 hours then I got a reponse.

once in restart your management agents, that will probably take a hour or 2 if it works

since your vms are up its not too bad, but you will prob ably have to reboot the server sometime soon.usually it indicates you might need a firmware update etc.

you might have to hard reboot

[Saltshaker]

Well I did check back about 5 hours later and still no response. I have no immediate need to do anything with the server yet but I will. I may have to hard reboot when the time comes but I dont know if there are any firmware updates on this server yet...its only a few months old but you never know. Plus I still want to update it to 5.1 regardless. Im hoping it will let me in on the local console so I'm going to give more time.

[Saltshaker]

Update: I was able to switch to the log screen...of course, now I can switch back but it throwing tons of Warnings:

Visor FSobj: 893 CANNOT CREATE FILE /etc/vmware/esx.conf .LOCK for process dcui because.....

Visor FSobj: 893 CANNOT CREATE FILE /etc/vmware/esx.conf .LOCK for process hostd because...

cant read the rest but it looks like its stuck. Anyone see this before?

[peetz]

Try to press Alt+F12 to switch to the log messages screen. Then scroll right to see the complete error message.

The part following "because ..." should be interesting.

- Andreas

[Saltshaker]

Ahh..duh. Ok thanks...so the full message is:

Visor FSobj: 893 CANNOT CREATE FILE /etc/vmware/esx.conf .LOCK for process dcui because the visorfs inode table is full

Visor FSobj: 893 CANNOT CREATE FILE /etc/vmware/esx.conf .LOCK for process hostd because the visorfs inode table is full

[peetz]

Using vCloud Director? Is this your issue: http://kb.vmware.com/kb/2031839 ?

You should open an SR with VMware in any case ...

- Andreas

[Saltshaker]

I do not use vClould director just have local storage but I did read that article. I think just may be forced to hard reboot this.

[peetz]

Yes. You could try to log in to the console and remove any files from /tmp, but even then the host would probably not recover from this without a reboot.

Try to shut down all VMs from within the guest OSs, then hard reset the host.

After that you should carefully monitor the /tmp directory if it is filling up with something.

This is strange, I have never encoutered such a problem before, and I run a lot of HP servers with exactly this ESXi build.

[Saltshaker]

Everything that I have read basically states to reboot. Obviously I will have to do this off hours since there are production VM's so I will log into each one, shut them down and hard reboot the server. But you are correct, this is odd. It makes me nervous in the sense that something is causing it. I dont want to chaulk it up as fluke but would like to make sure it does not happen again. Either way, I do want to update to 5.1 so I may try that later once I can log into it.

[Saltshaker]

So I just wanted to update you on this, I ended up logging into eash guest OS and shut them down. Once they were all down, I went to the console just to see if I could log in. Sure enough it let me right in, reset the management network and I was able to connect with vsphere client again. So I didnt have to hard reset the server. The only change prior to this issue was a server that I virtualized to this host. It had 24GB of RAM set on it so prior to starting it back up again I lowered it down in half. I checked the logs and I do not see any more of those entries since that day I shut everything down. Not sure if that had anything to do with it. Im going to keep an eye on it though and see what happens. Eventually I will upgrade it to 5.1 when I have time.