Getting error when opening console while remoteing to ESX

[JoelPx]

Hi all,

Finally got our little Virtual network off the ground, but trying to setup to remote to it using a SSL gateway, where we have to plug in the ports that have to be used.

Have 902 and 903 opened for the Virtual Client to use from a remote system.

So far I can remote to the ESX server perfectly using the Virtual Client, can even power on the virtual machines.

Problem comes in when I try to open a console to the virtual machines.

It will hang for a minute before timing out and coming up with this message:

Error connecting. Cannot connect to host x.x.x.x: A connection attempt failed because the connected party did not respond after a period of time, or established connection failed because connection host has failed to respond.

If I run netstat –a right after trying to open the console I see that my remote box is trying to talk to the ESX server.

TCP :903 SYN_SENT

But it just times out every time. If I remote to the server from internally it works great. So I’m fairly sure there is some other port I must be missing to put in our SSL gateway.

Anyone have any suggestions?

Thanks!

Joel

[kitcolbert]

If you give me the SR number I can take a look at the logs you uploaded. Since this appears to be a different problem than we've seen, I'd like to try and figure out what it is. Thanks.

[richard6121]

SR# 326914

[jrtjr]

We too were getting this issue. Our console was working just fine for a few months and then just stopped working one day (no known changes ... and only two of us able to make any changes). We were still able to get a console up if opening from a VI client on the same switch ... but soon even that stopped.

This fix has put us back in biz, thanks much!

[Lockwood]

Wanted to let you know that this has worked for me as well; we have approximately 40 VI3 instances and our latest additions to the VI3 world are the first this has happened to. We are running VC 2.0.1 patch 2 and ESX 3.0.1 build 39823.

Thank you very much for this workaround.

[Dick_Murray]

Thought I'd give some more info on the Console symptoms I've had. ESX 3.0.1 and couldn't connect to Console the first time I tried. Reading this post I saw the suggestion of resetting the vswitch of the Service Console. Instead I took a connection to the other Service Console I had created and the Console works. We have four NIC's in two pairs, with a Service Console on each pair. During build I tested the redundant links and the first Service Console was the last to go down. I'm assuming something has orphaned the original Service Console with link Up/Downs. Hope this helps..?

[Mork]

I've just stumbled on this thread as I've just experienced this issue.

Firstly, we've been running VI3 basically since it was released, and am now running 3.0.1 build 35804 in Production with VC 2.0.1 build 40644. My Test ESX servers are 3.0.1 build 39823 connected to the same VC.

I first had the issue of not being able to get a remote console open after I had changed our Test servers to a different subnet, and found I'd missed one setting in one configuration file which still specified the old subnet. Everything else was fine, just no remote console.

However, since then, all of a sudden, my Production ESX servers at one site are exhibiting the same issue, but nothing has changed with the exception of entering/exiting maintenance mode to repatch them to a new switching infrastructure.

The other Production servers at our other site are 100% fine, no issues experienced like this at all.

I have found the only way to fix this is to restart the service console networking (/sbin/service network restart) and they then work fine for a day before failing again.

I only worked out this was happening regularly yesterday, so haven't filed an SR or anything yet.

I'll try the workaround list earlier in this thread to see if it fixes the issue, and tomorrow I'll query our firewall guy to see if 903 is allowed as my VC is on a different subnet to all my ESX hosts and every internal subnet is firewalled.

[Mork]

Quick update... I just discovered one of our hosts has done it again since I fixed it yesterday.

I applied the proxy workaround from this thread and it instantly fixed the problem without needing to restart the service console networking.

I'll lodge an SR with VMware tomorrow after I confirm that 903 is being allowed through the firewalls.

[warkem]

WORKAROUND: bring down the vswif0 and back up again...Wouldn't this disconnect your ssh connection? I'm having the same issue, diff subnet, all machine built exactly the same, issue only occurs on 1 of the boxes. If I connect the VI client directly to the esx host rather than vc, works fine. The /etc/vmware/config work around did not work for me

Cheers,

Joe

[Dave_Mishchenko]

Hi Joe, it will take the service console offline so you would loose your SSH connection. You'd have to do it at the server or via remote card / KVM.

[warkem]

Hmm well I am using hp c-Class blades and I haven't worked out how to press Alt-F1 (to get to the console) in iLo. Not sure if its possible

Cheers,

Joe

[Dave_Mishchenko]

I haven't used HPs in a while, but you may need to use a hot - key - see page 90 - . Not sure if the manual will apply to your blades or not.

[warkem]

Thanks but found a shortcut

For anyone who is interested in iLo2 Alt+ left arrow will get you to the console

Also restarting the service console using ifdown and up worked for me

Cheers,

Joe

[EricSp]

Please forgive my lack of experience with this product, but when I SSH to the ESX server it won't let me login as root.

I can get to the local console, but how do I edit the config file as you reccomend?

- add the following to /etc/vmware/config:

- vmauthd.server.alwaysProxy=TRUE

Thanks for any help you can offer.

-Eric

[Dave_Mishchenko]

The root login is denied SSH access by default. You can change that, but it's best to leave it as is. Instead you'll want to create another login in the VI client and enable the grant shell access option. Then login to SSH with that user and run the command su - to switch to the root login. You can then edit the file.

[Aviso]

I had the same issue with the console and found more information. Running ESX 3.0.2 I had no problems until the MTU size on my network link was lowered. Then I had to apply the /etc/vmware/config fix to get things working again. We tested it a while and confirmed it was directly related to the MTU size. On another note, I noticed I couldn't ping the VM when I had the console up and was experiencing the problem.

So any ideas on what the MTU would have to do with it? My best guess is ESX (3.0.2 at least) can't dynamically lower it's MTU based on the icmp messages it recieves from the client workstation. I'm wondering if with the support for varying MTU sizes in 3.5 this is still an issue.

[cantplaygottawo]

It looks like we are getting this self same problem on our new ESX 3.5 VI3 setup

Problem is occuring with a single subnet 25 bit and occurs when attempting to console from both the VC system and a seperate VI client direct to the ESX server, all systems are on the same 25 bit subnet, and we can create a new virtual machine with no problems but as soon as we try to power on the system our console appears to hang before coming back with the common error.

Have tried the file change as noted here and once I got the typo's out of the way that resolved the problem.

Question is why this is still around in version 3.5 when it was known about in 2006 on 3.0 ?

[grex827]

adding vmauthd.server.alwaysProxy = "TRUE" worked for me. The weird thing is, I have multiple ESX servers on the same subnet, and only this one had this issue.

[v4nilla]

I ran into the same issue with ESX 3.5 Update 1 and VC 2.5 Build 64192. Adding the vmauthd.server.alwaysProxy = "TRUE" to the /etc/vmware/config file fixed the problem. I have VC and the ESX hosts on the same VLAN, while my machine is on a separate routed VLAN. This started happening all of the sudden after changing the VLAN the ESX hosts were on. Previously they were being routed through an ISA firewall, now they are routed through a Cisco 4506. They were always on a separate routed network, but after the change the problem occurred.

[bean3178]

The vmauthd.server.alwaysProxy=TRUE is not working for me. I have the VC server and both ESX nodes on the same subnet. I'm using the VI Client on my local workstation to connect into the VC server from a differenet network (I have multiple NIC's configured in my VC server). Even after the config change and restarting xinetd, I'm still getting the error. It looks like the "Open Console" window is still trying to connect to the nodes on the IP subnet that my workstation doesn't have access to. Help?

[greg_g_w]

Hello,

I had a similar issue (unable to connect to console on any VMs), tried all the fixes and workarounds found on these forums, and google results without success.

After poking around I was able to get things working. Hopefully my issue is similar to yours and this will get you sorted out.. least it is yet another thing to try. 😎

The short of it is it seems to be a network/mask length issue, at least in my case. After correcting it, things started working instantly after restarting the network (which you can do remotely via ssh contrary to what some believe here, just send it to the background. ( “/etc/init.d/network restart&” - millage may vary )) The net bits have to match that of which your network is configured to (whatever the router’s interface is configured as). it didn't matter if you were on the same local network or routed.

The file /etc/sysconfig/network-scripts/ifcfg-vswif0 contains the interface configuration. Ours orginaly had:

DEVICE=vswif0

BOOTPROTO=static

IPADDR=10.3.200.20

NETMASK=255.255.255.0

ONBOOT=yes

BROADCAST=10.3.200.255

I changed it to:

DEVICE=vswif0

BOOTPROTO=static

IPADDR=10.3.200.20

NETMASK=255.255.0.0

ONBOOT=yes

BROADCAST=10.3.255.255

The take away here is, confirm your network configuration is proper and matches (client,server,router)

A few points I have to mention,

I was trouble shooting this prior to finding the solution, I did leave and still have the “vmauthd.server.alwaysProxy” bit at the end of my /etc/vmware/config file. I’ve yet to remove this change to see if things still work.

from the vSwitch0 properties page:

I added a default route which seemed to be missing from here, but did exist within config files within /etc/sysconfig/net*. After adding this I noticed after restarting the network services it went back to 'blank', guess it doesn't want a default gatway there...

thought I should mention them for completeness.

hope this helps someone.

-greg