I have a bunch of ESX 3.0 servers and a Virtual Center server sitting in a /20 network. Access to these servers from other networks is via a one-to-one NAT for each of these servers (each server has its own NAT address).
From my PC in another network, I can connect to the Virtual Center server and do everything except open a console on any of the Guest VM's. When I try to open a console, I get an error dialog box appearing within a second or two that says "Error connecting: Connection terminated by server. Do you want to try again?"
I've tried setting "vmauthd.server.alwaysProxy=TRUE" as per the discussion in http://www.vmware.com/community/thread.jspa?threadID=46632 , but it doesn't change the problem or its symptoms.
If I do a tcpdump of traffic on port 902 on the ESX server when I try to connect to a Guest VM console, I get the following:
\# tcpdump -ni vswif2 port 902
tcpdump: listening on vswif2
12:56:54.307619 10.55.15.11.4735 > 10.55.15.18.vmware-authd: P 1184572066:1184572204(138) ack 1559722067 win 64997 (DF)
12:56:54.307802 10.55.15.18.vmware-authd > 10.55.15.11.4735: P 1:91(90) ack 138 win 62780 (DF)
12:56:54.308327 10.55.15.11.4735 > 10.55.15.18.vmware-authd: P 138:372(234) ack 91 win 64907 (DF)
12:56:54.308439 10.55.15.18.vmware-authd > 10.55.15.11.4735: P 91:181(90) ack 372 win 62780 (DF)
12:56:54.506796 10.55.15.11.4735 > 10.55.15.18.vmware-authd: . ack 181 win 64817 (DF)
12:56:54.506809 10.55.15.18.vmware-authd > 10.55.15.11.4735: P 181:719(538) ack 372 win 62780 (DF)
12:56:54.725531 10.55.15.11.4735 > 10.55.15.18.vmware-authd: . ack 719 win 64279 (DF)
12:56:55.409294 10.55.15.18.32771 > 10.55.15.11.902: udp 205 (DF)
8 packets received by filter
0 packets dropped by kernel
The ESX server is 10.55.15.18 and the Virtual Center server is 10.55.15.11.
I've tried disabling the ESX firewall, but it doesn't change the problem.
Any suggestions?
This is a bug.
Use this for a workaround:
Add the following to /etc/vmware/config:
vmauthd.server.alwaysProxy=TRUE[/b]
More info is available here:
http://www.vmware.com/community/thread.jspa?messageID=442403
I've tried that (as I wrote in the 3rd paragraph), but it didn't work.
Yes, I rebooted the ESX host and restarted the "VMware VirtualCenter Server" service on the Virtual Center server.
ok
Perhaps kitcolbert[/b]could provide some further insight. He's familiar with this area of code and has probably been waiting for a case like this.
You might want to lodge an SR and reference his handle in case he misses this post... Or reference this post from the one referred to above in case he is "watching" it...
Its odd that vmauthd.server.alwaysProxy isn't helping here (since that effectively pushes the remote console traffic through the same connection that the VC traffic is connecting through).
I wonder if there is some other problem with remote consoles on this setup. Do you know if the remote console works from client machines (i.e, from inside the NAT?)
To see if you really need the alwaysProxy work-around, you can try a telnet to the ESX host's port 903 (from the machine you're opening the remote console on). If you see a banner about the "VMware Authentication Daemon" you know there is no NAT problem.
(If you don't see the banner, there could be any of a number of things wrong ...)
Does /var/log/secure show anything "interesting" when you attempt to connect a remote console? (Do a 'tail -f /var/log/secure' and then attempt to connect a remote console. That should help filter out the noise from logging in directly, etc.)
As an aside: you shouldn't have to restart anything when adding/removing the 'vmauthd.server.alwaysProxy=TRUE' configuration option, because vmauthd is started for each incoming connection, and it re-reads the config file each time it starts up.
Remote consoles work fine from within the NAT. The NAT itself is working fine - I can ssh to the ESX host from outside the NAT.
When I telnet to port 903 on the ESX host from outside the NAT, I see the "VMware Authentication Daemon" banner.
There's nothing logged in /var/log/secure when I try to open a console from outside the NAT.
I've just tried connecting my VI client (from outside the NAT) direct to one of the ESX hosts, instead of connecting it to the Virtual Center server. I can open the guest consoles on that host!
The problem with opening guest consoles from the VI client connected to the Virtual Center server seems to be because of the one-to-one NAT (each server has its own NAT address). I'm guessing that the VI client contacts the VC server, which tells the ESX host to talk to the VI client, but the ESX host doesn't know how to get the packets to my VI client because it there isn't a NAT connection established between the VI client and the ESX host.
Ah! D'oh. I should've thought of this ...
What happens is the VI client connects to the VC server, and then (to open the remote console) the VC server passes an IP address back to the VI client that it should attempt to contact. Because the VC server sees a different (i.e,. internal) IP address for the ESX machine, the VI client will receive a useless IP address. (The same idea as what you have written, except that the VI client is always a client and always initiates the connection.)
Ok, that makes sense. There's no easy workaround then, other than connecting direct to each ESX host. ![]()
FWIW, Citrix addressed this issue in their Metaframe product with an "altaddr" command that allows you to specify an "alternate" (NAT) address to use. Maybe the same thing could be done for ESX in a future release?
This can also be a resolving problem. When you open a console, the VC server gives the ESX host name how it is in VC. Your computer outside the NAT maybe can't resolve that name. When this is the case you can create entry's in you hosts file with the names as the are in VC and the "natted" ip-addresses.
This can also be a resolving problem. When you open a
console, the VC server gives the ESX host name how it
is in VC. Your computer outside the NAT maybe can't
resolve that name. When this is the case you can
create entry's in you hosts file with the names as
the are in VC and the "natted" ip-addresses.
Has anyone gotten this to work? I had no luck; it appears that even if you modify the hosts file on the VC Server, when you open a remote console on the client VC is still handing the client the original (internal) IP address of the ESX server which VC first used to successfully connect to the ESX server.
Also, it stands to reason that even if this hosts file hack worked, then next time VC tried to connect to the ESX server (i.e., if you had to reboot VC or just temporarily disconnect/reconnect the ESX host) then VC would be using the incorrect external IP address, unless you continually modify the hosts file back and forth. Hence this seems a pretty ugly workaround, even if it worked. Or am I missing something?
To summarize, is there no way at all to get the VI3 client to open a console to a VM when the VC Server is behind a NATed firewall? (other than connecting directly to the ESX Server and using the "vmauthd.server.alwaysProxy=TRUE" workaround, which unfortunately is unacceptable to my client)
Thanks,
Tim Lang
Yeah, modifying the hosts file doesn't seem like the way to go.
Unfortunately there's no way to get this to work right now.
Has there been a solution to this problem yet? We have a situation very similar and cannot find a resolution.
tag
Log on to a VM using Remote Desktop and use the VI client from there.
Same issue, cannot connect via remote console to NATed VC2 server. Works fine only with direct connection to ESX3.
We loose many features and flexibility this way. Any workaround, patch, or solution?
Note that whenever the VIC is on a different subnet from the hardened ESX this issue persist. If the VIC is on the same subnet as the ESX...you are able to see the VMs. My experience has been that after the hardening of the system, the VIC can no longer make connection to the VMs. The challenge then ...is to identify exactly what was hardened too tightly....which causes this issue.
