VMware Cloud Community
Darwinian999
Contributor
Contributor

Guest Console access via NAT

I have a bunch of ESX 3.0 servers and a Virtual Center server sitting in a /20 network. Access to these servers from other networks is via a one-to-one NAT for each of these servers (each server has its own NAT address).

From my PC in another network, I can connect to the Virtual Center server and do everything except open a console on any of the Guest VM's. When I try to open a console, I get an error dialog box appearing within a second or two that says "Error connecting: Connection terminated by server. Do you want to try again?"

I've tried setting "vmauthd.server.alwaysProxy=TRUE" as per the discussion in http://www.vmware.com/community/thread.jspa?threadID=46632 , but it doesn't change the problem or its symptoms.

If I do a tcpdump of traffic on port 902 on the ESX server when I try to connect to a Guest VM console, I get the following:

\# tcpdump -ni vswif2 port 902

tcpdump: listening on vswif2

12:56:54.307619 10.55.15.11.4735 > 10.55.15.18.vmware-authd: P 1184572066:1184572204(138) ack 1559722067 win 64997 (DF)

12:56:54.307802 10.55.15.18.vmware-authd > 10.55.15.11.4735: P 1:91(90) ack 138 win 62780 (DF)

12:56:54.308327 10.55.15.11.4735 > 10.55.15.18.vmware-authd: P 138:372(234) ack 91 win 64907 (DF)

12:56:54.308439 10.55.15.18.vmware-authd > 10.55.15.11.4735: P 91:181(90) ack 372 win 62780 (DF)

12:56:54.506796 10.55.15.11.4735 > 10.55.15.18.vmware-authd: . ack 181 win 64817 (DF)

12:56:54.506809 10.55.15.18.vmware-authd > 10.55.15.11.4735: P 181:719(538) ack 372 win 62780 (DF)

12:56:54.725531 10.55.15.11.4735 > 10.55.15.18.vmware-authd: . ack 719 win 64279 (DF)

12:56:55.409294 10.55.15.18.32771 > 10.55.15.11.902: udp 205 (DF)

8 packets received by filter

0 packets dropped by kernel

The ESX server is 10.55.15.18 and the Virtual Center server is 10.55.15.11.

I've tried disabling the ESX firewall, but it doesn't change the problem.

Any suggestions?

Reply
0 Kudos
18 Replies
Quotient
Expert
Expert

This is a bug.

Use this for a workaround:

Add the following to /etc/vmware/config:

vmauthd.server.alwaysProxy=TRUE[/b]

More info is available here:

http://www.vmware.com/community/thread.jspa?messageID=442403

Reply
0 Kudos
Darwinian999
Contributor
Contributor

I've tried that (as I wrote in the 3rd paragraph), but it didn't work.

Reply
0 Kudos
Quotient
Expert
Expert

Best clean the sleep from my eyes... Smiley Happy

Since applying the workaround have you issued:

service network restart

service mgmt-vmware restart

service vmware-vpxa restart[/b]

...

or rebooted the ESX Host...?

Reply
0 Kudos
Darwinian999
Contributor
Contributor

Yes, I rebooted the ESX host and restarted the "VMware VirtualCenter Server" service on the Virtual Center server.

Reply
0 Kudos
Quotient
Expert
Expert

ok

Perhaps kitcolbert[/b]could provide some further insight. He's familiar with this area of code and has probably been waiting for a case like this.

You might want to lodge an SR and reference his handle in case he misses this post... Or reference this post from the one referred to above in case he is "watching" it...

Reply
0 Kudos
admin
Immortal
Immortal

Its odd that vmauthd.server.alwaysProxy isn't helping here (since that effectively pushes the remote console traffic through the same connection that the VC traffic is connecting through).

I wonder if there is some other problem with remote consoles on this setup. Do you know if the remote console works from client machines (i.e, from inside the NAT?)

To see if you really need the alwaysProxy work-around, you can try a telnet to the ESX host's port 903 (from the machine you're opening the remote console on). If you see a banner about the "VMware Authentication Daemon" you know there is no NAT problem.

(If you don't see the banner, there could be any of a number of things wrong ...)

Does /var/log/secure show anything "interesting" when you attempt to connect a remote console? (Do a 'tail -f /var/log/secure' and then attempt to connect a remote console. That should help filter out the noise from logging in directly, etc.)

As an aside: you shouldn't have to restart anything when adding/removing the 'vmauthd.server.alwaysProxy=TRUE' configuration option, because vmauthd is started for each incoming connection, and it re-reads the config file each time it starts up.

Reply
0 Kudos
Darwinian999
Contributor
Contributor

Remote consoles work fine from within the NAT. The NAT itself is working fine - I can ssh to the ESX host from outside the NAT.

When I telnet to port 903 on the ESX host from outside the NAT, I see the "VMware Authentication Daemon" banner.

There's nothing logged in /var/log/secure when I try to open a console from outside the NAT.

Reply
0 Kudos
Darwinian999
Contributor
Contributor

I've just tried connecting my VI client (from outside the NAT) direct to one of the ESX hosts, instead of connecting it to the Virtual Center server. I can open the guest consoles on that host!

The problem with opening guest consoles from the VI client connected to the Virtual Center server seems to be because of the one-to-one NAT (each server has its own NAT address). I'm guessing that the VI client contacts the VC server, which tells the ESX host to talk to the VI client, but the ESX host doesn't know how to get the packets to my VI client because it there isn't a NAT connection established between the VI client and the ESX host.

Reply
0 Kudos
admin
Immortal
Immortal

Ah! D'oh. I should've thought of this ...

What happens is the VI client connects to the VC server, and then (to open the remote console) the VC server passes an IP address back to the VI client that it should attempt to contact. Because the VC server sees a different (i.e,. internal) IP address for the ESX machine, the VI client will receive a useless IP address. (The same idea as what you have written, except that the VI client is always a client and always initiates the connection.)

Reply
0 Kudos
Darwinian999
Contributor
Contributor

Ok, that makes sense. There's no easy workaround then, other than connecting direct to each ESX host. Smiley Sad

FWIW, Citrix addressed this issue in their Metaframe product with an "altaddr" command that allows you to specify an "alternate" (NAT) address to use. Maybe the same thing could be done for ESX in a future release?

Reply
0 Kudos
MrData
Contributor
Contributor

This can also be a resolving problem. When you open a console, the VC server gives the ESX host name how it is in VC. Your computer outside the NAT maybe can't resolve that name. When this is the case you can create entry's in you hosts file with the names as the are in VC and the "natted" ip-addresses.

Reply
0 Kudos
tlang
Contributor
Contributor

This can also be a resolving problem. When you open a

console, the VC server gives the ESX host name how it

is in VC. Your computer outside the NAT maybe can't

resolve that name. When this is the case you can

create entry's in you hosts file with the names as

the are in VC and the "natted" ip-addresses.

Has anyone gotten this to work? I had no luck; it appears that even if you modify the hosts file on the VC Server, when you open a remote console on the client VC is still handing the client the original (internal) IP address of the ESX server which VC first used to successfully connect to the ESX server.

Also, it stands to reason that even if this hosts file hack worked, then next time VC tried to connect to the ESX server (i.e., if you had to reboot VC or just temporarily disconnect/reconnect the ESX host) then VC would be using the incorrect external IP address, unless you continually modify the hosts file back and forth. Hence this seems a pretty ugly workaround, even if it worked. Or am I missing something?

To summarize, is there no way at all to get the VI3 client to open a console to a VM when the VC Server is behind a NATed firewall? (other than connecting directly to the ESX Server and using the "vmauthd.server.alwaysProxy=TRUE" workaround, which unfortunately is unacceptable to my client)

Thanks,

Tim Lang

Reply
0 Kudos
kitcolbert
VMware Employee
VMware Employee

Yeah, modifying the hosts file doesn't seem like the way to go.

Unfortunately there's no way to get this to work right now.

Reply
0 Kudos
Algorithm
Contributor
Contributor

Has there been a solution to this problem yet? We have a situation very similar and cannot find a resolution.

Reply
0 Kudos
default-user
Contributor
Contributor

tag

Reply
0 Kudos
Henno
Contributor
Contributor

Log on to a VM using Remote Desktop and use the VI client from there.

Reply
0 Kudos
Gaborus
Contributor
Contributor

Same issue, cannot connect via remote console to NATed VC2 server. Works fine only with direct connection to ESX3.

We loose many features and flexibility this way. Any workaround, patch, or solution?

Reply
0 Kudos
Edt
Contributor
Contributor

Note that whenever the VIC is on a different subnet from the hardened ESX this issue persist. If the VIC is on the same subnet as the ESX...you are able to see the VMs. My experience has been that after the hardening of the system, the VIC can no longer make connection to the VMs. The challenge then ...is to identify exactly what was hardened too tightly....which causes this issue.

Reply
0 Kudos