Why does VMWare Fusion accept forwarded TCP connections when the app server socket isn't listening.

[JeepSnob]

Forgive me if I sound like I don't exactly understand this. I know enough to be dangerous

I am running VMWare Fusion 3.01 on Mac OS X 10.6.2.

I have a virtual machine running Windows XP Professional SP3

I have configured this VM to use NAT and I have configured the vmnet8/nat.conf file with the following entry:

10123 = 192.168.207.128:10123

I've done this because I have a server and client application that I am testing. The server is running on the Windows XP VM, listening on port 10123. The client is running on a laptop and makes a TCP connection to the server using the Mac's IP address. When I have my server and client running, the port forwarding "works" and my client and server applications talk with no problem.

Here's the rub....

I was troubleshooting an issue where my server application seemed to be accepting old connections, that is connections that the client already gave up on and closed. I fired up Wireshark to observe the traffic between the client (on the laptop) and the Mac (where the server is running on the Windows XP VM). I noticed that even when the server application is NOT running that the client's TCP connection attempts were being accepted by VMWare Fusion.

I see the normal SYN, SYN/ACK, ACK 3-way handshake that starts a TCP connection. This seems odd to me since there is no server application running. So I assume that the VMWare Fusion port forwarding software is actually accepting the connections on behalf of the VM. This is an assumption that needs to be confirmed.

Next, after my client realizes that nobody is home (i.e. the server isn't responding since it isn't running) my client application closes the connection. It sends a FIN/ACK and gets an ACK from, presumably, VMWare Fusion.

Twenty seconds later I see a FIN/ACK from VMWare and my laptop ACKs effectively completing the closure of the TCP connection.

This is creating havoc for my server application because when it starts it immediately receives and accepts a connection request from an address/port combination that was terminated several seconds prior.

What I think is happening is that VMWare Fusion is accepting TCP connections on behalf of the VM then attempting to make its own connection to the VM. Once such a connection is made it relays packets between the two connections. What I thought should happen is that VMWare Fusion would actually forward packets directly, only making the necessary changes to the address and port fields in the IP header. In other words, the SYN packet coming from the client should have gone directly to the listening process in the VM. If that process is not listening then the SYN/ACK packet would never be sent back to the client.

So, long story short, is this a case of either

1) the VMWare port forwarding function a poor man's port forwarder

2) this is how port forwarding is supposed to work

3) I've completely misinterpreted what is happening

I greatly appreciate your thoughts. I'm hoping to avoid switching to bridged mode or doing my testing elsewhere.

- JeepSnob

[JeepSnob]

I have collected some additional information.

I ran netstat -f inet -a -n | grep 8081 and got the following when my client and server were NOT running, just VMWare port forwarding to my VM

tcp4 0 0 *.8081 . LISTEN

When I run my TCP client I see the following (with no server application running)

tcp4 0 0 192.168.254.203.8081 192.168.254.205.50749 ESTABLISHED

tcp4 0 0 *.8081 . LISTEN

If I kill the client before it sends data the connection goes away as expected

tcp4 0 0 *.8081 . LISTEN

If I kill the client after it sends data the connection lingers in a CLOSE_WAIT state for 20 seconds before it is closed by the VMWare port forwarding software and the data is discarded.

tcp4 13 0 192.168.254.203.8081 192.168.254.205.50749 CLOSE_WAIT

tcp4 0 0 *.8081 . LISTEN

I still believe it is the VMWare Fusion port forwarding software that is making the decision to hold onto the data for those 20 seconds since CLOSE_WAIT, as I understand it, means that the IP stack is waiting for the local side to close the connection after the remote side has already done so.

You are probably wondering why this is a problem...well the application protocol I must implement doesn't deal well with immediately receiving data from the half-up connection. The data being received is only supposed to arrive some time after a connection is made. When it arrives immediately after the TCP connection is made bad things happen. And I can't change the protocol.

So I guess my question is can anyone confirm that it is VMWare Fusion that is controlling this behavior and is there any way to configure the port forwarding behavior to not hold a connection half open just because data has been received, especially since it was never able to establish a connection with the application that isn't running yet in the VM. Even if I'm lucky and there is, there will still be a small race condition where it could happen. The best thing would be if the connection was never allowed to be established without the server application running. In the end, I suppose switching to bridged mode may be my only solution.

Thanks for listening (reading) while I think this through out loud.

- JeepSnob

[asiw]

With Workstation 7, I notice VMware's NAT forwarder isn't transparent and terminates the TCP protocol on behalf of the guest. Between NAT and guest appears to be raw ethernet frame communication with translated fields.

Thus, I get a connect and disconnect from the NAT on a forwarded port, rather than a TCP connection refused, or timeout, when no VMs are running.

I suspect VMWare Fusion behaves the same way.

[rcardona2k]

If you think about how a Layer 4 port forwarding proxy works it accepts the inbound connection while attempting to bridge the connection to the guest. If the proxy gets a connection refused from the guest, the only alternative is to RST the already established connection. Trying to establish the backend TCP connection before calling accept() on the inbound connection is likely not feasible. Most application firewalls work the way VMware does. At the host level the port forwarder has no way of knowing a socket is listening on a guest port unless the Tools does inspection and it would have to keep monitoring which is intensive. Or on Windows you could register either an NDIS or TDI-layer driver that could react to winsock bind() calls, still it's messy to try to keep state on whether the guest has an active listener. The current solution is the simpler path that works mostly as intended.

If you need to have the true behavior of a single TCP stack, bridged networking is your best alternative.