I am running an application that is using RESTful APIs to access web services in a guest OS. If I have the network configured for the guest operating system in NAT mode, any TCP keepalive frames generated by the guest operating system stack are acknowledge locally and not communicated to external service. Below is a network trace on both the host system as well as the guest OS. If you look at the guest network trace, you can see that a TCP keep alive frame is generated at 10:17:50.995582 - approximately after 30 seconds of idle time on the session - which is what the guest OS is set to:
root@ubuntu:/home/user# cat /proc/sys/net/ipv4/tcp_keepalive_time
30
You will also notice that a response is received almost immediately at 10:17:50.995933 - within a few hundred microseconds. This is clearly not enough time for the initial keep alive packet to traverse the Internet and have a response received (sub 1ms latency). However if you look at the trace from the host system, you will see that the two TCP keep alive frames are totally missing. Thus the response must be being generated by the VMware NAT. This behavior will cause premature termination of many RESTful APIs and cause additional overhead on the network infrastructure due to having to restart connections. If however I run the guest OS in Bridge mode versus NAT mode, the TCP keepalive frames reach their destination as expected.
Question - does anyone know how to enable the generation of TCP keepalive frames in NAT mode of operations?
There is also a secondary question regarding the performance of the NAT environment. If you look at the Host OS trace, you will see that one of the frames from the host was retransmitted. This is either due to the NAT dropping the initial packet (it got to the host system at 10:17:19.688156) or an acknowledgement was not returned in time to the peer - see duplicate reception of frame at 10:17:19.847157 - approximately 150ms later. If the timestamps are anywhere near close, the guest OS reported receiving the packet at 10:17:19.875038, after the second transmit of the frame was received by the host system.
Question was the initial frame dropped somehow internally by the VMware network logic?
<<<< NETWORK TRACE SNIPPETS - IP ADDRESSES CHANGED TO protect infrastructure >>>>>>
// VMWARE GUEST UBUNTU 16.04
10:17:19.668584 IP 172.16.58.172.45472 > 8.8.8.8.443: Flags [S], seq 20383986, win 29200, options [mss 1460,sackOK,TS val 121525096 ecr 0,nop,wscale 7], length 0
10:17:19.711060 IP 8.8.8.8.443 > 172.16.58.172.45472: Flags [S.], seq 204427829, ack 20383987, win 64240, options [mss 1460], length 0
10:17:19.711160 IP 172.16.58.172.45472 > 8.8.8.8.443: Flags [.], ack 1, win 29200, length 0
10:17:19.724826 IP 172.16.58.172.45472 > 8.8.8.8.443: Flags [P.], seq 1:518, ack 1, win 29200, length 517
10:17:19.725206 IP 8.8.8.8.443 > 172.16.58.172.45472: Flags [.], ack 518, win 64240, length 0
10:17:19.776853 IP 8.8.8.8.443 > 172.16.58.172.45472: Flags [P.], seq 1:1369, ack 518, win 64240, length 1368
10:17:19.776908 IP 172.16.58.172.45472 > 8.8.8.8.443: Flags [.], ack 1369, win 31464, length 0
10:17:19.778510 IP 8.8.8.8.443 > 172.16.58.172.45472: Flags [P.], seq 1369:2737, ack 518, win 64240, length 1368
10:17:19.778533 IP 172.16.58.172.45472 > 8.8.8.8.443: Flags [.], ack 2737, win 34200, length 0
10:17:19.816835 IP 8.8.8.8.443 > 172.16.58.172.45472: Flags [P.], seq 2737:3481, ack 518, win 64240, length 744
10:17:19.816895 IP 172.16.58.172.45472 > 8.8.8.8.443: Flags [.], ack 3481, win 36936, length 0
10:17:19.820652 IP 172.16.58.172.45472 > 8.8.8.8.443: Flags [P.], seq 518:768, ack 3481, win 36936, length 250
10:17:19.821015 IP 8.8.8.8.443 > 172.16.58.172.45472: Flags [.], ack 768, win 64240, length 0
10:17:19.874549 IP 8.8.8.8.443 > 172.16.58.172.45472: Flags [P.], seq 3481:3588, ack 768, win 64240, length 107
10:17:19.875038 IP 172.16.58.172.45472 > 8.8.8.8.443: Flags [P.], seq 768:1013, ack 3588, win 36936, length 245
10:17:19.875230 IP 8.8.8.8.443 > 172.16.58.172.45472: Flags [.], ack 1013, win 64240, length 0
10:17:50.995582 IP 172.16.58.172.45472 > 8.8.8.8.443: Flags [.], ack 3588, win 36936, length 0 <<< TCP KEEP Alive sent by guest OS
10:17:50.995933 IP 8.8.8.8.443 > 172.16.58.172.45472: Flags [.], ack 1013, win 64240, length 0 <<< TCP Keep alive response received by guest OS
10:18:19.629828 IP 8.8.8.8.443 > 172.16.58.172.45472: Flags [R.], seq 3588, ack 1013, win 64240, length 0 <<< reset received from peer
// VMWARE FUSION HOST system High Sierra - 10.13.3
10:17:19.481865 IP 10.127.1.75.55687 > 8.8.8.8.443: Flags [S], seq 2843664308, win 65535, options [mss 1460,nop,wscale 5,nop,nop,TS val 1142390491 ecr 0,sackOK,eol], length 0
10:17:19.520206 IP 8.8.8.8.443 > 192.168.0.1.55687: Flags [S.], seq 2021796967, ack 2843664309, win 8192, options [mss 1380,nop,wscale 8,sackOK,TS val 1182037997 ecr 1142390491], length 0
10:17:19.520327 IP 192.168.0.1.55687 > 8.8.8.8.443: Flags [.], ack 1, win 4104, options [nop,nop,TS val 1142390529 ecr 1182037997], length 0
10:17:19.538072 IP 192.168.0.1.55687 > 8.8.8.8.443: Flags [P.], seq 1:518, ack 1, win 4104, options [nop,nop,TS val 1142390546 ecr 1182037997], length 517
10:17:19.589521 IP 8.8.8.8.443 > 192.168.0.1.55687: Flags [.], seq 1:1369, ack 518, win 513, options [nop,nop,TS val 1182038003 ecr 1142390546], length 1368
10:17:19.591280 IP 8.8.8.8.443 > 192.168.0.1.55687: Flags [.], seq 1369:2737, ack 518, win 513, options [nop,nop,TS val 1182038003 ecr 1142390546], length 1368
10:17:19.591353 IP 192.168.0.1.55687 > 8.8.8.8.443: Flags [.], ack 2737, win 4053, options [nop,nop,TS val 1142390597 ecr 1182038003], length 0
10:17:19.629372 IP 8.8.8.8.443 > 192.168.0.1.55687: Flags [P.], seq 2737:3481, ack 518, win 513, options [nop,nop,TS val 1182038008 ecr 1142390597], length 744
10:17:19.629460 IP 192.168.0.1.55687 > 8.8.8.8.443: Flags [.], ack 3481, win 4072, options [nop,nop,TS val 1142390635 ecr 1182038008], length 0
10:17:19.633870 IP 192.168.0.1.55687 > 8.8.8.8.443: Flags [P.], seq 518:768, ack 3481, win 4096, options [nop,nop,TS val 1142390639 ecr 1182038008], length 250
10:17:19.687206 IP 8.8.8.8.443 > 192.168.0.1.55687: Flags [P.], seq 3481:3588, ack 768, win 512, options [nop,nop,TS val 1182038013 ecr 1142390639], length 107
10:17:19.687324 IP 192.168.0.1.55687 > 8.8.8.8.443: Flags [.], ack 3588, win 4092, options [nop,nop,TS val 1142390692 ecr 1182038013], length 0
10:17:19.688156 IP 192.168.0.1.55687 > 8.8.8.8.443: Flags [P.], seq 768:1013, ack 3588, win 4096, options [nop,nop,TS val 1142390692 ecr 1182038013], length 245
10:17:19.847157 IP 192.168.0.1.55687 > 8.8.8.8.443: Flags [P.], seq 768:1013, ack 3588, win 4096, options [nop,nop,TS val 1142390851 ecr 1182038013], length 245 << Retransmit of TCP packet
10:17:19.887525 IP 8.8.8.8.443 > 192.168.0.1.55687: Flags [.], ack 1013, win 511, options [nop,nop,TS val 1182038034 ecr 1142390692,nop,nop,sack 1 {768:1013}], length 0
10:18:19.442628 IP 8.8.8.8.443 > 192.168.0.1.55687: Flags [R], seq 2021800555, win 0, length 0 <<< TCP reset only received - no TCP keep alives ever sent
