My Host machine can't connect to the guest VMs even though both are connected via vmnet8 (NAT)!

[SomuSysAdmin]

My guest VM has an ip of 10.0.99.99 (Static IP). This can ping the NAT device & gateway at 10.0.99.2. I have full internet functionality on this guest VM.

My physical host has an IP of 192.168.3.108 with a gateway IP of 192.168.3.1 on the interface wlo1 (further routers lead to the ISP). On vmnet8 host adapter, the IP is 10.0.99.1.

Now, while both can operate over the internet, they still can't talk to each other. I tried to see my Routing table on my host, and it says:

# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.3.1     0.0.0.0         UG    600    0        0 wlo1
10.0.99.0       0.0.0.0         255.255.255.0   U     0      0        0 vmnet8
192.168.3.0     0.0.0.0         255.255.255.0   U     600    0        0 wlo1
192.168.35.0    0.0.0.0         255.255.255.0   U     0      0        0 vmnet1

Is this okay? Shouldn't the route to 10.0.99.0 have a gateway of 10.0.99.2? When I try pinging the gateway for vmnet8 (10.0.99.2) I don't get any reply!!

# ping 10.0.99.2
PING 10.0.99.2 (10.0.99.2) 56(84) bytes of data.
^C
--- 10.0.99.2 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4126ms

I changed the route as :

# ip route add 10.0.99.0/24 via 10.0.99.2 dev vmnet8
RTNETLINK answers: File exists
[root@lappyPrimeF nat]# ip route change 10.0.99.0/24 via 10.0.99.2 dev vmnet8
[root@lappyPrimeF nat]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.3.1     0.0.0.0         UG    600    0        0 wlo1
10.0.99.0       10.0.99.2       255.255.255.0   UG    0      0        0 vmnet8
192.168.3.0     0.0.0.0         255.255.255.0   U     600    0        0 wlo1
192.168.35.0    0.0.0.0         255.255.255.0   U     0      0        0 vmnet1

This however, doesn't solve my problems (at least I get a reply now?):

# ping 10.0.99.2
PING 10.0.99.2 (10.0.99.2) 56(84) bytes of data.
From 10.0.99.1 icmp_seq=1 Destination Host Unreachable
From 10.0.99.1 icmp_seq=2 Destination Host Unreachable
From 10.0.99.1 icmp_seq=3 Destination Host Unreachable
^C
--- 10.0.99.2 ping statistics ---
5 packets transmitted, 0 received, +3 errors, 100% packet loss, time 4126ms
pipe 4
[root@lappyPrimeF nat]# ping 10.0.99.99
PING 10.0.99.99 (10.0.99.99) 56(84) bytes of data.
From 10.0.99.1 icmp_seq=1 Destination Host Unreachable
From 10.0.99.1 icmp_seq=2 Destination Host Unreachable
From 10.0.99.1 icmp_seq=3 Destination Host Unreachable
^C
--- 10.0.99.99 ping statistics ---
5 packets transmitted, 0 received, +3 errors, 100% packet loss, time 4120ms
pipe 4

Also, traceroute is at least producing a result now, instead of 30 columns full of *'s:

# traceroute 10.0.99.99
traceroute to 10.0.99.99 (10.0.99.99), 30 hops max, 60 byte packets
 1  lappyPrime.somuVMnet.local (10.0.99.1)  3075.113 ms !H  3074.881 ms !H  3074.773 ms !H

So, what's going on? Please help me so that my host and guest can communicate via the vmnet8 device? I'm pretty sure this can be done due to this article. It states:

The Host Computer and the NAT Network  

The host computer has an adapter on the NAT network (identical to the host-only adapter on the host-only network). This adapter allows the host and the virtual machines to communicate with each other for such purposes as file sharing. The NAT never forwards traffic from the host adapter.

So, why can't my host talk to the guest or even ping it, and vice versa?

Here's the /etc/vmware/vmnet8/nat/nat.conf file (on the physical host):

# VMware NAT configuration file
# Manual editing of this file is not recommended. Using UI is preferred.
[host]
# NAT gateway address
ip = 10.0.99.2
netmask = 255.255.255.0
# VMnet device if not specified on command line
device = /dev/vmnet8
# Allow PORT/EPRT FTP commands (they need incoming TCP stream ...)
activeFTP = 1
# Allows the source to have any OUI.  Turn this on if you change the OUI
# in the MAC address of your virtual machines.
allowAnyOUI = 1
# Controls if (TCP) connections should be reset when the adapter they are
# bound to goes down
resetConnectionOnLinkDown = 1
# Controls if (TCP) connection should be reset when guest packet's destination
# is NAT's IP address
resetConnectionOnDestLocalHost = 1
# Controls if enable nat ipv6
natIp6Enable = 0
# Controls if enable nat ipv6
natIp6Prefix = fd15:4ba5:5a2b:1008::/64
[tcp]
# Value of timeout in TCP TIME_WAIT state, in seconds
timeWaitTimeout = 30
[udp]
# Timeout in seconds. Dynamically-created UDP mappings will purged if
# idle for this duration of time 0 = no timeout, default = 60; real
# value might be up to 100% longer
timeout = 60
[netbios]
# Timeout for NBNS queries.
nbnsTimeout = 2
# Number of retries for each NBNS query.
nbnsRetries = 3
# Timeout for NBDS queries.
nbdsTimeout = 3
[incomingtcp]
# Use these with care - anyone can enter into your VM through these...
# The format and example are as follows:
#<external port number> = <VM's IP address>:<VM's port number>
#8080 = 172.16.3.128:80
[incomingudp]
# UDP port forwarding example
#6000 = 172.16.3.0:6001

Just in case it's relevant, here's the .vmx file for the host with IP : 10.0.99.99 (which can ping 10.0.99.2 Gateway for vmnet8):

#!/usr/bin/vmware
.encoding = "UTF-8"
config.version = "8"
virtualHW.version = "14"
mks.enable3d = "TRUE"
pciBridge0.present = "TRUE"
pciBridge4.present = "TRUE"
pciBridge4.virtualDev = "pcieRootPort"
pciBridge4.functions = "8"
pciBridge5.present = "TRUE"
pciBridge5.virtualDev = "pcieRootPort"
pciBridge5.functions = "8"
pciBridge6.present = "TRUE"
pciBridge6.virtualDev = "pcieRootPort"
pciBridge6.functions = "8"
pciBridge7.present = "TRUE"
pciBridge7.virtualDev = "pcieRootPort"
pciBridge7.functions = "8"
vmci0.present = "TRUE"
hpet0.present = "TRUE"
usb.vbluetooth.startConnected = "TRUE"
displayName = "vmInfra"
guestOS = "rhel7-64"
nvram = "vmInfra.nvram"
virtualHW.productCompatibility = "hosted"
powerType.powerOff = "soft"
powerType.powerOn = "soft"
powerType.suspend = "soft"
powerType.reset = "soft"
tools.syncTime = "FALSE"
sound.autoDetect = "TRUE"
sound.fileName = "-1"
sound.present = "TRUE"
vcpu.hotadd = "TRUE"
memsize = "2048"
mem.hotadd = "TRUE"
scsi0.virtualDev = "lsilogic"
scsi0.present = "TRUE"
sata0.present = "TRUE"
scsi0:0.fileName = "/vm/mac/vmInfra/HDD1/vmInfra.vmdk"
scsi0:0.present = "TRUE"
sata0:1.deviceType = "cdrom-raw"
sata0:1.fileName = "auto detect"
sata0:1.present = "TRUE"
usb.present = "TRUE"
svga.graphicsMemoryKB = "786432"
ethernet0.connectionType = "custom"
ethernet0.addressType = "generated"
ethernet0.virtualDev = "e1000"
serial0.fileType = "thinprint"
serial0.fileName = "thinprint"
ethernet0.present = "TRUE"
serial0.present = "TRUE"
extendedConfigFile = "vmInfra.vmxf"
floppy0.present = "FALSE"
scsi0:1.fileName = "/vm/mac/vmInfra/HDD2/localRepo.vmdk"
scsi0:1.present = "TRUE"
uuid.bios = "56 4d dc 8c 5d 05 94 57-36 e8 ed 84 34 79 8c 1f"
uuid.location = "56 4d dc 8c 5d 05 94 57-36 e8 ed 84 34 79 8c 1f"
migrate.hostlog = "./vmInfra-e6b36999.hlog"
scsi0:1.redo = ""
scsi0:0.redo = ""
pciBridge0.pciSlotNumber = "17"
pciBridge4.pciSlotNumber = "21"
pciBridge5.pciSlotNumber = "22"
pciBridge6.pciSlotNumber = "23"
pciBridge7.pciSlotNumber = "24"
scsi0.pciSlotNumber = "16"
usb.pciSlotNumber = "32"
ethernet0.pciSlotNumber = "33"
sound.pciSlotNumber = "34"
vmci0.pciSlotNumber = "35"
sata0.pciSlotNumber = "36"
ethernet0.generatedAddress = "00:0C:29:79:8C:1F"
ethernet0.generatedAddressOffset = "0"
vmci0.id = "880380959"
monitor.phys_bits_used = "43"
vmotion.checkpointFBSize = "8388608"
vmotion.checkpointSVGAPrimarySize = "134217728"
cleanShutdown = "FALSE"
softPowerOff = "FALSE"
usb:1.speed = "2"
usb:1.present = "TRUE"
usb:1.deviceType = "hub"
usb:1.port = "1"
usb:1.parent = "-1"
svga.guestBackedPrimaryAware = "TRUE"
tools.upgrade.policy = "useGlobal"
usb:0.present = "TRUE"
usb:0.deviceType = "hid"
usb:0.port = "0"
usb:0.parent = "-1"
sata0:1.autoDetect = "TRUE"
ethernet0.vnet = "/dev/vmnet8"

[continuum]

> Is this okay? Shouldn't the route to 10.0.99.0 have a gateway of 10.0.99.2?
No.
Unless you massively changed all defaults manually, the host should have an IP of 10.0.99.1 and should have a route to the 10.0.99.0 network via the gateway 10.0.99.1
Your host should be able to ping 10.0.99.1 and should get no a reply from 10.0.99.2
For the host the IP 10.0.99.2 should not be configured at all - only exception is in vmnat.conf.

[SomuSysAdmin · Accepted solution]

The NAT config as well as the VMX file was completely fine! However, the route should be configured on the host such that packets for the 10.0.99.0/24 subnet should merely use the vmnet10 device:

$ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.3.1     0.0.0.0         UG    600    0        0 wlo1
10.0.99.0       0.0.0.0         255.255.255.0   U     0      0        0 vmnet8

Turns out that the /etc/vmware/vmnet99/dhcpd/dhcpd.conf file needs an entry for each static IP that needs to be assigned, even if the DHCP server isn't running. That allows the host and the VMs to communicate. Of course, the static IPs shouldn't clash with the DHCP range (by default <net>.128 - <net>.253) and thus should lie within <net>.3 - <net>.127 if the DHCP range hasn't been adjusted. The entries should be inserted at the bottom of the dhcpd.conf file, OUTSIDE the DO NOT MODIFY section. They take the form:

host vmInfra.somuVMnet.local {

    hardware ethernet 00:0C:29:79:8C:1F;

    fixed-address 10.0.99.99;

}

The MAC address provided in the hardware ethernet value is the one assigned to the virtual NIC of the VM. Now, this file should only be edited once the VMs have been shut down, and need a restart of the dhcpd service running on the host for the vmnet. Since I couldn't find the individual service linked to the particular vmnet, I just restarted the vmware service itself, using:

systemctl restart vmware; systemctl status -l vmware

Now we ensure that with that last part that the vmware service started properly. If it's running, and the dhcp server is running on the vmnet, then the VM will be auto-assigned the requested IP and there is no need to explicitly setup the IP using NetworkManager. (I have no clue what happens if the IPs clash, bdw). My VMs can now successfully SSH into the host and vice versa.