CantVM
Contributor
Contributor

Networking Issue, physical device can only ping one VM, not the rest

This setup is not best practice, but it's the only one I'm able to do unforunately. I've setup OpenVPN on a PFSense box. That PFSense is connected to another physical NIC on a ESXi 5.0.0 server. On one of the VMs I can ping it's adapter 192.168.1.15, but not it's other adapter IP 192.168.1.162. Every VM can ping the 192.168.1.15 IP, but nothing can ping the LAN IP of the PFSense that's connected to vmnic1, it's IP is 192.168.1.223. I've attached a visio to explain the network, I'm confused because I've never had to do a workaround like this before. Why can't these devices talk and how can I make it so that they can? Please help me.

I've attached a diagram of the setup.

Network flow:

OpenVPN client -> connects to PFSense -> can ping PFSense LAN IP -> can ping VMs ip of 192.168.1.15 -> cannot ping anything else

VMs on server -> can ping the 192.168.1.15 -> cannot ping the LAN IP of PFSense 192.168.1.223

VM with both adapters -> can ping everything including PFSense LAN IP

Tags (1)
0 Kudos
9 Replies
harry89
Enthusiast
Enthusiast

Hey ,

Are you able to ping vmkernel port , since that is also part of same subnet ?

Can you run esxtop on esxi and press n and share the output which physical nics are being used by virtual eth of Virtual machine

cheers!

Harry
VCIX-DCV6.5 ,VCIX-NV6 , VCAP-CMA7
Mark answer as correct/helpful if it solves your query
0 Kudos
CantVM
Contributor
Contributor

Just tested, here's what I have:

- Every VM can ping the kernel port IP 192.168.1.161

- I can ping all the VMs through the ESXi CLI including the 192.168.1.15, but I cannot ping the LAN IP of the PFSense connected to VMnic1 192.168.1.223

ESXTOP results:

PORT-ID              USED-BY       TEAM-PNIC     DNAME              PKTTX/s  MbTX/s    PKTRX/s  MbRX/s %DRPTX %DRPRX

  16777217           Management        n/a        vSwitch0              0.00    0.00       0.00    0.00   0.00   0.00

  16777218               vmnic0            -               vSwitch0          22631.41   56.91   33813.72   44.91   0.00   0.00

  16777220                 vmk0           vmnic0           vSwitch0              1.67    0.00       3.10    0.00   0.00   0.00

  16777221  3426: <domain name>  vmnic0       vSwitch0          22278.07   55.19   33783.20   46.99   0.00   0.31

  16777222           3452:TSTER     vmnic1         vSwitch0              0.00    0.00       0.24    0.00   0.00   0.00

  16777223           3452:TSTER     vmnic1        vSwitch0              0.00    0.00       0.00    0.00   0.00   0.00

  16777225     3501:<VM Hostname>     vmnic0        vSwitch0              0.72    0.00       5.25    0.00   0.00   0.00

  16777233 13209303:<VM with both adapters>   vmnic0   vSwitch0            154.26    3.15     219.35    0.56   0.00   0.76

  16777234 13209303:<VM with both adapters>   vmnic1    vSwitch0              0.00    0.00       0.24    0.00   0.00   0.00

  16777235               vmnic1          - vSwitch0              0.00    0.00       1.43    0.00   0.00   0.00      <-------- NIC I'm trying to get traffic through

Does this help?

0 Kudos
harry89
Enthusiast
Enthusiast

One another finding is that why VM is having two network cards( both in same subnet )

This could be the reason .

pastedImage_1.png

Harry
VCIX-DCV6.5 ,VCIX-NV6 , VCAP-CMA7
Mark answer as correct/helpful if it solves your query
0 Kudos
CantVM
Contributor
Contributor

Okay, so on that vm adapter with IP 192.168.1.15 I could just set the gateway to 192.168.1.223 of the PFSense LAN port. I don't think any traffic is able to get out of the vmnic1, hence why only the 192.168.1.15 can ping that IP. Now, I've also changed the IPs to a 192.168.0.x network and still could not ping the other vms. Would assigned that vmnic1 to the host itself 192.168.1.161 work?

0 Kudos
harry89
Enthusiast
Enthusiast

Do the below test :

Keep same subnet on the VM adapters .

Mark one nic down and test connectivity

Then mark second nic down and check connectivity

Assign different subnet on network adapters

assign separate gateway to both

Allow the changes on the firewall and test

If in both the case it doesn't work its either issue with vmnic1 itself or connectivity from vmnic to phsycial switch (port or connecting cable)

Cheers!

Harry
VCIX-DCV6.5 ,VCIX-NV6 , VCAP-CMA7
Mark answer as correct/helpful if it solves your query
0 Kudos
CantVM
Contributor
Contributor

Test 1 results:

- I disconnected the vmnic1 on the VM with 2 adapters/IPs, vmnic1 is the IP of 192.168.1.15. No other VM could ping it after I unconnected that network adapter. Only the VM could internally ping it.

- I turned the adapter back on and change the gateway from 192.168.1.223 to 192.168.1.1  and cannot ping the 1.223 address on the VM with multiple adapters now. I changed it back and can ping it.

- Nothing else can reach the .1.233 IP, just the 192.168.1.15

Test 2 results:

- I changed the VM adapter with IP 192.168.1.15 to 192.168.0.15 and the LAN IP on the PFSense to .0.223 with that as the vms GW. The VM with both IPs (192.168.1.161/192.168.0.15 now) can reach the .0.223

- all other VMS cannot reach the 192.168.0.15 or 0.223 IP, subnetting issue most likely

So the switch going from the physical adapter on the VM Server and PFSense is not the issue.

I believe the issue is VMNic1 not being able to tell all other vms that 192.168.1.223 (set as 192.168.0.223 for the test) is reachable.

0 Kudos
CantVM
Contributor
Contributor

I'm thinking since I have the PFSense connected to vmnic1, the vm with multiple IPs with both adapters configured, so vmnic0 has IP 192.168.1.162 and vmnic1 as 192.168.1.15., and can only ping the .1.223 that's a cable to vmnic1 means that no traffic that goes in/out of vmnic0 can inherently see traffic from vmnic1, meaning that's why 192.168.1.15 can only talk to .1.223. How can I bridge this traffic so that the cable going into vmnic1 (PFSense LAN IP of 192.168.1.223)  can reach IPs on VMNic0 (192.168.1.162 on same VM and 192.168.1.162 on other VM)?

0 Kudos
harry89
Enthusiast
Enthusiast

Go to esxi host command line (ssh putty).

Mark vmnic1 down

localcli network nic down -n vmnic1

Check the connectivity.. both vm adapters will be mapped to  vmnic0

Then mark vmnic0 down

localcli network nic down -n vmnic0

Check the connectivity.. both vm adapters will be mapped to  vmnic1

*assuming both vmnic0 and vmnic1 are both active adapters on vSwitch

Harry
VCIX-DCV6.5 ,VCIX-NV6 , VCAP-CMA7
Mark answer as correct/helpful if it solves your query
0 Kudos
CantVM
Contributor
Contributor

VMnic0: this is the nic used for user services, I can't interact with the switch associated to this port. If i could I'd just plug the PFSense through that switch and my problem would be solved because I would not need multiple adapter.

VMNic1: this is connected to a switch that is connected to only a PFSense box. I need to be able to contact all vms on vSwitch0 through vmnic1

I can't assign that vm only one vmnic because all the services users need for it would be unavailable except for through the PFSense box, which would defeat the purpose of what I'm trying to do.

My biggest issue is that I can't talk between vmnic1 except on the vm with both adapters using both vmnics. How can I make it so all vms can communicate through vmnic1 while still being assigned to vmnic0

0 Kudos