This setup is not best practice, but it's the only one I'm able to do unforunately. I've setup OpenVPN on a PFSense box. That PFSense is connected to another physical NIC on a ESXi 5.0.0 server. On one of the VMs I can ping it's adapter 192.168.1.15, but not it's other adapter IP 192.168.1.162. Every VM can ping the 192.168.1.15 IP, but nothing can ping the LAN IP of the PFSense that's connected to vmnic1, it's IP is 192.168.1.223. I've attached a visio to explain the network, I'm confused because I've never had to do a workaround like this before. Why can't these devices talk and how can I make it so that they can? Please help me.
I've attached a diagram of the setup.
OpenVPN client -> connects to PFSense -> can ping PFSense LAN IP -> can ping VMs ip of 192.168.1.15 -> cannot ping anything else
VMs on server -> can ping the 192.168.1.15 -> cannot ping the LAN IP of PFSense 192.168.1.223
VM with both adapters -> can ping everything including PFSense LAN IP
Are you able to ping vmkernel port , since that is also part of same subnet ?
Can you run esxtop on esxi and press n and share the output which physical nics are being used by virtual eth of Virtual machine
Just tested, here's what I have:
- Every VM can ping the kernel port IP 192.168.1.161
- I can ping all the VMs through the ESXi CLI including the 192.168.1.15, but I cannot ping the LAN IP of the PFSense connected to VMnic1 192.168.1.223
PORT-ID USED-BY TEAM-PNIC DNAME PKTTX/s MbTX/s PKTRX/s MbRX/s %DRPTX %DRPRX
16777217 Management n/a vSwitch0 0.00 0.00 0.00 0.00 0.00 0.00
16777218 vmnic0 - vSwitch0 22631.41 56.91 33813.72 44.91 0.00 0.00
16777220 vmk0 vmnic0 vSwitch0 1.67 0.00 3.10 0.00 0.00 0.00
16777221 3426: <domain name> vmnic0 vSwitch0 22278.07 55.19 33783.20 46.99 0.00 0.31
16777222 3452:TSTER vmnic1 vSwitch0 0.00 0.00 0.24 0.00 0.00 0.00
16777223 3452:TSTER vmnic1 vSwitch0 0.00 0.00 0.00 0.00 0.00 0.00
16777225 3501:<VM Hostname> vmnic0 vSwitch0 0.72 0.00 5.25 0.00 0.00 0.00
16777233 13209303:<VM with both adapters> vmnic0 vSwitch0 154.26 3.15 219.35 0.56 0.00 0.76
16777234 13209303:<VM with both adapters> vmnic1 vSwitch0 0.00 0.00 0.24 0.00 0.00 0.00
16777235 vmnic1 - vSwitch0 0.00 0.00 1.43 0.00 0.00 0.00 <-------- NIC I'm trying to get traffic through
Does this help?
One another finding is that why VM is having two network cards( both in same subnet )
This could be the reason .
Okay, so on that vm adapter with IP 192.168.1.15 I could just set the gateway to 192.168.1.223 of the PFSense LAN port. I don't think any traffic is able to get out of the vmnic1, hence why only the 192.168.1.15 can ping that IP. Now, I've also changed the IPs to a 192.168.0.x network and still could not ping the other vms. Would assigned that vmnic1 to the host itself 192.168.1.161 work?
Do the below test :
Keep same subnet on the VM adapters .
Mark one nic down and test connectivity
Then mark second nic down and check connectivity
Assign different subnet on network adapters
assign separate gateway to both
Allow the changes on the firewall and test
If in both the case it doesn't work its either issue with vmnic1 itself or connectivity from vmnic to phsycial switch (port or connecting cable)
Test 1 results:
- I disconnected the vmnic1 on the VM with 2 adapters/IPs, vmnic1 is the IP of 192.168.1.15. No other VM could ping it after I unconnected that network adapter. Only the VM could internally ping it.
- I turned the adapter back on and change the gateway from 192.168.1.223 to 192.168.1.1 and cannot ping the 1.223 address on the VM with multiple adapters now. I changed it back and can ping it.
- Nothing else can reach the .1.233 IP, just the 192.168.1.15
Test 2 results:
- I changed the VM adapter with IP 192.168.1.15 to 192.168.0.15 and the LAN IP on the PFSense to .0.223 with that as the vms GW. The VM with both IPs (192.168.1.161/192.168.0.15 now) can reach the .0.223
- all other VMS cannot reach the 192.168.0.15 or 0.223 IP, subnetting issue most likely
So the switch going from the physical adapter on the VM Server and PFSense is not the issue.
I believe the issue is VMNic1 not being able to tell all other vms that 192.168.1.223 (set as 192.168.0.223 for the test) is reachable.
I'm thinking since I have the PFSense connected to vmnic1, the vm with multiple IPs with both adapters configured, so vmnic0 has IP 192.168.1.162 and vmnic1 as 192.168.1.15., and can only ping the .1.223 that's a cable to vmnic1 means that no traffic that goes in/out of vmnic0 can inherently see traffic from vmnic1, meaning that's why 192.168.1.15 can only talk to .1.223. How can I bridge this traffic so that the cable going into vmnic1 (PFSense LAN IP of 192.168.1.223) can reach IPs on VMNic0 (192.168.1.162 on same VM and 192.168.1.162 on other VM)?
Go to esxi host command line (ssh putty).
Mark vmnic1 down
localcli network nic down -n vmnic1
Check the connectivity.. both vm adapters will be mapped to vmnic0
Then mark vmnic0 down
localcli network nic down -n vmnic0
Check the connectivity.. both vm adapters will be mapped to vmnic1
*assuming both vmnic0 and vmnic1 are both active adapters on vSwitch
VMnic0: this is the nic used for user services, I can't interact with the switch associated to this port. If i could I'd just plug the PFSense through that switch and my problem would be solved because I would not need multiple adapter.
VMNic1: this is connected to a switch that is connected to only a PFSense box. I need to be able to contact all vms on vSwitch0 through vmnic1
I can't assign that vm only one vmnic because all the services users need for it would be unavailable except for through the PFSense box, which would defeat the purpose of what I'm trying to do.
My biggest issue is that I can't talk between vmnic1 except on the vm with both adapters using both vmnics. How can I make it so all vms can communicate through vmnic1 while still being assigned to vmnic0