I have a problem with our vDS. Let me explain our topology first:
a) two ESXi v5.1U2 servers (ESX1 and ESX2)
b) two virtual machines each on separate ESXi (DMZ, PROD)
c) distributed switch with uplink port group and two port gropus (dvPG_DMZ, dvPG_PROD)
e) each port group has it dedicated physical nic (dvPG_DMZ = vmnic1 and dvPG_PROD = vmnic2) connected to physical switches
f) for explanation purposes DMZ segment is 150.1.5.x and PROD segment is 15.155.130.x
g) between DMZ and PROD PHYSICAL switches we have a firewall because DMZ vms are forbidden to access PROD newtork
Virtual machine DMZ has IP address 18.104.22.168 and its vmnic mapped on dvPG_DMZ
VM PROD has IP 22.214.171.124 and vmnic on dvPG_PROD.
My problem is that if I put both VMs on same ESXi and change DMZ VM IP address to one from PROD segment (ie 126.96.36.199) those VMs can ping each other. In other words their traffic is not routed through physical nic and blocked by our FW. Instead it seems that all traffic is done on distributed switch. I must mention that during testing they stayed on different port groups. Are we missing some vDS option here?? As far as I understood all traffic between port groups on same distributed switch SHOULD go through physical cards.
Second issue is, I believe same reason, two VMs with SAME IP address but on different port groups have IP conflict while they are on same host, as soon as we separate them no IP conflict. Physical nics from port groups are connected to separate physical switches which are not connected with each other in any way. So it is impossible to have traffic going from one switch to another.
We have following options set on port groups
Promiscuous mode - Reject
MAC Address Changes - Accept
Forget Transmit - Accept