Communication with different portgroups using the same network?

[sandroalvesbras]

Hi,

I have a host with four physical NICs in trunk with VLANs for separate services.

Each VM created receives a specific portgroup for its network.

However, I need to use more than one portgroup on the same VM to create a windows cluster and replicate an SQL server for specific NICs.

I added two new NICs and each NIC is in a specific portgroup. In the image I added, the NICs (2 and 3) are in the same portgroup, but will be in different portgroups.

For the routinely to work it was necessary to add static routes in windows so that the new NICs have access to different networks that are on other servers in Azure, but I have another VM on the same network using the portgroup and the behavior I found super strange.

In the VM that is in the same VMware, using the same portgroups the behavior is identical.

What happens?

When I transfer I copy a file using the IP of the network to the other machine that is also an IP on the same network, in this case the main vNIC that has a gateway I see data transfer also in the other NICs.

The traffic should only occur on the IP network card of the network I am connecting to on the other virtual machine.

When I transfer data using the IP of the other network (porgroup) on vNIC2 I see traffic on vNIC1 too. This on Windows, ok ?!

So I went to look at VMware and all the traffic goes through vmnic0 only.

I'm adding some images for you to help me understand what I'm doing wrong.

Thank you.

[Lalegre]

Hey sandroalvesbrasil​,

So first of all the traffic from one VM to another VM will be created from the shortest path that is know by the routing table so in your case if they are in the same network then it will happen on the secondary vNICs. Regarding the traffic that you see going to the first one I assume here that you have a gateway configured so you will see traffic as probably your DNS are outisde the network same as NTP, etc.

Regarding all the traffic coming out using the vmnic0 could be related to the Teaming and Failover policy you are using in the portgroups. Please check that you have more than one vmnic marked as active and let us know also which method are you using for load balancing such as Route Based on virtual port or Route Based on Physical NIC Load.

From there we can do a good analysis as the placement of the VMs to vmnics changes depending of the load balancing policy you are using and maybe it is expected to work like that.

[sandroalvesbras]

Hi,

on vNIC1 yes i have a LAN network gateway, but on vNIC2 i don't use default gateway. I defined a specific gateway to ensure that the return to the destination uses vNIC2 and not vNIC1.

My fear is that looking at the transfer rate, I see consumption on vNIC1 and I understand that this is wrong, as I do not want to affect vNIC1 traffic since I made vNIC2 available for this specific data transfer.

I want to ensure that only data travels through vNIC2.

What I need to know is whether the fact that there is traffic passing through vNIC1 is wrong because it is generating consumption.

<-----

VM1

vNIC1: 172.16.12.40/24 - Gateway: 172.16.12.254

vNIC2: 172.16.11.40/24 - Gateway: Blank

route add 172.17.11.40 mask 255.255.255.255 172.16.11.254

VM2

vNIC1: 172.16.12.41/24 - Gateway: 172.16.12.254

vNIC2: 172.16.11.41/24 - Gateway: Blank

route add 172.17.11.40 mask 255.255.255.255 172.16.11.254

In this scenario I do not use any additional gateway configuration for the VMs to talk to the 172.16.11.0/24 network, as my L3 switch has internal routing and knows these networks.

Now to talk to another network, yes, I need to add a static route for the 172.16.11.0/24 network to talk to the 172.17.11.0/24 network.

VM3

vNIC1: 172.17.12.40/24 - Gateway: 172.16.12.254

vNIC2: 172.16.11.40/24 - Gateway: Blank

route add 172.16.11.40 mask 255.255.255.255 172.17.11.1

route add 172.16.11.41 mask 255.255.255.255 172.17.11.1

---->

Thank you.

[sandroalvesbras]

More interesting is that if I disable vNIC2 on Windows and VMware, I still access the other server via RPC using the vNIC2 IP that was disabled. Bizarre!

[Lalegre]

If the /32 route is in place the traffic you should expect is that one as is have more weight and is preferred over the default one. If you are currently facing some traffic of the vNIC1 then I would recommend you to open a Wireshark to see that is happening there.

In this case you have to think what traffic flows normally over there. Focusing in the first scenario I would recommend you to copy a file over that network and check what is happening in the background.

I am saying this to discard any application behavior that forces the connectivity over the other interface. I assuming here for example that you are not using FQDN for doing the test in case of having the DNS Record pointing to the IP in vNIC1.

[sandroalvesbras]

I did a few more tests.

When I copy a file to the other virtual machine that also has the two vNICs on the same networks, that is, with the same configurations I see the traffic also on vNIC1 and vNIC2.

In this case here as I have two VMs in the same VMware cluster with the same configuration as the ports / trunk / portgroup, all I had to do was add vNIC 2 and add the IPs of the VLANs to define in the portgroups that they communicated perfectly.

In this scenario I didn't have to add a static route.

I only needed to add static route when I have another server in another network segment.

[sandroalvesbras]

Can this affect my configuration by showing traffic at the vNIC level, even on different portgroups?

[Lalegre]

The teaming and failover policy will not affect the usage of the vNICs but the usage of the Physical NICs. As you have one uplink as active only all the traffic will go over that one.

if you are actually copying the file between two VMs over the same L2 segment and you are facing same amount of traffic on both vNICs then let me tell you that is weird.

I would highly recommend you the next:

• Running the Wireshark or any packet capture tool during a file copy operation (Focusing on vNIC1 and then on vNIC2)
• See also the performance tab in vSphere from the VM you are trying to connect.

Finally, sorry but i did not get this:

"In this case here as I have two VMs in the same VMware cluster with the same configuration as the ports / trunk / portgroup, all I had to do was add vNIC 2 and add the IPs of the VLANs to define in the portgroups that they communicated perfectly"

If you are trying to copy files over the same L2 should be over the same VLAN (unless you have an special configuration which i doubt here). If the communication happens over two different VLANs then must be routing somewhere and in this case you need to make sure that the routing path is the same from source to destination and the same when traffic goes back.