Network Traffic Not Staying Local To Host

[mlmbcal]

Hello,

We have a netapp storage appliance attached to our esx hosts. We have noticed that VMs on the same host using the same vswitch will intermittently use the vmnetwork vswitch to send data to each other rather than the iSCSI vswitch, thus sending the data out to our dell switch and back in. When this happens we get about 500 Mb/s throughput. Without any intervention on our part, the VMs will, at random, copy data to each other locally within the host using the iSCSI vswitch only and we will see a transfer rate of up to 3.5 Gb/s. This is obviously preferable as we have DRS rules in place to keep some large SQL databases and their app servers on the same host. The iSCSI vswitch and vmnetwork vSwitch each have two active NICs configured for round robin. My question is, why would our VMs send data to a VM on the same host/vswitch out to the network rather than keeping the transmission within the host? We have even seen, for example, VMserver-A copy to VMserver-B at a 3.5 Gb/s rate while, at the same time, VMserver-B copies to VMserver-A at a rate of 400 Mb/s. Give it 5 minutes or so and try the B to A copy again and it will fly along at 3.5 GB/s. Any suggestions on where to start looking? Our environment and Netapp were setup by the vendors using certified technicians so I am confident the configuration should be correct...

Thanks everyone!

[zying]

It looks like you have at least 2 vNICs per VM: one is connected the vmnetwork vswitch ,the other is connected to the iSCSI vswitch. If this is the case, then it's very likely the VM itself (GuestOS) decides which vNICs to use, rather than ESXi. Check the route talbe in the VM, see if it's configured correctly.

[mlmbcal]

Thanks Zying but no, only one nic per vm. Checked to be sure and our iSCSI vlan subnet is not in the routing table for any of our VMs. The iSCSI vlan is not even reachable from the within the guest VM, only our ESX hosts can connect to the iSCSI vlan. Its bizarre. I have taken turns setting one of the two nics the iSCSI vSwitch and vmnetwork vSwitch to standby in an attempt to isolate the issue to a particular nic or switchport (each nic on our vSwitches, iSCSI and vmnetwork, is connected to a different switch, two identically configured switches running together for redundancy) and each time results were the same. Sometimes it uses the iSCSI vSwitch to copy the data within the host and sometimes it uses the vmnetwork, out and right back into the host. Its bizarre.

[rickardnobel]

If possible, could you post a screenshot of the networking setup with the vSwitches for the iSCSI and the guest network?

[mlmbcal]

Hello all,

Sorry for the delay in replying, was off for the holidays.  I have limited my testing to 2 vms, both server 2008 x64, on the same host, on the same vswitch, on the same datastore. A robocopy from server A to server B runs at 30,000 MB/Min. Robocopy from B to A runs at 4000 MB/min. Both copies are a push copy, robocopy c:\temp\test \\otherserver\c$\temp\test /is. If I run a copy on server B and pull the file from the other server (robocopy \\otherserver\c$\temp\test c:\temp\test /is) it runs at 30000 MB/min. If I reverse the copy (robocopy c:\temp\test \\otherserver\c$\temp\test /is) it runs at 3000 MB/min. So regardless of what server I am on or how I run the copy, copy from server A to B is very fast, B to A is slow. And, by using a bandwidth monitor on the switch, I confirmed that the fast A to B copy does not use the host's vmnetwork vswitch interfaces and copies over the iSCSI switch ports only but copy B to A goes out the vmnetwork vswitch port and back in, traffic on the vmnetwork ports spike for the B to A copy but not on the A to B copy.

What screenshots would be helpfull? Not sure which settings in particular you would like to see. I am not 100% convinced that it is the host server config as in my testing both VMs are on the same host and vswitch. Both VMs are using the E1000 adapter and both have the same settings on the E1000 adapter.

Thanks again for your help.

[mlmbcal]

Attached is a screenshot of our vmnetwork config. vswitch1 is our LAN and vswitch2 is for our iSCSI network. In playing around with this some more I noticed that if I vmotioned the two servers I have been playing with to another host they both copy locally via the host, as expected. But other vms on that host do not. And when I vmotioned the vms back to the host they were on, the problem goes away but is present on other VMs. Its almost as if when a VM is vmotioned to a host it selects a path for local communication between VMs and sticks with it. Sometimes it transfers data locally and sometimes it does not. Please let me know if any other screenshots will help. The iSCSI-2 VMkernal port uses vmnic2, vmnic6 is disabled and iSCSI-1 uses vmnic6 with vmnic2 disabled. Our iSCSI network is on a seperate vlan from the LAN and I have confirmed the switch is configured properly although we never setup a vlanid for vmnetwork-12, it is set to none (0) but from what I understood that was optional.

Thanks again!

[rickardnobel]

mlmbcal wrote:

We have a netapp storage appliance attached to our esx hosts. We have noticed that VMs on the same host using the same vswitch will intermittently use the vmnetwork vswitch to send data to each other rather than the iSCSI vswitch, thus sending the data out to our dell switch and back in.

Thanks for the screen shot! Just some more clarification, you say above that the VMs sometimes does use the vmnetwork when sending traffic to each other. However, this should be expected. When you say that they does not always use the iSCSI vSwitch, what do you want to happen? Any traffic from the Virtual Machine will always just hit the first connected vSwitch. No "logical" VM traffic should be on the iSCSI vswitch (or network).