vMotion configurtions on a 10GB port channel pair of uplinks

[nabeelsayegh]

Hello...

I have a question that I am hoping someone can answer for me.  It has to do with a problem I am having with server-to-serve vmotion performance.  First, let me give you my setup info:

• 16 node cluster (Dell PowerEdge R820's using Intel x540 10GB dual port RJ45 network adapters.
• ESXi 6.5 Update 2 patched to latest greatest.
• All Host firmware also update to date.
• Each host is attached to a pair of Cisco Nexus 7K's (that are ISL'd together).
• Each host in cluster is configured in a standard (non-LACP) port channel.  Jumbo packets NOT enabled.
• a Single dvswitch (6.5) is configured for all hosts
• two vmkernel interfaces....one for management in VLAN 30 and one for vmotion VLAN 31.
• no gateway override configured for VLAN 31/vmotion vmkernel….everything stays layer 2
• There is about 100 VM's currently hosted in this cluster. each are 2 proc, 16GB, single 40GB HDD, 1 VMXNET3 all running hw version compatibility for 6.5
• Each host runs the same number of VM's....or as close to it as it can; about 6 each.

When I try and put a host in maintenance mode....some VM's fly and peak about 5-6GB/s of network throughput.  Others seem to be capped at 1GB and run MUCH slower.  They always complete, but I am struggling to figure out why.  Network ports do not show any packet drops or CRC errors.  ESXTOP also does not show any issues while vmotions are in progress.  I did notice one very interesting thing.  When a vm that is running 'slow' from one host to another, it will run slow with any vm….on these hosts....not just one.  When I look closer at ESX top, I noticed that traffic coming from source host is coming out of VMNIC 1 and being received by VMNIC 2 on the target.  VM migrations that run fast are sending and received traffic on the SAME VMNIC.  So I immediately thought that perhaps this is an issue with ISL saturation.....iow...the only way traffic that is going from one host (and the primary nic) to the second host (secondary nic) is through the ISL.   Then it occurred to me that esxi should be smart enough to figure this out on it's own and not attempt to send and receive traffic so arbitrarily (or at least, not give us the ability to confine or define a preferred link for vmotion traffic to eliminate this situation altogether.)   I am not looking to add any additional NIC's or go back to an active/standby configuration until I have exhausted all possibilities.

So, any suggestions/recommendations would be welcome.

[daphnissov]

Ok, so your problem is likely due to your LAG (passive). Before even getting into that, why are you using a bond in the first place?

[nabeelsayegh]

So the portchannel configuration was used to aggregate the uplinks to provide both fault tolerance and load distribution.  We use this portchanneled 10GB pair not just for vmotion, but consolidating management and VM traffic as well.

[daphnissov · Accepted solution]

Ok, that's a natural reason, but if you're using a vDS already you *do not need* to bond these links to achieve the same thing. By using LBT on the VM port groups it actually has a better, more effective chance at distributing that load across those NICs because it looks at saturation. Management doesn't need it. For vMotion, if you only have 2 vmnics per ESXi host (0 and 1), this is what I typically configure if you have Management and vMotion as kernel services and N number of VM networks:

• Management (0 active; 1 standby)
• vMotion (1 active; 0 standby)
• VM Network N (0, 1 active [LBT enabled[)

When you configure vMotion in this fashion, the ISL will not be traversed so long as vmnic1 is connected to the same switch on all hosts and so you'll see maximum evacuation effort that won't impinge upon management services. It's common that people want to use LAGs with ESXi but usually because they just don't know that vSphere offers actually better mechanisms to deal with such load. There are very few use cases where LAGs (even active) should be used in vSphere and they almost always end up causing more problems, confusion, and complexity than they alleviate.

[nabeelsayegh]

This is definitely doable.  I guess that would accomplish the same thing and also give me the flexibility to do more creative work with the NIC's.  Thank you!

BTW....one thing we never got into looking at was multi-nic vmotion (by defining multiple vmkernels pinned to a specific NIC) due to our portchannel configs.  Not sure I really need to do this anymore, but anyone have thoughts about its value and if it is worth really looking at.

[daphnissov]

You definitely could do multi-NIC vMotion and in 10 GbE designs I do that have more than 2 vmnics per host, I'll often use this (for example, with a pair of dedicated 10 GbE vMotion interfaces). I couldn't recommend that for you since I don't know all the functions that your management network has to perform. For example, some backup applications and appliances will pull VM data through the management vmkernel port (which isn't efficient, but that's another discussion), and in those cases you may not want to configure it in such circumstances. However, that said, if you only have ~6 VMs per host with ~16GB vRAM per VM, multi-NIC vMotion isn't *really* going to buy you much in return. It's most effective for large, in-memory database VMs or hosts with high degrees of saturation, or similar. So, while you may see some benefit from it, it isn't likely going to be night and day.

[nabeelsayegh]

yeah, it definitely would not buy me anything within this cluster.  I have many others with up to 50 VM's per host and other clusters (SQL specific) that have VM memory configs north of 500GB.  In those situations it may be worth looking into.

Circling back to my original problem.  I was putting more thought into this and I came to the sad realization that converting my clusters to the config you mentioned is going to be a pretty huge under taking.  The dvswitch has hosts from many clusters and all the port groups are configured in the same way (IP hash for port-channeled NIC's).  I would almost have to do this an entire cluster at a time.  Define a new dvswitch and port groups and then migrate the hosts/VM over at the simultaneously....or at least one host (and associated VM's) at a time.  The problem I am going to have is time....that is....getting the window to do this work in.

You live, your learn.