VMware Cloud Community
zmichl
Contributor
Contributor

vMotion and port-based load-balancing?

Hello,

I am planning our new vSphere installation at the moment and had thinked about how to configure networking for a while now. I have 12 physical nics in each esx-server and 2 physical switches. I want the network connectivity of the esx-servers to be redundant and load-balancing between the 2 switches. As my switches don't support distributed trunking (cross-stack etherchannel in cisco-language) I am going to use port-based load-balancing for my vSwitches. I've read about port based load-balancing and understood that outbound traffic alway goes out at the same physical nic based upon the port-id on the vSwitch the vm is connected to. This means, that a virtual machines network-speed is limited by the speed of the physical nic the traffic goes at. Is this correct?

However, this is not my main-problem with port-based load-balancing. I just asked myself how port-based "load-balancing" (as its not real load-balancing at all) works together with vMotion. As vMotion is realized by a vMotion-enabled vmkernel-port on a vSwitch, vMotion-traffic is always going out at the same physical nic, based upon the port-id, the vmkernel-port is at. Imo, this means that vMotion can't suffer from having multiple nics configured for the vswitch. Is this right?

So what should I do to speed up vMotion while still being redundant? I figured out, that I could set up 2 vMotion-enabled vmkernel-ports, each mapped to 2 physical nics. The 2 nics of the 1st vmkernel-port would be connected at the 1st switch and the 2 nics of the 2nd port would be connected to the 2nd switch. With this setup, I could enable 802.3ad aggregation separately on both switches and set up ip-based load-balancing for the vmkernel-ports.

But how does the esx distribute vMotion traffic across the 2 vmkernel-ports in this setup? Does the esx dynamically spread vMotion-traffic across the 2 ports at all?

I would be glad to read about comments on my idea and any other ideas to solve this problem...

Reply
0 Kudos
5 Replies
AndreTheGiant
Immortal
Immortal

This means, that a virtual machines network-speed is limited by the speed of the physical nic the traffic goes at. Is this correct?

Yes

this means that vMotion can't suffer from having multiple nics configured for the vswitch. Is this right?

Yes, for this reason you can have a single vSwitch for VMotion and FT (for example) with explicit failover for each vmkernel interface.

I figured out, that I could set up 2 vMotion-enabled vmkernel-ports, each mapped to 2 physical nics.

This does not work, only the first interface will be used.

Andre

Andre | http://about.me/amauro | http://vinfrastructure.it/ | @Andrea_Mauro
Reply
0 Kudos
Texiwill
Leadership
Leadership

Hello,

VMKernel NICs are limited to 'explicit' failover modes and not load balanced. In other words you have 2 pNICs attached to 1 vSwitch and 1 pNIC is active for vmkernel NICs at a time.

It is possible for some vmknics to get 'multipath' type configurations but it requires the use of multiple networks and subnets to make happen. vMotion is not one that allows this.


Best regards,
Edward L. Haletky VMware Communities User Moderator, VMware vExpert 2009

Now Available: 'VMware vSphere(TM) and Virtual Infrastructure Security'[/url]

Also available 'VMWare ESX Server in the Enterprise'[/url]

Blogging: The Virtualization Practice[/url]|Blue Gears[/url]|TechTarget[/url]|Network World[/url]

Podcast: Virtualization Security Round Table Podcast[/url]|Twitter: Texiwll[/url]

--
Edward L. Haletky
vExpert XIV: 2009-2023,
VMTN Community Moderator
vSphere Upgrade Saga: https://www.astroarch.com/blogs
GitHub Repo: https://github.com/Texiwill
Reply
0 Kudos
zmichl
Contributor
Contributor

Okay, so are there any other posibilities to speed up vMotion? I thought to build a 802.3ad trunk of 2 pnics per switch- one active (primary) trunk and one standby (secondary) trunk. But I am only allowed to add single pnics to the vMotion enabled vmkernel-port. So if one adapter in the primary trunk fails, the esx server would activate one of the adapter in the secondary trunk (which would be conected on another switch) in addition to the remaining adapter of the active trunk. That is not what I want. My intention is to make esx taking the whole primary trunk offline and using the secondary trunk. It seems that this type of configuration is not possible with vSphere...?

Reply
0 Kudos
Texiwill
Leadership
Leadership

Hello,

To Speed up vMotion Transport times I would use 10G over 1G pNICs and 1G over 10/100 pNICs. However, there is more to vMotion than just the transport. The transport on a 1G connection is sufficiently fast. But the wall clock time does not appear to be. I would also use a set of 1G pNICs dedicated to vMotion. In many cases it takes ESX longer to setup the VM on the new host than it does to transfer inuse memory! Remember, the vMotion transport is only for in-use memory not the entire memory stack.


Best regards,
Edward L. Haletky VMware Communities User Moderator, VMware vExpert 2009

Now Available: 'VMware vSphere(TM) and Virtual Infrastructure Security'[/url]

Also available 'VMWare ESX Server in the Enterprise'[/url]

Blogging: The Virtualization Practice[/url]|Blue Gears[/url]|TechTarget[/url]|Network World[/url]

Podcast: Virtualization Security Round Table Podcast[/url]|Twitter: Texiwll[/url]

--
Edward L. Haletky
vExpert XIV: 2009-2023,
VMTN Community Moderator
vSphere Upgrade Saga: https://www.astroarch.com/blogs
GitHub Repo: https://github.com/Texiwill
Reply
0 Kudos
DennisBray
Enthusiast
Enthusiast

So what should I do to speed up vMotion while still being redundant? I figured out, that I could set up 2 vMotion-enabled vmkernel-ports, each mapped to 2 physical nics. The 2 nics of the 1st vmkernel-port would be connected at the 1st switch and the 2 nics of the 2nd port would be connected to the 2nd switch. With this setup, I could enable 802.3ad aggregation separately on both switches and set up ip-based load-balancing for the vmkernel-ports.

Setting up 802.3ad won't give you a single larger pipe to run the vMotion transfer. And understand that IP hash load balancing only helps distribute the traffic when there is more than a single IP destination. vMotion is always just between a pair of ESX hosts. That is 1 source IP and 1 destination IP, therefore a single interface will be used, regardlesss of the number of physical nics you team together.

If you are determined that bandwidth is the limiting factor, then go with Edward's suggestion of using faster ethernet ports.

Dennis Bray VCI, VCP

Reply
0 Kudos