ESXi5 vMotion traffic limit

[SteveR123]

Hi, I have a question that I haven't seen answered as yet, apologies if I'm duplicating though.

I have a vSphere 5 environment that has run into network switch throughput issues. Apparently the uplinks on the switches that are in use with the ESXi hosts are congested when vMotion is active. This is leading to problems on other servers that share the switches.

We're using NETIOC to prioritise the various traffic types, but that's not the issue as the affected servers are not VM's. I have put a NETIOC limit in place for vMotion traffic at 1GB, but obviously that depends on what other workloads/servers are generating at the time on the switch. It turned out that even during single VM migrations we had issues with the uplinks and therefore other servers.

Now the question! ..... Will vMotion be stable if I put a lower limit in place? Reduce the limit to 500mbps possibly. My concern is that 1GB isolated has always been recommended for this traffic and that anything lower than that may compromise the vMotion task itself?

I understand that vMotion should be on an isolated network but that hasn't been possible in this environment. We have 4 x 1GB uplinks on each host,shared for our dVS with LBT. 10GB is in the pipeline but will be several months away.

Many thanks for any responses in advance

Steve

[Virtualinfra]

It Depends on the hardware and network set up in the environment.

VMware recommends to keep vmotion traffic seperatly, for not disturbing the virtual machine traffic and management traffic.

Since you have only 4X1GB uplink. I would suggest to configure as follows.

dvswitch1 with 4 Up link Nic.

DV port group1 with NIC1 active and NIC2 standby for management Network,

DV prot group2 with NIC2 active and NIC1 Standby for Vmotion.

DV port group3 with NIC3 and NIC4 for Virtual machine Network.

Try this..I might not be exact..

[SteveR123]

Thanks for the reply Dharshan. I undertsand the networks should be seperated as you suggested and they are indeed configured as seperate dVportgroups and NETIOC is configured to prioritise the different traffic. Unfortunately the issue is when the traffic 'leaves' the ESX host. Based on 4x1GB, the hosts can generate up to 4GB of traffic during a vMotion. The physical switch uplinks are limited to 2GB and therefore issues arise (dropped packets apparently).

I have put a NETIOC limit in place to cap/limit the vMotion traffic that is generated to 1GB, although we still have some issues even at this.

The question is, will I compromise vMotion tasks if this limit is reduced further (to say 500mbps)????

Thanks

Steve

[Techstarts]

chances are very high that you will face latency in vMotion traffic. technically vMotion traffic is purely ruled by what is going on in the VM when it is migrated. i.e. what is in the RAM? So if you have bigger VM say 4 GB RAM and above , you might face timeouts problem.

[SteveR123]

Thanks for the response Techstarts, we are indeed experiencing timeouts as initially mentioned.

Does anyone know what the official VMware line on NETIOC limits for vMotion traffic is? I think i'll have to raise it with VMware themselves.

Thanks and regards.

[danpalacios]

When you mentioned the 2GB uplink, are you referring to the connection to the core/other switches or the backplane within the switch?  It sounds like you are vMotioning from one switch to another and that is the source of your uplink contention issue.  Is there a way to put the hosts all on a single switch or perhaps create clusters which chare a single switch and vMotion within that group?  Also, do you run your storage access through that switch as well?  The additional iSCSI traffic could also be impacting your traffic volume.

[SteveR123]

Thanks for the response. Yes it's the uplink from the physcial edge switch(es) to the core, and the vMotion is the root of our congestion issue. Unfortunately the physical switching infrastructure is shared for all networks (although VLANs are in use), no physical isolation. It looks like the hardware just isn't up to the job anymore.

FYI VMware said that apart from the 1GB recommended there's no right answer as it will depend on the rate of change within memory on the guest, and latency also.

I'm hoping to isolate all vMotion traffic to one physical switch and that should sort us out.

Cheers

Steve

[dukelet]

Suggestion

With 4 nics: try the below:

2 nics - share management with your VMs (management does not use alot of traffic)

2 nics - vmotion (since you are using vcenter 5), configurate multi-nic vmotion

For multi-nic vmtion configuration:

2 dvport group:

With multi-nic vmtion (traffic for vmtion - 1 VM uses both nic 1 and 2 from example below).

example:

vmotion dvport group

nic 1 active, nic 2 standby

vmotion2 dvport group

nic 2 active, nic 1 standby

Chris

[dukelet]

Sorry, missed you last message.

Can you describe how you have your vDS configure (the nics) - the 4 nics for management, VMs and Vmotion?

(if possible screenshot)

Chris

[rickardnobel]

SteveR123 wrote:

Based on 4x1GB, the hosts can generate up to 4GB of traffic during a vMotion.

Do you have explicit configured the Multi-NIC vMotion feature of ESXi 5? If not, the vMotion traffic could not exceed a single ethernet card and only consume 1 Gbit/sec.

[dukelet]

Yes, you have to configure it (see video how to configure)

http://blogs.vmware.com/kbtv/2011/10/how-to-configure-multi-nic-vmotion-in-vmware-vsphere-5.html

The difference is if you have 2 nics, and if you are trying to vmotion 1 guest OS, the traditional vmotion will only use 1 nic. if you have 2 guest OS (vmotion),  it will use round-robin for the 2 nics to vmotion the guest OS

With multi-nic vmotion, if you have 2 nics and have 1 guest OS, it will use both nics (will use the max bandwith) which is faster to vmotion.

Chris

[RicardoAdao]

If you have a 2 gig uplink from the switch  it could be a bottleneck.

For what I see... one idea could be using a configuration like this:

    -> dvport1 - nicA+nicB (active) nicD->nicC (standby) -> VM traffic and esx management

    -> dvport2 - nicC (active) nicD->nicA->nicB (standby) -> Vmotion

    -> dvport3 - nicD (active) nicC->nicB->nicA (standby) -> FTolerance

    -> dvport4 - nicD+nicC (active) nicB->nicA (standby) -> iSCSI

And probably use load balance based in physical link capacity used, that probably could optimize link use... it's just an idea.