VMware Cloud Community
AaronDelp01
Enthusiast
Enthusiast
Jump to solution

2 Concurrent vMotions saturate a 10GB link with 4.1?

Hello all - I've been doing some thinking (and writing for an upcoming blog article) based on a peer's testing in our lab and I'm wondering if anybody can verify what we are seeing.

With the introduction of vSphere 4.1 the number of concurrent vMotions on 10GB has been increased to 8. Furthermore, the amount of bandwidth a vMotion can consume has been increased to 8Gbps. With no Traffic Shaping, NetIOC, or QoS, this sounds like it could lead to some pretty significant performance problems because the second (and subsequent) vMotions will saturate the link. Somebody suggested that the 8Gbps is the TOTAL bandwidth that the 8 vMotions would consume but that doesn't appear to be the case.

We did testing in our lab and we are seeing exactly the opposite. The first vMotion will spike to aruond 8 Gbps and the 2nd vMotion will peg the link at a little over 9 Gbps. Again, this is with no traffic shaping, QoS, NetIOC, etc.

Has anyone else seen this? If this is true it pretty much means that some form of Inbound and Outbound Traffic shaping MUST be utilized or you can get yourself in trouble pretty quick with a single pair of 10GB links.

Thank you for your time!






Aaron Delp

http://blog.aarondelp.com

http://blog.scottlowe.org/author/adelp/

Aaron Delp aarondelp.com // @aarondelp
0 Kudos
1 Solution

Accepted Solutions
dilpreet
VMware Employee
VMware Employee
Jump to solution

Hi Aaron,

I was just trying to answer your question regarding 8G per vMotion or 8G total for all vMotions going out of the box and I wanted folks on the thread to be able to get to the NIOC documents.

As of 4.1, we reserve one core for ALL vMotions which is just enough to be able to drive line rate on a 10G link. As you might have seen in VMworld with some 40G NICs being demoed we can take up more cores if they are available because we currently don't set a limit (the demos showed vMotion taking up 26G out of the 40G NIC, only limited by the PCIe fabric since the card was 8 lanes I believe). So another way to limit vMotion in particular is to set an explicit CPU limit for vMotion in the UI which is some percentage of a core rather than a full core or a full core to make sure we cannot exceed 10G.

You may also want to devote a blog on how to firewall the vMotion network such that it is isolated from the other networks, since vMotion is a spoiled child right now and doesn't share it's toys well.

Regards,

-Dilpreet

PS. I will work on getting the KB created.

View solution in original post

0 Kudos
20 Replies
vmroyale
Immortal
Immortal
Jump to solution

Hello.

I attended VMworld session TA8440 - "10Gb & FCoE Real World Design Considerations" and this was discussed. The presenter showed traffic graphs and the 8 Gbps was for a single VMotion. He recommended multiple times that controls be implemented to prevent VMotion(s) from saturating a 10 Gb link and impacting performance.

If you can get that session when it becomes available, there is a lot of great technical information in there.

Good Luck!

Brian Atkinson | vExpert | VMTN Moderator | Author of "VCP5-DCV VMware Certified Professional-Data Center Virtualization on vSphere 5.5 Study Guide: VCP-550" | @vmroyale | http://vmroyale.com
0 Kudos
AaronDelp01
Enthusiast
Enthusiast
Jump to solution

Hey There! I'm glad you brought that up! Yes, I actually work with Don Mann (who presented the session) and it based off of his testing that I'm asking the questions. I agree the session was great and I would really like to see if there is anybody out there that is seeing the same things that he did in testing in preparation for the session.

I feel like since 4.1 has been out for awhile somebody would have encountered this by now.

Thank you for your response!!






Aaron Delp

http://blog.aarondelp.com

http://blog.scottlowe.org/author/adelp/

Aaron Delp aarondelp.com // @aarondelp
0 Kudos
yorkie
Hot Shot
Hot Shot
Jump to solution

Hey Aaron, this is a good catch and I think we could all do with a paper that has data and guidance in it. What I'd love to see is some charts showing the Vmotion traffic AND it's impact on non-Vmotion traffic.

In Cisco UCS the default, as you know, is to bunch all LAN traffic in one class and all SAN (FCoE) in another, making sure they share bandwidth 50/50 should there be any contention, though both can burst above that, as long as they don't impact the other. This means if your FC traffic is only using 20%, then your Vmotion traffic if free to saturate the other 80% (at the expense of other LAN traffic).

Depending on the path of the Vmotion traffic (within the same fabric vs. from fabric to fabric via northbound switches) then the impact of Vmotion saturating the link will of course be different, but if we apply QoS consistently across the devices (if they can do QoS, not all can!) then we can protect all paths.

But what to set QoS to? Rate limiting is one way of focusing on Vmotion traffic, and QoS is the additional way (complementary to rate limiting, not instead of) to focus on non-Vmotion traffic. If we used Nexus 1K we could mark the VLANs (e.g. Data, Vmotion, Management, NFS etc) and then limit Vmotion and/or guarantee non-Vmotion. If we used the Cisco VIC then we could mark the Vmotion NICs. Some great choices here.

I've pinged BradHedlund already and would love to be involved in some kind of paper development on this. I think it (finally) raises a significant design aspect to virtualization that few architects take seriously, or understand, today. It's a classic area where two disciplines (virtualization and networking) collide and requires cross-domain skills.

Let's work on it this week!

Cheers

Steve

0 Kudos
vSeanClark
Enthusiast
Enthusiast
Jump to solution

Some good ole' best practices for isolating VMotion on it's own physical interfaces and switches could make this all a moot point. Smiley Wink

At least then the solution could span multiple network hardware providers and require less complex config, but at the expense of more ports and switches. Will be interesting to see further guidance come out.






Sean Clark - vExpert 2009, VCPx3 - http://twitter.com/vseanclark - http://seanclark.us - http://vmunderground.com

Sean Clark - http://twitter.com/vseanclark
0 Kudos
dgourlay
Contributor
Contributor
Jump to solution

a few random thoughts...

1) At 1GbE having dedicated segments for vMotion and other VMK traffic makes total sense, but it may be cost prohibitive at 10Gb to do the same, at least for the next 2-3 years until prices come down to GbE lovels and multiport 10GbE NICs are built into the mainboard.

2) If we then use 10Gb interfaces on the server connecting to a pair of upstream switches (a common config) I would suggest using a two/three-color policer on the physical switch port and the inherent capabilities of the vSwitch or vNDS.

a) On the vSwitch use the vSphere 4.1 controls for managing traffic and allocating bandwidth to each traffic types. It's a crying shame that .1p or DSCP are not supported for classification on the vSwitch, but it is what it is... This will control what a givenserver can send.

b) Mirror this policy on the physical switchport. Mark traffic that is above the watermark for a given traffic class to the lowest 'best effort' or discard eligible class of service so it is serviced last.

This will ensure that if you set 2Gb of bandwidth to vMotion that you can send and receive 2Gb guaranteed, and that if more bandwidth is available for vmotion it can/will be used; but if the data plane traffic for the aggregate of the VMs is sending/receiving it will not have to contend with vMotion.

dg

0 Kudos
yorkie
Hot Shot
Hot Shot
Jump to solution

That's a good view point to keep in mind, Sean. Sounds like there are at least the following options:

1) Use dedicated 1GbE links - the pro's being familiarity, simplicity; the con's being limited bandwidth, port sprawl.

2) Use shared 10GbE links - the pro's being high util, lower TCO; the con's being impact on other traffic, QoS complexity.

3) Use dedicated 10GbE links - the pro's being high bandwidth, simplicity; the con's being cost, port sprawl.

I think this discussion is about option 2, bookmarked by the others. I think (1) is going away and (3) might be coming, but (2) is definitely the now and immediate future for some time yet.

Like all things, I think QoS is complex if you start with a blank piece of paper. If you start with a good guide, with good data and good practice, then you have a way to mitigate that complexity.

What you can't change is the fact that this topic represents and overlap between server/desktop virtualization and network worlds, which is possibly one of the main reasons for uncomfortableness and, IMHO, the main reason we focus on this and fix it.

Cheers

Steve

AaronDelp01
Enthusiast
Enthusiast
Jump to solution

All good points! The closet thing I have seen to a good QoS "Cook Book" is the NetApp/Cisco/VMware SMT Implementation guide. Inside is all the steps needed to QoS on the 1kv option.

BTW, I have posted my thoughts on the ideas here: /[http://blog.aarondelp.com/2010/09/keeping-vmotion-tiger-in-10gb-cage-part.html|http://blog.aarondelp.com/2010/09/keeping-vmotion-tiger-in-10gb-cage-part.html]

What we really need is recommendations on how to handle both Inbound and Outbound traffic for all types of situations. I have the vSS and vDS pretty much under control but the 1kv I will need a BUNCH of help on.

Steve, I'll reach out to you this morning and get you introduced to Don Man..

Thank you!






Aaron Delp

http://blog.aarondelp.com

http://blog.scottlowe.org/author/adelp

Aaron Delp aarondelp.com // @aarondelp
0 Kudos
dilpreet
VMware Employee
VMware Employee
Jump to solution

Hi Aaron,

vMotion traffic is already capped at 10G for 4.1, only one core is reserved for vMotion traffic which will keep us at 10G. That being said we have new technologies like network IO control (released in 4.1) which allows you to assign shares to the various traffic types (VM network, management, vMotion, FT, etc) and will allow you to cap traffic based on the assigned shares when the uplink in question is congested. So if you assign the following shares 40% VM network, 40% vMotion and 20% management and the workload prior to vMotion is only using 6G out of the 10G, when vMotion is initiated it will capped at 4G such that it will not affect the current running network traffic. For more information: http://www.vmware.com/files/pdf/techpaper/VMW_Netioc_BestPractices.pdf.

Regards,

-dilpreet

AaronDelp01
Enthusiast
Enthusiast
Jump to solution

Dilpreet - We know the answer is use some form of control to the links, please see the link to my blog article. The point here is that WITHOUT traffic control you can now saturate the link as of 4.1 with vMotion traffic and it looks like it may cause unpredictable results to other types of traffic. This wasn't the case with 4.0 as the vMotion traffic was capped at around 5 Gbps. Due to this change, there is a fundamental difference in the design as traffic control should now be considered MANDATORY as of 4.1. In my experience, many didn't use it nor really understand it prior to this. VMware needs to publish more documentation around this subject to bring awareness to this fact.

As I'm researching this we also need a way to Inbound Traffic Shape with vDS in a priority kind of way ala NetIOC as opposed to Inbound Traffic Shaping. Priority control out and limit control in leads to some ugly solutions and situations.






Aaron Delp

http://blog.aarondelp.com

http://blog.scottlowe.org/author/adelp/

Aaron Delp aarondelp.com // @aarondelp
0 Kudos
dilpreet
VMware Employee
VMware Employee
Jump to solution

Hi Aaron,

I was just trying to answer your question regarding 8G per vMotion or 8G total for all vMotions going out of the box and I wanted folks on the thread to be able to get to the NIOC documents.

As of 4.1, we reserve one core for ALL vMotions which is just enough to be able to drive line rate on a 10G link. As you might have seen in VMworld with some 40G NICs being demoed we can take up more cores if they are available because we currently don't set a limit (the demos showed vMotion taking up 26G out of the 40G NIC, only limited by the PCIe fabric since the card was 8 lanes I believe). So another way to limit vMotion in particular is to set an explicit CPU limit for vMotion in the UI which is some percentage of a core rather than a full core or a full core to make sure we cannot exceed 10G.

You may also want to devote a blog on how to firewall the vMotion network such that it is isolated from the other networks, since vMotion is a spoiled child right now and doesn't share it's toys well.

Regards,

-Dilpreet

PS. I will work on getting the KB created.

0 Kudos
AaronDelp01
Enthusiast
Enthusiast
Jump to solution

Dilpreet - Thank you for your response! I'm sorry if the previous post came off the wrong way, it was not my intention at all. I wanted to confirm what we are seeing and I believe you have done that. We now need to figure out how to take this information and control vMotion going forward. By the way, your quote of: vMotion is a spoiled child right now and doesn't share it's toys well. is the best explanation I have heard!!

Thank you for your information and I will post follow up links to my blog articles and any white papers that Steve and the folks have Cisco author as well.

Thank you!






Aaron Delp

http://blog.aarondelp.com

http://blog.scottlowe.org/author/adelp/

Aaron Delp aarondelp.com // @aarondelp
0 Kudos
chadwickking
Expert
Expert
Jump to solution

Really good post, very informative as we are looking into this as well. I like how you higlight how if there is not QoS, or NOIC that this occurs compared to that of 4.0. If we find anything relevant in our labs I will be sure to post as well.

Cheers,

Chad King

VCP-410 | Server+

"If you find this post helpful in anyway please award points as necessary"

Cheers, Chad King VCP4 Twitter: http://twitter.com/cwjking | virtualnoob.wordpress.com If you find this or any other answer useful please consider awarding points by marking the answer correct or helpful
0 Kudos
AaronDelp01
Enthusiast
Enthusiast
Jump to solution

Here's a great link from Jason Boche's site that is a GREAT video by Dilpreet to further explain what is going on. Thank you to Jason for the link and to Dilpreet for putting together the video!!

Jason's Article: Meet the Engineer: Dilpreet




Aaron Delp

http://blog.aarondelp.com

http://blog.scottlowe.org/author/adelp

Aaron Delp aarondelp.com // @aarondelp
0 Kudos
dgourlay
Contributor
Contributor
Jump to solution

Does vMotion traffic use a well defined TCP port/port-range? Am wondering how I can create a QoS policy on a network port to WRR this traffic between Storage, Ethernet/IP Data Traffic and vMotion.

dg

0 Kudos
rmaureira
Contributor
Contributor
Jump to solution

Maybe more vendors should do something similar to HP's VirtualConnect. With their Flex10 and FlexFabric products you can create virtual nics on each 10GB link (4 per link) and manage the bandwidth allocated to each vnic dynamically. QoS seems like a major pain in the bottom.

0 Kudos
dgourlay
Contributor
Contributor
Jump to solution

Isn't fixed bandwidth allocation per vNIC a form of QoS? Plus from what I head here - http://bradhedlund.com/2010/08/16/cisco-ucs-qos-vs-hp-virtual-connect-rate-limiting/ it seems that the Flex-10 implementation is pretty static...

I would think a WRR based scheduler with a bucket per traffic class would be pretty darn simple and allow a much more fair bandwidth scheduling while, as you rightly point out, not being a pain in the bottom Smiley Happy

(agree completely on the assertion that most QoS implementations out there are overly complex for the task at hand- major room for improvement in most of these...)

dg

0 Kudos
yorkie
Hot Shot
Hot Shot
Jump to solution

Brad Hedlund has posted a great article on Cisco UCS and QoS: http://bradhedlund.com/2010/09/15/vmware-10ge-qos-designs-cisco-ucs-nexus/

0 Kudos
yorkie
Hot Shot
Hot Shot
Jump to solution

You have to split this problem into two parts, it's not just about the Host NICs - it's about (a) Host Bandwidth and (b) Network Bandwidth.

The days of many dedicated 1GbE ports are going, going gone. New thinking is required for 10GbE, so let's all take a step to the left.

Every platform (HP, Cisco) can do rate limiting on NICs, and now with VMware's NetIOC you can do it from the hypervisor. That bit is easy, but it's only the start of the story.

As Aaron has said elsewhere, this is not just about Vmotion! Vmotion is the obvious candidate for saturation and that saturation is not just for one host's bandwidth, it's also for the Rest of the Network. It takes two-hosts-to-tango to do Vmotion, so that 8Gbps is going via a network somewhere. Imagine 16 hosts doing 8 simultaneous Vmotions - the network has to now do tens of gigs of vmotion traffic. What if there are other busy traffic types that are business critical? What if you are running Unified Comms on the network? What about FCoE?

THere is also the problem that rate limiting is a hammer/nut solution. Allowing VMotion to consume as much bandwidth as possible is a Good Thing - AS LONG as it doesn't saturate it and hurt other traffic like UC, HTTP, DB, FCoE. With rate limiting you are hurting VMotion (or any other traffic) and in $$ terms you are not getting a good enough ROI from your expensive, virtual network. Do you do individual VM limits/reservations/affinities today? That is the equivalent of rate limiting a NIC on the network It's easy but not necessarily right.

QoS is only complicated when it's new, but it's a bit like Jumbo Frames in that it has to be done properly and consistently, but it needs more design work.

QoS is also inconsistently implemented by vendors, even within a vendor it can be done differently on different products.

So, as a Cisco employee I want to make this more transparent and easier for non-networking guys to get a handle on - and so does Brad (see hist post), and Aaron who's not a Cisco employee but is a rockstar working for a great partner.

But back to your point - yes you can Pipe Carve with HP (you can do it with Cisco UCS too) but it isn't the answer, more of putting a bandaid on a deep crack in the wall of a dam Smiley Happy

0 Kudos
AaronDelp01
Enthusiast
Enthusiast
Jump to solution

Steve - Perfectly put. I couldn't have said it better.






Aaron Delp

http://blog.aarondelp.com

http://blog.scottlowe.org/author/adelp/

Aaron Delp aarondelp.com // @aarondelp
0 Kudos