VMware Cloud Community
bjselman
Contributor
Contributor
Jump to solution

The Switch Stacker?

Would stacking my ProCurve 2900's provide a more stable redundant link and a better flow utilizing ip-hash load balancing? Concurrently, if it is NOT stacked, will both of my vmnic's still run in active/active if set as such? Would it be true load balancing (or sharing, whatever)?

(i have two server racks, each with dll380, nsm2060, pc2900 all links cross-thatched)

(6 pNics/server, 2-sc/iscsi, 2-vmotion, 2-vmnet)

(esx 3.5 / vc 2.5)

thanks in advance

Reply
0 Kudos
1 Solution

Accepted Solutions
ac57846
Hot Shot
Hot Shot
Jump to solution

IP hash

  • Only way to get one VM over multiple pNICs

  • Each VM uses only one pNIC for each conversation

  • vSwitch pNICs must all be connected to one physical switch, i.e. one config

  • physical switch must be aware & must multilink aggregate the ports, etherchannel in Cisco terms

MAC Hash

  • Exists for legacy reasons only

  • Do not use in al ESX 3.x environment

Source Port

  • Default and generally best

  • Each VM uses only one pNIC for all communications

  • Switches are unaware

Unless your servers are going to have multiple clients downloading multiple media files at once you should definitely use Port based load balancing.

If you port based doesn't give enough bandwidth then seriously consider 10GBE before anything else, this will increase the size of the shared pipe and benefit all connections on that vSwitch.

If you must use IP hash then you must stack the two physical switches and configure the ports as a multilink in the Procurve config. You will only get a performance boost for multiple concurrent connections to a single VM from different clients.

The forums definitely rock, I'm a contract VCI filling in lab time in a FastTrack course. If you haven't been to teh official VMware courses you should consider it as we discuss these issues in the course.

Al.

View solution in original post

Reply
0 Kudos
12 Replies
ac57846
Hot Shot
Hot Shot
Jump to solution

If you stack your 2900s you will introduce a fresh single point of failure, the two switches will share a single config, so messing a single switch config will stop all networking. With two switch configs you should only be able to screw one up at a time.

The best advice from VMware is to stick with source port based load balancing unless you have a single VM that floods a GBE link. Even then the single VM must be talking to multiple clients before IP hash will help.

bjselman
Contributor
Contributor
Jump to solution

Well, I will have a few VM's that will talk to many clients. I'll monitor activity with default port-based and maybe try ip-hash if I can figure out how to explicitely set the 2900 to src-dest-ip load balancing. Literature about that is so scarce or I'm not looking in the right place. I was under the impression that ip-hash LB didn't work too well with non-stacked switches because of the mac-duplicates.

Reply
0 Kudos
ac57846
Hot Shot
Hot Shot
Jump to solution

IP based load balancing only works where all of the NICs in the ESX server are attached to one logical switch, i.e. a stack that shares a single config.

By the way if you are using VLANs on the physical switches and all of the physical NICs connect to the same switches I would be inclined to only have one vSwitch & connect all six NICs to it. If you already have all of the traffic on one physical switch there is no change to security profile by having it all on one vSwitch, and then the VM traffic can be spread over more NICs.

If you want to guarantee GBE bandwidth for VMotion & SC then you can use active & standby NIC settings for each port or portgroup.

Attached is my start point network design with two Service console ports, a VMKernel port and a single production portgroup.

Key elements are fault toerance for all functions, guaranteed bandwidth for the VMK, Service console networking doesn't fail unless production VM networking fails (for HA functionality) and high bandwidth for VMs.

In your case I would have all six NICs on the vSwitch & assign four of them as active for VMs, the other two as active/standby for VMotion & Service Console

kukacz
Enthusiast
Enthusiast
Jump to solution

BJ, the ProCurve 2900 does src-dst-ip load balancing as part of their static non-protocol trunking (ie. not the LACP trunking). In the scope of a single switch, of course. HP doesn't describe the balancing algorithm in their manual but as you've already found on my blog, I've verified the balancing works in both directions with "IP hash" set on ESX.

I'm referencing my article Teaming NICs in ESX - configuration, describing the trunking with ProCurve 2900 used as an example switch.

--

Lukas Kubin

Reply
0 Kudos
bjselman
Contributor
Contributor
Jump to solution

we are using vlans, but the switches are cross-thatched for complete redundancy (bond0=vmnic0/vmnic1 -> vmnink0 goes to switch1, vmnic1 goes to switch2, etc). this goes for both pNics for iSCSI/SC, both pNics for vMotion and both pNics for VM's.

If i choose to use active/standby for my iSCSI/SC bond, will that be enough for the iSCSI traffic?

Al, what does A, S and U stand for in your design layout? So if your physical switch dies (i know, highly unlikely), then you are out in the water like Luca Brazi & Fredo?

It doesn't sound like there is a nice load-sharing scheme for the redundancy design handed down from above. (see attached JPG).

Lukas, per our discussions, I just got off the phone with HP. I was told that MAC SA/DA was the only algorithm configured and it was hard-coded (can't change). So what I got is hard-coded MAC l.b. on my switch. When I try MAC-hash on the vSwitch to a non-protocol Trunk, I can ping to and from my x64 guest (grafted from esx2 RP) and to my x86 guest (grafted from esx1 RP), but not from my x86 guest. Both esx1 and 2 have the same exact configs.

I'm gonna take a break and come back and try Port ID-hash and IP-hash with a.) no trunk, b.) trunk, and c.) lacp. I'll post my results. I would like to see at least two different configurations work properly, so I can run traffic tests and pick the best one.

Reply
0 Kudos
ac57846
Hot Shot
Hot Shot
Jump to solution

A = Active, S = Standby, U = unused

These are the NI teaming settings fro each port group. The aim is to dedicate some bandwidth to prticular functions

In my diagram I had a single physical switch, but if port based load balancing is used then multiple physical switches (with independent configs) are supported, attached is an updated picture

For iSCSI it doesn't matter what load balancing rule you use only a single pNIC will be used for a single iSCSI target.

If your switch is doing link aggregation then you will have to use IP hash balancing on the vSwitch, LACP is not supported, the trunking must be static and unconditional.

Reply
0 Kudos
bjselman
Contributor
Contributor
Jump to solution

ok, here's what i found.

If I try to trunk both switches alike (either LACP or Trunk), then I can only ping one x64 VM that was already there. other x64 VM's installed after reconfiguring didn't echo. I could also ping to one (already there) x86 machine, but i couldn't ping from it. the other x86 VM's i installed after configuring didn't echo. No M$ NLB and all client/service/protocols look the same across VM's.

If the ports are not trunked and not protocol'd then MAC has gives me good stability. IP hash works, but things don't seem as smooth. Port ID works too (i think - can't remember). I'm beginning to think that Trunking will not work on this design (thanks Lukas), with the cross-thatching of switches.

I guess I can instead take both iSCSI pNics from ESX1 and patch them to switch1 and vice versa for switch2. Do the same for the VM pNics but NOT vMotion. Then drop them in a large 3 or 4 port Trunk across the switches. If I leave vMotion crisscrossed and switch1 dies, then the vMotion vmknic going to switch2 should pick up the load and migrate succesfully, correct? got some testing to do ahead of me eh.

One thing seems odd. With vSwitches all set to MAC-hash, i can set my 2900's to active LACP (i gather from cryptic HP documentation that this is "Dynamic") and ALL OF MY VM's echo back and forth. So far things seem to be normal. I am compfused in all this vendor bender swahili brewaha.

The way I see it, I got 3 options (we move lots of large multimedia graphic files, so LB is the interest - redundancy can be had either way)

1.) no trunk, no lacp, no protocol, flow-control or not (design for complete redundancy).

2.) ditch the crisscrossing and take the vSwitch bonds to same switch - trunk between switches. Lukas see's balancing this way with IP-hash.

3.) leave it as is - even though its not supposed to work. ?

Of course, the ultimate test is when I have 10+ VM's contending for pipe.

Is it a good representation of testing vMotion by pulling the VM network's link cables from the VM's host machine?

These forums are groovy!

Reply
0 Kudos
bjselman
Contributor
Contributor
Jump to solution

Al,

if the trunk must be static, then how can i set my switches to Active LACP and still pass traffic via MAC or IP-hash? Are you saying it won't balance unless its IP-hash?

or that the vSwitches will just ignore the extra packet header? and if so, this would seem like unncessary overhead. and if so, then crisscrossing the links between the two switches seems futile, since i can't trunk that way.

Reply
0 Kudos
ac57846
Hot Shot
Hot Shot
Jump to solution

IP hash

  • Only way to get one VM over multiple pNICs

  • Each VM uses only one pNIC for each conversation

  • vSwitch pNICs must all be connected to one physical switch, i.e. one config

  • physical switch must be aware & must multilink aggregate the ports, etherchannel in Cisco terms

MAC Hash

  • Exists for legacy reasons only

  • Do not use in al ESX 3.x environment

Source Port

  • Default and generally best

  • Each VM uses only one pNIC for all communications

  • Switches are unaware

Unless your servers are going to have multiple clients downloading multiple media files at once you should definitely use Port based load balancing.

If you port based doesn't give enough bandwidth then seriously consider 10GBE before anything else, this will increase the size of the shared pipe and benefit all connections on that vSwitch.

If you must use IP hash then you must stack the two physical switches and configure the ports as a multilink in the Procurve config. You will only get a performance boost for multiple concurrent connections to a single VM from different clients.

The forums definitely rock, I'm a contract VCI filling in lab time in a FastTrack course. If you haven't been to teh official VMware courses you should consider it as we discuss these issues in the course.

Al.

Reply
0 Kudos
bjselman
Contributor
Contributor
Jump to solution

we've got about 12 clients down/uploading huge graphics files at once. in the future, we'll have many external clients downloading/uploading multiple large media files to a web server.

thanks al. that pretty much narrows it down.

if i want to cross-link the switches, then no trunking and no ip hashing.

if i want to trunk, then i can't cross-link the switches.

don't use mac hash.

that leaves cross-link & port id with current design. i have mac hash working with active lacp now. tomorrow i will run tests and check it against running with port id. i can't understand why i shouldn't mac-hash when my switch is mac-hashing and it seems to be working fine (besides the fact its not really aggregated).

if i want aggregation, i'll basically have to switch 2 cables and make a few more for interswitch trunks. vmotion performance will be a big factor in final design i suppose.

now i see the yellow brick road.

Reply
0 Kudos
kukacz
Enthusiast
Enthusiast
Jump to solution

BJ, I'm curious for your tests' result. I've done something similiar and was only able to confirm the trunking-iphash way to work (balancing in both directions).

Depending on your budget there might be another option for you: Make use of those 2900's two embedded CX-4 10GbE ports. You would have to purchase a 10GbE card or two for your ESX server. You can also use them to stack stack those switches together.

I've tested the 10GbE connection between ProCurve 2900 and a Chelsio S310 card in ESX together with 3 node LeftHand SAN/IQ storage cluster. This worked fine. You could avoid any balancing algorithms this way while reaching the increased throughput you request for the sequential traffic.

--

Lukas Kubin

Reply
0 Kudos
bjselman
Contributor
Contributor
Jump to solution

I'll post my findings.

Ah, The Budget. Let's not talk about the budget. Please.

Thanks again for everyone's insight!

Reply
0 Kudos