iSCSI multipath and round robin load balancing questions

[ryanv]

What do I need to do to get the esxcfg-mpath round robin commands to stick after a reboot?

Yes, I know it's "experimental" but my iscsi VMs utilize both NICs as long as I don't reboot so I'd like to figure out how prevent the multiplath settings from going back to defaults after a reboot.

While I'm at it what would would you recommend to be the best multipath policy settings for round robin and iSCSI VMs?

I read somewhere that in order for multipath settings to be persisant across reboots I have to configure multipath using the vml LUN name instead. Unfortunately, everything still goes back to defaults after a reboot. Here's the commands I'm using in case you can spot my problem:

esxcfg-mpath --lun=vml.020000000060003ff044ddd3159991c1025a8498e0566972747561 -p rr

esxcfg-mpath --lun=vml.020000000060003ff044ddd31591a7deb2dff75293566972747561 -p rr

I've also tried using the following policy values that were suggested in another thread:

esxcfg-mpath --lun=vml.020000000060003ff044ddd3159991c1025a8498e0566972747561 0 -H any -B 64 -C 64 -T any

esxcfg-mpath --lun=vml.020000000060003ff044ddd31591a7deb2dff75293566972747561 0 -H any -B 64 -C 64 -T any

The load isn't always evenly balanced using these settings so would you recommend different policy values instead?

In case it helps your answer, my setup is currently a SAN with 4 NICs teamed to a dedicated iscsi only switch, and 2 teamed NICs to our production network The quad port teamed NIC has 2 IPs assigned to the team, each of different subnets. I have two identical servers for ESX but only one is set up right now. It has two NICs teamed to the production network and the other dual port NIC is split between two vSwitches and set for different subnets to handle iscsi traffic.

Jumbo frames are enabled and working on the SAN, switch, and ESX server.

I was originally using teaming on iSCSI NICs on ESX but we weren't happy with only being able to use one path per server and I don't feel like paying for nor going through the hassle of setting up each vm using netboot/i. If I can get round robin load balancing working optimally and after rebooting, then I won't need to do anything special to the VMs.

So, any ideas? Any help is much appreciated.

[Yattong]

Hey,

Welcome to the forums.

Without more information on the setup of your environment, it difficult to know whether round robin will actually optimise your iscsi solution.

Have you read this article? http://virtualgeek.typepad.com/virtual_geek/2009/01/a-multivendor-post-to-help-our-mutual-iscsi-cust...

Its one of the best articles to come out of late on optimising vmware environments(on iscsi).

If you found this or any other answer useful please consider the use of the Helpful or correct buttons to award points

~y

[ryanv]

Thanks for the article. I'm not sure how it eluded my searches but it was very informative. So from what I gathered, even using load balancing in an active/active setup, you're only getting a single connection to the iSCSI target so I won't gain any speed out of it.

Because I hadn't noticed a certain bottleneck earlier, this claim went unnoticed in my original testing. In an active/active round robin setup and two VMs going writing from a local virtual disk to an iSCSI virtual disk, each transfer levelled off at about 50MB/s.

I reverted back to my original teamed setup with one vswitch and redid the test. Sure enough, only one NIC in the team was being used and each transfer was going at about 52MB/s writing to the iSCSI virtual disks. So slightly faster speeds but only using one NIC. Reading from the iscsi disk to the local virtual disk levelled off around 33MB/s per VM.

So.... until the next release comes out, I guess the only way to maximize my speeds would be to have the VMs boot from the SAN directly.

Are there any alternative's to Double-Take's Netboot/i? Also, does Netboot/i or any alternative still work normally with vmotion?

I guess I'm basically looking for the best way to have the VM's iSCSI initiator boot the OS directly from the SAN to be able to use multiple connections and increase my speeds, rather than having the single ESX initator do that and be stuck using only one connection to the SAN.

[Yattong]

beware though, if you start using the initiator inside a vm, you'll enter another realm of 'whats supported and whats not',

e.g. from recollection I dont think you will be able to use snapshots.

Why dont you add another iscsi target if you want additional throughput?

If you found this or any other answer useful please consider the use of the Helpful or correct buttons to award points

~y

[ryanv]

Is it even possible to add another target with Microsoft Unified Data Storage Server 2003 R2? I don't want to add another physical server into the equation.

I'm still trying to find out if it'll accept multiple connections per session if I use the MS initator but there's lack of documenation for the MS iSCSI target.

[Yattong]

I havent used mudss but if its like any other MS server, then you should be able to add additional nics, or even additional IP addresses to the same nic too.

If you found this or any other answer useful please consider the use of the Helpful or correct buttons to award points

~y

[ryanv]

I have a quad port NIC teamed and dedicated to iSCSI traffic on the SAN. The team has two IPs on two different subnets but the ESX initiator still seems to use one connection to the iSCSI target. ESX now sees double the LUNs and it groups the target LUNS together in an active/standby setup. That's why I was trying to use active/active load balancing to see if using both links would increase network speeds. Nope, that's not the case.

So what needs to be done for ESX to see two iSCSI targets and load balance the links to increase bandwidth? Or is that at all possible?

[Yattong]

How are you teaming the nics? Are you using link aggregation on the physical switch(es)?

As per article above, you need to 'not' use link aggregation. This will allow a distinct session per target per physical link.

If you found this or any other answer useful please consider the use of the Helpful or correct buttons to award points

~y

[ryanv]

I read somewhere earlier that teaming isn't supported for iSCSI. I guess what that means is it's not supported in the sense of increasing bandwidth for iSCSI load balancing purposed.

For some bizarre reason, I had to uninstall and reinstall the intel NIC software to edit the teaming but I broke the single teamed quad port NIC into two teams of two NICs each. Now that each team has a unique MAC address ESX is now uses both NICs for more multiple VMs.

I haven't set round robin up yet but as long as it's using both links. I don't really care that much because at this point I don't expect it to help much so I'll just leave it how it is. So on to the results...

Two active VMs copying from iSCSI virtual disk to local virtual disk: 48MB/s each

Two active VMs copying from local virtual disk to iSCSI virtual disk: 56MB/s each

Single VM copying from iSCSI virtual disk to local virtual disk: 75MB/s (to be fair the speed was still climbing before the 8GB transfer completed)

Single VM copying from local virtual disk to iSCSI virtual disk: 71MB/s

During all my tests I have been comparing the results to a physical windows server that originally had two teamed NICs going to the SAN and uses the MS initiator. The this last test, I broke the team in two and set the initiator to use multipath:

Copying from iSCSI volume to local RAID1 array using multipath initator: 119MS/s

Copying from local RAID1 array to iSCSI volume using multipath initator: 125MS/s

The VMs were originally far behind the physical windows box when it comes to transfer speeds even when I had the VMs using the MS intiator to connect iSCSI LUNs. The physical server originally did about 102MS/s reads on one NIC and the VMs were sometimes reached as high as 66MB/s. The physical box still has much faster network speeds than a single VM but with multiple VMs ESX seems to be closing the gap.

Keep in mind that all the speeds I mention are when the transfer speeds appear to level off. The burst rates are much higher. Sometimes it starts high and descends, sometimes it starts low and climbs. I haven't paid enough attention to notice the pattern but the speeds I listed are at the point where the transfer doesn't seem to be going any faster or slower.

Anyway, are these realistic numbers or should I be pushing for something more? Although at this point I'm not even sure what else is possible to increase the speeds.... I didn't really go into the details of my setup so here's some more info:

SAN:

Intel s5000sva board, 8GB of RAM, and supermicro 16bay chassis running Windows Unified Data Storage Server R2 Enterprise x64 (MS iSCSI target)

3ware 9650SE controller with 12 1TB 3.0Gbs drives in RAID 6in one big 9TB array holding my iSCSI virtual disks.

Onboard Intel RAID with 4 320GB 3.0Gb drives in RAID 10 for the OS

2port onboard Intel NIC teamed for regular network communication

4port Intel Pro 1000PT NIC split into two teams for iSCSI traffic only

The quad port NIC connects to a 3com switch that is being used exclusively to connect the SAN to ESX servers.

ESX Server #1 (haven't set up the others yet)

Lenovo x3650 with 8GB of RAM running ESX 3.5 fully updated.

2 300GB SAS drives in RAID 1 array.

2port onboard broadcom NIC teamed for regular network traffic

2port Intel Pro 1000PT NIC now split into two seperate vswitch/vmkernel groups for iSCSI traffic only

The switch is a Baseline 2916-SFP Plus.

All NICs/teams on the SAN network and the switch have 9000 byte jumbo frames enabled as does the vswitch and vmkernel. I believe flow control is enabled everywhere and speed settings are manually set on the switch ports and ESX ports going to and from ESX. I read somewhere that doing that avoids some potential issues but I forget exactly what those issues were.

Test Server:

HP Proliant ML150 G4 (i think) with 10GB of RAM running Server 2008 x64

2 striped 80GB SATA 1.5Gbs drives

2 1TB SATA 3.0Gbs drives in RAID1 array (used this array during testing)

2port Intel Pro 1000PT NIC teamed for iSCSI traffic only (same as the ESX server)

[ryanv]

I’d like to go back to my original question of how to get

the round robin commands to stick after a reboot.

Currently to get things somewhat balancing I need to havetwo targets and a LUN per VM and have the preferred paths split between my two

NICs to the SAN. Well, we’ve decided to

pursue getting round robin working so that we can split the traffic over two

paths but only use a single big iSCSI virtual disk for all VMs. If we run out of space, we’ll add another 2TB

LUN, but the first 2TB LUN will be plenty for now.

I have some more question though and the answers will determine

whether or not round robin is worth pursuing further.

If I have two targets and a single virtual disk, will round

robin use both links (separate subnets) together to theoretically double the

bandwidth or is it going to be switching packets between the paths and

ultimately only be as fast as one link?

If I stick with my current setup where the LUNs have

preferred paths balanced across the two targets, what happens when a link fails

and then recovers? Does the preferred

path get changed to the stable he reason I’m asking is I don’t want to have to

manually switch back the preferred path because that can cause traffic to

become unbalanced if I don’t notice a link had failed and recovered.

Finally, my very first question again… how do I get the

round robin commands to stick across reboots?