uninspired
Contributor
Contributor

iSCSI Hardware vs. Software Adapter

Jump to solution

Hey all-

I'm trying to figure out a vMotion issue (the issue being that it essentially stopped working). A lot of what I've read pins vMotion problems on storage, so I'm starting there. Anyway, while I've been poking through the storage a question occurred to me:

If you have pNICs that can handle iSCSI offloading, should the iSCSI Software Adapter be disabled? i.e., You should only need to use the Software Adapter if you don't have the offloading capability with your physical NICs, correct? Also, does anyone have a hunch as to whether or not there would be any disruption if I turn off the Software Adapter (as long as the LUNs are all showing up on the hardware HBA)?

Thanks

-Chris

0 Kudos
1 Solution

Accepted Solutions
dreamworkit
Contributor
Contributor

This is a good question. I had the same when configuring my storage. Ok think of it like this: If you have two ports on a 10Gb controller of a PS6010, you technically have 'two' paths from an ESXi Host, since each port carries an IP address. You can see which path is linked to what IP under the Manage Paths section, in the Target area (or below the paths section)  Now, say you have two members, and both carry the same specs, you now have four Active (I/O) paths. This is really regardless of the physical connection out of the Host. You could have one physical SFP+ NIC and it would register two paths since you have two target addresses, even though you have one 'physical' path. Kind of like a -< split if you want to think of it like that.

All of the below assumes you have a dedicated set of switches, with dedicated links to each member and switch, with failover best practices in place. It is recommended that you isolate your storage network from your normal network. If you are trying to share the EQL storage with normal network operations, you are adding undesired complexity.

  What you need to do is make sure that your iSCSI VMkernel denomations are linked up to the the right vmnics. Under your EQL iSCSI vSwitch properties make sure you have opted to do a switch override for the vmnic assignments (Put Host in maintenance mode so you can test everything without killing any VM connections):

  • Click on your vSwitch Properties, select iSCSI0 and hit 'Edit'

  • Under the NIC Teaming Tab, make sure the Failover Order is selected to 'Override vSwitch failover order

    • Active adapters: vmnic 0

    • Unused adapters: vmnic 1, vmnic 2

  • Follow these steps for the other two iSCSI designations, making sure you have only one per iSCSI port group.

Something I did notice was that you do not have an vMotion dedicated port group. This should be done just for consistency sake, but doesn't mean you have to burn a port just for vMotion. In my environment, the port assignment look like the attached screenshot (vSwitch assignments.png). Both of the vmnics are assigned to my vMotion port group.

Lastly, make sure MEM has bound all the vmnics associated with iSCSI traffic (above) to the iSCSI Software Adapter (vmhbaxx). You may just want to start over and delete the vSwitch, run the MEM install all over again, and then do the above steps. But I needed to show you what was happening before I just told you to nab the vmnics and target the vmhba for Software iSCSI initiator.

At the end of this, you will have a configuration like this:

  • NIC teaming override for one vmnic to each port group.
  • ALL vmnics that are used for iSCSI communication BOUND to the vmhba iSCSI Software Adapter (i.e. vmhba36) using the MEM setup 'config' wizard
  • All paths in your storage should be Active (I/O)

Does this make sense? Attached is a MEM cheat sheet as well. It's basic on purpose. I assume I'm calling on the sheet when I need to configure it in a hurry. You can see Dell's is attached too.

Hope that helps.

View solution in original post

0 Kudos
15 Replies
DSTAVERT
Immortal
Immortal

I would make sure that your iSCSI hardware is on the HCL. http://vmware.com/go/hcl There are/have been issues with some hardware iSCSI adapters.

-- David -- VMware Communities Moderator
0 Kudos
uninspired
Contributor
Contributor

Yep, it's supported. It's a pretty standard card (Broadcom NetXtreme II BCM5708).

0 Kudos
dreamworkit
Contributor
Contributor

Hardware offloading for iSCSI doesn't work very well. Using a HBA gets a small performance difference, if any, and should only be used for FC communication. In my experience, you should always use the iSCSI Software Adapter integrated with vSphere for any operations that involve iSCSI storage, because you need to map the 'vmnic' assignments to the Software iSCSI 'vmhba'. Physical NIC models don't really matter, short of the speed and making sure they are HCL certified by VMware. As for a vMotion issue, you need only to create the VMkernel Port that will handle the traffic, verify that the VMkernel IP is on the same subnet as the destination Host's, and you should be able to select 'vMotion' under the port properties to get that going.

If it fails, it is important to know at which step (%). I learned this one through a few failures myself. Jumbo Frames is probably the #1 cause of this. If you 'think' you enabled the Jumbo Frames option of 9024 (normal, though there are higher settings) on a switch, verify that you enabled it across all ports on the switch, not just the startup configuration option. Made that mistake myself. vMotion between two Hosts with Jumbo Frames enabled would fail, while vMotion between the default MTU of 1500 and MTU of 9024 would work because vSphere would automatically choose the lowest value. Also, Jumbo Frames should only be used for 10Gb communications, as 1Gb can easily get flooded with large packets.

Based on your post, I assume you are using FC HBAs. Turning off iSCSI should have no effect, but really isn't necessary because you need to setup the discovery options anyway. It's just a service effectively without the configuration.

Hope that helps.

-Jordan Shaw

www.dreamworkit.com

0 Kudos
DSTAVERT
Immortal
Immortal

http://kb.vmware.com/kb/1025644

-- David -- VMware Communities Moderator
0 Kudos
uninspired
Contributor
Contributor

Thanks for the post, but I don't think it's applicable.

I guess I didn't articulate my question well enough. So, here's my concern: All of my iSCSI LUNs show up under BOTH the hardware iSCSI adapter (BCM5708) HBAs *AND* the Software Adapter HBA. If they're all connected to the Hardware HBAs, can/should I disable the Software Adapter altogether?

Thanks for your help.

0 Kudos
dreamworkit
Contributor
Contributor

4.0 or 4.1? 4.1 differentiates between your hardware connections differently. Did you map using a discovery method? Are these NICs seperated and dedicated for storage while others are dedicated for LAN traffic? Important as it dictates whether you'll kill your Datastore connections. Sounds like they are interconnected somehow. See attached how mine appears in ESXi 4.1

uninspired
Contributor
Contributor

The hosts and vCenter have all been brought up to v4.1 several months ago. These NICs (one per each of 3 hosts within the cluster) were previously not utilized. They were present, but unused. When I was first put in charge of this cluster, the storage was all SAS and local. Each of the 3 hosts had a Broadcom NetXtreme II BCM5708 that was not being used for anything. So, this NIC (2x ports) is entirely dedicated to just this iSCSI storage.

I was charged with moving all of our VMs to our Dell Equallogic iSCSI storage system. I created 5x 2TB iSCSI LUNs to be shared within the cluster. After creating the volumes and adding them to the cluster (on the Broadcom HBAs), everything worked perfectly for around 6 months. I used the recently released Dell MEM and everything was kosher for a while. I moved all of the VMs from the SAS and local storage to the iSCSI, and  performance improved dramatically and vMotion worked 100% of the time.  Recently, however, I've noticed some errors and this most recent problem with vMotion. It works a VERY small percentage of the time, now (trying to figure out when/why it works), and I think it depends on the host. One of my three hosts looks the way I think it should: All of the iSCSI storage is attached to the Broadcom HBAs and the software HBA is only connected to the old SAS storage.

Attached images are the host i *THINK* is configured correctly. ("ESXi3" in a 3-host cluster). I'll post the contrasting hosts in a moment. NOTICE: The iSCSI LUNs on hba35/36 are called "EQLOGIC iSCSI" while the ones on hba38 are "DELL iSCSI" --- totally different arrays.

EDIT: THESE ARE 2TB LUNs, as VMWare can't handle 5TB LUNs

0 Kudos
dreamworkit
Contributor
Contributor

Ok that helps clear it a little. Thanks. Did you run the vCLI Pearl script for the MEM plugin? Or did you only install it via the Update Manager?

**Looking over your screenshots now...

0 Kudos
uninspired
Contributor
Contributor

I used the vCLI script, which worked like a dream initially. Everything installed and configured properly, and everything worked great for several months.

It's only now that things started to act funky. I attached a screenshot for host number 2 in the 3-host cluster. On this host, the two hardware HBAs are idenitcally configured and list all of the Equallogic LUNs. However, this Software Adapter lists the Equallogic LUNs, while I *think* it should only list the old (non-Equallogic) storage like host #3 (screenshots in previous post).

0 Kudos
uninspired
Contributor
Contributor

Host #1 has the most unusual configuration. I have to reiterate that for several months, all three hosts were configured identically. 2 hardware HBAs on each of the three pointed to the same Equallogic iSCSI LUNs, and NONE of the three Software Adapters were attaching to the Equallogic LUNs.

Now, host #1 looks extremely peculiar:

  1. Now only one of the two hardware HBAs on host #1 is connected to any of the Equallogic LUNs. The other shows zero connections.
  2. Like host #2, however, the Software Adapter shows all of the old LUNs PLUS the Equallogic LUNs that should only be attached to the Broadcom HBAs
0 Kudos
dreamworkit
Contributor
Contributor

Ok, I have a good idea what is going on. Technically, everything is using iSCSI and what you are referring to as HBAs (5708) are not really HBAs. Their NICs, through and through. That's why you are seeing both LUN and Volumes groups under that iSCSI adapter. What you should be doing is utilizing the NICs in a configuration like this:

2 Broadcom 5708/5709> iSCSI Software to both Dell EQL and misc. storage / vMotion

2 Broadcom 5708/5709> LAN traffic / Management Network

<stop me anywhere along here if I'm assuming something that isn't right>

What you need to do is configure everything to use the iSCSI network for failover and point the vmnic assignments to use the iSCSI Software Adapter via MEM wizard (which is awesome I agree). You're going to need to go back to the drawing board and reconfigure your Hosts network setup basically. Host 1 looks like it has a failure point in your Storage Network which could render you up the creek. If you don't need the old SAN group, I'd start by just configuring your systems to connect and use the EQL SANs first. I'm going to type something up here and attach it (in the next day or so) that may help you navigate this, because I unfortunately can't see the other end of your environment (physical).

Talk with you soon.

0 Kudos
uninspired
Contributor
Contributor

dreamworkit - Thanks for the help. I ended up having multiple problems. The biggest problem was that some of my ports were still configured as an etherchannel on the switch, so none of them were working. Now that that's cleared up, however, I'm still left with my original question: What's the best way to configure multiple pNICs/vmhbas connecting via iSCSI to Dell Equallogic storage?

Now that things are working, I can focus on how to make them work optimally. I've attached a screenshot of what my iSCSI vswitch looks like. The original question was, should I attach the three vmhbas (three ports across two Broadcom 5708s) to the EQL, or should I attach the single iSCSI software adapter?

I've done some cursory read/write tests, and - in the speed department - I was definitely getting better speeds using the single software adapter vs. the three separate broadcom vhbas. However, when I check the paths to the array, there are 7 or 8 when I'm connecting over the three broadcom vmhbas, whereas there are only two paths when utilizing the single iSCSI software adapater for connection. Even more concerning is that with either configuration, the storage view shows "Partial/No Redundancy" under Multipathing Status.

Does anyone have any documentation related to ESXi v4.1 iSCSI/Dell Equallogic/Dell MEM (MPIO) configuration?

0 Kudos
dreamworkit
Contributor
Contributor

This is a good question. I had the same when configuring my storage. Ok think of it like this: If you have two ports on a 10Gb controller of a PS6010, you technically have 'two' paths from an ESXi Host, since each port carries an IP address. You can see which path is linked to what IP under the Manage Paths section, in the Target area (or below the paths section)  Now, say you have two members, and both carry the same specs, you now have four Active (I/O) paths. This is really regardless of the physical connection out of the Host. You could have one physical SFP+ NIC and it would register two paths since you have two target addresses, even though you have one 'physical' path. Kind of like a -< split if you want to think of it like that.

All of the below assumes you have a dedicated set of switches, with dedicated links to each member and switch, with failover best practices in place. It is recommended that you isolate your storage network from your normal network. If you are trying to share the EQL storage with normal network operations, you are adding undesired complexity.

  What you need to do is make sure that your iSCSI VMkernel denomations are linked up to the the right vmnics. Under your EQL iSCSI vSwitch properties make sure you have opted to do a switch override for the vmnic assignments (Put Host in maintenance mode so you can test everything without killing any VM connections):

  • Click on your vSwitch Properties, select iSCSI0 and hit 'Edit'

  • Under the NIC Teaming Tab, make sure the Failover Order is selected to 'Override vSwitch failover order

    • Active adapters: vmnic 0

    • Unused adapters: vmnic 1, vmnic 2

  • Follow these steps for the other two iSCSI designations, making sure you have only one per iSCSI port group.

Something I did notice was that you do not have an vMotion dedicated port group. This should be done just for consistency sake, but doesn't mean you have to burn a port just for vMotion. In my environment, the port assignment look like the attached screenshot (vSwitch assignments.png). Both of the vmnics are assigned to my vMotion port group.

Lastly, make sure MEM has bound all the vmnics associated with iSCSI traffic (above) to the iSCSI Software Adapter (vmhbaxx). You may just want to start over and delete the vSwitch, run the MEM install all over again, and then do the above steps. But I needed to show you what was happening before I just told you to nab the vmnics and target the vmhba for Software iSCSI initiator.

At the end of this, you will have a configuration like this:

  • NIC teaming override for one vmnic to each port group.
  • ALL vmnics that are used for iSCSI communication BOUND to the vmhba iSCSI Software Adapter (i.e. vmhba36) using the MEM setup 'config' wizard
  • All paths in your storage should be Active (I/O)

Does this make sense? Attached is a MEM cheat sheet as well. It's basic on purpose. I assume I'm calling on the sheet when I need to configure it in a hurry. You can see Dell's is attached too.

Hope that helps.

0 Kudos
uninspired
Contributor
Contributor

Thanks for the info -- This is dead-on! I had just found the same PDF  about an hour ago so your timing is impeccable (Original source for  anyone interested:  http://www.equallogic.com/resourcecenter/assetview.aspx?id=8453).

Thanks for all of your help -- I appreciate it!

0 Kudos
dreamworkit
Contributor
Contributor

No problem. Took me a while to type that up, making sure if was relevant to what you were going through. I've had that PDF for a while but it isn't exactly 100% correct (hence my supplemental). Their original Word document is really messy and even encourages the installation using Update Manager. DO NOT do it that way. Bad things happen to good people. It doesn't bind the vmhbas to the iSCSI Software Adapter but only installs the MEM plug in. That PDF is significantly an easier read if you are so inclined to do so.

Something that may not be clear in my cheat sheet is where VAAI is located. That's under Advanced Settings under each Host's 'Configuration' tab. If you don't have MEM setup, they recommend that setting be off. This shows you how to turn it on basically (by default it is on). Basically, it's the hardware acceleration piece that 4.1 is known for, offloading tasks to your SAN. Be ready for a performance bump if you don't have enough SAN IOPS.

In answer to your request to other information, hit http://www.virtualizationbuster.com/ for great info on EqualLogic and vSphere subjects, which includes how MPIO works. It's a go to source for me.

Have fun!

0 Kudos