VMware Cloud Community
moneta82
Contributor
Contributor

Configuring iSCSI dependent hardware

Hi to everybody I've a small question...

My system is now configured with an iSCSI software adapter in this way:

- Virtual switch called iSCSI with a VMKernel port for the iSCSI with two NICs in teaming.
- iSCSI software adapter

Now I want to use my two new iSCSI broadcom dependent hardware adpater.

Following the iSCSI configuration guide official by VMware I must to configure the system in this way but I have
perplexity about it.


- two VMkernel Ports in same subnet (one for each iSCSI adapter).
- each VMkernel port linked to it phy_NIC
- both VMkernel ports bound to the relative iSCSI hardware adapter.


Is that right?
Can I have two VMKernel port in the same subnet?

Thanks in advance for your advice!

Reply
0 Kudos
14 Replies
EXO3AW
Enthusiast
Enthusiast

Hi moneta82,

http://www.vmware.com/pdf/vsphere4/r41/vsp_41_iscsi_san_cfg.pdf explains the necessary steps in a good way, but basically you're right with your questions:

Two VMkernel Ports in the same subnet ?

Sure, why not ? As long as you want to provide redundant storage paths that's the way to go.

For a deeper understanding read the aforementioned document, it provides tons of useful info.

Especially page 32ff. "Setting Up and Configuring Dependent Hardware iSCSI Adapters" and 37ff "Bind iSCSI Ports to iSCSI Adapters" are necessary to get the most out if it.

Reply
0 Kudos
idle-jam
Immortal
Immortal

have a look at this: http://www.yellow-bricks.com/2009/03/18/iscsi-multipathing-with-esxcliexploring-the-next-version-of-... it might be useful for your understanding as well as multipath setup.

Reply
0 Kudos
moneta82
Contributor
Contributor

OK... Thanks for your reply... So next week I try to make this changes. I've only this doubt yet.

I have to create the two VMKernel ports linked with a nic and then I have to bind the vmk1 and vmk2 with the iSCSI adapter.

My question is: Is there a correlation between the iSCSI adapter and the physical nic?

thanks a lot for the answer

Reply
0 Kudos
anjaneshbabu
Contributor
Contributor

having spent quite some time figuring this out, I believe the following might help.

We use broadcom 5709 adapters with TOE and ISCSI offload, but the fine print did not mention that these are dependent adpaters which means they are driven by the VMware iscsi s/w adapter . and forget the option to set targets in the bios of the HBA - it does not work with ESXi.

Despite this workaroundw its  still means that iscsi encapsulations occurs at hardware layer saving precious cpu cycles available to vms with the trade off that performance metrics are not available since the isci processing takes place beyond the vmkernel stack.

This method will work best if the dependent HBA is visible in the list of storage adapters and requires diving into command line via local or SSH console.

The overview of the process

1. Enable the built in s/w iscsi adapter and make a note of the adapter name (eg vmhba32)

2. Create a new vSwitch with the dependent HBA nics as uplinks

3. Create one vm kernel port for each uplink ( this is important else the process will fail).

4. Associate each vm kernel port with a single uplink moving the rest to unused (not standby).This is done via the vSwitch propeties(vSwitch properties-->Edit-->Nic Teaming tab and failover order)

5. Open the console and check out if any physical links are currently associated with the s/w iscsi adapter (should be none)

     esxcli swiscsi vmnic list -d vmhba32

6. Run thru the list of physical nics on the host
    esxcfg-vmknic -l
7. Associate the target network inteface (dependant HBA uplinks) with the s/w iscsi adaptor
    esxcli swiscsi nic add -n vmk0 -d vmhba32

8. Repeat for each HBA that you want to associate  (in our case it was 4)

9. Double check association

esxcli swiscsi vmnic list -d vmhba32

10. And then open up s/w iscsi adaptor and enter the ip configuration of the target server.Rescan to detect luns and thats it.

My reference was

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=102564...

However, the process does not work as decribed in the kb without enabling the sw adapter first. And setting targets in the HBA bios did not work.

Hope the attached figure helps


http://www.itgeeks.info/
moneta82
Contributor
Contributor

thanks for your answer but following this document: http://www.vmware.com/pdf/vsphere4/r41/vsp_41_iscsi_san_cfg.pdf at page 36 fig. 2-1.

I have to bind the vmk port directly to my iSCSI hardware adapter and not to the iSCSI software adapter.... Am I wrong?

thanks

Reply
0 Kudos
anjaneshbabu
Contributor
Contributor

"I have to bind the vmk port directly to my iSCSI hardware adapter and not to the iSCSI software adapter.... Am I wrong?"

- you are right in the interpretation of the guide .

However,binding the underlying nic to the iscsi hardware adpater  did not work for me for example

esxcli swiscsi nic add -n vmk1 -d vmhba34

-->  always threw up an 'invalid adapter' error .  I think this works with iscsi independent HBAs not dependent HBAs. it certainly did not work with Broadcom 5709 adapters we have. 

After a bit of fuddling about, i settled on the process is my last post to get it working.

From what I could gather  (following on from KB 1025644) when i associated the underlying nic with the s/w adapter, the associated dependent iscsi hardware becomes linked with the storage stack.

What model is your iscsi HBA ? and does it appear in the storage devices list of host ?

http://www.itgeeks.info/
Reply
0 Kudos
moneta82
Contributor
Contributor

Attached the iSCSI adapter see by vClient. I've installed HP NC382m that is drive by broadcom 5709S.

My guess was to bind the first nic (vmk port) with vmhba32 and the second with vmhba33 and then select for each LUN the round robin path selection.

Reply
0 Kudos
anjaneshbabu
Contributor
Contributor

"My guess was to bind the first nic (vmk port) with vmhba32 and the second with vmhba33 and then select for each LUN the round robin path selection."

yep - that is correct at least in theor.

Give it a go and would be interested on how it turns out.

In our case, for reasons unknown, binding vmk port with the broadcom vmhba resulted in an error 'invalid adpater' . But  this was probably an exception .

try your method first and do update with results

http://www.itgeeks.info/
Reply
0 Kudos
moneta82
Contributor
Contributor

I'm running with 4.1... You?

Reply
0 Kudos
anjaneshbabu
Contributor
Contributor

Running 4.1 update 1

http://www.itgeeks.info/
Reply
0 Kudos
opbz
Hot Shot
Hot Shot

actually the subnet bit depends on your storage.

if you have EMC you need to use a different subnet for every SP port that is configured for iscsi.

while with equallogics you can have multiple vmkernels on the same subnet.

main recomendation would be to configure as per the vendors best practices. Otherwise you will have disconects or bad performance.

Reply
0 Kudos
EXO3AW
Enthusiast
Enthusiast

I've seen the 'invalid adapter' error some times as well with extremely strange behaviors ...

Sometimes a simple host reboot made the trick happen, other times i had to delete the vmk ports completely, reboot and recreate them from the CLI - GUI didn't work. Quite strange.

Actually the KB shows 2 configurations with iSCSI:

1. Let the BCM handle everything (no SW iSCSI)

2. Let SW iSCSI handle everything except Offloading.

Good Luck in sorting everything out Smiley Wink

Reply
0 Kudos
moneta82
Contributor
Contributor

As soon as possible I will update the thread.... Thanks to everybody!!

Reply
0 Kudos
Tomek24VMWARE
Contributor
Contributor

with which the matrix works Broadcom - offloading? in your case, or with what you've done the test if it works?

Reply
0 Kudos