iSCSI MPIO (Multipath) with Nexus 1000v

[joshio]

Anyone out there have iSCSI MPIO successfully working with Nexus 1000v? I have followed the Cisco Guide to the best of my understanding and I have tried a number of other configurations without success - vSphere still shows the same number of Paths as it shows Targets.

The Cisco document states the following:

Before starting the procedures in this section you must know or do the following.

•You have already configured the host with one port channel that includes two or more physical NICs.

•You have already created VMware kernel NICs to access the SAN external storage.

•A Vmware Kernel NIC can only be pinned or assigned to one physical NIC.

•A physical NIC can have multiple VMware Kernel NICs pinned or assigned to it.

What does " A Vmware Kernel NIC can only be pinned or assigned to one physical NIC" mean in regard to Nexus 1000v? I know how to pin a physical NIC with standard vDS, but how does that work with 1000v? The only thing related to "pinning" I could find inside of 1000v was with port channel sub-groups. I tried creating a port channel with manual sub-groups, assigning sub-group-id values to each uplink, then assigning a pinning id to my two VMkernel port profiles (and directly to the vEthernet ports as well). But, that didn't seem to work for me

I can ping both of the iSCSI VMkernel ports from the upstream switch and from inside the VSM, so I know Layer 3 connectivity is there. One odd thing, however, is that I only see one of the two VMkernel MAC addresses bound on the upstream switch. Both addresses show bound from inside the VSM.

What am I missing here?

[lwatta]

+What does " A Vmware Kernel NIC can only be pinned or assigned to one

physical NIC" mean in regard to Nexus 1000v?+

It means that when you assign a vmk nic to a port-profile it will get pinned to one of the uplink ports out of the ESX host. We will not load balance traffic from that VMK nic across all the available uplinks.

+I know how to pin a physical NIC with standard vDS, but how does that

work with 1000v?+

Your uplink (type eth) port-profile will need a channel group directive. We highly recommend using our vPC-Mac Pinning based port channel for simple configurations when your upstream switches don't support lacp or you don't have access to your upstream switches. We are trying to get away from manual sub group and cdp based sub group created port-channels.

To create vPC MAC pinning based port channel add the below command to your uplink (type eth) port-profile

channel-group auto mode on mac-pinning

What this will do is everytime you add physical nic from the server to the uplink port-profile is assign that physical nic a new sub group id. So each nic becomes it's own sub group. When veth ports are assigned we round robin assign them to sub groups that make up the uplink port-profile.

Now on to iSCSI pinning. You need to create a port profile just for your iSCSI connections. This port-profile type should type veth. Make sure to have "system vlan" defined and "capability iscsi-multipath". Assign your VMK interfaces to this port-profile. What the "capability iscsi-multipath" command does it is attempts to keep the VMK interfaces on different uplink ports. Since the vmk interfaces could get added at different times it is possible that they could get assigned to the same uplink port. This command tells the VEM that they must be on different uplink ports and to keep them seperate.

You can run "vemcmd show iscsi" on the esx host to see the vmk pinning after you add them to the iscsi port-profile you create.

Give the above a try and let us know how you make out.

[joshio]

Thank you for those clear instructions. I have made the changes that you recommended, and "vemcmd show iscsi" shows each vmk interface pinned to a separate uplink.

However, In VMware, when I Rescan the HBAs, I still see the same number of paths as there are targets:

I have set a couple of the Devices to Round Robin, but a Rescan still does nothing; when doing this with vSS, this worked as expected and caused the additional paths to show up:

Here are my two port profiles from 1000v:

And here are the uplinks from VMware:

And the vmk interfaces from VMware:

I can still ping both vmk interfaces from the connected switch, but still only one shows in the MAC Address table for some reason. Not sure if that is causing my problem, however.

I am clearly missing another link somewhere. Thanks in advance for your help.

[lwatta]

Are you using one IP address on your iSCSI array to host both paths? If so thats the issue. We assume that you will have an IP address for iSCSI A fabric and another IP address to access the iSCSI B fabric.

Also make sure to try a rediscovery.

louis

[joshio]

Yes, there is only one IP Address on the EqualLogic storage group. Is this a limitation of Nexus 1000v? With a VMware Standard Switch, VMware was able to see multiple paths to the targets even though there was only a single target IP Address.

[lwatta]

Yeah I'm pretty sure its a problem on our side in this case. Let me check with engineering to be sure I beleive there is a workaround.

[joshio]

Thank you for your help!

[lwatta · Accepted solution]

Just to close the loop in case someone stumbles on this thread.

This is indeed a bug on the Cisco Nexus 1000v. The bug is only relevent to ESX hosts that have been patched to 4.0u2(and now 4.1). Short term work around is to back rev to 4.0u1. Medium term the fix will be rolled up into a maintenance release for the Nexus 1000V.

Our code implementation for getting the iSCSI multipath info was wrong but allowed in 4.0U1. 4.0U2 no longer allowed our wrong implementation.

So iSCSI multipathing and N1KV stay at 4.0U1 until we have a maintenance release for the Nexus 1000V

[crlindahl]

we are running into the same problem and have upgraded N1kV to the latest code which was released just a few days ago and see multipathing working now. But with the storage only on 1 subnet, we have noticed that only 1 physical NIC between VMWare ESXi box and physical network is really utilized for storage. If you fail the one NIC, traffic moves to the other one, but with these 2 NICs in a LACP configuration I would assume that they would load share if they were both operational?

[joshio]

As a followup to this issue, after upgrading to ESX 4.1, then upgrading to Nexus 1000v 4.0(4)SV1(3a), everything worked as expected. However, we did a fresh install of ESX 4.1 to one of our hosts today, and after installing Nexus 1000v 4.0(4)SV1(3a), vCenter only shows a single path again. Pinnings look fine, and it's using the same profiles that are working on other hosts.

root@lgv10 ~# vemcmd show iscsi pinning

LTL Device IfIndex Pinned_Uplink

48 vmk0 1b130000 25

49 vmk1 1b130010 20

I can ping both of the iSCSI vmk interfaces. Everything else I've checked looks the same between the 4.1 upgraded hosts and the 4.1 fresh install host. Is there anything specific I should look at?

About the only thing I can think to do is reload the host with ESX 4.0 and try the 4.1 Upgrade, since I know it works that way.