vmn00by
Contributor
Contributor

Establish multiple iSCSI paths to EqualLogic PS4000

Jump to solution

I have two Dell PowerEdge M610 in a PowerEdge M1000e that was given to my team as seed units (thanks Dell) and we ended up buying the storage they sent with it, an EqualLogic PS4000. The Dell rep came in and helped get the basic setup running and how iSCSI worked and showed us a doc (the same Configuring_VMware_vSphere_Software_iSCSI_with_Dell_EqualLogic_PS_Series_Storage.pdf doc noted in many threads here) on how to get ESX(i) v4.1 running with that storage. We got everything up and running for the most part before he left, but after going over the documentation again, I saw that we should be able to establish mutable connections and "Round Robin" to improve performance. As I see it so far in both vSphere Client and in the management console of the EqualLogic console, I'm only getting one per server (as seen in the attached screen shots). I have been beating my head over this for the past few days trying to get the connections going. I actually saw it work once but for other reasons, I had to re-do the raid on the host and reload and haven't been able to get it working again... :`-(

So, here is how the hardware is setup. From the PS4000 control module 0, I have eth0 going to 1 Dell PowerConnect M6220 in the chassis in slot C2, eth1 going to another M6220 port in C2. So the storage is hooked directly to the chassis. And the same setup on the control modal 1. Those ports hook up to vmnic4 & 5.

Switch Name Num Ports Used Ports Configured Ports MTU Uplinks

vSwitch0 128 4 128 1500 vmnic0,vmnic1

PortGroup Name VLAN ID Used Ports Uplinks

VM Network 0 0 vmnic0,vmnic1

Management Network 0 1 vmnic0,vmnic1

Switch Name Num Ports Used Ports Configured Ports MTU Uplinks

vSwitch2 128 6 128 9000 vmnic4,vmnic5

PortGroup Name VLAN ID Used Ports Uplinks

iSCSIMgnt 0 1 vmnic5,vmnic4

iSCSI2 0 1 vmnic5

iSCSI1 0 1 vmnic4

~ # esxcfg-vmknic -l

Interface Port Group/DVPort IP Family IP Address Netmask Broadcast MAC Address MTU TSO MSS Enabled Type

vmk0 Management Network IPv4 10.131.172.74 255.255.255.0 10.131.172.255 00:26:b9:31:d6:a9 1500 65535 true STATIC

vmk1 iSCSI1 IPv4 192.168.1.21 255.255.255.0 192.168.1.255 00:50:56:71:e7:f5 9000 65535 true STATIC

vmk2 iSCSI2 IPv4 192.168.1.22 255.255.255.0 192.168.1.255 00:50:56:75:92:76 9000 65535 true STATIC

~ # esxcfg-nics -l

Name PCI Driver Link Speed Duplex MAC Address MTU Description

vmnic0 0000:01:00.00 bnx2 Up 1000Mbps Full 00:26:b9:31:d6:a9 1500 Broadcom Corporation Broadcom NetXtreme II BCM5709 1000Base-SX

vmnic1 0000:01:00.01 bnx2 Up 1000Mbps Full 00:26:b9:31:d6:ab 1500 Broadcom Corporation Broadcom NetXtreme II BCM5709 1000Base-SX

vmnic2 0000:03:00.00 bnx2 Up 1000Mbps Full 00:26:b9:31:d6:ad 1500 Broadcom Corporation Broadcom NetXtreme II BCM5709 1000Base-SX

vmnic3 0000:03:00.01 bnx2 Up 1000Mbps Full 00:26:b9:31:d6:af 1500 Broadcom Corporation Broadcom NetXtreme II BCM5709 1000Base-SX

vmnic4 0000:05:00.00 bnx2 Up 1000Mbps Full 00:26:b9:31:d6:b1 9000 Broadcom Corporation Broadcom NetXtreme II BCM5709 1000Base-SX

vmnic5 0000:05:00.01 bnx2 Up 1000Mbps Full 00:26:b9:31:d6:b3 9000 Broadcom Corporation Broadcom NetXtreme II BCM5709 1000Base-SX

...OK, past all of that, here is another finding after tinkering with the network settings for vSwitch2. If I change:

iSCSI2 0 1 vmnic5

iSCSI1 0 1 vmnic4

...to...

iSCSI2 0 1 vmnic4

iSCSI1 0 1 vmnic5

The connection to the storage is lost...for a time. About 5min later I see the connection established again on the EqualLogic, but on the other port (eth0 to eth1). So I know the connection CAN work, but just isn't doing both at once.

So what am I doing wrong? I have tried to read up on the other threads, but haven't read an answer to this issue. Thanks for your help.

0 Kudos
1 Solution

Accepted Solutions
VMmatty
Virtuoso
Virtuoso

There is a step in that configuration guide where you bind all of your VMkernel ports to the iSCSI initiator. Do you remember doing that step?

What does the output of the following command look like: esxcli swiscsi nic list -d vmhba39

You should see both vmk1 and vmk2 in there.

Matt

My blog:

Matt | http://www.thelowercasew.com | @mattliebowitz

View solution in original post

0 Kudos
16 Replies
VMmatty
Virtuoso
Virtuoso

There is a step in that configuration guide where you bind all of your VMkernel ports to the iSCSI initiator. Do you remember doing that step?

What does the output of the following command look like: esxcli swiscsi nic list -d vmhba39

You should see both vmk1 and vmk2 in there.

Matt

My blog:

Matt | http://www.thelowercasew.com | @mattliebowitz

View solution in original post

0 Kudos
vmn00by
Contributor
Contributor

That was it. I forgot the bind step. So yes I can now establish multiple connections to the storage. Thanks!

Sort of another question off that. Now I have the multiple connections going, I kinda had to fight it to use both adapters at once. I all the odd iSCSI# would get the vmnic5, and even iSCSI# vmnic4. I would see only the odd iSCSI IP addresses connect, and not even. So I made all the iSCSI adapters vmnic4. ESX would freek out, then a min or so later all those connections would come back. I put the even/odd scheme back and still only even connections then. After going back and forth a few times I got both NICs working at once.

So the question is, am I just making something out of nothing and the Round Robin will kick in when needed and use available connection if the other if full?

0 Kudos
VMmatty
Virtuoso
Virtuoso

I'm glad that worked for you.

I'm having some trouble understanding exactly what you're seeing. I don't really follow what you mean when you say you "all odd iSCSI# would get vmnic5"? How are you assigning vmnic5 specifically?

In my experience when setting this up there isn't anything you have to do other than the steps listed in that document. You configure the networking and then set the VMFS datastore to use Round Robin as the path selection policy. You should see two active paths listed as "Active (I/O)" and subsequently you should see the two connections in the EqualLogic web interface. Doing any operation should utilize both NICs but a good test would be to clone a VM on that datastore. You should see traffic spread somewhat evenly both on the EqualLogic page as well as in the vCenter performance graphs for your network adapters.

If I'm not understanding you correctly can you provide some screenshots?

Matt

My blog:

Matt | http://www.thelowercasew.com | @mattliebowitz
0 Kudos
vmn00by
Contributor
Contributor

Sorry on the even odd thing. iSCSI adapter to NIC assignments:

iSCSI1 - vmnic5 - 192.168.1.11

iSCSI2 - vmnic4 - 192.168.1.12

iSCSI3 - vmnic5 - 192.168.1.13

iSCSI4 - vmnic4 - 192.168.1.14

iSCSI5 - vmnic5 - 192.168.1.16

iSCSI6 - vmnic4 - 192.168.1.17

But on EqualLogic the only connections I see is for IPs .11, 13, and 15. So I change all the adapters to just 4...

iSCSI1 - vmnic4 - 192.168.1.11

iSCSI2 - vmnic4 - 192.168.1.12

iSCSI3 - vmnic4 - 192.168.1.13

iSCSI4 - vmnic4 - 192.168.1.14

iSCSI5 - vmnic4 - 192.168.1.15

iSCSI6 - vmnic4 - 192.168.1.16

I then lose all connection to the storage for a min. Then it magically connects with all addresses. I put it back to the even/odd setup as before and all connection for IPs 11, 13 and 16 are lost again. Doesn't seem like it likes to use both ethernet ports at the same time. I have to play and fight with it to get it to work with both ports. I'm hoping when things get too loaded up that it will pick up on the other connections.

0 Kudos
VMmatty
Virtuoso
Virtuoso

It looks like you're using more than one VMkernel port group per physical NIC. Unfortunately that iSCSI setup guide that you're using is incorrect in the recommendation to use anything more than a 1:1 mapping of VMkernel ports per physical NIC. I've spoken to both VMware and Dell about that issue several times and each time I've been told to disregard that and just use one VMkernel port per physical NIC.

I would make that change and see if that helps. Just use the following (for example - use whichever IPs and NICs make sense):

VMkernel port #1:

iSCSI1 - vmnic5 active, vmnic4 unused, 192.168.1.11

VMkernel port #2:

iSCSI2 - vmnic4 active, vmnic5 unused, 192.168.1.12

You can then delete all of the other VMkernel ports that you created. You'll need to go into the initiator setup and remove the VMkernel ports after you remove them from the GUI. It isn't going to let you remove them from the initiator without rebooting, but the command is: esxcli swiscsi nic remove -n vmk3 -d vmhba39 (replace vmk3 with the actual vmk of your VMkernel ports).

Does this make sense? I would try this before doing anything else.

Matt

My blog:

Matt | http://www.thelowercasew.com | @mattliebowitz
0 Kudos
vmn00by
Contributor
Contributor

I ran that command (failed so I had to set the volume offline for it to work) to removed the extra vmk3-6 and deleted the iSCSI3 - 6 connections from vmhba36. That worked, but I still only get one connection to the storage:

Switch Name Num Ports Used Ports Configured Ports MTU Uplinks

vSwitch2 128 6 128 9000 vmnic4,vmnic5

PortGroup Name VLAN ID Used Ports Uplinks

iSCSIMgnt 0 1 vmnic5,vmnic4

iSCSI2 0 1 vmnic5

iSCSI1 0 1 vmnic4

Storage screen shot attached. And I only did it to u060dsehst4 for now...

And thanks again for your help.

0 Kudos
VMmatty
Virtuoso
Virtuoso

It looks like (at least from here) that connections on vmnic5 aren't making it to the SAN but vmnic4 are. That seems to be like a problem on the switch. Maybe vmnic5 isn't on the correct VLAN or some other switching problem?

Have you ever been able to make any connection work from vmnic5? From the screenshots you attached here and your previous description it seems like vmnic5 is the problem here. I don't fully understand why the EQL connections tab is showing mutiple connections from the same IP address but maybe those are legacy connections that just haven't been cleared.

Matt

My blog:

Matt | http://www.thelowercasew.com | @mattliebowitz
0 Kudos
Yps
Enthusiast
Enthusiast

Are you using ESX 4.1 ?

EQL have released a multipath IO plugin called EQL MEM for 4.1. Download it from their supportsite and use it to configure EQL ISCSI connection from ESX to EQL

Regards, Magnus

0 Kudos
vmn00by
Contributor
Contributor

I just don't get it. I came back in this morning and both connections were up and running as seen in the screen shots. I only set that up on one server since I had to remove most of the VMkernel Ports. I guess it just took some time to figure out it had and could use both. I don't know if this is unique to my situation or this kind of lag is typical.

Still the question I have might have confused me on the "connections" and made me think I need to over commit the phisical ports with a more than 2:1 VMKernel Port to phisical port config. I just found a more complete but somewhat more generic document on setting up ESX/ESXi to iSCSI storage.

So the question I have for that doc is based of this statement.: "EqualLogic storage systems impose a maximum limit of 512 iSCSI connections per storage pool and 2048 connections per storage group." Are those connections limited per NIC from the ESX box (in my case 2 per host), or per VM? Thinking it was per VM, thats why I over committed the NICs. If this issue is only per NIC, then I'm OK. I just didn't want to paint myself into a corner and have to dink with setting up new kernel ports when everything was up and live.

Regardless of that, Thanks so much for your help VMmatty.

0 Kudos
vmn00by
Contributor
Contributor

I am running ESXi 4.1. This isn't exactly a "Sanctioned" box by our server team as we are the desktop team...darn server people with there nose in the air... Oh sorry. But thats why I'm stuck with the "i" version without all the fun vMotion funnes or even virtual center. Kind of a pain, but the setup we have gets the job done.

So would this third party plugin work with ESXi 4.1? Beyond that, how would I get my hands on this?

0 Kudos
Yps
Enthusiast
Enthusiast

ESXi 4.1 are supported for EQL MEM plugin. You can download it from Equallogic support site.

Regards, Magnus

vmn00by
Contributor
Contributor

I went to the download and drivers section, put in my service tag and there wasn't any downloads. Is that where I'm supposed to go to get the plugin?

0 Kudos
Yps
Enthusiast
Enthusiast

You can download it from here: https://www.equallogic.com/secure/login.aspx?ReturnUrl=/Default.aspx

Regards, Magnus

0 Kudos
VMmatty
Virtuoso
Virtuoso

I think part of the confusion comes from the fact that Dell themselves printed a document that recommends overcommitting VMkernel ports to physical NICs and they say to do it for performance reasons. Like I said I have been told time and time again by Dell and VMware that that document is inaccurate and that advice should not be followed. It should always be a 1:1 mapping.

The MEM is a good option provided you're on ESX 4.1 and have at least a vSphere Enterprise license. The performance that you'll get should be better than the native multipathing plug-in provided by VMware. But don't mess with the MEM until you've got your issues with connectivity sorted, which it seems like you do.

The connection max on the EqualLogic would be per VMkernel port, so that is why overcommitting becomes limiting. If you have 5 hosts and each has a 2:1 overcommit w/ 6 LUNs, you're talking about 12 x 5 = 60 connections. It doesn't seem like much but it can add up quickly as you add hosts and datastores. I believe that EqualLogic raised the connection limit to 1024 in the latest release of their firmware though so it isn't really that much of a concern.

Hope things are working out well for you.

Matt

My blog:

Matt | http://www.thelowercasew.com | @mattliebowitz
0 Kudos
Johnnyk1
Contributor
Contributor

I have a similar situation but my question is the use of Jumbo Frames.

I'm using the BCM 5709 iSCSI offload cards and I see them when I boot and look under Storage Adapters, I can get them all working fine at 1500 MTU but I cannot get them to work using Jumbo Frames. I see that in your config print out that you have a MTU of 9000 and you are using BCM 5709 cards. Did you get Jumbo Frames working from end to end?

Here is the clip from another posting did looking for help on this

I have a brand new installation

Dell R710 using BCM5709 dual port with iSCSI offload

PS4000xv Equalogic iSCSI SAN

ESXi4 260247, fresh default install

For the sake of testing I have both ports of the active controller on the PS4000XV plugged directly into theBCM5709 card on the R710.

Under config and storage adapters I see both of the "Broadcom iSCSI Adapters" vmhba32 and vmhb33 and a displayed IQN

Steps taken to enable iSCSI, I've already created seperate vSwitches with one vMkernel port with IP and one NIC for both physical adapters (one to one)

(using cli) esxcfg-swiscsi nic add -n vmk1 -d vmhba32 step to bind port to nic

(using cli) esxcfg-swiscsi nic add -n vmk2 -d vmhba33 step to bind port to nic

(done to both adapters) right click>Properties>Dynamic Discover>Add and then list Group IP in PS4000XV

Rescan storage adapters and now I can see the available storage at the adapter level that I already presented

Click on Storage under configuration tab and then Add Storage>Disk/LUN> and pick LUN I need

Ok so far everything works and I can create VM's and use them just fine. The challenge is now to use Jumbo Frames

Steps taken to enable Jumbo Frames, please PS4000XV is directly attached to this single servers BCM5709 dual port card, no switch in between.

(using cli) esxcfg-vswitch -m 9000 vSwitch1 step to enable jumbo frame on vswitch

(using cli) esxcfg-vswitch -m 9000 vSwitch2 step to enable jumbo frame on vswitch (at this point I lost view of my storage completely from the server)

(using cli) esxcfg-vmknic -m 9000 iSCSI1 step to enable vmkernel with Jumbo Frames

(using cli) esxcfg-vmknic -m 9000 iSCSI2 step to enable vmkernel with Jumbo Frames

can anyone help me identify what step I've done wrong, or where in the end-to-end Jumbo Frames I missed out?

0 Kudos
Yps
Enthusiast
Enthusiast

fyi, jumboframes are not supported with Broadcom 5709 hw initator. See known issue with ESX and i EQL MEM userguide.

I compared Broadcom initator without jumboframes with ESX software initator with jumboframes, and I got about 25% more IO (512bytes 100% read) with ESX software initiator, but CPU utilization are higher, but I think it´s not a problem with todays CPUs.

Thanks, Magnus

0 Kudos