ESXi 4.1 + EqualLogic PS6500E MPIO

[timjwatts]

Hi,

Hmm - busy night...

I've followed the MPIO guides in some detail and done the following:

Dell R610, ESXi 4.1, 4x Broadcomm BCM5790 which include iSCSI offload.

Those are configured as 4 portgroups on  vSwitch1 with teaming set so each of the portgroups goes to 1 NIC only, all others in "Unused".

IPs are 10.223.10.110, 111, 112, 113

They map to HBAs vmhba33,34,35,36 (type Broadcomm iSCSI Adaptor) and I've carefully checked the MAC addressed to ensure I then do the bind between the vmk1,2,3,4 nics correctly to the correct HBA.

THe SAN (proven working with MPIO from linux host) has IPs:

10.223.10.10 (Group IP)

10.223.10.11,12,13,14 (the 4 NICs).

Everything is pingable and the switch VLAN config is sound.

I then did a Dynamic Discovery on both the group IP and one of the iSCSI taget IPs (tried various).

In all cases, the paths coming back in the vClient Static Discovery tab are both LUNs (I have 2 volumes), but are showing as the SAN group IP only (10.223.10.10)

Under linux, running and iscsiadm "discovery" command came back with all combinations of the targetIP and paths (in this case that should be 8 combinations, not 2).

Please please tell me the SAN target IPs do not have to be on different VLANs 😐

Otherwise, I'm not sure what gives here. Can't really proceed withMPIO pathing as there are no multipaths.

SAN auth is very simple - no CHAP, just initiators 10.223.10.* allowed, very simple...

As always, any ideas would be most welcome

Cheers

Tim

[timjwatts]

Eeek!

Sorry folks, put this on hold for a mo...

I had reset the EQL SAN since the last time I tested from a linux host. I've just retested, and the linux openiscsi tools are only returning single sets of targets.

Hmm - something's up with the EQL box then...

<scuttles off to kick something>

[timjwatts]

Scrub that last remark..

The linux host in question only has one uplink and one IP. The original test one had 4 uplinks and 4 IPs on the iSCSI VLAN. I don;t suppose the current one would shouw multipaths, because it is meaningless...

Hmm, have to assume the EQL box is OK - back to the documents...

[virtualdutch]

On the ESXi host you've four iSCSI NIC's available, but you've mounted each iSCSI vmkernel to another vmhba?

You should mount each iSCSI vmkernel to the same vmhba and set an default path selection policy to round robin with storage array type VMW_SATP_EQL.

Then you will see an equal load on each iSCSI NIC (in our enviroment I changed the round robin every 1000 iops to 3).

The other way is to connect from the virtual machine, but the linux open iscsi initiator is not developing anymore (at least my oppinion, the last update is from 2005). And active/active connection is not the most stabile way with this initiator only active/failover.

[timjwatts]

Hi,

No - I did a "bind" to the same vmhba. The Broadcom iSCSI offload feature complicates things as you automatically get a vmhba, but you then have to play "hunt the corresponding vmnic" and bind one to the other - at least that's what the documenation seems to say.

I'll have another play this morning. It's getting confusing that the EQL box was quite happy to offer me 4 sets of paths, using the iSCSI IPs when I last had a multipathed linux host on it, but now, ESX is discovering only the SAN group (or master) IP. That may be a "feature" of course - it's hard to know what's supposed to happen as I've not done this before.

Just found another document: "iSCSI SAN Configuration Guide" and that's helpful.

Thanks for the round robing tuning trick

Cheers

Tim

[timjwatts]

Oh for fuck's sake (not you, Broadcom)!!!

http://communities.vmware.com/message/1673309

It turns out that the BCM5709 does not support jumbo frames with ESX4.1 (or 5 apparantly!!!)

The thread above notes that I'd probably be better off using the E1000 NICs with a SW initiator without iSCSI offload but with jumbo frames.

That explains the erratic results - when it worked, it was because I had done a quick manual setup without jumbo frames.

The confusing things was that vmkping -s 9000 <SAN> worked from the ESXi in tech support mode.

I found that EQL have an MPIO plugin that contains a handy setup.pl script that configures everything. Running it with --mtu=1500 worked, but even --mtu=2000 broke iSCSI.

Sigh...

Oh well, on Monday I will repatch the NICs and use the E1000 chips. Bloody Broadcom. How long has everyone else been using jumbo frames?...

Cheers Guys,

Tim

[timjwatts]

Ok - one more point...

The BCM5709 chips work OK with a SW initiator at MTU 9000.

Hmm, I wonder whether to leave them like that or swap. I'll go and have a look at the chip datasheet and see if the TOE can work at MTU 9000. If not, I will rededicate them to the main uplinks to the public internet that will be at MTU 1500 anyway, then the TOE engine can do something useful (maybe).

Re-confirms my gut feeling that Broadcom are a buch of cheap sods and Intel E1000s will always be better...

Cheers

Tim

[timjwatts]

http://www.vmadmin.co.uk/vmware/35-esxserver/252-esxihwswiscsijumbo

Ok - ^^^ that is worth a read.