VMware Cloud Community
Saturnous
Enthusiast
Enthusiast

1 Equallogic PS6000VS 2 PowerConnect 6224 4 NICs / Host - HowTo?

Hi all,

i read several community and blog posts + official documentation about best practices when connecting Equallogic to ESX. But some open questions remain. No one see a "use as much ports as possible" and the "8 paths max per iscsi LUN" dilema. Equallogic documentation suggest to connect the iscsi switches, but this means when i utilize the 4 storage ports i can't utilize more than 2 NIC ports. I want to keep it "simple and stupid" + high available. I want to avoid trunks/bondings in any form (LAG on the Equallogic or static on the ESX or interconnects between the two switches).

Here the config :

          NIC1     -     SW1

          NIC2     -     SW2

ESX   NIC3     -     SW1

          NIC4    -     SW2

They are configured in a 1:1 NIC to vmk port relation.

                    NIC1     -     SW1

PS60000       NIC2     -     SW2

                    NIC3     -     SW1

                    NIC4     -     SW2

All NICs/Ports have IPs from the same Subnet. The 2 switches will have poor uplinks to the production network for managment purposes, the iscsi network will be locked in seperate vlans.

I have to avoid more than 8 paths (as a limit from ESX side).

I want to keep the paths as short as possible (NIC - Switch - EQ Port).

The easiest way seams for me is to keep the switches un-stacked. Would offer the following 8 paths.

ESX NIC1 - EQ NIC1

ESX NIC1 - EQ NIC3

ESX NIC2 - EQ NIC2

ESX NIC2 - EQ NIC4

ESX NIC3 - EQ NIC1

ESX NIC3 - EQ NIC3

ESX NIC4 - EQ NIC2

ESX NIC4 - EQ NIC4

Having 2 seperated networks but only one subnet let me some questions open.

Would this config work ?

Does the Group IP binds on all the ports (and is so reachable from all ESX NICs) ?

Does the discover process try to reach the group IP over all vmkernel interfaces ?

Would the iscsi gateway IP (the Group IP) propagate all IPs from the port to all vmkernel ports ?

Would there be any spamming on the iscsi network by e.g. ESX NIC1 trying to reach EQ NIC 2, or would it give up after a reasonable timeframe ?

Is the config a good idea ?

How a better idea would look like ?

0 Kudos
7 Replies
Phokay
Enthusiast
Enthusiast

If you just follow the best practices mentioned in Equallogic document, you will get the best performance. The way Equallogic works is when an iSCSI initiator request data from EQL group IP, it returns an eth# port in round robin fashion starting from eth0. It's also important to setup round robin NMP policy on ESX servers. You can connect 2 switches with uplinks without configuring LAGs.

Hope this helps.

0 Kudos
AndreTheGiant
Immortal
Immortal

See this document:

http://www.equallogic.com/resourcecenter/assetview.aspx?id=8453

If you use the 1:1 mapping (requided also for the native multipath module) you will have only 4 paths.

Andre

Andrew | http://about.me/amauro | http://vinfrastructure.it/ | @Andrea_Mauro
0 Kudos
Saturnous
Enthusiast
Enthusiast

A automatic discovery on the group IP deliver not all the IPs from the port back, just 2 of them so that i get 8 paths. But i dont see any pattern what IPs are messaged back, on all my 3 hosts i got a different combination - not even load balanced.

Using the first Equallogic port instead the group IP discovered all 4 ports in a equal manner.

0 Kudos
AndreTheGiant
Immortal
Immortal

It must discover all the connections.

Have you test with vmkping if you can ping group and member IPs?

Andre

Andrew | http://about.me/amauro | http://vinfrastructure.it/ | @Andrea_Mauro
0 Kudos
bmorbach
Enthusiast
Enthusiast

I am afraid you are completely misunderstanding how the EqualLogic product works.

If I understand your setup correct you are using 1 ESX server with 4 NICs for exclusive iSCSI traffic.
You are also using 1 PS6000.

Server as well as storage are connected via 2 switches.

A SAN should always be a dedicated network just for storage data. Hence you are correct about using a dedicated LAN segment for iSCSI traffic.

It is also critical for performance that traffic will not go via a router.

To achieve a multipathed connection between an ESX server and a storage device multiple iSCSI sessions must be established using multiple VMkernels and the ESX can then use round robin multipathing.

So far so good.

Most storage products that support multipathing will present an iSCSI target via different controller ports and for multipathing to work you must establish an iSCSI session to each controller IP address involved.

That is NOT the case with an EqualLogic array. EqualLogic arrays are a virtualization approach completely hiding the physical array NICs behind a virtual group IP. The group IP is up on one of the array ports but NOT on all of them. It will switch to another array NIC should that become necessary. To establish an iSCSI session you must always connect to the group IP only. The array will then decide (load balancing) which of its physical NIC should take that session. That NIC will then answer with an iSCSI redirect and tell the ESX server to connect to its IP address instead of the group IP. You should NEVER open an iSCSI session to any other IP than the group IP. Also there is no way for you to force a connection to a certain storage NIC!


The array itself has an active and a passive controller. The active controller can be configured with up to 4 NICs for iSCSI traffic. The passive controller will take over these connections should the active controller fail. All 4 NICs on the active coontroller support active/active multipathing.

For this whole mechanism to work every ESX NIC must be able to physically reach every array port. With a redundant switch setup both switches either MUST be stacked or they MUST be connected using an ISL with at least 4 GB bandwidth.

You can then create up to 4 iSCSI sessions from a maximum of 4 VMkernels (since you have 4 ESX NICs) using the group IP in every iSCSI session. By default ESX will create these as active/passive which you can change on the ESX server.

Hence you will never get more than 4 paths to an iSCSI target. Should you only use 2 iSCSI sessions you will only use 2 paths.

Hope this helps.

0 Kudos
Saturnous
Enthusiast
Enthusiast

Using a interconnect is what i wanted avoid as both switches have a (2 port LAG) uplink to 2 another storage-only Switches with 2 other eql storages connected, these have also a 2 port LAG each other. If i interconnect the two PC 6224's i need to take care about path finding - i didnt designed it.

Connecting over the group IP gives all my 3 ESX hosts * 4 NICs a loadbalancing, i just dont understand why the storage ports are not spread equaly over all LUNs and hosts. It just assign without any patterns 8 paths - it equaly use the vmkX NICs but not the ports on the EQL array, some LUNs only run over 2 EQL ports some over 3 but they have all 7 same size same usage pattern.

LUN 1 :

vmk1 - eql port1

vmk2 - eql port3

vmk3 - eql port2

vmk4 - eql port1

vmk1 - eql port1

vmk2 - eql port2

vmk3 - eql port1

vmk4 - eql port3

LUN 2 :

just another chaotic combo

If i dont use the group IP but for the LUNs switching plain EQL port IPs it utilize only 2 EQL ports.

Anyway the array runs fast as hell - and in some chaotic way it loadbalance - so i see the issue as resolved. I would be happy if the equallogic guys build in some path managing to provide some flexibilty. I imagine how to deal with this beasts in a 10Gb / 1Gb enviroment.

I ran 2 iometer instances 60% write 4kb 100% random and archived ca 2200 IOPS on the spindles. One instance was on the physical backup machine one on a virtual machine with a RDM LUN.

BUT when i used the VAAI init function (creating a zeroed out vmdk) on ANY LUN the overall IOPS breaks down to under 100, VMs on the EQL nearly freezed. The VAAI datamover took it down to 1000 each. If you have freezes in your EQL SAN disable VAAI init, without init function it tooks ca 2,5 times more to zero out a vmdk, which is ok.

0 Kudos
depping
Leadership
Leadership

@Saturnous: Is this with an EQL that is supported for VAAI? if so please report it so this can be fixed / looked at.

Duncan (VCDX)

Available now on Amazon: vSphere 4.1 HA and DRS technical deepdive

0 Kudos