Failover problems between esx an Dell EQL PS4000

[s_buerger]

Hi,

I already have a case opened with vmware about this issue for some time with still no solution, but I hope someone has seen this behavior.

We have a news PS4000 and I did some failover tests. I setup as suggested in the TR1049 guide from Dell.

Setup:

1 ESX 4.0u1 (HP DL380G5, 2 Dual Port Intel PT1000 for iSCSI)

2 Dell 5224 Gigabit Switches dedicated for iSCSI (newest firmware)

1 PS4000

actual configuration (already tried different setups with only two vmks and only two pnics or 4vmks with 2 pnics):

4vmks through 4 vswitches to 4 pnics

2 pnics to switch1, 2 pnics to switch2

the switches are connected through a LAG with 6gbit

The san is connected with one interface to switch1, one to the other (passive controller same, not used for this failover test)

Now when I connect to a volume I get 4 logical connections.

It is possible that some connections goes through the lag, because the connection first goes to the group ip and the array says to which interface ip the client should connect.

For example:

source: esx interface on switch2

destination: san interface on switch1

This connection goes esx <-> switch2 <-> switch1 <-> san

Now when I power off switch1 all connections that goes through switch1 are dead which is okay.

The connection that goes from esx interface on switch2 to san interface on switch1 should failover and reconnect to the group ip which should tell the esx it should now connect to the interface ip on switch2, but it didn't, not after seconds, not after hours.

Now when I power on switch1 again all connections are reconnected without the connections that goes through the lag, they are still marked dead.

If I did a rescan they are gone. They are back when I restart the esx server.

So failover doesnt work as expected. And failback works only for some connections.

My question is:

Have someone seen this behavior in failover tests or does it worked for you?

If it worked, are you sure the connections where going through the LAG, because direct connections are no problem.

The worst case (which is more likely with 2 logical connections as with 4) is that all connections to a volume goes through one switch and when this switch goes off, all connections are dead with no failover.

Some additional behavior:

When switch1 is off, I can't vmkping anymore. I should still vmkping the second interface and the group ip but there is no response. there is a command line switch where I can tell which interface vmkping should use, but no luck, still no vmkping. It seems that vmkping only uses the first vmkernel interface. From a management server (win2k3) I still can ping the interface on switch2 and the group ip.

[s_buerger]

a switch power off / on test will show no errors (aside the link down and up again) if all of your connections are not going through the LAG.

[s_buerger]

just to be shure i did the whole test again, setup like tr1049.

192.168.55.50 vmk1

192.168.55.51 vmk2

Vmnic4 <-> switch1

Vmnic7 <-> switch2

# esxcli swiscsi nic list -d vmhba33

vmk1

pNic name: vmnic4

ipv4 address: 192.168.55.50

ipv4 net mask: 255.255.255.0

ipv6 addresses:

mac address: 00:15:17:54:a4:f2

mtu: 1500

toe: false

tso: true

tcp checksum: false

vlan: true

link connected: true

ethernet speed: 1000

packets received: 263335

packets sent: 12494

NIC driver: e1000e

driver version: 0.4.1.7-NAPI

firmware version: 5.11-2

vmk2

pNic name: vmnic7

ipv4 address: 192.168.55.51

ipv4 net mask: 255.255.255.0

ipv6 addresses:

mac address: 00:15:17:54:ab:11

mtu: 1500

toe: false

tso: true

tcp checksum: false

vlan: true

link connected: true

ethernet speed: 1000

packets received: 226449

packets sent: 295

NIC driver: e1000e

driver version: 0.4.1.7-NAPI

firmware version: 5.11-2

Vmhba33:C0:T1:L0 and :T4:L0 going through LAG (vmk1 -> switch1 -> switch2 -> eql eth1)

Vmhba33:C1:T1:L0 and :T4:L0 going through LAG (vmk2 -> switch2 -> switch1 -> eql eth0)

17:23 Power off switch1

Severity Date Time Member Message

-

---

-

---

-

---

-

---

-

---

INFO 22.07.10 17:21:50 ps2 iSCSI session to target '192.168.55.22:3260, iqn.2001-05.com.equallogic:0-8a0906-decf80507-c8e000d06b24c341-s2-esx-vm5' from initiator '192.168.55.50:52842, iqn.1998-01.com.vmware:esx1-41360ad8' was closed. iSCSI initiator connection failure. No response on connection for 6 seconds.

INFO 22.07.10 17:21:50 ps2 iSCSI session to target '192.168.55.22:3260, iqn.2001-05.com.equallogic:0-8a0906-11ef80507-558000d06a44c333-s2-esx-vm2' from initiator '192.168.55.50:52838, iqn.1998-01.com.vmware:esx1-41360ad8' was closed. iSCSI initiator connection failure. No response on connection for 6 seconds.

INFO 22.07.10 17:21:50 ps2 iSCSI session to target '192.168.55.21:3260, iqn.2001-05.com.equallogic:0-8a0906-dccf80507-032000d06af4c341-s2-esx-vm4' from initiator '192.168.55.50:52841, iqn.1998-01.com.vmware:esx1-41360ad8' was closed. iSCSI initiator connection failure. Network link state changed to down.

INFO 22.07.10 17:21:50 ps2 iSCSI session to target '192.168.55.21:3260, iqn.2001-05.com.equallogic:0-8a0906-79ff80507-0aa000d06ac4c341-s2-esx-vm3' from initiator '192.168.55.50:52839, iqn.1998-01.com.vmware:esx1-41360ad8' was closed. iSCSI initiator connection failure. Network link state changed to down.

INFO 22.07.10 17:21:50 ps2 iSCSI session to target '192.168.55.21:3260, iqn.2001-05.com.equallogic:0-8a0906-11ef80507-558000d06a44c333-s2-esx-vm2' from initiator '192.168.55.51:52843, iqn.1998-01.com.vmware:esx1-41360ad8' was closed. iSCSI initiator connection failure. Network link state changed to down.

INFO 22.07.10 17:21:50 ps2 iSCSI session to target '192.168.55.21:3260, iqn.2001-05.com.equallogic:0-8a0906-decf80507-c8e000d06b24c341-s2-esx-vm5' from initiator '192.168.55.51:52846, iqn.1998-01.com.vmware:esx1-41360ad8' was closed. iSCSI initiator connection failure. Network link state changed to down.

INFO 22.07.10 17:21:50 ps2 iSCSI session to target '192.168.55.21:3260, iqn.2001-05.com.equallogic:0-8a0906-059050507-e2c000000264c20b-s2-esx-vm1' from initiator '192.168.55.50:52837, iqn.1998-01.com.vmware:esx1-41360ad8' was closed. iSCSI initiator connection failure. Network link state changed to down.

Still the same problem. The connections through the LAG are booth dead and I now have two not reachable volumes.

And the Vmhba33:C1:T1:L0 and :T4:L0 does not failover to .22 (eth1).

The worst, they still didnt even fail back when switch1 is powered on again.

They are gone from the esx configuration page when I hit rescan.

They are dead until I reboot the esx.

[s_buerger]

installed esxi 4.1 today.

It have still the same problem.

No change at all without this weird configuration through vcli.

Next step is try the same with 2 simple office switches connected together with 1gbit. It should be the same problem.

And then I will try this EqualLogic Multipathing Extension Module.

2 months since we buy this stuff for more then 20k eur and it still does not work as expected.

I can't understand why no one have seen this before...

And our productive will be more complex with 4 switches, 3 esx, 2 eql. Can't really imagine what will happen then...

[s_buerger]

weird thing. the failback worked this time:

Vmhba33:C0:T1:L0 and T3:L0 via LAG

Vmhba33:C1:T1:L0 and T3:L0 via LAG

switch1 off:

switch1 on again:

Weird thing, now no connection goes through the LAG. So it didnt just failback it also switched over to the other interfaces.

This time I didn't wait hours before powered on switch1. Could be the solution.

But still no failover...

[MarekZdrojewski]

Hi S.Buerger,

I saw this almost the same issue with the Dell 6224 switches. Could you post the running config of your Dell 5224 switches?

Thanks.

[s_buerger]

current configuration on the switches:

current configuration on the switches:

spanning-tree mode rstp

interface port-channel 1

spanning-tree portfast

flowcontrol on

exit

interface port-channel 2

spanning-tree portfast

spanning-tree cost 200000

flowcontrol on

exit

interface range ethernet all

spanning-tree disable

exit

interface range ethernet all

flowcontrol on

exit

interface range ethernet g(1-6)

channel-group 1 mode on

exit

interface range ethernet g(7-10)

channel-group 2 mode on

exit

iscsi target port 860 address 0.0.0.0

iscsi target port 3260 address 0.0.0.0

no iscsi enable

interface vlan 1

ip address 192.168.55.36 255.255.255.0

exit

We have spanning tree with port fast on the lags because in production we would use 4 switches with a "ring" connection and the stp disables one of the connections. If one link between the switches fails the disabled one will be enabled.

It's because in production the SAN has two switches in one server room and two in another, two esx and one eql (productive) on one side and another esx + eql (test machines and backup for productive) on the other side.

The vmfs volumes with the productive vms gets then replicated to the second eql every x hours for backup.

With the cost setup only one of the "long" connections between the server rooms is disabled.

This setup was already reviewed with Dell and it does work as expected (on LAG between the serverrooms gets disabled).

No stp on the esx and eql ports.

[MarekZdrojewski]

Ok, are the 2 switches at each side stacked? If not, you should stackt them (solved my problem) and consider enabling jumbo frames for you iSCSI traffic. Also, did you disable the storm control on the ports? You can find additional information about the setup of my Dell 6224 switches here.

hth

Marek.Z

[s_buerger]

they cant be stacked. only the 62xx can be stacked.

the command "no storm-control unicast" does not exist on 54xx.

There is a broadcast (multicast and unknown unicast) storm control but it is disabled by default (and in our configuration):

port storm-control broadcast enable

The port storm-control broadcast enable Interface Configuration mode command enables Broadcast

storm control. Use the no form of this command to disable Broadcast storm control.

Syntax

port storm-control broadcast enable

no port storm-control broadcast enable

Default Configuration

Broadcast storm control is disabled.

console# show ports storm-control

Port State Rate Included

-

---

-

---

-

---

-

---

g1 Disabled 3500 Broadcast

g2 Disabled 3500 Broadcast

...

[s_buerger]

did a whole set of tests with vmware support.

as soon as i kill the vmk on the pnic which is connected to the switch i powered off the failover works.

it looks like that esx still tries to reach the array on the first vmk so it cant reconnect the dead paths.

more next week...

before the removal:

esxcli --server esx1.saxsys.de --username root network neighbor list

192.168.55.20 00:09:8a:07:05:f9 vmk1 586

192.168.55.22 00:09:8a:07:05:f9 vmk1 865

.21 was already removed from the arp cache.

the removal:

C:\Program Files (x86)\VMware\VMware vSphere CLI\bin>esxcli --server esx1.saxsys.de --username root swiscsi nic remove -n vmk1 -d vmhba33

Failed to Remove NIC.

Removed iSCI1 vmkernel port through the vi client.

C:\Program Files (x86)\VMware\VMware vSphere CLI\bin>esxcli --server esx1.saxsys.de --username root swiscsi nic list -d vmhba33

Enter password:

vmk2

pNic name: vmnic7

ipv4 address: 192.168.55.51

ipv4 net mask: 255.255.255.0

ipv6 addresses:

mac address: 00:15:17:54:ab:11

mtu: 1500

toe: false

tso: true

tcp checksum: false

vlan: true

vlanId: 0

ethernet speed: 1000

packets received: 91657

packets sent: 17511

NIC driver: e1000e

driver version: 0.4.1.7.1-1vmw-NAPI

firmware version: 5.11-2

vmk1

pNic name:

ipv4 address: ::

ipv4 net mask: ::

ipv6 addresses:

mac address: 00:00:00:00:00:00

mtu: 0

toe: false

tso: false

tcp checksum: false

vlan: false

vlanId: 0

ethernet speed: 0

packets received: 0

packets sent: 0

NIC driver:

driver version:

firmware version:

not nice but now:

192.168.55.20 00:09:8a:07:05:f9 vmk2 1172

192.168.55.22 00:09:8a:07:05:f9 vmk2 1166

And it reconnected the dead paths.

[s_buerger]

from the dell eql MEM-User_Guide:

4 Known Issues and Limitations

The following are known issues for this release.

Failure On One Physical Network Port Can Prevent iSCSI Session Rebalancing

In some cases, a network failure on a single physical NIC can affect kernel traffic on other NICs. This occurs if the physical NIC with the network failure is the only uplink for the VMKernel port that is used as the default route for the subnet. This affects several types of kernel network traffic, including ICMP pings which the EqualLogic MEM uses to test for connectivity on the SAN. The result is that the iSCSI session management functionality in the plugin will fail to rebuild the iSCSI sessions to respond to failures of SAN changes.

Could it be the same problem I have? So they already know about this problem?

Aside this it looks like the Dell MEM makes only sense in setups with more then one array per psgroup, because the PSP selects a path to a interface of the array where the data of the volume is stored. And it have a lot of limitations. We only have one array per group for now, so I think I skip this.

Still dont understand why there is no way to prevent that the connections go through the LAG in the first place, it should be possible to prefer direct connections...

[s_buerger]

Dell acknowledged that the known issue they reporting in the manual of the EqualLogic Multipathing Extension Module is the same I get.

They didn't open a ticket at vmware for now, but they will, after some more tests.

I think this issue is there since esx 4.0. In VI3 they used only one vmkernel for swiscsi with redundancy on layer1/2, so there it should not be the case.

My case number for this issue at vmware is 1544311161, the case number at dell is 818688246.

If vmware acknowledge this as a bug in 4.1, and don't have a workaround, we will go with at least 4 logical paths for each volume and hope that at least one path is still connected after switch1 fails, until they fix it.

[MarekZdrojewski]

OK, good luck and thanks for sharing.

Cheers!

[fede_pg]

Hello, I've implemented the same storage solution (PS4000 + 2x Powerconnect 5424) but with two DELL R710 and Hyper-V on W2008 R2. All is running the latest firmware (PS4000 and switches) or software (DELL HIT for Windows with MPIO driver). Each server use two different nics for iSCSI.

However I've had the same issues with iSCSI connections through the LAG, so I don't think it's a VMware-related problem.

After some days of unsuccessful configs (I've tried any kind of switch config!), now the whole thing seems more stable.

What I have done:

1) Switch config

spanning-tree mode rstp

interface range ethernet all

flowcontrol on

exit

port jumbo-frame

interface port-channel 1

switchport mode trunk

exit

vlan database

vlan 100

exit

interface range ethernet g(5-20)

switchport access vlan 100

exit

interface port-channel 1

switchport trunk allowed vlan add 100

exit

interface vlan 100

name iSCSI-Vlan

exit

voice vlan oui-table add 0001e3 Siemens_AG_phone________

voice vlan oui-table add 00036b Cisco_phone_____________

voice vlan oui-table add 00096e Avaya___________________

voice vlan oui-table add 000fe2 H3C_Aolynk______________

voice vlan oui-table add 0060b9 Philips_and_NEC_AG_phone

voice vlan oui-table add 00d01e Pingtel_phone___________

voice vlan oui-table add 00e075 Polycom/Veritel_phone___

voice vlan oui-table add 00e0bb 3Com_phone______________

interface range ethernet g(21-24)

channel-group 1 mode auto

exit

iscsi target port 860 address 0.0.0.0

iscsi target port 3260 address 0.0.0.0

no iscsi enable

interface vlan 1

ip address ##########

exit

hostname iscsi-switch-up

username admin password ########### level 15 encrypted

2) PS4000 config (with Group Manager)

In the volume config, I've added in the access tab not only a row with the iqn of my server, but also two more rows with the ip addresses used by the server's initiator. After that, the DELL MPIO driver immediately restored the second connection to the iSCSI target.

HTH, regards.

[s_buerger]

Update from Dell:

"We did reproduce this issue internally, and a fix will be released in

the 5.0.2, and 4.3.8 firmware."

"I do not

have a date yet as to when this version will be available."

[s_buerger]

Fix-List from v5.0.2 Firmware:

iSCSI Connections may be redirected to Ethernet ports without valid network links.

We will upgrade as soon as possible and then I will make the failover tests again...

[poweredge2010]

Hi Buerger,

We are having the same problem as yours, that by switch off master switch will lead to vmkping non functional to EQL or other ESX hosts's VMKernel, but our iSCSI can still connnect fortunately, as we use 4 paths (PS6000XV), just the vmkping part doens't work, all other VMotion/FT VMKernel still working (strange, only iSCSI VMKernel couldn't be vmkpingable).

However, in another case, if we just turn off the iSCSI ports connecting to EQL eth0 and eth1 on PC5448 master switch using Open Manage web interface, then vmkping works fine as designed.

So this means A) turn off the iSCSI ports using web interface on master switch and B) TURN OFF the master switch

ARE TWO different things although they seem to bring the same effect (ie, blocking EQL eth0 and eth1 path on master switch)

As you mentioned, it seems the eth0 and eth1 on EQL active controller is fixed to a path somehow on master switch.

Already using FW5.0.2 and MEM installed, I thought the problem is fixed in FW5.0.2.

Anyway, we contacted EQL and they said

======================

There was an issue with MEM and FW 5.0.0 and 5.0.1 but it is now fixed in FW 5.0.2. (seemed NOT), so EQL is pointing to VMWare, seemed there is a bug in ESX 4.1 now?

As far as Dell case 818688246 you referenced goes it is in fact similar to your problem. That case is closed and the solution to that case was:

1. some unspecified changes made by Vmware (in their case 1544311161) . A problem with ESX network stack is mentioned.

======================

May I know what do we need to change in ESX setting in order to fix this problem please? Is this actually an ESX 4.1 bug?

Thank you very much.

[s_buerger]

Funny, the vmware support status for SR 1544311161 says: "Fix:

3rd party issue - Dell will investigate".

If its only the vmkping that did not work on failover, then I think this is no big deal and can be ignored.

Its important that all possible iscsi connections failover and failback.

We still doesnt upgrade to vmware 4.1 or fw 5.0.2.

[poweredge2010]

My last reply to EQL Support today:

A Possible Bug in VMware 4.1 or EQL MEM 1.0 Plugin

Some updates, may be you can pass them to L3 for further analysis.

The problem seemed to be due to EQL MEM version 1.0 Known Issue. (User Manual 4-1)

==================================================

Failure On One Physical Network Port Can Prevent iSCSI Session Rebalancing

In some cases, a network failure on a single physical NIC can affect kernel traffic on other NICs. This occurs if the physical NIC with the network failure is the only uplink for the VMKernel port that is used as the default route for the subnet. This affects several types of kernel network traffic, including ICMP pings which the EqualLogic MEM uses to test for connectivity on the SAN. The result is that the iSCSI session management functionality in the plugin will fail to rebuild the iSCSI sessions to respond to failures of SAN changes.

==================================================

I’ve performed the test again and it showed your prediction is correct that the path is always fixed to vmk2.

1. I’ve restarted the 2nd ESX host, and found the path C0 to C1 showed correctly.

2. The reboot master switch, from 1st or 2nd ESX Host, I cannot ping vCenter or vmkping EQL or iSCSI vmks on other ESX hosts, and CANNOT PING OTHER VMKernel such as VMotion of FT this time as well.

3. But VM did not crash or restart, so underneath, all iSCSI connections stay on-line, that’s good news.

4. After master swtich comes back, under storage path on ESX Hosts, I see those C4, C5 paths generated.

Could you confirm with VMware and EQL if they have this bug in their ESX 4.1 please? (ie, Path somehow is always fixed to vmk2)

I even did a further test by switching off iSCSI ports on the master switch one by one, problem ONLY HAPPENS when I switch off ESX Host VMK2 port. (which is the first vmk iScsi port, ie, the default route for the subnet?)

It confirmed the vmkping IS BINDED to the 1st iSCSI vmk port which is vmk2 and this time, all my vmkping dead including VMotion as well as FT.

The good news is underneath the surface everything works perfectly as expected, iSCSI connections are working fine, VMs are still working and they can be VMotioned around, FT is also working fine.

We do hope VMware and EQL can release a patch sometime soon so they fix this vmkping problem that vmkping always has to go out of the default vmk2 which is the 1st iSCSI VMKernel in the vSwitch, but not any other vmk connections, so when the master switch dead, vmkping also died with vmk2 as vmkping uses vmk2 for ICMP ping to other VMKernel IP address.

[poweredge2010]

FYI, as of today Oct 19, 2010 Equallogic confirmed with me that this is actually a bug in VMware ESX 4.1, but not bug related to Equallogic.

"Yes, this is a bug

At this time VMware is working the issue but we do not have status from their end."

" After applying FW5.0.2 The ESX problem is still there. You cannot ping the vmknic primary.

However in 5.0.2 we now extend ping to return a “link down” rather than a

ICMP failure. On link down we allow a redirect to other array NICs . "

ICMP failure is what you were having, after appying FW5.0.2, the problem should be gone, but you still cannot ping to any of the vmkernel NICs or EQL nics, and this doesn't matter as at least it's working correctly, no more downtime.