Hi,
I already have a case opened with vmware about this issue for some time with still no solution, but I hope someone has seen this behavior.
We have a news PS4000 and I did some failover tests. I setup as suggested in the TR1049 guide from Dell.
Setup:
1 ESX 4.0u1 (HP DL380G5, 2 Dual Port Intel PT1000 for iSCSI)
2 Dell 5224 Gigabit Switches dedicated for iSCSI (newest firmware)
1 PS4000
actual configuration (already tried different setups with only two vmks and only two pnics or 4vmks with 2 pnics):
4vmks through 4 vswitches to 4 pnics
2 pnics to switch1, 2 pnics to switch2
the switches are connected through a LAG with 6gbit
The san is connected with one interface to switch1, one to the other (passive controller same, not used for this failover test)
Now when I connect to a volume I get 4 logical connections.
It is possible that some connections goes through the lag, because the connection first goes to the group ip and the array says to which interface ip the client should connect.
For example:
source: esx interface on switch2
destination: san interface on switch1
This connection goes esx <-> switch2 <-> switch1 <-> san
Now when I power off switch1 all connections that goes through switch1 are dead which is okay.
The connection that goes from esx interface on switch2 to san interface on switch1 should failover and reconnect to the group ip which should tell the esx it should now connect to the interface ip on switch2, but it didn't, not after seconds, not after hours.
Now when I power on switch1 again all connections are reconnected without the connections that goes through the lag, they are still marked dead.
If I did a rescan they are gone. They are back when I restart the esx server.
So failover doesnt work as expected. And failback works only for some connections.
My question is:
Have someone seen this behavior in failover tests or does it worked for you?
If it worked, are you sure the connections where going through the LAG, because direct connections are no problem.
The worst case (which is more likely with 2 logical connections as with 4) is that all connections to a volume goes through one switch and when this switch goes off, all connections are dead with no failover.
Some additional behavior:
When switch1 is off, I can't vmkping anymore. I should still vmkping the second interface and the group ip but there is no response. there is a command line switch where I can tell which interface vmkping should use, but no luck, still no vmkping. It seems that vmkping only uses the first vmkernel interface. From a management server (win2k3) I still can ping the interface on switch2 and the group ip.
a switch power off / on test will show no errors (aside the link down and up again) if all of your connections are not going through the LAG.
just to be shure i did the whole test again, setup like tr1049.
192.168.55.50 vmk1
192.168.55.51 vmk2
Vmnic4 <-> switch1
Vmnic7 <-> switch2
# esxcli swiscsi nic list -d vmhba33
vmk1
pNic name: vmnic4
ipv4 address: 192.168.55.50
ipv4 net mask: 255.255.255.0
ipv6 addresses:
mac address: 00:15:17:54:a4:f2
mtu: 1500
toe: false
tso: true
tcp checksum: false
vlan: true
link connected: true
ethernet speed: 1000
packets received: 263335
packets sent: 12494
NIC driver: e1000e
driver version: 0.4.1.7-NAPI
firmware version: 5.11-2
vmk2
pNic name: vmnic7
ipv4 address: 192.168.55.51
ipv4 net mask: 255.255.255.0
ipv6 addresses:
mac address: 00:15:17:54:ab:11
mtu: 1500
toe: false
tso: true
tcp checksum: false
vlan: true
link connected: true
ethernet speed: 1000
packets received: 226449
packets sent: 295
NIC driver: e1000e
driver version: 0.4.1.7-NAPI
firmware version: 5.11-2
Vmhba33:C0:T1:L0 and :T4:L0 going through LAG (vmk1 -> switch1 -> switch2 -> eql eth1)
Vmhba33:C1:T1:L0 and :T4:L0 going through LAG (vmk2 -> switch2 -> switch1 -> eql eth0)
17:23 Power off switch1
Severity Date Time Member Message
-
-
-
-
-
INFO 22.07.10 17:21:50 ps2 iSCSI session to target '192.168.55.22:3260, iqn.2001-05.com.equallogic:0-8a0906-decf80507-c8e000d06b24c341-s2-esx-vm5' from initiator '192.168.55.50:52842, iqn.1998-01.com.vmware:esx1-41360ad8' was closed. iSCSI initiator connection failure. No response on connection for 6 seconds.
INFO 22.07.10 17:21:50 ps2 iSCSI session to target '192.168.55.22:3260, iqn.2001-05.com.equallogic:0-8a0906-11ef80507-558000d06a44c333-s2-esx-vm2' from initiator '192.168.55.50:52838, iqn.1998-01.com.vmware:esx1-41360ad8' was closed. iSCSI initiator connection failure. No response on connection for 6 seconds.
INFO 22.07.10 17:21:50 ps2 iSCSI session to target '192.168.55.21:3260, iqn.2001-05.com.equallogic:0-8a0906-dccf80507-032000d06af4c341-s2-esx-vm4' from initiator '192.168.55.50:52841, iqn.1998-01.com.vmware:esx1-41360ad8' was closed. iSCSI initiator connection failure. Network link state changed to down.
INFO 22.07.10 17:21:50 ps2 iSCSI session to target '192.168.55.21:3260, iqn.2001-05.com.equallogic:0-8a0906-79ff80507-0aa000d06ac4c341-s2-esx-vm3' from initiator '192.168.55.50:52839, iqn.1998-01.com.vmware:esx1-41360ad8' was closed. iSCSI initiator connection failure. Network link state changed to down.
INFO 22.07.10 17:21:50 ps2 iSCSI session to target '192.168.55.21:3260, iqn.2001-05.com.equallogic:0-8a0906-11ef80507-558000d06a44c333-s2-esx-vm2' from initiator '192.168.55.51:52843, iqn.1998-01.com.vmware:esx1-41360ad8' was closed. iSCSI initiator connection failure. Network link state changed to down.
INFO 22.07.10 17:21:50 ps2 iSCSI session to target '192.168.55.21:3260, iqn.2001-05.com.equallogic:0-8a0906-decf80507-c8e000d06b24c341-s2-esx-vm5' from initiator '192.168.55.51:52846, iqn.1998-01.com.vmware:esx1-41360ad8' was closed. iSCSI initiator connection failure. Network link state changed to down.
INFO 22.07.10 17:21:50 ps2 iSCSI session to target '192.168.55.21:3260, iqn.2001-05.com.equallogic:0-8a0906-059050507-e2c000000264c20b-s2-esx-vm1' from initiator '192.168.55.50:52837, iqn.1998-01.com.vmware:esx1-41360ad8' was closed. iSCSI initiator connection failure. Network link state changed to down.
Still the same problem. The connections through the LAG are booth dead and I now have two not reachable volumes.
And the Vmhba33:C1:T1:L0 and :T4:L0 does not failover to .22 (eth1).
The worst, they still didnt even fail back when switch1 is powered on again.
They are gone from the esx configuration page when I hit rescan.
They are dead until I reboot the esx.
installed esxi 4.1 today.
It have still the same problem.
No change at all without this weird configuration through vcli.
Next step is try the same with 2 simple office switches connected together with 1gbit. It should be the same problem.
And then I will try this EqualLogic Multipathing Extension Module.
2 months since we buy this stuff for more then 20k eur and it still does not work as expected.
I can't understand why no one have seen this before...
And our productive will be more complex with 4 switches, 3 esx, 2 eql. Can't really imagine what will happen then...
weird thing. the failback worked this time:
Vmhba33:C0:T1:L0 and T3:L0 via LAG
Vmhba33:C1:T1:L0 and T3:L0 via LAG
switch1 off:
switch1 on again:
Weird thing, now no connection goes through the LAG. So it didnt just failback it also switched over to the other interfaces.
This time I didn't wait hours before powered on switch1. Could be the solution.
But still no failover...
Hi S.Buerger,
I saw this almost the same issue with the Dell 6224 switches. Could you post the running config of your Dell 5224 switches?
Thanks.
current configuration on the switches:
current configuration on the switches:
spanning-tree mode rstp
interface port-channel 1
spanning-tree portfast
flowcontrol on
exit
interface port-channel 2
spanning-tree portfast
spanning-tree cost 200000
flowcontrol on
exit
interface range ethernet all
spanning-tree disable
exit
interface range ethernet all
flowcontrol on
exit
interface range ethernet g(1-6)
channel-group 1 mode on
exit
interface range ethernet g(7-10)
channel-group 2 mode on
exit
iscsi target port 860 address 0.0.0.0
iscsi target port 3260 address 0.0.0.0
no iscsi enable
interface vlan 1
ip address 192.168.55.36 255.255.255.0
exit
We have spanning tree with port fast on the lags because in production we would use 4 switches with a "ring" connection and the stp disables one of the connections. If one link between the switches fails the disabled one will be enabled.
It's because in production the SAN has two switches in one server room and two in another, two esx and one eql (productive) on one side and another esx + eql (test machines and backup for productive) on the other side.
The vmfs volumes with the productive vms gets then replicated to the second eql every x hours for backup.
With the cost setup only one of the "long" connections between the server rooms is disabled.
This setup was already reviewed with Dell and it does work as expected (on LAG between the serverrooms gets disabled).
No stp on the esx and eql ports.
Ok, are the 2 switches at each side stacked? If not, you should stackt them (solved my problem) and consider enabling jumbo frames for you iSCSI traffic. Also, did you disable the storm control on the ports? You can find additional information about the setup of my Dell 6224 switches here.
hth
Marek.Z
they cant be stacked. only the 62xx can be stacked.
the command "no storm-control unicast" does not exist on 54xx.
There is a broadcast (multicast and unknown unicast) storm control but it is disabled by default (and in our configuration):
port storm-control broadcast enable
The port storm-control broadcast enable Interface Configuration mode command enables Broadcast
storm control. Use the no form of this command to disable Broadcast storm control.
Syntax
port storm-control broadcast enable
no port storm-control broadcast enable
Default Configuration
Broadcast storm control is disabled.
console# show ports storm-control
-
-
-
-
g1 Disabled 3500 Broadcast
g2 Disabled 3500 Broadcast
...
did a whole set of tests with vmware support.
as soon as i kill the vmk on the pnic which is connected to the switch i powered off the failover works.
it looks like that esx still tries to reach the array on the first vmk so it cant reconnect the dead paths.
more next week...
before the removal:
esxcli --server esx1.saxsys.de --username root network neighbor list
192.168.55.20 00:09:8a:07:05:f9 vmk1 586
192.168.55.22 00:09:8a:07:05:f9 vmk1 865
.21 was already removed from the arp cache.
the removal:
C:\Program Files (x86)\VMware\VMware vSphere CLI\bin>esxcli --server esx1.saxsys.de --username root swiscsi nic remove -n vmk1 -d vmhba33
Failed to Remove NIC.
Removed iSCI1 vmkernel port through the vi client.
C:\Program Files (x86)\VMware\VMware vSphere CLI\bin>esxcli --server esx1.saxsys.de --username root swiscsi nic list -d vmhba33
Enter password:
vmk2
pNic name: vmnic7
ipv4 address: 192.168.55.51
ipv4 net mask: 255.255.255.0
ipv6 addresses:
mac address: 00:15:17:54:ab:11
mtu: 1500
toe: false
tso: true
tcp checksum: false
vlan: true
vlanId: 0
ethernet speed: 1000
packets received: 91657
packets sent: 17511
NIC driver: e1000e
driver version: 0.4.1.7.1-1vmw-NAPI
firmware version: 5.11-2
vmk1
pNic name:
ipv4 address: ::
ipv4 net mask: ::
ipv6 addresses:
mac address: 00:00:00:00:00:00
mtu: 0
toe: false
tso: false
tcp checksum: false
vlan: false
vlanId: 0
ethernet speed: 0
packets received: 0
packets sent: 0
NIC driver:
driver version:
firmware version:
not nice but now:
192.168.55.20 00:09:8a:07:05:f9 vmk2 1172
192.168.55.22 00:09:8a:07:05:f9 vmk2 1166
And it reconnected the dead paths.
from the dell eql MEM-User_Guide:
4 Known Issues and Limitations
The following are known issues for this release.
Failure On One Physical Network Port Can Prevent iSCSI Session Rebalancing
In some cases, a network failure on a single physical NIC can affect kernel traffic on other NICs. This occurs if the physical NIC with the network failure is the only uplink for the VMKernel port that is used as the default route for the subnet. This affects several types of kernel network traffic, including ICMP pings which the EqualLogic MEM uses to test for connectivity on the SAN. The result is that the iSCSI session management functionality in the plugin will fail to rebuild the iSCSI sessions to respond to failures of SAN changes.
Could it be the same problem I have? So they already know about this problem?
Aside this it looks like the Dell MEM makes only sense in setups with more then one array per psgroup, because the PSP selects a path to a interface of the array where the data of the volume is stored. And it have a lot of limitations. We only have one array per group for now, so I think I skip this.
Still dont understand why there is no way to prevent that the connections go through the LAG in the first place, it should be possible to prefer direct connections...
Dell acknowledged that the known issue they reporting in the manual of the EqualLogic Multipathing Extension Module is the same I get.
They didn't open a ticket at vmware for now, but they will, after some more tests.
I think this issue is there since esx 4.0. In VI3 they used only one vmkernel for swiscsi with redundancy on layer1/2, so there it should not be the case.
My case number for this issue at vmware is 1544311161, the case number at dell is 818688246.
If vmware acknowledge this as a bug in 4.1, and don't have a workaround, we will go with at least 4 logical paths for each volume and hope that at least one path is still connected after switch1 fails, until they fix it.
OK, good luck and thanks for sharing.
Cheers!
Hello, I've implemented the same storage solution (PS4000 + 2x Powerconnect 5424) but with two DELL R710 and Hyper-V on W2008 R2. All is running the latest firmware (PS4000 and switches) or software (DELL HIT for Windows with MPIO driver). Each server use two different nics for iSCSI.
However I've had the same issues with iSCSI connections through the LAG, so I don't think it's a VMware-related problem.
After some days of unsuccessful configs (I've tried any kind of switch config!), now the whole thing seems more stable.
What I have done:
1) Switch config
spanning-tree mode rstp
interface range ethernet all
flowcontrol on
exit
port jumbo-frame
interface port-channel 1
switchport mode trunk
exit
vlan database
vlan 100
exit
interface range ethernet g(5-20)
switchport access vlan 100
exit
interface port-channel 1
switchport trunk allowed vlan add 100
exit
interface vlan 100
name iSCSI-Vlan
exit
voice vlan oui-table add 0001e3 Siemens_AG_phone________
voice vlan oui-table add 00036b Cisco_phone_____________
voice vlan oui-table add 00096e Avaya___________________
voice vlan oui-table add 000fe2 H3C_Aolynk______________
voice vlan oui-table add 0060b9 Philips_and_NEC_AG_phone
voice vlan oui-table add 00d01e Pingtel_phone___________
voice vlan oui-table add 00e075 Polycom/Veritel_phone___
voice vlan oui-table add 00e0bb 3Com_phone______________
interface range ethernet g(21-24)
channel-group 1 mode auto
exit
iscsi target port 860 address 0.0.0.0
iscsi target port 3260 address 0.0.0.0
no iscsi enable
interface vlan 1
ip address ##########
exit
hostname iscsi-switch-up
username admin password ########### level 15 encrypted
2) PS4000 config (with Group Manager)
In the volume config, I've added in the access tab not only a row with the iqn of my server, but also two more rows with the ip addresses used by the server's initiator. After that, the DELL MPIO driver immediately restored the second connection to the iSCSI target.
HTH, regards.
Update from Dell:
"We did reproduce this issue internally, and a fix will be released in
the 5.0.2, and 4.3.8 firmware."
"I do not
have a date yet as to when this version will be available."
Fix-List from v5.0.2 Firmware:
iSCSI Connections may be redirected to Ethernet ports without valid network links.
We will upgrade as soon as possible and then I will make the failover tests again...
Hi Buerger,
We are having the same problem as yours, that by switch off master switch will lead to vmkping non functional to EQL or other ESX hosts's VMKernel, but our iSCSI can still connnect fortunately, as we use 4 paths (PS6000XV), just the vmkping part doens't work, all other VMotion/FT VMKernel still working (strange, only iSCSI VMKernel couldn't be vmkpingable).
However, in another case, if we just turn off the iSCSI ports connecting to EQL eth0 and eth1 on PC5448 master switch using Open Manage web interface, then vmkping works fine as designed.
So this means A) turn off the iSCSI ports using web interface on master switch and B) TURN OFF the master switch
ARE TWO different things although they seem to bring the same effect (ie, blocking EQL eth0 and eth1 path on master switch)
As you mentioned, it seems the eth0 and eth1 on EQL active controller is fixed to a path somehow on master switch.
Already using FW5.0.2 and MEM installed, I thought the problem is fixed in FW5.0.2.
Anyway, we contacted EQL and they said
======================
There was an issue with MEM and FW 5.0.0 and 5.0.1 but it is now fixed in FW 5.0.2. (seemed NOT), so EQL is pointing to VMWare, seemed there is a bug in ESX 4.1 now?
As far as Dell case 818688246 you referenced goes it is in fact similar to your problem. That case is closed and the solution to that case was:
1. some unspecified changes made by Vmware (in their case 1544311161) . A problem with ESX network stack is mentioned.
======================
May I know what do we need to change in ESX setting in order to fix this problem please? Is this actually an ESX 4.1 bug?
Thank you very much.
Funny, the vmware support status for SR 1544311161 says: "Fix:
3rd party issue - Dell will investigate".
If its only the vmkping that did not work on failover, then I think this is no big deal and can be ignored.
Its important that all possible iscsi connections failover and failback.
We still doesnt upgrade to vmware 4.1 or fw 5.0.2.
My last reply to EQL Support today:
A Possible Bug in VMware 4.1 or EQL MEM 1.0 Plugin
Some updates, may be you can pass them to L3 for further analysis.
The problem seemed to be due to EQL MEM version 1.0 Known Issue. (User Manual 4-1)
==================================================
Failure On One Physical Network Port Can Prevent iSCSI Session Rebalancing
In some cases, a network failure on a single physical NIC can affect kernel traffic on other NICs. This occurs if the physical NIC with the network failure is the only uplink for the VMKernel port that is used as the default route for the subnet. This affects several types of kernel network traffic, including ICMP pings which the EqualLogic MEM uses to test for connectivity on the SAN. The result is that the iSCSI session management functionality in the plugin will fail to rebuild the iSCSI sessions to respond to failures of SAN changes.
==================================================
I’ve performed the test again and it showed your prediction is correct that the path is always fixed to vmk2.
1. I’ve restarted the 2nd ESX host, and found the path C0 to C1 showed correctly.
2. The reboot master switch, from 1st or 2nd ESX Host, I cannot ping vCenter or vmkping EQL or iSCSI vmks on other ESX hosts, and CANNOT PING OTHER VMKernel such as VMotion of FT this time as well.
3. But VM did not crash or restart, so underneath, all iSCSI connections stay on-line, that’s good news.
4. After master swtich comes back, under storage path on ESX Hosts, I see those C4, C5 paths generated.
Could you confirm with VMware and EQL if they have this bug in their ESX 4.1 please? (ie, Path somehow is always fixed to vmk2)
I even did a further test by switching off iSCSI ports on the master switch one by one, problem ONLY HAPPENS when I switch off ESX Host VMK2 port. (which is the first vmk iScsi port, ie, the default route for the subnet?)
It confirmed the vmkping IS BINDED to the 1st iSCSI vmk port which is vmk2 and this time, all my vmkping dead including VMotion as well as FT.
The good news is underneath the surface everything works perfectly as expected, iSCSI connections are working fine, VMs are still working and they can be VMotioned around, FT is also working fine.
We do hope VMware and EQL can release a patch sometime soon so they fix this vmkping problem that vmkping always has to go out of the default vmk2 which is the 1st iSCSI VMKernel in the vSwitch, but not any other vmk connections, so when the master switch dead, vmkping also died with vmk2 as vmkping uses vmk2 for ICMP ping to other VMKernel IP address.
FYI, as of today Oct 19, 2010 Equallogic confirmed with me that this is actually a bug in VMware ESX 4.1, but not bug related to Equallogic.
"Yes, this is a bug
At this time VMware is working the issue but we do not have status from their end."
" After applying FW5.0.2 The ESX problem is still there. You cannot ping the vmknic primary.
However in 5.0.2 we now extend ping to return a “link down” rather than a
ICMP failure. On link down we allow a redirect to other array NICs . "
ICMP failure is what you were having, after appying FW5.0.2, the problem should be gone, but you still cannot ping to any of the vmkernel NICs or EQL nics, and this doesn't matter as at least it's working correctly, no more downtime.