HP/Broadcom 5709 iSCSI issue?

[Titans99]

I have two DL385 G7's and for iSCSI connectivity on ESXi 5, I am using two of the internal Broadcom 5709 NIC's with hardware initiator.  On my existing ESXi 4.1 hosts, I am using the software iSCSI adapter (NC360T's) which easily connects to an HP p2000 iscsi array and also connects to both Enhance-tech iscsi arrays (no CHAP).

For some reason the 5709 NIC's in the new servers will NOT connect to the enhance-tech SAN.  I have upgraded to the latest HP firmware and the latest ESXi 5 build (Dec 15th build 515841)

I get the below error in the vmware syslog.log.  Anyone have any ideas? (I can post more of the log if needed)

2012-01-06T23:08:46Z iscsid: Login Target: iqn.1997-10.com.enhance-tech:storage02:target01 if=bnx2i-984be10a54f6@vmk1 addr=10.1.10.5:3260 (TPGT:65536 ISID:0x1)
2012-01-06T23:08:46Z iscsid: Notice: Assigned (H34 T1 C0 session=15, target=5/9)
2012-01-06T23:08:46Z iscsid: SessionResolve for 10.1.10.5 (via vmk1) started)
2012-01-06T23:08:46Z iscsid: Login Target: iqn.1997-10.com.enhance-tech:storage02:target01 if=bnx2i-984be10a54f6@vmk1 addr=10.1.10.6:3260 (TPGT:65536 ISID:0x2)
2012-01-06T23:08:46Z iscsid: Notice: Assigned (H34 T1 C1 session=16, target=5/a)
2012-01-06T23:08:46Z iscsid: SessionResolve for 10.1.10.6 (via vmk1) started)
2012-01-06T23:08:46Z iscsid: DISCOVERY: transport_name=bnx2i-984be10a54f6 Pending=2 Failed=0
2012-01-06T23:08:47Z iscsid: DISCOVERY: transport_name=bnx2i-984be10a54f6 Pending=2 Failed=0
2012-01-06T23:08:48Z iscsid: DISCOVERY: transport_name=bnx2i-984be10a54f6 Pending=2 Failed=0
2012-01-06T23:08:49Z iscsid: DISCOVERY: transport_name=bnx2i-984be10a54f6 Pending=2 Failed=0
2012-01-06T23:08:50Z iscsid: DISCOVERY: transport_name=bnx2i-984be10a54f6 Pending=2 Failed=0
2012-01-06T23:08:50Z iscsid: ep_poll failed rc-1
2012-01-06T23:08:50Z iscsid: Login Failed: iqn.1997-10.com.enhance-tech:storage02:target01 if=bnx2i-984be10a54f6@vmk1 addr=10.1.10.5:3260 (TPGT:65536 ISID:0x1) Reason: 00040000 (Initiator Connection Failure)
2012-01-06T23:08:50Z iscsid: Notice: Reclaimed Channel (H34 T1 C0 oid=5)
2012-01-06T23:08:50Z iscsid: ep_poll failed rc-1
2012-01-06T23:08:50Z iscsid: Login Failed: iqn.1997-10.com.enhance-tech:storage02:target01 if=bnx2i-984be10a54f6@vmk1 addr=10.1.10.6:3260 (TPGT:65536 ISID:0x2) Reason: 00040000 (Initiator Connection Failure)
2012-01-06T23:08:50Z iscsid: Notice: Reclaimed Channel (H34 T1 C1 oid=5)
2012-01-06T23:08:50Z iscsid: Notice: Reclaimed Target (H34 T1 oid=5)
2012-01-06T23:08:51Z iscsid: DISCOVERY: transport_name=bnx2i-984be10a54f6 Pending=0 Failed=2

[elgreco81]

Hi,

Which exact model is? (this is an extract from vmware hcl page)

Broadcom	Broadcom NetXtreme II BCM5709 1000Base-T	iSCSI	ESXi 5.0 ESX / ESXi 4.1 U2 ESX / ESXi 4.1 U1 ESX / ESXi 4.1
Broadcom	Broadcom NetXtreme II BCM5709S 1000Base-SX	iSCSI	ESX / ESXi 4.1 U2 ESX / ESXi 4.1 U1 ESX / ESXi 4.1

I don't know the answer to your question but doing some search, I found this (sadly the web page to this KB seems not to work properly...)

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=200710...

From this page, this is the "interesting" part http://itblog.rogerlund.net/

ESXi 5.x boot delays when configured for Software iSCSI

Symptoms

• ESXi 5.0 experiences a delay when booting during the the software-iscsi step.
• After the boot process completes, the sysboot.log file contains entries similar to:
  [01:57:50.925338] sysboot: software-iscsi
  [02:28:22.330320] sysboot: restore-paths
• After the boot process completes, the syslog.log file contains entries similar to:
  iscsid: cannot make a connection to 192.168.1.20:3260 (101,Network is unreachable)
  iscsid: Notice: Reclaimed Channel (H34 T0 C1 oid=3)
  iscsid: session login failed with error 4,retryCount=3
  iscsid: Login Target Failed: iqn.1984-05.com.dell:powervault.md3000i.6002219000a14a2b00000000495e2886 if=iscsi_vmk@vmk8 addr=192.168.1.20:3260 (TPGT:1 ISID:0xf) err=4
  iscsid: Login Failed: iqn.1984-05.com.dell:powervault.md3000i.6002219000a14a2b00000000495e2886 if=iscsi_vmk@vmk8 addr=192.168.1.20:3260 (TPGT:1 ISID:0xf) Reason: 00040000 (Initiator Connection Failure)

Cause

This issue occurs because ESXi 5.0 attempts to connect to all configured or known targets from all configured software iSCSI portals. If a connection fails, ESXi 5.0 retries the connection 9 times. This can lead to a lengthy iSCSI discovery process, which increases the amount of time it takes to boot an ESXi 5.0 host.

Resolution

To minimize the amount of time the boot process spends discovering iSCSI targets, you can reduce the number of network portals and the number of targets.
To list the current number and configuration of an ESX host's network portals, run the command:
esxcli iscsi networkportal list
The output is similar to:
vmhba34:
Adapter: vmhba34
Vmknic: vmk6
MAC Address: 00:1b:21:59:16:e8
MAC Address Valid: true
IPv4: 192.168.1.206
IPv4 Subnet Mask: 255.255.255.0
IPv6:
MTU: 1500
Vlan Supported: true
Vlan ID: 10
Reserved Ports: 63488~65536
TOE: false
TSO: true
TCP Checksum: false
Link Up: true
Current Speed: 10000
Rx Packets: 656558
Tx Packets: 111264
NIC Driver: ixgbe
NIC Driver Version: 2.0.84.8.2-10vmw-NAPI
NIC Firmware Version: 0.9-3
Compliant Status: compliant
NonCompliant Message:
NonCompliant Remedy:
Vswitch: dvSwitch0
PortGroup: DvsPortset-0
VswitchUuid: 26 46 30 50 c0 cf df 1e-52 ef ab d7 a2 ab 96 f9
PortGroupKey: dvportgroup-78003
PortKey: 1731
Duplex:
Path Status: active
Note: This is an example of one network portal (HBA34).
To list currently running targets, run the command:

vmkiscsi-tool -T vmhba##

For more information on reducing number of network portals and the number of targets, contact your array vendor”

Hope this throw some light to your problem!!!

Regards,

Seb - elgreco81

[Titans99]

Thanks for your input.  The NIC is the Broadcom NetXtreme II BCM5709 1000Base-T, and the wierd thing is ... if I use the iSCSI software intiator all LUN's show up fine.  It's only when using the hardware initiator that I get nothing (other than errors in the log).  I may need to open a ticket with VMware.

[lombardialess]

Have you got receive an answer from vmware support? I've got a similar issue with a hp dl385 g7 and 1 hp p2000 g3

It drives me crazy!!!!

[Titans99]

Nothing from them yet. I actually have a HP P2000 G3 as well and the hardware initiator works fine with that unit.  I ended up having to use the software initiator for the enhance-tech unit because I couldn't wait any longer.  So I'm using both now ugh.

[JRedmond]

I've had the same issue.  What a surprise I am running HP ProLiant DL385 G7 with a P2000 G3.  I'm actually dealing with some freezing issues that I am trying to resolve as well.  Going to switch to the iSCSI Software Adapter and see if that helps.

[Titans99]

I was using both for a few weeks but support told me using both hardware and software initiator simultaneously was not supported.  Last week I switched totally over to the software initiator.  It seems to work fine but it would be nice to utilize the offload capabilities.

By the way JRedmond, are you using Jumbo Frames?  I have not enabled it yet but thinking about it.

[JRedmond]

I do not have Jumbo Frames enabled.  From my knowledge it is not worth using Jumbo Frames unless you are utilitizing 10Gb iSCSI.  This is from what I have read and from HP engineers that I've worked with in the past.  I wonder if people with an Intel processor G7 have the same issue or it is with the P2000 G3 and ToE in general.

[Titans99]

Same here, I read mixed reviews and there is just no clear cut answer.  Seems wierd to me for such a mature technology.

[lombardialess]

After all of my tests i found a good workaround....seems to be an esxi iscsi bug and the vmware support don't give an official answer.

Try to make 2 vswitch, one for each vmk port.

Configure in the software iscsi adapter all the ip of the host port of HP P2000 g3 in the dynamic discovery section.

Don't bind the vmk port....it do automatically!

It's not the best practice....but the best practice don't work!!!!

Performance is ok and no errors are logged.

look also here:

http://vmtoday.com/2012/02/vsphere-5-networking-bug-affects-software-iscsi/

and here:

http://vmtoday.com/2012/02/vsphere-5-networking-bug-2-affects-management-network-connectivity/

Bye

Alex

[Titans99]

Thanks lombardialess... excellent links.  That is exactly what I ended up doing - creating two separate vSwitches.

[eryk81]

There is defiantly a bug connecting to an iSCSI array on a second subnet but I found a solution that fits my needs. Maybe it will help.

Background:

I was able to ping the iSCSI array from both test host. I was able to 'nc' to port 3260. The host logs showed error 4, transportation error.

I update all firmware on all servers to the most current. It fixed some performance and boot issue but not the iSCSI issue.

Updated the hosts to ESXi 5.0 U2 and installed the recommended NIC drivers. Issue persisted.

Moved the servers to the same subnet and iSCSI worked. Moved back and issue came back.

Installed Windows and Linux guest on the hosts and setup iSCSI initiator software. I was able to connect through the host to and iSCSI target on a different subnet but I couldn’t connect from the host itself.

Tested with a different server and ESXi 5.0 build. Issue persisted.

To resolve the issue, I turned off NIC teaming/failback for all of the configurations sets in the vSwitch that is going to communicate with the iSCSI array. Not just the switch but the VMkernel too.

Next, I made sure that no NIC/vSwitches where bound to the iSCSI software adapter.

This resolved my issue. I tested vMotion, failover, reboot, etc. and everything is working well.

Reading this thread, this won’t work for some of the more complicated setups but for simple setups, this will hopefully resolve your connection issues.