Broadcom iSCSI Adapter - BCM 5709 not working with... - Page 2

Tomek24VMWARE · ‎07-14-2010

Hi All

I testing new ESXi on my server Dell R710 with 4 TOE port BroadCom Ethernet adapter.

000:001:00.0 Network controller: Broadcom Corporation PowerEdge R710 BCM5709 Gigabit Ethernet

000:001:00.1 Network controller: Broadcom Corporation PowerEdge R710 BCM5709 Gigabit Ethernet

000:002:00.0 Network controller: Broadcom Corporation PowerEdge R710 BCM5709 Gigabit Ethernet

000:002:00.1 Network controller: Broadcom Corporation PowerEdge R710 BCM5709 Gigabit Ethernet

This ethernet device have support for dependent BroadCom iSCSI Adapter.

when I configure vmk and NIC with vmhba34 - then iSCSI not working

when I configure vmk and NIC with Software iSCSI Adapter iSCSI working for me.

pleas see image:

and esxcli configuration for vmhba34

Does anyone have a similar problem and knows how to solve this?</span>

Tom

Andy_Banta · ‎08-04-2010

Aug 4 15:15:31 iscsid: ep_poll failed rc-1
Aug 4 15:15:31 vmkernel: 0:00:10:28.824 cpu9:4853)bnx2i::0x41000d404558: bnx2i_ep_disconnect: vmnic0: disconnecting ep 0x4100b00223d0 {1, 10a400}, conn 0x0, sess 0x0, hba-state 1, num active conns 1
Aug 4 15:15:31 iscsid: Login Failed: iqn.2010-05.pl.iri:p300q-d424-fff9055d8:dev1.ctr1 if=bnx2i-0026b98aecbd@vmk1 addr=10.1.1.1:3260 (TPGT:1 ISID:0x1) Reason: 00040000 (Initiator Connection Failure)
Aug 4 15:15:31 iscsid: Notice: Reclaimed Channel (H34 T1 C0 oid=2)
Aug 4 15:15:31 iscsid: Notice: Reclaimed Target (H34 T1 oid=2)
Aug 4 15:15:31 iscsid: DISCOVERY: Pending=0 Failed=2

So in this case, discovery is actually failing. The Broadcom driver is not making a connection to the target and therefore you're never seeing any sessions. It's as if there isn't connectivity to the target.

Aug 4 15:15:26 iscsid: SessionResolve for 10.1.1.1 (via vmk1) started)

Since the SessionResolve started and did not fail, it means there's something sitting at 10.1.1.1, but it isn't necessarily an iSCSI target. Anything else on this network that might be using that address?

Can you send your esxcfg-vmknic -l (lower-case 'L') output to see how you vmkNICs are configured?

Would it be possible to generate a vm-support dump of this?

Andy

Tomek24VMWARE · ‎08-04-2010

~ # esxcfg-vmknic -l

Interface Port Group/DVPort IP Family IP Address Netmask Broadcast MAC Address MTU TSO MSS Enabled Type

vmk0 Management Network IPv4 10.10.1.11 255.255.248.0 10.10.7.255 00:1b:21:63:c9:90 1500 65535 true STATIC

vmk1 VMkernel IPv4 10.1.1.250 255.255.255.0 10.1.1.255 00:50:56:79:d5:56 1500 65535 true STATIC

You are say

Aug 4 15:15:31 iscsid: ep_poll failed rc-1
Aug 4 15:15:31 vmkernel: 0:00:10:28.824 cpu9:4853)bnx2i::0x41000d404558: bnx2i_ep_disconnect: vmnic0: disconnecting ep 0x4100b00223d0 {1, 10a400}, conn 0x0, sess 0x0, hba-state 1, num active conns 1
Aug 4 15:15:31 iscsid: Login Failed: iqn.2010-05.pl.iri:p300q-d424-fff9055d8:dev1.ctr1 if=bnx2i-0026b98aecbd@vmk1 addr=10.1.1.1:3260 (TPGT:1 ISID:0x1) Reason: 00040000 (Initiator Connection Failure)
Aug 4 15:15:31 iscsid: Notice: Reclaimed Channel (H34 T1 C0 oid=2)
Aug 4 15:15:31 iscsid: Notice: Reclaimed Target (H34 T1 oid=2)
Aug 4 15:15:31 iscsid: DISCOVERY: Pending=0 Failed=2So in this case, discovery is actually failing. The Broadcom driver is not making a connection to the target and therefore you're never seeing any sessions. It's as if there isn't connectivity to the target.
But I discover matrix when I add IP adres 10.1.1.1 in ISCSI Initiator vmhba34 Propertis (windows):
When I add Ip on tab Static Discovery I see:

Your Question is:
Since the SessionResolve started and did not fail, it means there's something sitting at 10.1.1.1, but it isn't necessarily an iSCSI target. Anything else on this network that might be using that address?

No, 10.1.1.1 use only by matrix, lan cable is connected direcclty between vmnic0 on Dell server and first port in matrix.

This is one Cat 5e cable between dell server and matrix, without switch.

In Matrix Log's I See only this message:

<div class="yui-dt-liner">INFO</div>	<div class="yui-dt-liner">Wed, 04 Aug 2010 15:17:29</div>	<div class="yui-dt-liner">[CTR1] iSCSI logout from iqn.1998-01.com.vmware:localhost:1394183217:34 (10.1.1.250:62380) was received, reason .</div>
<div class="yui-dt-liner">INFO</div>	<div class="yui-dt-liner">Wed, 04 Aug 2010 15:17:29</div>	<div class="yui-dt-liner">[CTR1] iSCSI login from iqn.1998-01.com.vmware:localhost:1394183217:34 (10.1.1.250:62380) succeeds.</div>
<div class="yui-dt-liner">INFO</div>	<div class="yui-dt-liner">Wed, 04 Aug 2010 15:17:29</div>	<div class="yui-dt-liner">[CTR1] iSCSI logout from iqn.1998-01.com.vmware:localhost:1394183217:34 (10.1.1.250:63083) was received, reason .</div>
<div class="yui-dt-liner">INFO</div>	<div class="yui-dt-liner">Wed, 04 Aug 2010 15:17:29</div>	<div class="yui-dt-liner">[CTR1] iSCSI login from iqn.1998-01.com.vmware:localhost:1394183217:34 (10.1.1.250:63083) succeeds.</div>
<div class="yui-dt-liner">INFO</div>	<div class="yui-dt-liner">Wed, 04 Aug 2010 15:17:20</div>	<div class="yui-dt-liner">[CTR1] iSCSI logout from iqn.1998-01.com.vmware:localhost:1394183217:34 (10.1.1.250:52524) was received, reason .</div>
<div class="yui-dt-liner">INFO</div>	<div class="yui-dt-liner">Wed, 04 Aug 2010 15:17:20</div>	<div class="yui-dt-liner">[CTR1] iSCSI login from iqn.1998-01.com.vmware:localhost:1394183217:34 (10.1.1.250:52524) succeeds.</div>
<div class="yui-dt-liner">INFO</div>	<div class="yui-dt-liner">Wed, 04 Aug 2010 15:17:20</div>	<div class="yui-dt-liner">[CTR1] iSCSI logout from iqn.1998-01.com.vmware:localhost:1394183217:34 (10.1.1.250:54458) was received, reason .</div>
<div class="yui-dt-liner">INFO</div>	<div class="yui-dt-liner">Wed, 04 Aug 2010 15:17:20</div>	<div class="yui-dt-liner">[CTR1] iSCSI login from iqn.1998-01.com.vmware:localhost:1394183217:34 (10.1.1.250:54458) succeeds.</div>

When I configure vSwitch1 for iSCSI vmk1 I put Ip adress 10.1.1.1 netmask 255.255.255.0 but default gateway i could't change from 10.10.7.254 to 10.1.1.254.

This is My iSCSI vSwitch1

How generate vm-support dump??

Andy_Banta · ‎08-04-2010

Tom,

The esxcfg-vmknic -l output looks good.

The logs from the Matrix show the two discovery sessions that would happen with each rescan, and they close immediately, as expected.

It doesn't look like there's an attempt to start a regular, full-feature session with the target.

Based on the ESX-side logs, the attempt to open a connection through the Broadcom driver fails.

Aug 4 15:15:31 vmkernel: 0:00:10:28.569 cpu23:4119)bnx2i::0x41000d404558: bnx2i_cm_connect_cmpl: cid 0 failed to connect 10000000

The return code simply means "Connection Failed" from the Broadcom firmware.

At this point, it looks like a negotiation problem between the Broadcom iSCSI initiator and the storage.

Do you have any capability to get a wire trace from between the two systems? I'd recommend opening a support request with VMware. I'll conjecture what you'll find is that the HCL certification for the Matrix didn't include Broadcom adapters.

How generate vm-support dump??

Type "vm-support" in a console or ssh troubleshooting session on the ESXi host. However, I think the logs pretty clearly point out that the connection setup from the Broadcom driver to the storage is failing. I don't see any configuration of your host that should cause that.

Andy

Tomek24VMWARE · ‎08-04-2010

Andy,

When i use Software iSCSI Adapter on port Broadcom or intel Card I see Lun and all is OK.

Only on dependent Broadcom iSCSI Adapter I have problem.

I did not have support in VMware I use only free edition of vSphere and I could'n open a support request in VMware.

Thanks for your Help

Andy_Banta · ‎08-04-2010

Tom,

When i use Software iSCSI Adapter on port Broadcom or intel Card I see Lun and all is OK.

Understood. This is clearly the Broadcom iSCSI driver not establishing the connection, not an SW initiator problem.

I did not have support in VMware I use only free edition of vSphere and I could'n open a support request in VMware.

If you know of anyone else using the same type of storage, it might be worth seeing if they have the same problem or if they're in a position to open a support request. Is it anything that can be pursued with Matrix?

If you can get a wire trace, that would still be useful information. Many intelligent switches allow port mirroring, and a wireshark download is free.

Andy

c3techteam · ‎10-04-2010

Hi Andy,

This link has another angle not looked at is the latency of a connection without jumbo frame:

http://www.vmadmin.co.uk/vmware/35-esxserver/252-esxihwswiscsijumbo

I had the same issue with our servers Dell R710 and decided to go with software iSCSI after reading the post above as CPU should be more than sufficient over the IOPS required for our guest machines.

Regards,

BigRob201110141 · ‎03-17-2011

Make sure you have your Vkernel configured for the same VLAN as the iSCSI on the switch.

BigRob

DaIceMan · ‎03-29-2011

Is there any update on this problem? We are having a similar issue. We have an MSA2324i G3 with 10Gbit interfaces to 2 procurve 2910al switches with relevant 10Gbit SFP+ modules and 3 DL380G7 with 2 dual Broadcom 5709 NICs (4 ports). The vmhba33-36 appear under the storage adapters as they should, using the bnx2i driver (dependant hardware). After configuring the vmk network with the relevant IPs for LUN detection, the kernel fails to detect all the LUNs (there are 3) showing only 1. The Storage and LUNs are being accessed correctly with the 4 blades (BL490) from our C3000 through the Qlogic QMH4602 (Hardware Independant) and the relevant LUNs have be correctly mapped to the new WWNs (tried explicit or default, no change).

No jumbo frames are enabled on switches, vswitches or interfaces (neither the Qlogic). Flow control is enabled on all Storage related ports.

Tomek24VMWARE · ‎12-01-2011

Hi All

I'm use now ESXi 5 patch2 but still have problem with dependent iscsi.

Software iscsi works but if You have controllers in Storage configured in 2 different subnets and differents vlan.

And You Have configured 2 or more vmkernels on differents subnets. Then Software iscsi try connect all vmkernels (from different subnets) to all Storage targets.

For Example:

If you have vmkernel:

vmk1 10.1.1.10/21

vmk2 10.2.1.10/21

2 different subnets in 2 different vlan

Storage with Active/Active

controller1 10.1.1.1/21

controller2 10.2.1.1/21

2 different subnets and different Vlan.

Software ISCSI try connect from vmk1 to controller 1 and controller2 (controller2 is in different subnets) and software shouldn't connect from vmk1 to controller2.

for vmk2 is exactly the same.

Software iscsi try connect from vmk2 to controller 1 and 2 (controller 1 is not available for vmk2 - becouse this is different subnet on separated VLAN)

Then boot ESXi process using iscsi software take more time to connect wrong controller!!!!!

Software ISCSI should look on mask nad vlan settings on vSwitch and connet only to available subnet mask network vmk1=controller1 and vmk2=controller2

BUG BUG BUG BUG BUG in ESXi Software ISCSI!!!!!

QuankLei · ‎01-09-2012

Hi Everyone,

Anyone has a solution for this problem yet?

I have a similar problem here. We have 2 R710 both have a add-on BCM 5709 Dual Port NIC (same as the build-in). I applied the lastest patch and Broadcom driver from VMWare. It is still not working well if we use HARDWARE iSCSI adapter. Every time when I try to create a new VM or access to NAS via iSCSI, It will lost connection to NAS and cause the host freeze. Then I have to reboot the host in order to make connection again. I even tried the Dell Customized Esxi 5.0, and it come out to be the same. But IFF I use SOFTWARE iSCSI adapter to connect, it works perfectly.

Can Anyone help! Thanks a lot!

Tomek24VMWARE · ‎01-17-2012

Any update in this issue?

I think VMware has somewhere, which depends iscsi controller users. Broadcom has its own protocol iscsi and it has nothing to do with the iscsi RFC

QuankLei · ‎01-18-2012

In these couple days, I updated the servers' BIOS and NICs' FW, but the problem is persisted. And I found a link, I am not sure if this help, but it does not work for me.

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=102564...

BTW, have you ever contact VMWare Tech Support for this issue?

Thx!

resteves · ‎02-03-2012

I have the same problem with NC532i and NC382m.

First, with a fresh 4.1 U2 installation it only detected the NC382m HBAs, then i upgraded the bnx2i driver to 1.9.1k.v41.1 and now it detects the 4 HBAs, but i can't make connection to the storage:

Feb 3 12:30:28 iscsid: Login Target: iqn.1986-03.com.hp:storage.msa2324i.0931d8bf7e if=bnx2i-d8d385590c04@vmk1 addr=192.168.11.1:3260 (TPGT:65536 ISID:0x5)
Feb 3 12:30:28 iscsid: Notice: Assigned (H33 T0 C0 session=7, target=2/5)
Feb 3 12:30:28 iscsid: SessionResolve for 192.168.11.1 (via vmk1) started)
Feb 3 12:30:28 iscsid: DISCOVERY: Pending=1 Failed=0
Feb 3 12:30:28 vmkernel: 0:00:05:36.765 cpu0:4772)<6>bnx2i::0x41000b403548: [02:00.01]: ISCSI_INIT passed
Feb 3 12:30:29 iscsid: DISCOVERY: Pending=1 Failed=0
Feb 3 12:30:30 iscsid: DISCOVERY: Pending=1 Failed=0
Feb 3 12:30:31 iscsid: DISCOVERY: Pending=1 Failed=0
Feb 3 12:30:32 iscsid: DISCOVERY: Pending=1 Failed=0
Feb 3 12:30:33 iscsid: DISCOVERY: Pending=1 Failed=0
Feb 3 12:30:34 iscsid: DISCOVERY: Pending=1 Failed=0
Feb 3 12:30:35 iscsid: DISCOVERY: Pending=1 Failed=0
Feb 3 12:30:36 iscsid: DISCOVERY: Pending=1 Failed=0
Feb 3 12:30:37 iscsid: DISCOVERY: Pending=1 Failed=0
Feb 3 12:30:38 iscsid: DISCOVERY: Pending=1 Failed=0
Feb 3 12:30:39 iscsid: DISCOVERY: Pending=1 Failed=0
Feb 3 12:30:40 iscsid: DISCOVERY: Pending=1 Failed=0
Feb 3 12:30:41 iscsid: DISCOVERY: Pending=1 Failed=0
Feb 3 12:30:42 iscsid: DISCOVERY: Pending=1 Failed=0
Feb 3 12:30:43 iscsid: DISCOVERY: Pending=1 Failed=0
Feb 3 12:30:44 iscsid: DISCOVERY: Pending=1 Failed=0
Feb 3 12:30:45 iscsid: DISCOVERY: Pending=1 Failed=0
Feb 3 12:30:46 iscsid: DISCOVERY: Pending=1 Failed=0
Feb 3 12:30:47 iscsid: DISCOVERY: Pending=1 Failed=0
Feb 3 12:30:48 iscsid: DISCOVERY: Pending=1 Failed=0
Feb 3 12:30:49 vmkernel: 0:00:05:57.332 cpu0:4772)bnx2i::0x41000b403548: bnx2i_ep_disconnect: vmnic1: disconnecting ep 0x41000d00c000 {0, 800020}, conn 0x0, sess 0x0, hba-state 1, num active conns 1
Feb 3 12:31:04 vmkernel: 0:00:06:12.334 cpu0:4104)<1>bnx2i::0x41000b403548: [9172]: ofld_timer: CONN_DISCON timeout
Feb 3 12:34:21 vmkernel: 0:00:09:29.272 cpu0:4772)<6>cnic: vmnic1: cnic_abort_prep stuck on CID 800020, aborting
Feb 3 12:34:21 iscsid: Login Failed: iqn.1986-03.com.hp:storage.msa2324i.0931d8bf7e if=bnx2i-d8d385590c04@vmk1 addr=192.168.11.1:3260 (TPGT:65536 ISID:0x5) Reason: 00080000 (Initiator Connection Failure)
Feb 3 12:34:21 vmkernel: 0:00:09:29.272 cpu0:4772)<1>bnx2i::0x41000b403548: please submit GRC Dump, NW/PCIe trace, driver msgs to developers for analysis
Feb 3 12:34:21 iscsid: Notice: Reclaimed Channel (H33 T0 C0 oid=2)
Feb 3 12:34:21 iscsid: Notice: Reclaimed Target (H33 T0 oid=2)
Feb 3 12:34:21 iscsid: DISCOVERY: Pending=0 Failed=1

resteves · ‎02-03-2012

Correction, i can connect with the HBAs from NC382m, but not with the ones from NC532i

resteves · ‎02-03-2012

I think the problem with NC532i (Broadcom 57711E) is that the iSCSI ports have different macaddress from the regular nic, the HBAs that now show up with the new driver have the identification correspondent to the nic macaddress and not the iSCSI port.

Information from HP Onboard Administrator:

Mezzanine Slot	Mezzanine Device	Mezzanine Device Port	Port Status	Interconnect Bay	Interconnect Bay Port	Device ID
Embedded	Flex-10 Embedded Ethernet
		Port 1	OK	Bay 1	Port 3	D8:D3:85:59:0C:00
		iSCSI Port 1	OK	Bay 1	Port 3	D8:D3:85:59:0C:01
		Port 2	OK	Bay 1	Port 11	D8:D3:85:59:0C:04
		iSCSI Port 2	OK	Bay 1	Port 11	D8:D3:85:59:0C:05
1	HP NC382m Dual Port 1GbE Multifunction BL-c Adapter
		Port 1	OK	Bay 2	Port 3	d8:d3:85:bb:af:48
		Port 2	OK	Bay 2	Port 11	d8:d3:85:bb:af:4a

ESXi Broadcom iSCSI Adaptors show up like this:

iSCSI Alias:bnx2i-d8d385590c00

iSCSI Alias:bnx2i-d8d385590c04

iSCSI Alias:bnx2i-d8d385bbaf48

iSCSI Alias:bnx2i-d8d385bbaf4a

Anyone with NC532i has been able to make the iSCSI HBAs to connect to a storage?

Tomek24VMWARE · ‎02-03-2012

I reported the problem to Broadcom, after a few emails I got a proposal for solving the problem. But the problem is not resolved, and I worked very hard with Broadcom. The problem is not resolved, and Broadcom closed the bug.

I have no words.

Together, we must report the problem to Broadcom, may then fix the Broadcom drivers.

Tom

resteves · ‎02-06-2012

Just opened a case with Broadcom, now let's see what they say.

tsimons · ‎02-06-2012

All-

I'm on the outskirts of this, but I'll toss in two cents... When I started this thread the dependent iscsi adapter for broadcom did NOT support Jumbo Frames. We did a side by side comparision of Dependent iSCSI w/o Jumbo vs iSCSI Initiator with Jumbo. We were shocked and moved OFF of the dependent iSCSI adapters back to Software iSCSI to gain Jumbo Frame functionality for the increased throughput per Jumbo packet

~Todd

resteves · ‎02-07-2012

Ok, acording with Broadcom, the HBA's from NC532i card don't show up because for LOM cards is necessary a special license. I must contact HP regarding this license.

QuankLei · ‎02-07-2012

Thats a good news! How about BCM5709 iSCSI offload card?

All

Broadcom iSCSI Adapter - BCM 5709 not working with iSCSI