Broadcom 57711 + ESX/ESXi 4.1 + Jumbo Frames

[Box293]

Having some issues getting Broadcom 57711 NIC working on ESX/ESXi 4.1 with MTU 9000.

On ESX/ESXi 4.0 U2 I have no problems getting and MTU of 9000 working, I can push our EQL PS6010 SAN pretty hard and get about 550,000 KBps with IO Meter running inside 4 x VM's.

I have read documentation and have found that only an MTU of 1500 is supported for the 57711 NIC's on ESX/ESXi 4.1. This has somethig to do with the fact that these NICs have hardware iSCSI offloading, there are additional iSCSI adapters that appear under the storage adapters section on 4.1 hosts.

I have it all properly configured with 1:1 binding of the iSCSI nics to the iSCSI VMK ports using an MTU of 1500. When configured this was the max I can get out of the SAN is about 280,000 KBps. If I try the same process using an MTU of 9000 the VM's / host seem to stop responding.

While I understand that there is a limitation of 1500 when using the hardware iSCSI NICs I am unable to get the software iSCSI to work with an MTU of 9000 on ESX/ESXi 4.1. If I try software iSCSI using an MTU of 9000 the VM's / host seem to stop responding.

Is this also a limitation or am I missing something.

Is there anyone else out there experiencing the same problems as me?

VCP3 & VCP4 32846

VSP4

VTSP4

[milanmbk]

This Product Guide gives information about TCP Offload. Recommend everyone involved with throubleshooting to read this Guide and understand the technology before troubleshooting your environment. just my 2 cents

[rykelley]

thanks for posting the obvious config guides....any way. We are on HP BL490c G6's and seeing similar latency issues. the NC364m mezz. cards perform just fine. Using the Equallogic MEM you can disable the offloading. so i'm trying that next...I'll post results.

[jeffa35 · Accepted solution]

Looks like an updated Broadcom Driver CD (1.60.50.v41.2) was recently posted in the vsphere downloads/drivers&tools area. I have not tried it yet, hopefully it will resolve this issue.

[rykelley]

i'm hoping also but the driver also says this " (Note: The driver has not been certified to support iSCSI, and VMware does not provide support in iSCSI application unitl iSCSI certification is complete"

not that that means it wont fix the issues.

[Box293]

Awesome great pickup.

Have installed these drivers on my ESXi 4.1 test box and it seems to have fixed the problem!!!

I installed all four drivers that came in the bundle.

One thing I've noticed now is that when watching the ESXi host's performance graphs the iSCSI NICs are now showing traffic. Previously they did not and this indicated that the traffic was being handed to the hardware iSCSI (as per some of the EqualLogic documentation that comes with the multipathing module).

The tests I have run are pretty basic. 4 x VM's running IOMeter, 3 Outstanding IO queues and 32K 100% read tests. ESXi host would crash in the past when I tried powering on the second VM. Have not had one problem running with the new drivers. Will leave the tests running all weekend to give it a good work out.

On ESXi 4.0 Update 2 I was getting about 549,000 KBps read rate on the ESXi host. My EqualLogic PS6010XV SAN was on firmware 4.3.7.

Since then the SAN has been upgraded to firmware 5.0.2. With ESXi 4.1 and the latest Broadcom driver I am getting about 611,000 KBps read rate (peaking at 640,000).

I have not loaded the EqualLogic Multipathing Module on my ESXi host yet, will look into this next week.

Now I just need to wait until the drivers are certified by VMware before we put this into production.

Glad we finally got to a resolution with this, I wasn't going crazy :smileylaugh:

VCP3 & VCP4 32846

VSP4

VTSP4

[Box293]

Unfortunately we couldn't get our Dell blade servers with Intel NICs

I've never been a Broadcom fan.

VCP3 & VCP4 32846

VSP4

VTSP4

[Box293]

Just an update to say that the test VMs have been running IOMeter for 2.5 days solid without any issues.

SAN shows no disconnected iSCSI sessions, have been up for 2 days and 18 hours.

VCP3 & VCP4 32846

VSP4

VTSP4

[Box293]

One last update. Now there are new drivers I was curious if the hardware iSCSI would work with jumbo frames (documentation calls these Dependent Hardware iSCSI Adapters).

I'm not having much luck getting it to work. I have had this working before with an MTU of 1500 (without the newer drivers) but now it's not even consistently working with an MTU of 1500 on the newer drivers. Sometimes it presents the EQL volumes, other times it doesn't.

I'm not going to go down this path any more. Perhaps I will revisit it with the next major release of ESX once the technology has become more mature, but until then I'll stick to the good old fashioned VMware Software iSCSI initiator

VCP3 & VCP4 32846

VSP4

VTSP4

[randybw1]

Box293, I'm so confused, probably because I've been up jacking with this stuff a long time this a.m.

I thought after you updated the drivers, you had Jumbo Frames working with the Broadcom cards? The last post says you don't even have them stable with regular 1500 MTU? I've updated one of mine to the newer drivers and still can't get Jumbo working. I'll post more info on that later.

Thanks

[Box293]

Hey randybw1,

These Broadcom 57711 nics have an iSCSI offloading engine. The "engine" was not active in ESX 4.0.x however in ESX 4.1.x they introduced support for it. How this works is that under the storage adapters section of the vSphere client you would see additional Broadcom iSCSI hardware initiators (hwiSCSI), there will be as many of these hwiSCSI adapters as there are physical 57711 NICs. VMware refers to these hwiSCSI adapters as Dependent Hardware iSCSI Adapters because technically they are not a dedicated hardware iSCSI adapter.

This is different to the VMware software iSCSI initiator (swiSCSI). When using swiSCSI the ESX hosts CPU resources are used to process the traffic sent over the iSCSI network. When looking at the ESX hosts performance graphs for the NICs used for iSCSI you will actually see this traffic graphed.

When using hwiSCSI this traffic is passed onto the iSCSI offloading engine and hence the ESX hosts CPU is not used. When looking at the ESX hosts performance graphs for the NICs used for iSCSI you will NOT see this traffic graphed as the traffic has been handed to the offloading engine.

So the original problem I had when I created this thread is that when I used the swiSCSI initiator with jumbo frames on ESX 4.1.x the ESX host would lose connectivity to the SAN (under a load). Additionally I could NOT see any traffic being graphed for my hosts NICs that were used for the iSCSI traffic.

In my opinion I could see that the problem here is that the drivers for the 57711 NICs that came as part of ESX 4.1 260247 were broken and were always passing the traffic onto the iSCSI offloading engine. This is why it all worked ok with an MTU of 1500 as it was documented that jumbo frames were not supported on the 57711 NICs when using the hwISCSI.

After I installed the newer drivers the ESX hosts performance graphs for the NICs used for iSCSI now had traffic graphed.

So my most recent post was about me trying to see if the hwiSCSI supported jumbo frames with the 57711 NICs. I didn't think so but I thought I would try as I do like to try the newer technologies. However in saying that I will probably wait until the next major release of ESX before attempting to use hwiSCSI again. Additionally I may not have configured them correctly so waiting a while will help as others will post blogs about the correct way to set them up etc.

So to answer your question, the new drivers do work with jumbo frames when using swiSCSI.

Here are some tips to help you resolve your problem.

1) Does all of this work on ESX 4.0 Update 2? You need to install updated drivers on ESX 4.0 Update 2 however there is no hwiSCSI so this removes any of that as being a cause of your problems. Confirming it all works on ESX 4.0 Update 2 will confirm in your mind that all of the equipment is correctly configured to support jumbo frames (switch port settings etc).

2) Did you install all 4 drivers that came with the download?

3) Here is how I can quickly get a host up and running talking to the SAN with the new drivers. You will obviously need to change IP addresses, vmnic names and vmhba?? name to reflect your environment but this is the minimum amount of steps required to make it work (my iSCSI SAN is on the 192.168.102.* network):

• Install ESXi 4.1 from scratch on your host

• Give the host an IP address (I'll use 192.168.110.63)

• Download the latest drivers from the VMware website

• You will need to upload the extracted drivers a vMA VM (say /tmp/drivers)

• login to the vMA vm

• vifp addserver 192.168.110.63

• vifptarget -s 192.168.110.63

• vihostupdate --server 192.168.110.63 -i -b /tmp/drivers/offline-bundle/BCM-bnx2-2.0.15g.8.v41.1-offline_bundle-320302.zip

• Wait while the package is installed

• vihostupdate --server 192.168.110.63 -i -b /tmp/drivers/offline-bundle/BCM-bnx2i-1.9.1k.v41.1-offline_bundle-320302.zip

• Wait while the package is installed

• vihostupdate --server 192.168.110.63 -i -b /tmp/drivers/offline-bundle/BCM-bnx2x-1.60.50.v41.2-offline_bundle-320302.zip

• Wait while the package is installed

• vihostupdate --server 192.168.110.63 -i -b /tmp/drivers/offline-bundle/BCM-cnic-1.10.2m.v41.1-offline_bundle-320302.zip

• Wait while the package is installed

• logoff the vMA vm

• Reboot your ESXi host

• Once ESXi host has rebooted logon to the tech support console

• Type these commands in:

• esxcfg-vswitch -a vSwitch1

• esxcfg-vswitch -m 9000 vSwitch1

• esxcfg-vswitch -A iSCSI-vmnic6 vSwitch1

• esxcfg-vmknic -a -i 192.168.102.98 -n 255.255.255.0 -m 9000 iSCSI-vmnic6

• esxcfg-vswitch -A iSCSI-vmnic7 vSwitch1

• esxcfg-vmknic -a -i 192.168.102.99 -n 255.255.255.0 -m 9000 iSCSI-vmnic7

• esxcfg-vswitch -L vmnic6 vSwitch1

• esxcfg-vswitch -L vmnic7 vSwitch1

• esxcfg-vswitch -p iSCSI-vmnic6 -N vmnic7 vSwitch1

• esxcfg-vswitch -p iSCSI-vmnic7 -N vmnic6 vSwitch1

• esxcfg-swiscsi -e

• esxcli swiscsi nic add -n vmk1 -d vmhba44

• esxcli swiscsi nic add -n vmk2 -d vmhba44

• Now open the vSphere client

• Goto the storage adapters section and do a refresh

• Select the VMware Software iSCSI adapter and click properties

• On the Dynamic Discovery tab add your SANs IP address

• Click OK and do a rescan when prompted

You should now be connected to your iSCSI SAN using jumbo frames.

Hopefully this should get you on the right track. Let me know if you need any more assistance.

VCP3 & VCP4 32846

VSP4

VTSP4

[randybw1]

Box293,

Thanks for the detailed info, great post and wealth of info there. I completely understand now. I'm new to all this and we are starting fresh with ESXi 4.1, new servers (Dell 810s), switches (Cisco Nexus 5ks), and San (Equallogic 6010xv).

We got everything working with the Broadcom 57711s hwiSCSI at 1500 MTU. Everything I read said Jumbo Frames, so that was the next step, I found tons of info but couldn't get it working either. I still haven't tried the swiSCSI with anything settings, I had it in my head that hwiSCSI was the way to go to keep as much off the CPUs as possible.

I got the Jumbo Frames setup on the switches, verified everything, from end to end as all the documentation states to do, but still couldn't get them to work. Found this thread and a few others and figured out it was the Broadcom cards. Thought the new drivers were the answer, saw your post, got excited, updated the drivers and I'm sad once again.

Anyway, we are looking at getting Intel cards to replace them, if that doesn't happen, I guess I will try to figure out the best setup for us.

[rykelley]

Hi randybw1

have you seen a massive slow down and huge amount of latency when cloning and deploying?

[randybw1]

No I have not, I'm not doing much with it, but I have moved and cloned and all seems good. This a.m. I changed to swiSCSI and Jumbo Frames, we'll see how this works for now.

[David_Ulbrich]

-Broadcom 57711.

-Can be used in two modes regarding iSCSI:

1) As a NIC, carrying swISCSI traffic.

2) As a dependent hwiSCSI initiator.

Please note VMware KB Article: 1007654

"

You cannot use Jumbo Frames on a Broadcom card that is configured as a hardware initiator performing iSCSI offload functions. You can either use Jumbo Frames or iSCSI Offload and you cannot use both together with the Broadcom adapters.

"

http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&externalId=1007654&sl...

So, you choose option 2 above (hsiSCSI offload), you cannot use jumbo frames per the KB.

[Box293]

I cannot understand why you would use iSCSI offloading at the expense of jumbo frames. In my testing this results in a 50% performance drop in SAN throughput.

I don't understand why Broadcom would invest their resources in technolgoy that isn't going to be used by the consumer.

My biggest winge is that when buying Dell BLADE servers you don't get a choice between Intel and Broadcom NICs. I can understand that they can't offer you a choice with the onboard NICs on Fabric A however Fabric B and C are daughterboard cards so any type of I/O technology can be impletemented. I suppose it all comes down to which manufacturer will sell their technology to Dell for the smallest price.

Now I have this problem where I need to get Dell to find out when these drivers will be certified and unitl then I cannot put ESX 4.1 into production.

And while I am having a winge one other thing. Broadcoms statement on the drivers download page is as follows:

Note: This driver CD release includes version 1.60.50.v41.2 of the bnx2x driver on ESX/ESXi 4.1. The bnx2x driver supports products based on the Broadcom NetXtreme II BCM57710/BCM57711/BCM57711E 10/100/1000/2500/10000 Mbps PCIE Ethernet Network Controllers. (Note: The driver has not been certified to support iSCSI, and VMware does not provide support in iSCSI application unitl iSCSI certification is complete).

Issue 1) No release notes stating what is fixed

Issue 2) It only talks about the bnx2x driver. There are four drivers included in the download. Is this just the 10 GB NIC driver (I am assuming so)? Do we need to install all four drivers in the download or is only one of them an update.

Issue 3) The driver has not been certified to support iSCSI, and VMware does not provide support in iSCSI application unitl iSCSI certification is complete. iSCSI what ... hardware iSCSI or software iSCSI? Let's be a little more specific Broadcom.

Broadcom need to stop hiding behind being an OEM and provide real support for the products they sell. Even their support page FAQ's doesn't have any mention about ESX 4.1 and it has been released for about four months now.

Anyway it's good to have a rant every now and then!

VCP3 & VCP4 32846

VSP4

VTSP4

[Box293]

rykelley,

We will be upgrading our vCenter to 4.1 shortly and then we'll be able to add our ESXi 4.1 hosts and do some more testing.

When you talk about a massive slow down and huge latency, exactly how do you measure these? Are there some performance graphs you are looking at which indicate the latency and if so what sort of numbers are you seeing?

I'll be able to compare results with you.

VCP3 & VCP4 32846

VSP4

VTSP4

[HFELDMAN]

@box293 Great Post!

I have been following this post and tried your suggestion below however I was unable to negotiate jumbo frames. Did you flash your 57711 with a firmware update or anything else or did you just do the driver update. Did you see your Equallogic actually connect successfully using Jumbo Frames? I am going to open up case with all the vendor's today. Thanks for the input.

Install ESXi 4.1 from scratch on your host

Give the host an IP address (I'll use 192.168.110.63)

Download the latest drivers from the VMware website

You will need to upload the extracted drivers a vMA VM (say /tmp/drivers)

login to the vMA vm

vifp addserver 192.168.110.63

vifptarget -s 192.168.110.63

vihostupdate --server 192.168.110.63 -i -b /tmp/drivers/offline-bundle/BCM-bnx2-2.0.15g.8.v41.1-offline_bundle-320302.zip

Wait while the package is installed

vihostupdate --server 192.168.110.63 -i -b /tmp/drivers/offline-bundle/BCM-bnx2i-1.9.1k.v41.1-offline_bundle-320302.zip

Wait while the package is installed

vihostupdate --server 192.168.110.63 -i -b /tmp/drivers/offline-bundle/BCM-bnx2x-1.60.50.v41.2-offline_bundle-320302.zip

Wait while the package is installed

vihostupdate --server 192.168.110.63 -i -b /tmp/drivers/offline-bundle/BCM-cnic-1.10.2m.v41.1-offline_bundle-320302.zip

Wait while the package is installed

logoff the vMA vm

Reboot your ESXi host

Once ESXi host has rebooted logon to the tech support console

Type these commands in:

esxcfg-vswitch -a vSwitch1

esxcfg-vswitch -m 9000 vSwitch1

esxcfg-vswitch -A iSCSI-vmnic6 vSwitch1

esxcfg-vmknic -a -i 192.168.102.98 -n 255.255.255.0 -m 9000 iSCSI-vmnic6

esxcfg-vswitch -A iSCSI-vmnic7 vSwitch1

esxcfg-vmknic -a -i 192.168.102.99 -n 255.255.255.0 -m 9000 iSCSI-vmnic7

esxcfg-vswitch -L vmnic6 vSwitch1

esxcfg-vswitch -L vmnic7 vSwitch1

esxcfg-vswitch -p iSCSI-vmnic6 -N vmnic7 vSwitch1

esxcfg-vswitch -p iSCSI-vmnic7 -N vmnic6 vSwitch1

esxcfg-swiscsi -e

esxcli swiscsi nic add -n vmk1 -d vmhba44

esxcli swiscsi nic add -n vmk2 -d vmhba44

Now open the vSphere client

Goto the storage adapters section and do a refresh

Select the VMware Software iSCSI adapter and click properties

On the Dynamic Discovery tab add your SANs IP address

Click OK and do a rescan when prompted

[Box293]

Hi HFELDMAN,

I've not needed to perform any flash upgrades on the 57711 NICs to make this work.

Yes, looking at the EqualLogic event logs I can see events that say iSCSI login using Jumbo Frame length.

Some suggestions to help you resolve your issue:

1) Try making it work on ESX 4.0 Update 2. You will need to install newer drivers to make this work from the VMware website (1.52.12.v40.3). I suggest this because 4.0 it doesn't have the hwiSCSI component and I was able to get jumbo frames working relatively quickly. The additional purpose here is that it confirms that you have enabled jumbo frames end to end.

2) Jumbo frames end to end. I originally had a problem where I thought I had enabled jumbo frames end to end. I have 2 x 8024F switches connected together and I discovered that the LAG between the two switches did not have the jumbo frames enabled. Even though these ports had jumbo frames enabled the LAG settings override the port settings. Additionaly when you make a change to the LAG you need to physically disconnect the all the LAG port members for these settings to be applied to the NICs.

That's all I can think of now. I hope these can be of help.

VCP3 & VCP4 32846

VSP4

VTSP4

[HFELDMAN]

Box293,

Actually your suggestion did it we found we were missing an MTU on one of the LAGs. What I also subsequently found is that we were able to connect at jumbo frames even without the updated driver. For us it was a simple switch misconfiguration and the way we tested that was to ping through each point with a 8000 datagram size or larger.

On a side note when we opened the case with Dell they immediately wanted to swap out the 57711 with intels x520s no questions asked (20 in total). They know there is an issue with the iSCSI offload and 57711 somewhere.

[Box293]

Glad to be of help, that MTU LAG issue cost me about 4 days.

I can only dream of Intel x520s NIC ... my blades don't have an option for Intel NIC daughterboard cards :_|

VCP3 & VCP4 32846

VSP4

VTSP4