Highlighted
Enthusiast
Enthusiast

HP Flex 10 - Received Dropped Packets

VMware Community,

We are having a strange issue, which so far VMware/HP support has not been able to solve.

We have an environment where we are using 2 x 10GBps and 2 x 9Gbps adapters on a single DVS.

The DVS supports ESxi Management, vMotion, and Data traffic for the VMs.

We have Resource allocation turned on as well.

The blades are G7s.

Our problem is we are constantly seeing dropped received packets in this cluster. After a lot of research we thought it may be firmware/driver related and went down a whole path of updating our chassis, hosts, drivers, etc to the latest supported config by HP.

Still we are getting dropped received packets. We can recreate it quite easily, by initiating a vMotion.

We've tried running on just the 2 x 10gig and just the 2 x 9gig adapters but this made no difference.

Anyone got any suggestions?

Nick

Tags (4)
43 Replies
Highlighted
Contributor
Contributor

SvenHerren,

Thanks for the update...I will update the drivers once HP releases them to the public.

0 Kudos
Highlighted
Contributor
Contributor

Glad I found this thread... I have a very similar issue with bl460 G7 blades with tagged VLANs connected to FlexFabric 10Gb/24.  The Gen8 blades do not have any problems and they are running ESXi 5.1.  I found that the issue seems to be worse when you install ESXi 5.1 on the G7's.  This firmware update needs to come out ASAP.

0 Kudos
Highlighted
Contributor
Contributor

Are you guys seeing the dropped packets in esxtop?  I seem to be having the same issue and i see the constant dropped packets in vCenter performance charts.  However esxtop shows that all is good.  So which one is correct and do I really have a problem?

Stock drivers with 5.1 on a Bl465c G7 blade:

driver: be2net
version: 4.1.334.0
firmware-version: 4.1.450.7
bus-info: 0000:04:00.2
0 Kudos
Highlighted
Enthusiast
Enthusiast

Hi,

I blogged about this issue:

New Emulex Drivers for HP VMware ESXi 5.0 - 4.2.327.0

http://wp.me/p2iAYI-b1

Highlighted
Virtuoso
Virtuoso

A few days ago HP released new firmware and drivers, including firmware/driver updates to the Emulex-based CNAs and a firmware update for Virtual Connect modules. I haven't seen anything obvious in the releases notes pertaining dropped packets. But seriously, they releases two Emulex firmware updates with a minimal version difference at the same time?!

Edit 2013/02/25:

HP removed the Emulex CNA firmware 4.2.401.6 so now there's only 4.2.401.605 available. Got me vibes like these again: http://communities.vmware.com/message/2200359#2200359


HP BladeSystem c-Class Virtual Connect Firmware
http://www.hp.com/swpublishing/MTX-a94c23fca5ac4a21a3b88aa931 VC 3.75
3.75 (19 Feb 2013)


OFFLINE Firmware image (.iso) for HP Emulex 10GbE Converged Network Adapters and 10GbE Network Adapters
http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=32...
4.2.401.6 (19 Feb 2013)

OFFLINE Firmware image (.iso) for HP Emulex 10GbE Converged Network Adapters and 10GbE Network Adapters
http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=32...
4.2.401.605 (19 Feb 2013)


VMware ESXi 5.0/5.1 Driver CD for Emulex OneConnect OCe10102 10GbE Adapter (Network Function)
http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=32...
4.2.327.0 (12 Feb 2013)

Btw. the NIC driver is exactly the same version and binary (md5sum matching) that was already released back in 2012-09-12 and not some newer version or whatever.

-- http://alpacapowered.wordpress.com
0 Kudos
Highlighted
Enthusiast
Enthusiast

Same problem here, HP DL580G7 server with NC550SFP NICs.

driver: be2net
version: 4.2.327.0
firmware-version: 4.2.401.6
bus-info: 0000:11:00.1

4 x 10ge ports and I see fewer drops on the iscsi interfaces with jumbo frames than on the interfaces used for vmotion, vmnetwork etc.

0 Kudos
Highlighted
Contributor
Contributor

Hi,


I just made a list with all the Firmware relases I know off:

4.2.401.605 (19 Feb 2013) -> unbekannt ob dieser nun alle Probleme lösen wird.

4.1.450.16  (8 Oct 2012) -> Fixed a VMware ESXi 5.0 PSOD -> HP link

4.1.450.7 (4 Sep 2012) -> div. Bugs / package drops -> HP link -> HP Advisory link

4.1.402.20 (26 Jun 2012) -> div. Bugs -> HP link

4.0.493.0a (11 Apr 2012) -> div. Bugs bei reboot HosT PSOD/ HP link verschwunden hier aber weiterhin Dokumentiert: -> wooditwork -> Link

4.0.360.15b (5 Mar 2012) -> VMware PSOD -> HP Advisory Link

2.104.281.0 (12 May 2011) -> VMware PSOD -> HP Link

2.102.440.0  (30 Sep 2010) -> Network Loose / HP link verschwunden  aber weiterhin bei der VMware Dokumentiert -> Link

Does soembody tried the newest release (4.2.401.605 (19 Feb 2013)) does this one still drops the packet's?

btw. I have the same problem

cheers michael

0 Kudos
Highlighted
Enthusiast
Enthusiast

Emulex Firmware 4.2.401.605

  • Download
  • Resolved Issues from Firmware Version 4.2.401.6:
    1. Fixed VMware PSOD seen on AMD based servers with BE2 based cards
    2. Fixed FW memory leak with multiple logins to a redirected target
Highlighted
Contributor
Contributor

This sound's good. I'm willing to try it once more. Believe me I was installing drivers/firmwares for the last 2 years and always something new came up. I hope the journey is over for a good long time..

cheers michael

0 Kudos
Highlighted
Hot Shot
Hot Shot

0 Kudos
Highlighted
Enthusiast
Enthusiast

Please remember that the new Emulex drivers that were released are currently not supported by HP.

0 Kudos
Highlighted
Contributor
Contributor

I'm running 12 CPU (Intel) BL460c G7 hosts (10 of them) and trying to install the correct H/W firmware and drivers for ESXi 5.1.

Attached is what I have configured. Though my processors are INTEL and not AMD should the firmware of my Emulex NC553i 10 Gb 2-port FlexFabric

Net Adapter be at  4.2.401.605 instead of 4.2.401.6 ?  Also has anyone  tried the newest Emulex driver mentioned  2 posts earlier?

Thanks!

0 Kudos
Highlighted
Enthusiast
Enthusiast

You are correct, the HP NC553i CNA (Emulex) should be running 4.2.401.605 instead of 4.2.401.6.

Resolved Issues in version 4.2.401.605 from Firmware Version 4.2.401.6:

  1. Fixed VMware PSOD seen on AMD based servers with BE2 based cards
  2. Fixed FW memory leak with multiple logins to a redirected target

The newly released Emulex drivers that were mentioned before (4.4.231.0) are currently not supported by HP.

Highlighted
Enthusiast
Enthusiast

Hi,

same here.

I have seen this issue on BL680c G7, also on BL680cG5 with VirtualConnect behind, but WITHOUT FLEX-10.

On the G5 servers I´m using Broadcom Mezz-Cards.

AND: This issues are since the upgrade from 5.0 to 5.1! VDS version (5.0/5.1) is equal.

If I try to connect my VDS with only one uplink per host, the drops are gone!

I have opened a SR@VMware about that.

I will keep you up2date.

Jo!



Johannes Strasser / SDDC Architect @ Porsche Informatik GmbH
Twitter: @jo_strasser
0 Kudos
Highlighted
Contributor
Contributor

We have absolutelly the same issue.

Not only with Flex10. I got this case because of vmotion performance troubles. Nevertheless the average migration duration is absolutely OK, particularly with our DL980G7 hosts we notice dramatical vm freezes during vmotions. Allthough we use a dedicated 10Gb NIC-Team directly connected to the NEXUS Cores for vMotion some VMs experience sync lags up to 40 seconds and worse. That's absolutelly inaccaptable. We are only allowed to migrate VMs at late evening.

I'm not really sure this problem refers to the "RX Packet Drop" topic, but after some days of debugging time and no outcome I have to clutch at every straw Smiley Wink

BTW: Greetings from Bergheim @Jo

Manuel

Highlighted
Enthusiast
Enthusiast

Hi at all,

my SR at VMwareSupport is closed now.

My issue is only a cosmetic one, drops are shown but not really there Smiley Wink

I can ignore the shown drops at the vCenter side.

On the host side himself I can't see this drops.

This bugfix is addressed for 5.1 Update2, at the moment the release will be at the end of the year.

Greetz back, Manuel Smiley Happy

CU/JO



Johannes Strasser / SDDC Architect @ Porsche Informatik GmbH
Twitter: @jo_strasser
0 Kudos
Highlighted
Contributor
Contributor

Hi Jo,

How did you conclude that the errors where just cosmetic?

We are seeing the same kind of errors, but we cannot find any solution.

Thanks.

0 Kudos
Highlighted
Contributor
Contributor

Hi all,

as  many of us apparently, I had the same errors on ESXi 5.0 (same on different builds...)   :  plenty of "Receive packets dropped" ,  both on  BL 460c G7 and Gen8 since  several months. (All blades are in 2 stacked C7000 enclosures and members of a VC Flex10 domain,  10GbE back end)

this driver be2net-4.4.231.0-offline_bundle-1028063.zip (https://my.vmware.com/group/vmware/details?downloadGroup=DT-ESXI5X-EMULEX-BE2NET-442310&productId=28... )  fixed the problem for me  (+ HP firmwares up to date  with lastest HPSUM iso.)

Name    Version                    Vendor  Release Date  Acceptance Level  Status  

----------  -----------------------------  ------  ------------  ----------------  ---------

net-be2net  4.4.231.0-1OEM.500.0.0.472560  Emulex  2013-01-12VMwareCertified   Installed

PS : ESX update procedure,  follow this : Updating ESXi 5 - Single use esxcli How To

hope it can help Smiley Happy

Franck.

0 Kudos
Highlighted
Virtuoso
Virtuoso

yes, I'm running HP BL 465c G7 and G8 and here's the detailed version of firmware and the hardware model from vSphere console and from SSH console:

NIC: Emulex Corporation NC551i

HBA: ISP2532


~ # ethtool -i vmnic0

driver: be2net

version: 4.2.327.0

firmware-version: 4.2.401.6

bus-info: 0000:04:00.0


may I know what was the fix for this issue here ?

I'm detecting intermittent TCP resets and TCP retransmissions event from my Sniffer appliance (Riverbed SteelCentral / Cascade).

/* Any kind of comment or input would be greatly appreciated */
0 Kudos
Highlighted
Virtuoso
Virtuoso

ragmon the link that you specified on your blog or the post above is not working ?

Could not open page

The service or information you requested is not available at this time.

Please try again later.

(Error: system-websrv_unavail)

i'm trying to get the detailed information before I apply the firmware and driver to my production ESXi 5.1 Update 1 hosts.

/* Any kind of comment or input would be greatly appreciated */
0 Kudos