ESX4 with HP NC510F 10Gbit adapter

[Poort443]

I have a number of issues with the HP NC510F adapter. NFS datastores remain in "Inactive" state permanently. The datastores can be browsed, but that's all. VM's can't be started off it, and VMotions fail. The issues are resolved when I use a different adapter on the same vSwitch.

I've also had nothing but trouble with this adapter on ESX3.5. The NFS stores were properly connected, but VM's would freeze or be otherwise unreachable after a few hours or maybe a day. This is why I wanted to upgrade to ESX4, hoping that these problems would be resolved. All these issues happen on several servers, but all with the NC510F.

I'm working with VMware support to resolve these problems, but my question really is: Is anybody using this NC510F succesfully on ESX4?

This adapter is on the I/O Compatibilty Guide, and I'm using the driver that's stated there, the nx_nic 4.0.301

[Poort443]

Anybody using the NC510F? Don't tell me I'm the only one..:|

[d3aj]

Please paste the vmkernel log output when loading the nx_nic driver module

[Poort443]

Attached the vmkernel log. There's actually nothing in it related to loading nx_nic. There also isn't anything in it about loading bnx2 (the broadcom adapter that's working fine). esxcfg-module -l does show it as loaded.

Could you please tell me if you're using the NC510F? That's as important to me as resolving the issue itself. I'm still working with VMware support for that.

Thanks.

[d3aj]

WARNING: NFS: 898: RPC error 12 (RPC failed) trying to get port for Mount Program (100005) Version (3) Protocol (TCP) on Server

That's not right...

Anyhow, could you check if you have backed up logs (vmkernel.1 perhaps?)

If not then reload the driver and there should be nx_nic related output in the vmkernel.

[Poort443]

I rebooted the host to make sure the module was loaded again. It does show some messages:

<4>nx_nic: Unable to destroy the rx context, code 17 Error: Timed Out**

The message is repeated for my other nc510f (vmnic3).

[d3aj]

You have not provided the logs that show the nx_nic module being loaded. I need to look at that to make an assessment. You mentioned you rebooted the system but looking at the attached logs I do not see the system being rebooted. Watchdog is kicking, and there are tx timeouts.. There is certainly something wrong with the netxen card/driver. Until I see the logs with the module being loaded I wont be able to help you.

[Poort443]

The only reference in a vmkernel log to loading the nx_nic is from a week ago, in vmkernel.27 The other more recent logs show nothing for nx_nic. If I run esxcfg-modules -l the nx_nic does however show up as loaded..

In the meantime I installed a vmware driver CD, the VMware ESX/ESXi 4.0 Driver CD for QLogic Intelligent Ethernet Adapters. This changed my vmware driver and firmware versions. The vmware driver is newer (now: 4.0.404 before: 4.0.301), and oddly enough the firmware seems older (now: 3.4.339 before:4.0.222. HP however reports that firmware and driver are matched: I quote from another thread:

Information

The HP website is not containing any firmware for the NetXen based HP 10 GB network cards (NIC) like the NC510 and the NC512.

Reports have been submitted to HP support where people where trying to use the Redhat or SLES firmware package from the HP website in combination with the NetXen boot ISO /CD. The HP NetXen based 10 GB network card firmware was placed on a USB key or was added to the NetXen ISO / CD and was used for trying to flash and upgrade the HP 10 GB NIC firmware

NetXen based 10 GB network cards are using host-based firmware. This means that the Vmware driver consits of a driver part and a firmware image part. The moment the Vmware driver is loaded the host based firmware on teh NetXen card is automaticcally running the firmware level as supplied in the Vmware driver.

Due to the above it is not required to update the HP 10 GB based NetXen cards with a firmware package meant for Redhat or SLES

It all makes no difference unfortunately, the card still does not work at all.

[d3aj]

The discrepancy with the firmware versions happens due to the firmware being dynamically loaded onto the card. Part of the firmware resides in the driver.

[Poort443]

I've loaded the driver manually again, there are now some entries in the vmkernel log. Loading the driver was done at november 15, 11:40. I'd appreciate if you could look at these.

[d3aj]

Thanks Poort. I will look into this.

Here's a more technical explanation of the firmware versions:

"The vmware driver is newer (now: 4.0.404 before: 4.0.301), and oddly

enough the firmware seems older (now: 3.4.339 before:4.0.222."

Since your NX2031 card is flashed with an

older firmware 3.4.336 the firmware is loaded from the driver (fw file). The older

driver 4.0.301 has version NX2031 firmware file 4.0.222 and the newer

driver 4.0.404 has firmware version 3.4.339. This is not an issue as it behaves correctly, only a difference in versions.

[Poort443]

I managed to get a bit further after months of VMware support, new firmware, new drivers and ESX4 in stead of ESX3.5. In the end I never got this card working reliably.:( I've finally given up on it. It either should not have been on the HCL, or there's something else completely in my infrastructure that messes up the 10G.

We've decided to go with another 10G card. An expensive decision that I really didn't want, and it made us look bad in the eyes of the customer. I'm researching now which 10G adapter works reliably with ESX4. I'd be happy with suggestions. My advice to anybody looking into 10G is to stay away from the NC510F.

-Poort

[smora]

Did you test the HP NC510F in 4.1 U1 or finally you go away from that adapter? I am suffering similar problems:

http://communities.vmware.com/message/1732120#1732120

and nobody tell me some clue to point to the right way.