ixgbe updated module not loading

[lackhead]

Hey everybody-

Well, I've dived into ESX with both feet and have gotten my first server up and running. However, I have an Intel AF DA 10G card which I'd like to get working. I read up on how to install the updated ixgbe driver, including how to just install the driver rpm and avoid wedging your machine by doing a full "esxupdate update" (yes, I did try this too and verified that it does indeed wedge the box on boot). However, the updated ixgbe driver fails to load. Here's what I've tried:

1) fresh, virgin install of ESX 3.5 U3

2) copy over update iso, mount via loopback

3) rpm -ivh /cdmnt/VMupdates/RPMS/ VMware-esx-drivers-net-ixgbe-350.1.3.31.6-73033.i386.rpm

4) At this point, I try to load the module but get this:

# vmkload_mod ixgbe
Using /usr/lib/vmware/vmkmod/ixgbe.o
Failed to load module 'ixgbe': Invalid argument
Check system logs for more information.
#

/var/log/vmkernel reports the following failure:

Mar 6 11:15:19 gamma vmkernel: 0:00:13:41.782 cpu0:1024)Loading module ixgbe ...
Mar 6 11:15:19 gamma vmkernel: 0:00:13:41.782 cpu0:1024)Mod: 936: Starting load for module: ixgbe R/O length: 0x12000 R/W length: 0x7000 Md5sum: ba4ad736a6ad9fd56b2bd20a37ff0b04
Mar 6 11:15:19 gamma vmkernel: 0:00:13:41.914 cpu3:1039)Mod: 1373: Module ixgbe: initFunc: 0x9a8ec8 text: 0x9a3000 data: 0x2961480 bss: 0x2961e80 (writeable align 32)
Mar 6 11:15:19 gamma vmkernel: 0:00:13:41.914 cpu3:1039)<6>ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 1.3.31.5
Mar 6 11:15:19 gamma vmkernel: 0:00:13:41.914 cpu3:1039)<6>ixgbe: Copyright (c) 1999-2008 Intel Corporation.
Mar 6 11:15:19 gamma vmkernel: 0:00:13:41.914 cpu3:1039)PCI: driver ixgbe is looking for devices
Mar 6 11:15:19 gamma vmkernel: 0:00:13:41.914 cpu3:1039)PCI: driver ixgbe claimed 0 device
Mar 6 11:15:19 gamma vmkernel: 0:00:13:41.914 cpu3:1039)ALERT: Mod: 1413: Initialization for ixgbe failed with -19.
Mar 6 11:15:19 gamma vmkernel: 0:00:13:41.914 cpu3:1039)ixgbe failed to load with status 0, -19, 0xbad0001.

Now, I found another posting out there that talking about adding noapic as a kernel boot option, which is what I had to do to get ESX installed in the first place (booting off of an Adaptec raid volume), but if I leave noapic in as a boot option, the boot hangs when it tries to load the Adaptec driver.

Any ideas, suggestions, help, or condolensces would be most welcome!

-c

[mike_laspina]

Hi,

Did you follow this KB?

http://kb.vmware.com/kb/1004278

vExpert 2009

[lackhead]

Yes, I believe I have. At least, I have enabled Netqueue in the kernel (rebooted to do so), and set what I think are the appropriate options for my dual-port card:

# grep -i netque /etc/vmware/esx.conf
/vmkernel/netNetqueueEnabled = "TRUE"
# esxcfg-module -g ixgbe
ixgbe enabled = 1 options = 'InterruptType=2,2 VMDQ=16,16'

But I still can't load the module:

# vmkload_mod ixgbe
Using /usr/lib/vmware/vmkmod/ixgbe.o
Failed to load module 'ixgbe': Invalid argument
Check system logs for more information.
#

And I still get the -19 error on load:

Mar 9 04:16:32 gamma vmkernel: 0:00:03:07.700 cpu4:1041)ALERT: Mod: 1413: Initialization for ixgbe failed with -19.
Mar 9 04:16:32 gamma vmkernel: 0:00:03:07.700 cpu4:1041)ixgbe failed to load with status 0, -19, 0xbad0001.

Thanks for your help with this!

-c

[mike_laspina]

Hi,

Try changing the single ' to double " , many of the module load calls use perl which can have some parsing issues.

Regards,

Mike

vExpert 2009

[lackhead]

You mean for the esxcfg-module -s call? I did put them on the command line as double quotes..the single quotes are what is returned from the -g option:

# esxcfg-module -s "InterruptType=2,2 VMDQ=16,16" ixgbe
# esxcfg-module -g ixgbe
ixgbe enabled = 1 options = 'InterruptType=2,2 VMDQ=16,16'
#

I also checked /etc/vmware/esx.conf:

# grep -i ixgb /etc/vmware/esx.conf
/vmkernel/module/ixgbe.o/options = "InterruptType=2,2 VMDQ=16,16"
#

-c

[mike_laspina]

Yes that is what I was thinking.

It definetly looks like the driver is not accepting those options.

Does it load if you disable Netqueue and clear the options field in esx.conf?

If it does I would then add one at a time to see which one it fails on.

I suspect there is a patch/driver mismatch and you will probably have to work with the driver engineers.

Regards,

Mike

vExpert 2009

[lackhead]

Yeah, I tried that at first...I didn't find that KB article you referenced until things didn't work and I started poking around.

<sigh>

Ok, I will open up a support ticket and see what the engineers say. I'll post back what that turns up.

Thanks again for your help with this! It's nice to know I'm totally off-base in how this stuff is supposed to work.

-c

[mike_laspina]

Your welcome,

I was thinking you could exclude the InterruptType=2,2 since thats actually the default value.

The VMDQ is a custom parameter so it needs to be there for the vmk based version of ixgbe with netqueue enabled.

What I think they will probably tell you is that you must have Jumbo Frames enabled for it to work, but I'm not sure so it's better to talk to the SME's.

Regards,

Mike

vExpert 2009

[lackhead]

Just a quick update- I called tech support, and just got a jr. admin who wasn't much help. I've called back and will try to escalate the issue.

In the meantime, I installed 3.5 U2 and installed the ixgbe udpate to that and am running into the same errors. This makes me think that I'm running into something different than what has been reported with a bad update driver for this module in 3.5 U3. <sigh>

Anyway, just putting this in for posterity's sake.

-c

[mike_laspina]

While your waiting may I suggest trying the following.

First disable the module.

esxcfg-module -d ixgbe

Also run

rmmod ixgbe

Reboot ( if it detects new hardware do not configure it)

Run

esxcfg-module -e ixgbe

esxcfg-module -s "VMDQ=16,16" ixgbe

Reboot and run dmesg to see the hardware results.

Regards,

Mike

vExpert 2009