Driver for Intel X540-AT2 and Intel Corporation 4-Port SATA/SAS Controller

[MAndree]

Hello together,

i'm currently evaluating VMWare vSphere for our company and got stuck on the installation of two devices in an ESXi 5.1 host running on an Intel S2600GZ server board, where both are essential for the evalution:

The first is an Intel X540-AT2 network card, that was installed after the installation of the ESXi host (perhabs the problem?). Looking into the dmesg output, here is what i got:

/etc/vmware # lspci | grep X540
00:07:00.0 Network controller: Intel Corporation Ethernet Controller X540-AT2 [vmnic6]
00:07:00.1 Network controller: Intel Corporation Ethernet Controller X540-AT2 [vmnic7]

And for getting VID and DID:

/etc/vmware # lspci -p | grep 00:07
00:07:00.0 8086:1528 8086:0001  10/ 10/0xd8 B V              vmnic6
00:07:00.1 8086:1528 8086:0001  11/ 11/0x21 A V              vmnic7

I saw that there was no driver attached, so i went to the VMware Compatibility Guide and found this ixgbe driver, what perfectly matched my device.

But after downloading and installing the new driver nothing changed. I checked the installation to be sure:

/etc/vmware # esxcli software vib list | grep ixgbe
net-ixgbe                      3.11.32-1OEM.500.0.0.472560         Intel   VMwareCertified   2012-12-20

and as a last approach i thought that looking into dmesg would bring more light into the problem:

/etc/vmware # dmesg
...
2012-12-20T15:49:33.953Z cpu25:8786)<6>Intel(R) 10 Gigabit PCI Express Network Driver - version 3.11.32iov
2012-12-20T15:49:33.953Z cpu25:8786)<6>Copyright (c) 1999-2012 Intel Corporation.
2012-12-20T15:49:33.953Z cpu25:8786)PCI: driver ixgbe is looking for devices
2012-12-20T15:49:33.953Z cpu25:8786)DMA: 609: DMA Engine 'vmklnxpci-0:7:0.0' created using mapper 'DMANull'.
2012-12-20T15:49:33.953Z cpu25:8786)DMA: 609: DMA Engine 'vmklnxpci-0:7:0.0' created using mapper 'DMANull'.
2012-12-20T15:49:33.953Z cpu25:8786)DMA: 609: DMA Engine 'vmklnxpci-0:7:0.0' created using mapper 'DMANull'.
2012-12-20T15:49:33.953Z cpu25:8786)DMA: 654: DMA Engine 'vmklnxpci-0:7:0.0' destroyed.
2012-12-20T15:49:33.953Z cpu25:8786)LinPCI: LinuxPCI_DeviceUnclaimed:147: Device 0000:07:00 unclaimed.
2012-12-20T15:49:33.954Z cpu25:8786)DMA: 609: DMA Engine 'vmklnxpci-0:7:0.1' created using mapper 'DMANull'.
2012-12-20T15:49:33.954Z cpu25:8786)DMA: 609: DMA Engine 'vmklnxpci-0:7:0.1' created using mapper 'DMANull'.
2012-12-20T15:49:33.954Z cpu25:8786)DMA: 609: DMA Engine 'vmklnxpci-0:7:0.1' created using mapper 'DMANull'.
2012-12-20T15:49:33.954Z cpu25:8786)DMA: 654: DMA Engine 'vmklnxpci-0:7:0.1' destroyed.
2012-12-20T15:49:33.954Z cpu25:8786)LinPCI: LinuxPCI_DeviceUnclaimed:147: Device 0000:07:01 unclaimed.
2012-12-20T15:49:33.954Z cpu25:8786)PCI: driver ixgbe claimed 0 device
2012-12-20T15:49:33.954Z cpu25:8786)Mod: 4485: Initialization of ixgbe succeeded with module ID 27.
2012-12-20T15:49:33.954Z cpu25:8786)ixgbe loaded successfully.
...

Didn't help me much, but i hope you can get something from that.

On the second device, an Intel Corporation Patsburg 6 Port SATA AHCI Controller, I ran the same approach:

/etc/vmware # lspci -p | grep 00:09
00:09:00.0 8086:1d69 8086:3583  11/ 11/0xd0 A V              vmhba3
00:09:00.3 8086:1d70 8086:3582   5/   /     C V

found driver rste (build in into 5.1), but as well, this driver didn't claim the device (perhabs because it is running in sata mode?):

/etc/vmware # dmesg
...
2012-12-20T15:49:42.137Z cpu2:8852)Loading module rste ...
2012-12-20T15:49:42.138Z cpu2:8852)Elf: 1852: module rste has license BSD
2012-12-20T15:49:42.143Z cpu2:8852)Device: 90: Registered driver 'RSTe' from 68
2012-12-20T15:49:42.143Z cpu2:8852)Mod: 4485: Initialization of rste succeeded with module ID 68.
2012-12-20T15:49:42.143Z cpu2:8852)rste loaded successfully.
...

And i found this:

/etc/vmware # cat /var/log/vmkdevmgr.log | grep rste
2012-12-20T15:49:42.499Z| vthread-3| I120: Found driver rste for device bus=pci addr=00:09:00.0 id=8086:1d69 8086:3583.
2012-12-20T15:49:42.502Z| vthread-3| I120: Error binding driver rste for bus=pci addr=00:09:00.0 id=8086:1d69 8086:3583: No such file or directory

I really hope you guys get a clue on this, or you have another point i can look at.

Greets,

Markus Andree

[davez0r]

Hello Markus,

In the BIOS for the S2600GZ there should be a setting under Advanced => PCI Configuration => Memory Mapped I/O above 4 GB

Is that set to Enabled? If so, try setting it to Disabled and see if the ethernet controllers become available to ESXi.

I'm curious if this will solve your problem...

[MAndree]

Thank you very much davez0r,

that fixed the issue with the network card. Now we can test the system with full speed and there is only the RAID Controller, that isn't recognized yet. Any idea about that?

Greets,

Markus

[Alphabits]

Markus,

I am working with the same server model as you. It wasn't until I changed the "AHCI Capable SATA Controller" to "ENHANCED" that my drives on the onboard SAS/SATA controller became visible to ESXi.

Do not set this to "COMPATIBILITY" mode because that will use PATA emulation and you will suffer a noticable decrease in performance.

It is strange because it feels like I am changing a setting that should have no effect on whether or not ESXi can detect my drives. I understand that it cannot support the RAID because it is technically a Software RAID, but as I saw (and you showed with your error logs) it appears to try and associate the "scsi-rste" driver and fails to do so when the setting I mentioned is in AHCI mode.

-Ben

[MAndree]

Thanks Ben,

your suggestion works like a charm. After i have set the AHCI Capable SATA Controller to ENHANCED mode and switched the SAS/SATA Capable Controller back to INTEL(R) RSTe, the drives get successfully recognized by the hypervisor.

If you have any further suggestions on this specific hardware setup, feel free to share it with me. As i'm very new on the ESXi platform, i'm happy for any help I can get. Therefor I have to thank you both again for your good help.

Greets, Markus

PS: I think we will stick with ESXi and buy the VMWare vSphere Standard Edition in the next 2 weeks, as the evaluation went very well, although the onboard controller wasn't functional.

[BarrySmoke]

I have this same configuration.  what is strange for me, is that I had both the 10 gig nic, and the sas/sata controller working before I did a firmware update(trying to fix other issues). For the life of me, i could not figure out why I lost seeing the drives, and the nic.

I could see the nic in the advanced/passthrough screen, but just not in configuring network.

This post helped tremendously!

Thanks guys...

[CMS201110141]

I know this reply is a little late, but I have to express a BIG Thank You to davez0r!

We did encounter a similar problem with a Fujitsu D2755 SFP+ 10 Gbit NIC (based on Intel 82599) and were pretty clueless until I saw the "LinPCI: LinuxPCI_DeviceUnclaimed:147:" messages in syslog and then found this post!

VMotion is real fun now 🙂

Thanks again,

CM

[kecbfc]

Hi, I can confirm the same problem and solution with an Intel S2400BB chassis, Intel X540-T2 NIC and "ixgbe version 3.21.4" device driver. As said, disabling "Advanced => PCI Configuration => Memory Mapped I/O above 4 GB" fixed the problem.

[nicolasross]

Thanks to google for this ! I've added a 10 gig 540x to an esxi host that didn't registered it. I would never have found out that I needed to change the AHCI controler to enhanced !!