ESX 3.5 Update 3 - ICH7 storage

[markiusuk]

Hi all

I'm having a problem with my lab system ever since installing Update 3... I can't access one of my local disks any more.

The disk is attached to the ICH7 SATA controller on my system (Dell PowerEdge 840). Since Update 3, this controller is managed by the IDE driver ("Uniform Multi-Platform E-IDE driver Revision: 7.00beta4-2.4") instead of the ATA_PIIX driver.

I've added the appropriate entry to /etc/vmware/pciid/ata_piix.xml and run esxcfg-pciid which has correctly updated the various files (pci.xml, pcitable etc) and rebooted but still no disk.

I've had a look through the logs and it seems as though the problem is now that the kernel IDE driver has already got hold of the ICH7 controller before the ATA_PIIX driver starts:

/var/log/messages

Feb 11 19:58:53 maggie kernel: Uniform Multi-Platform E-IDE driver Revision: 7.00beta4-2.4

Feb 11 19:58:53 maggie kernel: ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx

Feb 11 19:58:53 maggie kernel: ICH7: IDE controller at PCI slot 00:1f.1

Feb 11 19:58:53 maggie kernel: ICH7: chipset revision 1

Feb 11 19:58:53 maggie kernel: ICH7: not 100% native mode: will probe irqs later

Feb 11 19:58:53 maggie kernel: ide0: BM-DMA at 0xfc00-0xfc07, BIOS settings: hda:DMA, hdb:pio

Feb 11 19:58:53 maggie kernel: ICH7-SATA: IDE controller at PCI slot 00:1f.2

Feb 11 19:58:53 maggie kernel: ICH7-SATA: chipset revision 1

Feb 11 19:58:53 maggie kernel: ICH7-SATA: 100% native mode on irq 20

Feb 11 19:58:53 maggie kernel: ide1: BM-DMA at 0xcc60-0xcc67, BIOS settings: hdc:DMA, hdd:pio

Feb 11 19:58:53 maggie kernel: ide2: BM-DMA at 0xcc68-0xcc6f, BIOS settings: hde:pio, hdf:pio

Feb 11 19:58:53 maggie kernel: hda: TSSTcorp CD-RW/DVD-ROM TS-H492C, ATAPI CD/DVD-ROM drive

Feb 11 19:58:53 maggie kernel: hdc: ST3500630AS, ATA DISK drive

Feb 11 19:58:53 maggie kernel: blk: queue c04b4fd8, I/O limit 4095Mb (mask 0xffffffff)

Feb 11 19:58:53 maggie kernel: ide0 at 0x1f0-0x1f7,0x3f6 on irq 14

Feb 11 19:58:53 maggie kernel: ide1 at 0xcc98-0xcc9f,0xcc92 on irq 20

Feb 11 19:58:53 maggie kernel: hdc: attached ide-disk driver.

Feb 11 19:58:53 maggie kernel: hdc: host protected area => 1

Feb 11 19:58:53 maggie kernel: hdc: 976773168 sectors (500108 MB) w/16384KiB Cache, CHS=60801/255/63

Feb 11 19:58:53 maggie kernel: hda: attached ide-cdrom driver.

Feb 11 19:58:53 maggie kernel: hda: ATAPI 48X DVD-ROM CD-R/RW drive, 1536kB Cache

Feb 11 19:58:53 maggie kernel: Uniform CD-ROM driver Revision: 3.12

Feb 11 19:58:53 maggie kernel: Partition check:

Feb 11 19:58:53 maggie kernel: hdc: hdc1

/var/log/vmkernel

Feb 11 19:59:17 maggie vmkernel: 0:00:00:02.475 cpu0:1024)Chipset: 695: 0-1:00 busIRQ= 0 on 00-02

Feb 11 19:59:17 maggie vmkernel: 0:00:00:02.475 cpu0:1024)IntrVector: 279: 0x91

Feb 11 19:59:17 maggie vmkernel: 0:00:00:02.475 cpu0:1024)IOAPIC: 957: vector 0x91 to 00-02

Feb 11 19:59:17 maggie vmkernel: 0:00:00:02.475 cpu0:1024)PCI: 2376: failed for IDE 000:31.1

Feb 11 19:59:17 maggie vmkernel: 0:00:00:02.475 cpu0:1024)PCI: 1268: 000:31.2 8086:27c0 1028:01e7 added

Feb 11 19:59:17 maggie vmkernel: 0:00:00:02.475 cpu0:1024)PCI: 1270: classCode 0101 progIFRevID 8f01

Feb 11 19:59:17 maggie vmkernel: 0:00:00:02.475 cpu0:1024)PCI: 1273: intPIN C intLine 11

Feb 11 19:59:17 maggie vmkernel: 0:00:00:02.475 cpu0:1024)Chipset: 695: 000:31 C busIRQ=126 on 00-20

Feb 11 19:59:17 maggie vmkernel: 0:00:00:02.475 cpu0:1024)PCI: 1281: irq 20 vector 0x89

Feb 11 19:59:17 maggie vmkernel: 0:00:00:02.734 cpu2:1037)PCI: 1975: 000:31.2 named 'vmhba0' (was '')

Feb 11 19:59:17 maggie vmkernel: 0:00:00:02.734 cpu3:1035)PCI: 1975: 002:14.0 named 'vmhba1' (was '')

Feb 11 19:59:17 maggie vmkernel: 0:00:00:02.734 cpu2:1037)PCI: 1975: 006:00.0 named 'vmnic0' (was '')

Feb 11 19:59:17 maggie vmkernel: 0:00:00:02.766 cpu0:1024)VMNIX: CosSup: 663: Found 000:31.2 on the console

Feb 11 19:59:17 maggie vmkernel: 0:00:00:02.766 cpu0:1024)VMNIX: CosSup: 753: Unplugging 000:31.2

Feb 11 19:59:17 maggie vmkernel: 0:00:00:02.766 cpu1:1034)PCI: 1624: 000:31.2 to vmkernel

Feb 11 19:59:17 maggie vmkernel: 0:00:00:02.766 cpu1:1034)LinPCI: 43: 000:31.2 vmhba0

Feb 11 19:59:17 maggie vmkernel: 0:00:00:02.766 cpu0:1024)VMNIX: Mod: 467: Adding 000:31.2

Feb 11 19:59:17 maggie vmkernel: 0:00:00:02.774 cpu0:1024)VMNIX: CosSup: 663: Found 002:14.0 on the console

Feb 11 19:59:17 maggie vmkernel: 0:00:00:02.774 cpu0:1024)VMNIX: CosSup: 753: Unplugging 002:14.0

Feb 11 19:59:17 maggie vmkernel: 0:00:00:02.774 cpu1:1034)PCI: 1624: 002:14.0 to vmkernel

Feb 11 19:59:17 maggie vmkernel: 0:00:00:02.774 cpu1:1034)LinPCI: 43: 002:14.0 vmhba1

Feb 11 19:59:17 maggie vmkernel: 0:00:00:02.774 cpu0:1024)VMNIX: Mod: 467: Adding 002:14.0

Feb 11 19:59:17 maggie vmkernel: 0:00:00:02.779 cpu0:1024)VMNIX: CosSup: 663: Found 006:00.0 on the console

Feb 11 19:59:17 maggie vmkernel: 0:00:00:02.779 cpu0:1024)VMNIX: CosSup: 753: Unplugging 006:00.0

Feb 11 19:59:17 maggie vmkernel: 0:00:00:02.779 cpu1:1034)PCI: 1624: 006:00.0 to vmkernel

Feb 11 19:59:17 maggie vmkernel: 0:00:00:02.779 cpu1:1034)LinPCI: 43: 006:00.0 vmnic0

Feb 11 19:59:17 maggie vmkernel: 0:00:00:02.779 cpu0:1024)VMNIX: Mod: 467: Adding 006:00.0

Feb 11 19:59:17 maggie vmkernel: 0:00:00:02.838 cpu0:1024)Loading module ata_piix ...

Feb 11 19:59:17 maggie vmkernel: 0:00:00:02.840 cpu0:1024)Mod: 936: Starting load for module: ata_piix R/O length: 0xb000 R/W length: 0x4000 Md5sum: e8c795333929be433664377d59def006

Feb 11 19:59:17 maggie vmkernel: 0:00:00:02.921 cpu3:1035)Mod: 1373: Module ata_piix: initFunc: 0x89ed28 text: 0x89e000 data: 0x28e9400 bss: 0x28e98a0 (writeable align 32)

Feb 11 19:59:17 maggie vmkernel: 0:00:00:02.921 cpu3:1035)Mod: 1389: modLoaderHeap avail before: 7806552

Feb 11 19:59:17 maggie vmkernel: 0:00:00:02.921 cpu3:1035)Initial heap size : 1048576, max heap size: 8388608

Feb 11 19:59:17 maggie vmkernel: 0:00:00:02.921 cpu3:1035)<7>libata version 1.20 loaded.

Feb 11 19:59:17 maggie vmkernel: 0:00:00:02.921 cpu3:1035)PCI: driver ata_piix is looking for devices

Feb 11 19:59:17 maggie vmkernel: 0:00:00:02.921 cpu3:1035)PCI: Trying 00:1f.2

Feb 11 19:59:17 maggie vmkernel: 0:00:00:02.921 cpu3:1035)PCI: Announcing 00:1f.2

Feb 11 19:59:17 maggie vmkernel: 0:00:00:02.921 cpu3:1035)PCI: Trying 02:0e.0

Feb 11 19:59:17 maggie vmkernel: 0:00:00:02.921 cpu3:1035)PCI: Announcing 02:0e.0

Feb 11 19:59:17 maggie vmkernel: 0:00:00:02.921 cpu3:1035)PCI: Trying 06:00.0

Feb 11 19:59:17 maggie vmkernel: 0:00:00:02.921 cpu3:1035)PCI: Announcing 06:00.0

Feb 11 19:59:17 maggie vmkernel: 0:00:00:02.921 cpu3:1035)PCI: driver ata_piix claimed 0 device

Feb 11 19:59:17 maggie vmkernel: 0:00:00:02.921 cpu3:1035)ata_piix: pci_module_init failed (rc=-19)

Feb 11 19:59:17 maggie vmkernel: 0:00:00:02.921 cpu3:1035)Mod: 1436: Initialization for ata_piix succeeded with module ID 2.

Feb 11 19:59:17 maggie vmkernel: 0:00:00:02.921 cpu3:1035)ata_piix loaded successfully.

Is my assesment correct?

Can I prevent the basic IDE driver from grabbing the disk?

I don't mind losing access to the CD-ROM drive so perhaps the E-IDE driver can be completely disabled?

This has been working well under Update 2 and I've been using the disk for backups (my main storage is on a PERC 5 controller). I'd prefer to keep the lab system up-to-date but at the moment it looks like I'm going to be stuck on update 2

Please help!

Mark

[rilliam]

Have you found a solution to this problem yet? I am using a supermicro PDSML-LN2+ board with ICH7. ESX3.5 U3 will install but in the VIC I cannot see any storage adapter.

lspci :

00:1f.1 IDE interface: Intel Corporation: Unknown device 27df (rev 01)

00:1f.2 IDE interface: Intel Corporation: Unknown device 27c0 (rev 01)

[markiusuk]

My current workaround is not to apply Update 3 - this is a lab system so

it's not massively important that I keep it up to date.

When I get some spare time I'll figure out which of the specific patch

bundles affects the ICH7 and patch around it. I'll never have a

completely patched system but at least the bulk of patches will still be

applied.

I'll post back with my results

Mark

[markiusuk]

I was finally able to devise a workaround for this issue and get access to my ICH7 storage again.

I did this...

Replaced the following files with versions found on an ESX 3.5 Update 2 host:

/etc/vmware/pciid/ata_piix.xml

/usr/lib/vmware/vmkmod/ata_piix.o

/usr/lib/vmware-debug/vmkmod/ata_piix.o

Ran "esxcfg-pciid" and rebooted the host

Manually rebuilt the initrd image and replaced the ata_piix.o file in there too.

After rebooting again my ICH7 disks re-appeared in /vmfs/volumes

I expect that this workaround will get broken by future updates to the ata_piix.o module and will need to be re-applied.

Hope this helps someone.

Mark

[X_Burner]

Hi markiusuk!

You workaround sounds good...but where can I get the files when I didnt got a Update 2 Host?

May be you can send me a Mail?

Can you describe more detailed how you manually rebuld initrd image and replaced the ata_piix.o?

THX

X_Burner

[markiusuk]

Hi

I'm not 100% clear on the legality of me sending you the specific files involved so I'm reluctant to do that.

<EDIT>

You can find the files you need inside update ESX350-200803213-UG. Inside the update you'll find this file:

VMware-esx-drivers-scsi-ata_piix-350.1.05-82663.i386.rpm

You may even be able to simply install the RPM.... that would be nice and easy. Let me know if that works for you.

</EDIT>

Rebuilding the initrd image is relatively straightforward... the image is a gzipped file that once unzipped can be mounted as a loop file system. Once you've mounted it you can replace the files. It's a holiday weekend here in the UK so I don't have a lot of tinkering time over the next few days but I'll see if I can put together a howto over the next week or so. In the meantime you may want to have a look at this:

...and this...

http://communities.vmware.com/thread/116768

As I mentioned earlier in the thread, I'm not really a linux tech but something in the back of my mind tells me that there may be a way to trigger a automatic rebuild of the initrd image and have it pick up the appropriate (downgraded) modules from the live ESX installation.

Good luck

Mark

[X_Burner]

Hi markiusuk!

Thank you for the informations.

Im also not sure if its 100% legal to polst files like these...I will download the U2 Release at vmware.com.

I will post it here when I got time to test the rpm.

I agree with you that there should be an easier way to downgrade drivers within ESX.

A howto would be nice...but Im not in a hurry

CU

X_Burner