journal commit I/O error when boot from SAN with E...

josh_909 · ‎11-20-2008

I am having this error after the installation of ESX when doing a boot from san, had zero problems during installation (mounting ISO and installing on mapped LUN). This error shows up in the console login screen almost immediately after boot. Not sure if its involved, but we are multipathing... dual Virtual Connects, dual Fabric switches to a single SAN.

"journal commit I/O error" (screenshot attached)

Hardware:

HP c-7000 Chassis

Proliant BL460c G1 diskless blades

Qlogic QMH2462 4Gb FC HBA

Virtual Connect (Fiber and Ethernet)

Cisco 9134 Fabric switches

Compellent SAN (15k fiber drive enclosures for LUN 0)

Any help or direction on this one would be much appreciated...Thanks in advance!

Texiwill · ‎11-21-2008

Hello,

Review the contents of the file /var/log/vmkernel, you may see other errors that show up about the storage. They are most likely related.

Best regards,

Edward L. Haletky

VMware Communities User Moderator

====

SearchVMware Blog: http://itknowledgeexchange.techtarget.com/virtualization-pro/

Blue Gears Blogs - http://www.itworld.com/ and http://www.networkworld.com/community/haletky

As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization

--
Edward L. Haletky
vExpert XIV: 2009-2023,
VMTN Community Moderator
vSphere Upgrade Saga: https://www.astroarch.com/blogs
GitHub Repo: https://github.com/Texiwill

josh_909 · ‎11-21-2008

Perhaps this will help some...

/var/log/vmkwarning:

Nov 20 03:01:57 esxch4bl1 vmkernel: TSC: 85401 cpu0:0)WARNING: Cpu: 1321: BSP variable MTRR #0 mask too small; resetting from ff0000800 to 3ff0000800 to match PCPU phys addr size (38).

Nov 20 03:01:57 esxch4bl1 vmkernel: TSC: 7389657 cpu0:0)WARNING: Cpu: 1321: BSP variable MTRR #1 mask too small; resetting from fe0000800 to 3fe0000800 to match PCPU phys addr size (38).

Nov 21 09:06:52 esxch4bl1 vmkernel: TSC: 84942 cpu0:0)WARNING: Cpu: 1321: BSP variable MTRR #0 mask too small; resetting from ff0000800 to 3ff0000800 to match PCPU phys addr size (38).

Nov 21 09:06:52 esxch4bl1 vmkernel: TSC: 7340076 cpu0:0)WARNING: Cpu: 1321: BSP variable MTRR #1 mask too small; resetting from fe0000800 to 3fe0000800 to match PCPU phys addr size (38).

/var/log/vmkernel:

Nov 20 03:01:57 esxch4bl1 vmkernel: TSC: 0 cpu0:0)Init: 386: cpu 0: early measured tsc speed 3000106176 Hz

Nov 20 03:01:57 esxch4bl1 vmkernel: TSC: 25173 cpu0:0)Cpu: 341: id1.version 10676

Nov 20 03:01:57 esxch4bl1 vmkernel: TSC: 42597 cpu0:0)Cpu: 400: APIC ID mask: 0xff000000

Nov 20 03:01:57 esxch4bl1 vmkernel: TSC: 47862 cpu0:0)Cpu: 826: initial APICID=0x0

Nov 20 03:01:57 esxch4bl1 vmkernel: TSC: 53091 cpu0:0)MicrocodeIntel: 375: Update signature 60c00000000, Platform ID 18000088e40920.

Nov 20 03:01:57 esxch4bl1 vmkernel: TSC: 62073 cpu0:0)Cpu: 1022: 38 physical bits, 48 virtual bits

Nov 20 03:01:57 esxch4bl1 vmkernel: TSC: 67518 cpu0:0)Cpu: 1592: MTRR: cap=0xd08 with 8 variable MTRRs on BSP 0

Nov 20 03:01:57 esxch4bl1 vmkernel: TSC: 74529 cpu0:0)Cpu: 1628: MTRR: deftype=0xc06, enabled bit=1 and default type=6

Nov 20 03:01:57 esxch4bl1 vmkernel: TSC: 85401 cpu0:0)WARNING: Cpu: 1321: BSP variable MTRR #0 mask too small; resetting from ff0000800 to 3ff0000800 to match PCPU phys addr size (38).

Nov 20 03:01:57 esxch4bl1 vmkernel: TSC: 2131659 cpu0:0)Cpu: 1347: Variable MTRR 0: start=0x0 mask=0x3ff0000800 end=0x0 type=0

Nov 21 09:06:52 esxch4bl1 vmkernel: TSC: 25326 cpu0:0)Cpu: 341: id1.version 10676

Nov 21 09:06:52 esxch4bl1 vmkernel: TSC: 42318 cpu0:0)Cpu: 400: APIC ID mask: 0xff000000

Nov 21 09:06:52 esxch4bl1 vmkernel: TSC: 47583 cpu0:0)Cpu: 826: initial APICID=0x0

Nov 21 09:06:52 esxch4bl1 vmkernel: TSC: 53298 cpu0:0)MicrocodeIntel: 375: Update signature 60c00000000, Platform ID 18000088e40920.

Nov 21 09:06:52 esxch4bl1 vmkernel: TSC: 61587 cpu0:0)Cpu: 1022: 38 physical bits, 48 virtual bits

Nov 21 09:06:52 esxch4bl1 vmkernel: TSC: 67257 cpu0:0)Cpu: 1592: MTRR: cap=0xd08 with 8 variable MTRRs on BSP 0

Nov 21 09:06:52 esxch4bl1 vmkernel: TSC: 74376 cpu0:0)Cpu: 1628: MTRR: deftype=0xc06, enabled bit=1 and default type=6

Nov 21 09:06:52 esxch4bl1 vmkernel: TSC: 84942 cpu0:0)WARNING: Cpu: 1321: BSP variable MTRR #0 mask too small; resetting from ff0000800 to 3ff0000800 to match PCPU phys addr size (38).

Nov 21 09:06:52 esxch4bl1 vmkernel: TSC: 2132775 cpu0:0)Cpu: 1347: Variable MTRR 0: start=0x0 mask=0x3ff0000800 end=0x0 type=0

Nov 21 09:06:52 esxch4bl1 vmkernel: TSC: 7340076 cpu0:0)WARNING: Cpu: 1321: BSP variable MTRR #1 mask too small; resetting from fe0000800 to 3fe0000800 to match PCPU phys addr size (38).

Nov 21 09:06:52 esxch4bl1 vmkernel: TSC: 14761188 cpu0:0)Cpu: 1347: Variable MTRR 1: start=0x0 mask=0x3fe0000800 end=0x0 type=0

Nov 21 09:06:52 esxch4bl1 vmkernel: TSC: 20000826 cpu0:0)Cpu: 1500: start=0x0 type=0x606060606060606

Nov 21 09:06:52 esxch4bl1 vmkernel: TSC: 25572582 cpu0:0)Cpu: 1500: start=0x80000 type=0x606060606060606

Nov 21 09:06:52 esxch4bl1 vmkernel: TSC: 30400578 cpu0:0)Cpu: 1500: start=0xa0000 type=0x0

Nov 21 09:06:52 esxch4bl1 vmkernel: TSC: 34179966 cpu0:0)Cpu: 1500: start=0xc0000 type=0x505050505050505

Nov 21 09:06:52 esxch4bl1 vmkernel: TSC: 38747412 cpu0:0)Cpu: 1500: start=0xc8000 type=0x505050505050505

Nov 21 09:06:52 esxch4bl1 vmkernel: TSC: 43318602 cpu0:0)Cpu: 1500: start=0xd0000 type=0x505050505050505

Nov 21 09:06:52 esxch4bl1 vmkernel: TSC: 47900016 cpu0:0)Cpu: 1500: start=0xd8000 type=0x505050505050505

Nov 21 09:06:52 esxch4bl1 vmkernel: TSC: 52475085 cpu0:0)Cpu: 1500: start=0xe0000 type=0x505050505050505

Nov 21 09:06:52 esxch4bl1 vmkernel: TSC: 57053052 cpu0:0)Cpu: 1500: start=0xe8000 type=0x505050505050505

Nov 21 09:06:52 esxch4bl1 vmkernel: TSC: 61619913 cpu0:0)Cpu: 1500: start=0xf0000 type=0x505050505050505

Nov 21 09:06:52 esxch4bl1 vmkernel: TSC: 66194208 cpu0:0)Cpu: 1500: start=0xf8000 type=0x505050505050505

Nov 21 09:06:52 esxch4bl1 vmkernel: TSC: 73933398 cpu0:0)Cpu: 168: type 2

Nov 21 09:06:52 esxch4bl1 vmkernel: TSC: 73951344 cpu0:0)Cpu: 169: numCores 4, numHT 1

Nov 21 09:06:52 esxch4bl1 vmkernel: TSC: 73955853 cpu0:0)Cpu: 171: L2 cache : size 6291456, assoc. 24

Nov 21 09:06:52 esxch4bl1 vmkernel: TSC: 73961730 cpu0:0)Cpu: 173: L3 cache : size -1, assoc. -1

Nov 21 09:06:52 esxch4bl1 vmkernel: TSC: 74929302 cpu0:0)ACPI: 1299: found RSDP @ 0x3254f00, RSDT @ 0xcfe54000

josh_909 · ‎11-21-2008

Part of /var/log/dmesg:

Adding Swap: 557032k swap-space (priority -1)

qla2x00_set_info starts at address = d212d060

scsi_register_dev_mod starting finish

scsi_register_dev_mod done with finish

kjournald starting. Commit interval 5 seconds

EXT3 FS 2.4-0.9.19, 19 August 2002 on sd(8,1), internal journal

EXT3-fs: mounted filesystem with ordered data mode.

kjournald starting. Commit interval 5 seconds

EXT3 FS 2.4-0.9.19, 19 August 2002 on sd(8,6), internal journal

EXT3-fs: mounted filesystem with ordered data mode.

microcode: No suitable data for cpu 0

ip_conntrack version 2.1 (2176 buckets, 17408 max) - 308 bytes per conntrack

ipmi message handler version 39

IPMI System Interface driver version 39, KCS version 39, SMIC version 39, BT ver

sion 39

ipmi_si: Found SMBIOS-specified state machine at I/O address 0xca2

IPMI kcs interface initialized

ipmi device interface version 39

journal_bmap: journal block not found at offset 1036 on sd(8,6)

Aborting journal on device sd(8,6).

journal commit I/O error

ext3_abort called.

EXT3-fs abort (device sd(8,6)): ext3_journal_start: Detected aborted journal

Remounting filesystem read-only

EXT3-fs error (device sd(8,6)) in start_transaction: Journal has aborted

ppdev: user-space parallel port driver

ppdev0: claim the port first

lp: driver loaded but no devices found

ppdev1: claim the port first

ppdev2: claim the port first

ppdev3: claim the port first

ppdev4: claim the port first

ppdev5: claim the port first

ppdev6: claim the port first

ppdev7: claim the port first

EXT3-fs error (device sd(8,6)) in start_transaction: Journal has aborted

josh_909 · ‎11-21-2008

# ls /vmfs/devices/disks/ vmhba0:2:0:0 vmhba1:2:0:4

vmhba0:2:0:1 vmhba1:2:0:5

vmhba0:2:0:2 vmhba1:2:0:6

vmhba0:2:0:3 vmhba1:2:0:7

vmhba0:2:0:4 vml.02000000006000d310000b5300000000000000058b436f6d70656c

vmhba0:2:0:5 vml.02000000006000d310000b5300000000000000058b436f6d70656c:1

vmhba0:2:0:6 vml.02000000006000d310000b5300000000000000058b436f6d70656c:2

vmhba0:2:0:7 vml.02000000006000d310000b5300000000000000058b436f6d70656c:3

vmhba1:2:0:0 vml.02000000006000d310000b5300000000000000058b436f6d70656c:4

vmhba1:2:0:1 vml.02000000006000d310000b5300000000000000058b436f6d70656c:5

vmhba1:2:0:2 vml.02000000006000d310000b5300000000000000058b436f6d70656c:6

vmhba1:2:0:3 vml.02000000006000d310000b5300000000000000058b436f6d70656c:7

# esxcfg-vmhbadevs

vmhba0:2:0 /dev/sda

Texiwill · ‎11-22-2008