SCSI disk error, Reservation Conflict

madcult · ‎03-27-2007

Yesterday a disk crashed in one of our ESX servers. Therefore I had to go to our computing center and exchange the disk. Once connected to the console by our KVM-switch I realized that there were strange things happening on this ESX server. I checked all our other ESX servers and all had the same "problem". Everything works well but I doubt that these messages are good:

scsi0 (0,0,1) : RESERVATION CONFLICT

SCSI disk error : host 0 channel 0 id 0 lun 1 return code = 18

I/O error: dev 08:10, sector 0

scsi0 (0,0,2) : RESERVATION CONFLICT

SCSI disk error : host 0 channel 0 id 0 lun 2 return code = 18

I/O error: dev 08:20, sector 0

scsi0 (0,0,2) : RESERVATION CONFLICT

SCSI disk error : host 0 channel 0 id 0 lun 2 return code = 18

I/O error: dev 08:20, sector 2

scsi0 (0,0,2) : RESERVATION CONFLICT

SCSI disk error : host 0 channel 0 id 0 lun 2 return code = 18

I/O error: dev 08:20, sector 4

scsi0 (0,0,2) : RESERVATION CONFLICT

SCSI disk error : host 0 channel 0 id 0 lun 2 return code = 18

I/O error: dev 08:20, sector 6

scsi0 (0,0,4) : RESERVATION CONFLICT

SCSI disk error : host 0 channel 0 id 0 lun 4 return code = 18

I/O error: dev 08:40, sector 0

Usually we connect to our ESX servers by PuTTY so nobody has ever seen these messages.

We have 4 ESX servers in production environment and one in testing environment. All ESX servers are connected to the same 6 LUNs on a EMC CX3-40 storage. Maybe the VMFS is corrupted?

What's going wrong? Any suggestions?

I fear the worst!

affoliveira · ‎03-27-2007

It seems to be a SAN problem.

Can run "dmesg" on the service console and post the result?

Paul_Lalonde · ‎03-27-2007

Also, post the relevant portions of /var/log/vmkernel

Paul

madcult · ‎03-27-2007

That is the last part of the "dmesg" command...

But here is the complete log:

\[root@ESX-SERVER root]# dmesg

: (B0,I29,P1) -> 19

PCI->APIC IRQ transform: (B0,I29,P2) -> 18

PCI->APIC IRQ transform: (B0,I29,P3) -> 20

PCI->APIC IRQ transform: (B0,I31,P0) -> 18

PCI->APIC IRQ transform: (B0,I31,P1) -> 18

PCI->APIC IRQ transform: (B1,I0,P0) -> 16

PCI->APIC IRQ transform: (B3,I0,P0) -> 16

PCI->APIC IRQ transform: (B6,I0,P0) -> 16

PCI->APIC IRQ transform: (B6,I0,P1) -> 17

PCI->APIC IRQ transform: (B8,I0,P0) -> 16

PCI->APIC IRQ transform: (B8,I0,P1) -> 17

PCI->APIC IRQ transform: (B11,I2,P0) -> 21

PCI->APIC IRQ transform: (B12,I2,P0) -> 22

PCI->APIC IRQ transform: (B12,I2,P1) -> 23

PCI->APIC IRQ transform: (B15,I14,P0) -> 24

PCI->APIC IRQ transform: (B16,I6,P0) -> 25

PCI->APIC IRQ transform: (B16,I6,P1) -> 26

PCI->APIC IRQ transform: (B17,I0,P0) -> 16

VMNIX: vmnixPCIInfo=1:;3:;6:;8:;11:;12:;15:;16:;

VMNIX is hiding PCI device (B1,I0,P0)

VMNIX is hiding PCI device (B3,I0,P0)

VMNIX is hiding PCI device (B6,I0,P0)

VMNIX is hiding PCI device (B6,I0,P1)

VMNIX is hiding PCI device (B8,I0,P0)

VMNIX is hiding PCI device (B8,I0,P1)

VMNIX is hiding PCI device (B11,I2,P0)

VMNIX is hiding PCI device (B12,I2,P0)

VMNIX is hiding PCI device (B12,I2,P1)

VMNIX is hiding PCI device (B15,I14,P0)

VMNIX is hiding PCI device (B16,I6,P0)

VMNIX is hiding PCI device (B16,I6,P1)

Linux NET4.0 for Linux 2.4

Based upon Swansea University Computer Society NET3.039

Initializing RT netlink socket

IA-32 Microcode Update Driver: v1.13 <tigran@veritas.com>

Starting kswapd

aio_setup: num_physpages = 32768

aio_setup: sizeof(struct page) = 60

Journalled Block Device driver loaded

pty: 256 Unix98 ptys configured

Serial driver version 5.05c (2001-07-08) with MANY_PORTS MULTIPORT SHARE_IRQ SERIAL_PCI enabled

ttyS0 at 0x03f8 (irq = 4) is a 16550A

ttyS1 at 0x02f8 (irq = 3) is a 16550A

FDC 0 is a National Semiconductor PC87306

RAMDISK driver initialized: 256 RAM disks of 64000K size 1024 blocksize

Uniform Multi-Platform E-IDE driver Revision: 7.00beta4-2.4

ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx

hda: HL-DT-ST DVDRAM GSA-T10N, ATAPI CD/DVD-ROM drive

ide0 at 0x1f0-0x1f7,0x3f6 on irq 14

hda: attached ide-cdrom driver.

hda: ATAPI 24X DVD-ROM DVD-R-RAM CD-R/RW drive, 2048kB Cache

Uniform CD-ROM driver Revision: 3.12

SCSI subsystem driver Revision: 1.00

kmod: failed to exec /sbin/modprobe -s -k scsi_hostadapter, errno = 2

Initializing Cryptographic API

NET4: Linux TCP/IP 1.0 for NET4.0

IP: routing cache hash table of 4096 buckets, 32Kbytes

TCP: Hash tables configured (established 131072 bind 65536)

NET4: Unix domain sockets 1.0/SMP for Linux NET4.0.

RAMDISK: Compressed image found at block 0

Freeing initrd memory: 5879k freed

EXT2-fs warning: maximal mount count reached, running e2fsck is recommended

VFS: Mounted root (ext2 filesystem).

Mod: 698: vmnix module init_module=0xe080f828 STACK_TOP_LA=0xfd402ff0

Mod: 3216: hb timer on.

Mod: 3283: Started hb thread.

IRQ: 291: COS is using IOAPIC

Mod: 2305: Setting 0xfce00000 to read-only

Mod: 2305: Setting 0xfd000000 to read-only

Mod: 2305: Setting 0xfd200000 to read-only

VMNIX: BSP APIC ID: 0x0

IRQ: 950: irq 0 is used

IRQ: 954: irq 0 is enabled

IRQ: 950: irq 1 is used

IRQ: 954: irq 1 is enabled

IRQ: 950: irq 2 is used

IRQ: 954: irq 2 is enabled

IRQ: 950: irq 14 is used

IRQ: 954: irq 14 is enabled

IRQ: 1134: Usurping irq 0 (from IO-APIC-edge to vmnix-edge)

IRQ: 1134: Usurping irq 1 (from IO-APIC-edge to vmnix-edge)

IRQ: 1134: Usurping irq 2 (from XT-PIC to vmnix-edge)

IRQ: 1134: Usurping irq 3 (from IO-APIC-edge to vmnix-edge)

IRQ: 1134: Usurping irq 4 (from IO-APIC-edge to vmnix-edge)

IRQ: 1134: Usurping irq 5 (from IO-APIC-edge to vmnix-edge)

IRQ: 1134: Usurping irq 6 (from IO-APIC-edge to vmnix-edge)

IRQ: 1134: Usurping irq 7 (from IO-APIC-edge to vmnix-edge)

IRQ: 1134: Usurping irq 8 (from IO-APIC-edge to vmnix-edge)

IRQ: 1134: Usurping irq 12 (from IO-APIC-edge to vmnix-edge)

IRQ: 1134: Usurping irq 13 (from IO-APIC-edge to vmnix-edge)

IRQ: 1134: Usurping irq 14 (from IO-APIC-edge to vmnix-edge)

IRQ: 1134: Usurping irq 15 (from IO-APIC-edge to vmnix-edge)

IRQ: 1134: Usurping irq 16 (from IO-APIC-level to vmnix-level)

IRQ: 1134: Usurping irq 17 (from IO-APIC-level to vmnix-level)

IRQ: 1134: Usurping irq 18 (from IO-APIC-level to vmnix-level)

IRQ: 1134: Usurping irq 19 (from IO-APIC-level to vmnix-level)

IRQ: 1134: Usurping irq 20 (from IO-APIC-level to vmnix-level)

IRQ: 1134: Usurping irq 21 (from IO-APIC-level to vmnix-level)

IRQ: 1134: Usurping irq 22 (from IO-APIC-level to vmnix-level)

IRQ: 1134: Usurping irq 23 (from IO-APIC-level to vmnix-level)

IRQ: 1134: Usurping irq 24 (from IO-APIC-level to vmnix-level)

IRQ: 1134: Usurping irq 25 (from IO-APIC-level to vmnix-level)

IRQ: 1134: Usurping irq 26 (from IO-APIC-level to vmnix-level)

VGA: 297: VGA start b8000 end c0000 mapped start c00b8000 char height 16

VGA: 397:

VGA: 425: 0

Console: switching to colour VMNIX-VGA 80x25

scsi0 : lpfc

scsi1 : lpfc

Vendor: DGC Model: RAID 10 Rev: 0322

Type: Direct-Access ANSI SCSI revision: 04

Vendor: DGC Model: RAID 10 Rev: 0322

Type: Direct-Access ANSI SCSI revision: 04

Vendor: DGC Model: RAID 10 Rev: 0322

Type: Direct-Access ANSI SCSI revision: 04

Vendor: DGC Model: RAID 10 Rev: 0322

Type: Direct-Access ANSI SCSI revision: 04

Vendor: DGC Model: RAID 10 Rev: 0322

Type: Direct-Access ANSI SCSI revision: 04

Vendor: DGC Model: RAID 10 Rev: 0322

Type: Direct-Access ANSI SCSI revision: 04

VMWARE SCSI Id: Supported VPD pages for sda : 0x0 0x80 0x83 0x85 0xc0

VMWARE SCSI Id: Device id info for sda: 0x1 0x3 0x0 0x10 0x60 0x6 0x1 0x60 0x9b 0x20 0x1a 0x0 0xb4 0x7e 0xbe 0xca 0x6e 0x84 0xdb 0x11 0x1 0x0 0x0 0x10 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x6 0x0 0x0 0x0 0x0 0x0 0x0 0x1 0x14 0x0 0x4 0x0 0x0 0x0 0x7 0x1 0x15 0x0 0x4 0x0 0x0 0x0 0x2

VMWARE SCSI Id: Id for sda 0x60 0x06 0x01 0x60 0x9b 0x20 0x1a 0x00 0xb4 0x7e 0xbe 0xca 0x6e 0x84 0xdb 0x11 0x52 0x41 0x49 0x44 0x20 0x31

VMWARE: Unique Device attached as scsi disk sda at scsi0, channel 0, id 0, lun 0

Attached scsi disk sda at scsi0, channel 0, id 0, lun 0

VMWARE SCSI Id: Supported VPD pages for sdb : 0x0 0x80 0x83 0x85 0xc0

VMWARE SCSI Id: Device id info for sdb: 0x1 0x3 0x0 0x10 0x60 0x6 0x1 0x60 0x9b 0x20 0x1a 0x0 0x5a 0xda 0xb9 0x24 0x90 0x8e 0xdb 0x11 0x1 0x0 0x0 0x10 0x0 0x1 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x2 0x0 0x0 0x0 0x0 0x0 0x0 0x1 0x14 0x0 0x4 0x0 0x0 0x0 0x7 0x1 0x15 0x0 0x4 0x0 0x0 0x0 0x2

VMWARE SCSI Id: Id for sdb 0x60 0x06 0x01 0x60 0x9b 0x20 0x1a 0x00 0x5a 0xda 0xb9 0x24 0x90 0x8e 0xdb 0x11 0x52 0x41 0x49 0x44 0x20 0x31

VMWARE: Unique Device attached as scsi disk sdb at scsi0, channel 0, id 0, lun 1

Attached scsi disk sdb at scsi0, channel 0, id 0, lun 1

VMWARE SCSI Id: Supported VPD pages for sdc : 0x0 0x80 0x83 0x85 0xc0

VMWARE SCSI Id: Device id info for sdc: 0x1 0x3 0x0 0x10 0x60 0x6 0x1 0x60 0x9b 0x20 0x1a 0x0 0x4e 0x75 0x62 0x65 0x59 0x84 0xdb 0x11 0x1 0x0 0x0 0x10 0x0 0x2 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x3 0x0 0x0 0x0 0x0 0x0 0x0 0x1 0x14 0x0 0x4 0x0 0x0 0x0 0x2 0x1 0x15 0x0 0x4 0x0 0x0 0x0 0x1

VMWARE SCSI Id: Id for sdc 0x60 0x06 0x01 0x60 0x9b 0x20 0x1a 0x00 0x4e 0x75 0x62 0x65 0x59 0x84 0xdb 0x11 0x52 0x41 0x49 0x44 0x20 0x31

VMWARE: Unique Device attached as scsi disk sdc at scsi0, channel 0, id 0, lun 2

Attached scsi disk sdc at scsi0, channel 0, id 0, lun 2

VMWARE SCSI Id: Supported VPD pages for sdd : 0x0 0x80 0x83 0x85 0xc0

VMWARE SCSI Id: Device id info for sdd: 0x1 0x3 0x0 0x10 0x60 0x6 0x1 0x60 0x9b 0x20 0x1a 0x0 0x24 0x77 0x2c 0x7d 0x90 0x8e 0xdb 0x11 0x1 0x0 0x0 0x10 0x0 0x3 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x4 0x0 0x0 0x0 0x0 0x0 0x0 0x1 0x14 0x0 0x4 0x0 0x0 0x0 0x7 0x1 0x15 0x0 0x4 0x0 0x0 0x0 0x2

VMWARE SCSI Id: Id for sdd 0x60 0x06 0x01 0x60 0x9b 0x20 0x1a 0x00 0x24 0x77 0x2c 0x7d 0x90 0x8e 0xdb 0x11 0x52 0x41 0x49 0x44 0x20 0x31

VMWARE: Unique Device attached as scsi disk sdd at scsi0, channel 0, id 0, lun 3

Attached scsi disk sdd at scsi0, channel 0, id 0, lun 3

VMWARE SCSI Id: Supported VPD pages for sde : 0x0 0x80 0x83 0x85 0xc0

VMWARE SCSI Id: Device id info for sde: 0x1 0x3 0x0 0x10 0x60 0x6 0x1 0x60 0x9b 0x20 0x1a 0x0 0x68 0x8b 0x93 0xd1 0x59 0x84 0xdb 0x11 0x1 0x0 0x0 0x10 0x0 0x4 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x5 0x0 0x0 0x0 0x0 0x0 0x0 0x1 0x14 0x0 0x4 0x0 0x0 0x0 0x2 0x1 0x15 0x0 0x4 0x0 0x0 0x0 0x1

VMWARE SCSI Id: Id for sde 0x60 0x06 0x01 0x60 0x9b 0x20 0x1a 0x00 0x68 0x8b 0x93 0xd1 0x59 0x84 0xdb 0x11 0x52 0x41 0x49 0x44 0x20 0x31

VMWARE: Unique Device attached as scsi disk sde at scsi0, channel 0, id 0, lun 4

Attached scsi disk sde at scsi0, channel 0, id 0, lun 4

VMWARE SCSI Id: Supported VPD pages for sdf : 0x0 0x80 0x83 0x85 0xc0

VMWARE SCSI Id: Device id info for sdf: 0x1 0x3 0x0 0x10 0x60 0x6 0x1 0x60 0x9b 0x20 0x1a 0x0 0xe0 0xd3 0x32 0xe4 0x91 0x8e 0xdb 0x11 0x1 0x0 0x0 0x10 0x0 0x5 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0xa 0x0 0x0 0x0 0x0 0x0 0x0 0x1 0x14 0x0 0x4 0x0 0x0 0x0 0x2 0x1 0x15 0x0 0x4 0x0 0x0 0x0 0x1

VMWARE SCSI Id: Id for sdf 0x60 0x06 0x01 0x60 0x9b 0x20 0x1a 0x00 0xe0 0xd3 0x32 0xe4 0x91 0x8e 0xdb 0x11 0x52 0x41 0x49 0x44 0x20 0x31

VMWARE: Unique Device attached as scsi disk sdf at scsi0, channel 0, id 0, lun 5

Attached scsi disk sdf at scsi0, channel 0, id 0, lun 5

SCSI device sda: 2147483648 512-byte hdwr sectors (1048576 MB)

Partition check:

sda: sda1

SCSI device sdb: 3939702144 512-byte hdwr sectors (1923682 MB)

sdb: sdb1

SCSI device sdc: 3939702144 512-byte hdwr sectors (1923682 MB)

sdc: sdc1

SCSI device sdd: 3939702144 512-byte hdwr sectors (1923682 MB)

sdd: sdd1

SCSI device sde: 3939702144 512-byte hdwr sectors (1923682 MB)

sde: sde1

SCSI device sdf: 1125629184 512-byte hdwr sectors (549622 MB)

sdf: sdf1

scsi2 : aic7xxx

Vendor: SEAGATE Model: DAT DAT72-000 Rev: A1A0

Type: Sequential-Access ANSI SCSI revision: 03

scsi3 : megaraid

Vendor: MegaRAID Model: LD 0 RAID1 139G Rev: 516G

Type: Direct-Access ANSI SCSI revision: 02

VMWARE SCSI Id: Supported VPD pages for sdg :

VMWARE SCSI Id: Could not get disk id for sdg

:VMWARE: Unique Device attached as scsi disk sdg at scsi3, channel 0, id 0, lun 0

Attached scsi disk sdg at scsi3, channel 0, id 0, lun 0

SCSI device sdg: 286310400 512-byte hdwr sectors (139800 MB)

sdg: sdg1 sdg2 sdg3 sdg4 < sdg5 sdg6 sdg7 >

kjournald starting. Commit interval 5 seconds

EXT3-fs: mounted filesystem with ordered data mode.

Freeing unused kernel memory: 376k freed

usb.c: registered new driver usbdevfs

usb.c: registered new driver hub

usb-uhci.c: $Revision: 1.275 $ time 22:22:54 Sep 25 2006

usb-uhci.c: High bandwidth mode enabled

PCI: Setting latency timer of device 00:1d.0 to 64

usb-uhci.c: USB UHCI at I/O 0x9060, IRQ 16

usb-uhci.c: Detected 2 ports

usb.c: new USB bus registered, assigned bus number 1

hub.c: USB hub found

hub.c: 2 ports detected

PCI: Setting latency timer of device 00:1d.1 to 64

usb-uhci.c: USB UHCI at I/O 0x9040, IRQ 19

usb-uhci.c: Detected 2 ports

usb.c: new USB bus registered, assigned bus number 2

hub.c: USB hub found

hub.c: 2 ports detected

PCI: Setting latency timer of device 00:1d.2 to 64

usb-uhci.c: USB UHCI at I/O 0x9020, IRQ 18

usb-uhci.c: Detected 2 ports

usb.c: new USB bus registered, assigned bus number 3

hub.c: USB hub found

hub.c: 2 ports detected

usb-uhci.c: v1.275:USB Universal Host Controller Interface driver

PCI: Setting latency timer of device 00:1d.7 to 64

ehci-hcd 00:1d.7: Intel Corp. 82801EB USB2

ehci-hcd 00:1d.7: irq 20, pci mem e10b6000

usb.c: new USB bus registered, assigned bus number 4

ehci-hcd 00:1d.7: enabled 64bit PCI DMA

PCI: 00:1d.7 PCI cache line size set incorrectly (0 bytes) by BIOS/FW.

PCI: 00:1d.7 PCI cache line size corrected to 128.

ehci-hcd 00:1d.7: USB 2.0 enabled, EHCI 1.00, driver 2003-Jan-22

hub.c: USB hub found

hub.c: 6 ports detected

usb.c: registered new driver hiddev

usb.c: registered new driver hid

hid-core.c: v1.8.1 Andreas Gal, Vojtech Pavlik <vojtech@suse.cz>

hid-core.c: USB HID support drivers

mice: PS/2 mouse device common for all mice

EXT3 FS 2.4-0.9.19, 19 August 2002 on sd(8,98), internal journal

Adding Swap: 24579440k swap-space (priority -1)

hub.c: connect-debounce failed, port 1 disabled

hub.c: new USB device 00:1d.7-3, assigned address 2

hub.c: USB hub found

hub.c: 3 ports detected

hub.c: new USB device 00:1d.2-1, assigned address 2

input0: USB HID v1.10 Keyboard \[Avocent USB DSRIQ] on usb3:2.0

input1: USB HID v1.10 Mouse \[Avocent USB DSRIQ] on usb3:2.1

hub.c: new USB device 00:1d.2-2, assigned address 3

input2: USB HID v1.10 Keyboard \[Agilent Technologies GmbH Agilent Technologies Remote Management Card] on usb3:3.0

input3: USB HID v1.10 Mouse \[Agilent Technologies GmbH Agilent Technologies Remote Management Card] on usb3:3.1

kjournald starting. Commit interval 5 seconds

EXT3 FS 2.4-0.9.19, 19 August 2002 on sd(8,97), internal journal

EXT3-fs: mounted filesystem with ordered data mode.

kjournald starting. Commit interval 5 seconds

EXT3 FS 2.4-0.9.19, 19 August 2002 on sd(8,101), internal journal

EXT3-fs: mounted filesystem with ordered data mode.

kjournald starting. Commit interval 5 seconds

EXT3 FS 2.4-0.9.19, 19 August 2002 on sd(8,102), internal journal

EXT3-fs: mounted filesystem with ordered data mode.

kjournald starting. Commit interval 5 seconds

EXT3 FS 2.4-0.9.19, 19 August 2002 on sd(8,1), internal journal

EXT3-fs: mounted filesystem with ordered data mode.

st: Version 20040102, bufsize 32768, max init. bufs 4, s/g segs 16

Attached scsi tape st0 at scsi2, channel 0, id 6, lun 0

microcode: No suitable data for cpu 0

Intel(R) PRO/1000 Network Driver - version 7.0.33

scsi_register_dev_mod starting finish

scsi_register_dev_mod done with finish

loop: loaded (max 8 devices)

EXT2-fs warning: maximal mount count reached, running e2fsck is recommended

scsi0 (0,0,1) : RESERVATION CONFLICT

SCSI disk error : host 0 channel 0 id 0 lun 1 return code = 18

I/O error: dev 08:10, sector 0

scsi0 (0,0,2) : RESERVATION CONFLICT

SCSI disk error : host 0 channel 0 id 0 lun 2 return code = 18

I/O error: dev 08:20, sector 0

scsi0 (0,0,2) : RESERVATION CONFLICT

SCSI disk error : host 0 channel 0 id 0 lun 2 return code = 18

I/O error: dev 08:20, sector 2

scsi0 (0,0,2) : RESERVATION CONFLICT

SCSI disk error : host 0 channel 0 id 0 lun 2 return code = 18

I/O error: dev 08:20, sector 4

scsi0 (0,0,2) : RESERVATION CONFLICT

SCSI disk error : host 0 channel 0 id 0 lun 2 return code = 18

I/O error: dev 08:20, sector 6

scsi0 (0,0,4) : RESERVATION CONFLICT

SCSI disk error : host 0 channel 0 id 0 lun 4 return code = 18

I/O error: dev 08:40, sector 0

affoliveira · ‎03-27-2007

Check your LUN presentation in your Storage because the problem appears to be in the SAN

DFATAnt · ‎03-27-2007

I've seen these messages in my last job just after we went to VI3.0. It turned out that our SAN was way behind on the firmware level. After the firmware was updated, the messages stopped happening.

Ant