4 Replies Latest reply on Oct 2, 2009 11:22 AM by rubensluque

    SCSI Reservation Conflict on multiple storage(iSCSI) environment, need help!

    Donghwan Kang Novice

      Hi there,

       

      I have SCSI Resevation problem since new storage added in my VI3(ESX3.5) environment.

       

      BACKUPGROUND:

       

       

      • 10 ESX 3.5 Hosts(SunFire x4150/Intel Xeon/32GB/8 GigE Ports)

      • 3 Sun StorageTek 2510 iSCSI Storage

      • Software iSCSI connection

      • VMkernel Ports for iSCSI is Teaming by 2 NICs

      • VMware HA/DRS/VMotion

      • 250 Windows XP VM(VDI)

      • ESX 3.5 U4 and VC 2.5 U4

      • Successfully operating for 6 months with no SCSI Reservation Error

       

       

       

       

       

      CHANGES :

       

       

      • Added new iSCSI storage

      • Sun Unified Storage 7410 (iSCSI)

       

      Each Windows XP VMs has 2 virtual disks. One(C:) is in the STK2510 Storage, the other(D:) is 7410 storage.

      Each VMFS volume has about 20 VMs and size is 500~600GB.

       

      Since added Sun 7410 storage, too many SCSI Reservation Conflict error has occurred, degrade I/O performance or

      operation failed such as VM power-on/off, VMotion(operations that need SCSI Reservation)

       

       

      /var/log/vmkwarning

      Jul 15 15:29:35 esx01 vmkernel: 0:01:31:52.787 cpu2:1051)WARNING: SCSI: 5350: vml.0200000000600144f04a5de86e00000c29593bf000534f4c415249: Too many failed retries 33 (32), Returning I/O failure. 0x12 1/0x0 0x2 0x8 0x0

      Jul 16 02:27:34 esx01 vmkernel: 0:06:26:50.747 cpu1:1085)WARNING: SCSI: 119: Failing I/O due to too many reservation conflicts

      Jul 16 02:28:39 esx01 vmkernel: 0:06:27:55.471 cpu1:1085)WARNING: SCSI: 119: Failing I/O due to too many reservation conflicts

      Jul 16 02:28:44 esx01 vmkernel: 0:06:28:00.615 cpu1:1085)WARNING: SCSI: 119: Failing I/O due to too many reservation conflicts

      Jul 16 02:28:49 esx01 vmkernel: 0:06:28:05.782 cpu0:1085)WARNING: SCSI: 119: Failing I/O due to too many reservation conflicts

      Jul 16 02:29:11 esx01 vmkernel: 0:06:28:27.387 cpu0:1085)WARNING: SCSI: 119: Failing I/O due to too many reservation conflicts

      Jul 16 02:29:16 esx01 vmkernel: 0:06:28:32.295 cpu0:1085)WARNING: SCSI: 119: Failing I/O due to too many reservation conflicts

      Jul 16 02:29:24 esx01 vmkernel: 0:06:28:40.299 cpu2:1086)WARNING: J3: 2151: Aborting txn to slot 0 because can't verify locks

      Jul 16 02:30:41 esx01 vmkernel: 0:06:29:57.349 cpu2:1039)WARNING: FS3: 4787: Reservation error: Timeout

      Jul 16 02:30:41 esx01 vmkernel: 0:06:29:57.349 cpu2:1039)WARNING: FS3: 4981: Reclaiming timed out heartbeat http://HB state abcdef02 offset 3258368 gen 76 stamp 23352554968 uuid 4a5db706-ae5a83ca-edb2-00144f8dd1ce jrnl <FB 12403> drv 4.31 failed: Timeout

       

       

      When SCSI Reservation Conflict error occured, I checked "Pending Reservation" of volumes using esxcfg-info.(Resolving SCSI reservation conflicts(KB1002293))

      In that situation one or some volumes(in the Sun 7410) are "Pending Reservation=1"

      |----Console Device................................../dev/sdd

      |----Devfs Path....................................../vmfs/devices/disks/vml.0200000000600144f04a5e892100000c29667ceb00534f4c415249

      |----SCSI Level......................................6

      |----Queue Depth.....................................32

      |----Is Pseudo.......................................false

      |----Is Reserved.....................................false

      |----Pending Reservations............................1

       

       

      In my opinion, there is a problem about release reservation, bt I'm not sure what is root cause. Additionally, Sun 2510 and Sun 7410 storage are linsted in VMware HCL list.

       

      Any idea?

       

       

      Thanks in advance.

       

       

      DH,