1 2 Previous Next 17 Replies Latest reply on Mar 8, 2010 10:35 AM by joergriether

    esx 4 on nehalem vkernel suddenly went down, all iSCSI gone, unresponsive in vsphere center

    joergriether Hot Shot
    vExpert

      Dear Group,

       

      yesterday we encountered something really bad. One of our nehalem esx4 (latest patches) machine with a software iscsi initiator, target is equallogic, suddenly went totally offlline with the vkernel ip adress. In addition, the machine became unresponsive in vsphere center, but the main ip adress was still pingable.

       

      the vmkernel logs shows interesting infos, take a look:

      at 13:45 all was OK but then suddenly at 14:42 when the handler "world 6303/2" was started (what is this handler???) the catastrophe begun.

       

      I hat to hard reset the esx machine to get online again.

       

      Any ideas?

       

      best,

      Joerg

       

       

       

      Aug 27 13:45:25 esx7 vmkernel: 6:22:51:50.681 cpu0:4111)FSS: 3647: No FS driver claimed device '4a16ef91-b7c8669d-ac80-002219ccd2a1': Not supported Aug 27

       

      14:42:50 esx7 vmkernel: 6:23:49:15.841 cpu5:6303)ScsiCore: 95: Starting taskmgmt handler world 6303/2 Aug 27 14:42:53 esx7 vmkernel: 6:23:49:19.105 cpu2:4239)WARNING: iscsi_vmk: iscsivmk_StopConnection: vmhba33:CH:0 T:3 CN:0: iSCSI connection is being marked "OFFLINE"

       

       

      Aug 27 14:42:53 esx7 vmkernel: 6:23:49:19.105 cpu2:4239)WARNING: iscsi_vmk: iscsivmk_StopConnection: Sess ISID: 00023d000001 TARGET: iqn.2001-05.com.equallogic:0-8a0906-35e2f2304-bf3000000524a951-eql4-esx-lowpriovol3 TPGT: 1 TSIH: 0 Aug 27 14:42:53 esx7 vmkernel: 6:23:49:19.105 cpu2:4239)WARNING: iscsi_vmk: iscsivmk_StopConnection: Conn CID: 0 L: 172.16.150.131:56447 R: 172.16.150.222:3260 Aug 27 14:42:53 esx7 vmkernel: 6:23:49:19.105 cpu2:4239)WARNING: iscsi_vmk: iscsivmk_StopConnection: vmhba33:CH:0 T:1 CN:0: iSCSI connection is being marked "OFFLINE"

       

       

      Aug 27 14:42:53 esx7 vmkernel: 6:23:49:19.105 cpu2:4239)WARNING: iscsi_vmk: iscsivmk_StopConnection: Sess ISID: 00023d000001 TARGET: iqn.2001-05.com.equallogic:0-8a0906-4f32f2304-e21000000574a951-eql4-esx-lowpriovol1 TPGT: 1 TSIH: 0 Aug 27 14:42:53 esx7 vmkernel: 6:23:49:19.105 cpu2:4239)WARNING: iscsi_vmk: iscsivmk_StopConnection: Conn CID: 0 L: 172.16.150.131:58041 R: 172.16.150.223:3260 Aug 27 14:42:53 esx7 vmkernel: 6:23:49:19.105 cpu2:4239)WARNING: iscsi_vmk: iscsivmk_TaskMgmtIssue: vmhba33:CH:0 T:1 L:0 : Task mgmt "Abort Task" with itt=0x133c23 (refITT=0x133c20) timed out.

       

       

      Aug 27 14:42:54 esx7 vmkernel: 6:23:49:19.778 cpu3:5991)WARNING: NMP: nmp_IssueCommandToDevice: I/O could not be issued to device "naa.6090a04830f2324f51a97405000010e2" due to Not found Aug 27 14:42:54 esx7 vmkernel: 6:23:49:19.778 cpu3:5991)WARNING: NMP: nmp_DeviceRetryCommand: Device "naa.6090a04830f2324f51a97405000010e2": awaiting fast path state update for failover with I/O blocked. No prior reservation exists on the device.

       

       

      Aug 27 14:42:54 esx7 vmkernel: 6:23:49:19.778 cpu3:5991)WARNING: NMP: nmp_DeviceStartLoop: NMP Device "naa.6090a04830f2324f51a97405000010e2" is blocked. Not starting I/O from device.

       

       

      Aug 27 14:42:55 esx7 vmkernel: 6:23:49:20.779 cpu7:4207)WARNING: NMP: nmp_DeviceAttemptFailover: Retry world failover device "naa.6090a04830f2324f51a97405000010e2" - issuing command 0x410007148540 Aug 27 14:42:55 esx7 vmkernel: 6:23:49:20.779 cpu7:4207)WARNING: NMP: nmp_DeviceAttemptFailover: Retry world failover device

        1 2 Previous Next