Hello,
I am under vSphere 4.1 free version and I have a problem with datastores.
ESXi is installed in the same disk as datastore1. It is the second time that my VMs are in read only mode. The cause is that my datastore1 is mentionned as inactive in vSphere.
I can connect to vSphere, I can connect to my ESXi host through SSH, I can access to my VMs through SSH too but they are in read only mode because of datstore1 is inactive.
Do you have any idea about what the problem is and how to solve it ?
I have nothing done as specific configuration in vSphere or ESXi.
Thank you for your help.
I've checked logs and I've found that, in /var/log/messages :
How to retrieve my datastore1 ?
Hi,
We have the same issue. Our datastore becomes inactive and there are the same warning/errors messages. If the system is power off and then on, the datastore will be active but it will become again inactive in 1-2 days.
Have you resolved your problem? Incampatible hardware? problems with hardware?
Thanks.
Hi,
No i have to resinstall all the ESXi server...
Same problem here but on ESXi 5.1.
vmkernel.log
2013-06-05T12:03:00.761Z cpu0:4100)ScsiDeviceIO: 2331: Cmd(0x4124007cd0c0) 0x2a, CmdSN 0x1d83d from world 4100 to dev "t10.ATA_____WDC_WD20EFRX2D68AX9N0_________________________WD2DWCC1T0407884" failed H:0x5 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2013-06-05T12:03:04.761Z cpu0:4100)ScsiDeviceIO: 2331: Cmd(0x4124007cd0c0) 0x2a, CmdSN 0x1d840 from world 4100 to dev "t10.ATA_____WDC_WD20EFRX2D68AX9N0_________________________WD2DWCC1T0407884" failed H:0x5 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2013-06-05T12:03:04.833Z cpu2:4113)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x2a (0x4124007d8640, 8191) to dev "t10.ATA_____WDC_WD20EFRX2D68AX9N0_________________________WD2DWCC1T0407884" on path "vmhba34:C0:T0:L0" Failed: H:0x8 D:0x0 P:0x0 Possible sens$
2013-06-05T12:03:04.833Z cpu2:4113)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "t10.ATA_____WDC_WD20EFRX2D68AX9N0_________________________WD2DWCC1T0407884" state in doubt; requested fast path state update...
2013-06-05T12:03:04.833Z cpu2:4113)ScsiDeviceIO: 2300: Cmd(0x4124007d8640) 0x2a, CmdSN 0x800e0040 from world 8191 to dev "t10.ATA_____WDC_WD20EFRX2D68AX9N0_________________________WD2DWCC1T0407884" failed H:0x8 D:0x0 P:0x0
2013-06-05T12:03:04.833Z cpu2:4113)ScsiDeviceIO: 2300: Cmd(0x4124007fb780) 0x2a, CmdSN 0x800e0027 from world 8191 to dev "t10.ATA_____WDC_WD20EFRX2D68AX9N0_________________________WD2DWCC1T0407884" failed H:0x8 D:0x0 P:0x0
2013-06-05T12:03:04.833Z cpu2:4113)<6>ahci_scsi_abort: cmd 0x2a (0x412401a8a2c0), entering...
2013-06-05T12:03:04.833Z cpu2:4113)<7>ata3: ahci_port_reset, entering...
2013-06-05T12:03:04.833Z cpu2:4113)ScsiDeviceIO: 2300: Cmd(0x41240080c0c0) 0x2a, CmdSN 0x800e0000 from world 8191 to dev "t10.ATA_____WDC_WD20EFRX2D68AX9N0_________________________WD2DWCC1T0407884" failed H:0x8 D:0x0 P:0x0
2013-06-05T12:03:07.053Z cpu0:4113)<6>ata3: ahci_port_reset, hard reseting port
2013-06-05T12:03:12.155Z cpu0:4222)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "t10.ATA_____WDC_WD20EFRX2D68AX9N0_________________________WD2DWCC1T0407884" state in doubt; requested fast path state update...
2013-06-05T12:03:12.155Z cpu0:4222)ScsiDeviceIO: 2331: Cmd(0x4124007b91c0) 0x2a, CmdSN 0x1d83e from world 4100 to dev "t10.ATA_____WDC_WD20EFRX2D68AX9N0_________________________WD2DWCC1T0407884" failed H:0x5 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2013-06-05T12:03:12.570Z cpu3:4113)<4>ata3: port is slow to respond, please be patient (Status 0x80)
2013-06-05T12:03:18.544Z cpu1:4222)ScsiDeviceIO: 2331: Cmd(0x4124007dab40) 0x2a, CmdSN 0x1d841 from world 4100 to dev "t10.ATA_____WDC_WD20EFRX2D68AX9N0_________________________WD2DWCC1T0407884" failed H:0x5 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2013-06-05T12:03:37.068Z cpu0:4113)<3>ata3: COMRESET failed (errno=-16)
2013-06-05T12:03:40.042Z cpu2:4222)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x2a (0x4124007c3c00, 4157) to dev "t10.ATA_____WDC_WD20EFRX2D68AX9N0_________________________WD2DWCC1T0407884" on path "vmhba34:C0:T0:L0" Failed: H:0x5 D:0x0 P:0x0 Possible sens$
2013-06-05T12:03:40.042Z cpu2:4222)ScsiDeviceIO: 2331: Cmd(0x4124007c3c00) 0x2a, CmdSN 0x1d83f from world 4157 to dev "t10.ATA_____WDC_WD20EFRX2D68AX9N0_________________________WD2DWCC1T0407884" failed H:0x5 D:0x0 P:0x0 Possible sense data: 0x5 0x20 0x0.
2013-06-05T12:03:40.042Z cpu0:4157)WARNING: J3: 2990: Error committing txn callerID: 0xc1d0000f to slot 0: Timeout
2013-06-05T12:03:43.044Z cpu1:4157)HBX: 2441: Waiting for timed out [HB state abcdef02 offset 3883008 gen 49 stampUS 101905355158 uuid 51ad9aff-49ab3864-7ad4-001e67340ea1 jrnl <FB 1858000> drv 14.58] on vol 'wd_red1'
2013-06-05T12:03:45.376Z cpu2:73484)VMW_SATP_LOCAL: satp_local_updatePathStates:454: Failed to update path "vmhba34:C0:T0:L0" state. Status=Transient storage condition, suggest retry
2013-06-05T12:03:57.039Z cpu0:69489)WARNING: VSCSI: 3481: handle 8222(vscsi4:0):WaitForCIF: Issuing reset; number of CIF:1
2013-06-05T12:03:57.039Z cpu0:69489)VSCSI: 2374: handle 8222(vscsi4:0):Reset request on FSS handle 1011988 (1 outstanding commands) from (vmm0:magnet5xp)
2013-06-05T12:03:57.039Z cpu1:4170)VSCSI: 2653: handle 8222(vscsi4:0):Reset [Retries: 0/0] from (vmm0:magnet5xp)
2013-06-05T12:03:57.039Z cpu1:4170)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "t10.ATA_____WDC_WD20EFRX2D68AX9N0_________________________WD2DWCC1T0407884" state in doubt; requested fast path state update...
2013-06-05T12:03:57.039Z cpu1:4170)ScsiDeviceIO: 2300: Cmd(0x4124007c83c0) 0x2a, CmdSN 0x1 from world 69489 to dev "t10.ATA_____WDC_WD20EFRX2D68AX9N0_________________________WD2DWCC1T0407884" failed H:0x8 D:0x0 P:0x0
2013-06-05T12:03:57.039Z cpu1:4170)VSCSI: 2450: handle 8222(vscsi4:0):Completing reset (0 outstanding commands)
2013-06-05T12:04:26.378Z cpu1:4192)VMW_SATP_LOCAL: satp_local_updatePathStates:454: Failed to update path "vmhba34:C0:T0:L0" state. Status=Transient storage condition, suggest retry
2013-06-05T12:04:36.050Z cpu1:4157)HBX: 2441: Waiting for timed out [HB state abcdef02 offset 3883008 gen 49 stampUS 101905355158 uuid 51ad9aff-49ab3864-7ad4-001e67340ea1 jrnl <FB 1858000> drv 14.58] on vol 'wd_red1'
2013-06-05T12:04:41.545Z cpu3:4113)<6>ata3: ahci_port_reset: FAILED
2013-06-05T12:04:41.545Z cpu3:4113)<6>ahci_scsi_abort: cmd 0x2a (0x412401a8a2c0), FAILED
2013-06-05T12:04:41.545Z cpu3:4113)WARNING: LinScsi: SCSILinuxAbortCommands:1816:Failed, Driver ahci, for vmhba34
2013-06-05T12:04:41.545Z cpu3:4113)<6>ahci_scsi_abort: cmd 0x2a (0x412401a44a00), entering...
2013-06-05T12:04:41.545Z cpu3:4113)<7>ata3: ahci_port_reset, entering...
2013-06-05T12:04:43.767Z cpu3:4113)<6>ata3: ahci_port_reset, hard reseting port
2013-06-05T12:05:00.041Z cpu0:69489)WARNING: VSCSI: 3481: handle 8222(vscsi4:0):WaitForCIF: Issuing reset; number of CIF:1
2013-06-05T12:05:00.041Z cpu0:69489)VSCSI: 2374: handle 8222(vscsi4:0):Reset request on FSS handle 1011988 (1 outstanding commands) from (vmm0:magnet5xp)
2013-06-05T12:05:00.041Z cpu1:4170)VSCSI: 2653: handle 8222(vscsi4:0):Reset [Retries: 0/0] from (vmm0:magnet5xp)
2013-06-05T12:05:00.041Z cpu1:4170)VSCSI: 2450: handle 8222(vscsi4:0):Completing reset (0 outstanding commands)
2013-06-05T12:05:07.380Z cpu0:73891)VMW_SATP_LOCAL: satp_local_updatePathStates:454: Failed to update path "vmhba34:C0:T0:L0" state. Status=Transient storage condition, suggest retry
2013-06-05T12:05:19.485Z cpu2:4098)ScsiDeviceIO: 2331: Cmd(0x4124007e3e40) 0x12, CmdSN 0x2722 from world 0 to dev "t10.ATA_____WDC_WD20EFRX2D68AX9N0_________________________WD2DWCC1T0407884" failed H:0x5 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2013-06-05T12:05:29.055Z cpu2:4157)HBX: 2441: Waiting for timed out [HB state abcdef02 offset 3883008 gen 49 stampUS 101905355158 uuid 51ad9aff-49ab3864-7ad4-001e67340ea1 jrnl <FB 1858000> drv 14.58] on vol 'wd_red1'
2013-06-05T12:05:48.788Z cpu0:4113)<6>ata3: ahci_port_reset: FAILED
2013-06-05T12:05:48.788Z cpu0:4113)<6>ahci_scsi_abort: cmd 0x2a (0x412401a44a00), FAILED
2013-06-05T12:05:48.788Z cpu0:4113)WARNING: LinScsi: SCSILinuxAbortCommands:1816:Failed, Driver ahci, for vmhba34
2013-06-05T12:05:48.788Z cpu0:4113)<6>ahci_scsi_abort: cmd 0x2a (0x412401a41a00), entering...
2013-06-05T12:05:48.788Z cpu0:4113)<7>ata3: ahci_port_reset, entering...
2013-06-05T12:05:51.011Z cpu3:4113)<6>ata3: ahci_port_reset, hard reseting port
2013-06-05T12:05:51.845Z cpu2:6001)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x2a (0x4124007ee280, 6001) to dev "t10.ATA_____WDC_WD20EFRX2D68AX9N0_________________________WD2DWCC1T0407884" on path "vmhba34:C0:T0:L0" Failed: H:0x5 D:0x0 P:0x0 Possible sens$
2013-06-05T12:05:51.845Z cpu2:6001)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "t10.ATA_____WDC_WD20EFRX2D68AX9N0_________________________WD2DWCC1T0407884" state in doubt; requested fast path state update...
2013-06-05T12:05:51.845Z cpu2:6001)ScsiDeviceIO: 2318: Cmd(0x4124007ee280) 0x2a, CmdSN 0x800e0024 from world 6001 to dev "t10.ATA_____WDC_WD20EFRX2D68AX9N0_________________________WD2DWCC1T0407884" failed H:0x5 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0$
2013-06-05T12:05:59.488Z cpu2:4222)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "t10.ATA_____WDC_WD20EFRX2D68AX9N0_________________________WD2DWCC1T0407884" state in doubt; requested fast path state update...
2013-06-05T12:05:59.488Z cpu2:4222)ScsiDeviceIO: 2331: Cmd(0x4124007e3e40) 0x12, CmdSN 0x2723 from world 0 to dev "t10.ATA_____WDC_WD20EFRX2D68AX9N0_________________________WD2DWCC1T0407884" failed H:0x5 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2013-06-05T12:06:06.501Z cpu2:8231)VSCSI: 2374: handle 8219(vscsi0:0):Reset request on FSS handle 186460 (7 outstanding commands) from (vmm0:magnet3backup)
2013-06-05T12:06:06.501Z cpu2:8231)VSCSI: 2374: handle 8220(vscsi0:2):Reset request on FSS handle 219229 (0 outstanding commands) from (vmm0:magnet3backup)
2013-06-05T12:06:06.501Z cpu1:4170)VSCSI: 2653: handle 8219(vscsi0:0):Reset [Retries: 0/0] from (vmm0:magnet3backup)
2013-06-05T12:06:06.501Z cpu2:8231)VSCSI: 2374: handle 8221(vscsi0:3):Reset request on FSS handle 743518 (29 outstanding commands) from (vmm0:magnet3backup)
2013-06-05T12:06:06.501Z cpu1:4170)ScsiDeviceIO: 2300: Cmd(0x4124007f4700) 0x2a, CmdSN 0x800e003d from world 8191 to dev "t10.ATA_____WDC_WD20EFRX2D68AX9N0_________________________WD2DWCC1T0407884" failed H:0x8 D:0x0 P:0x0
2013-06-05T12:06:06.501Z cpu1:4170)ScsiDeviceIO: 2300: Cmd(0x4124007de140) 0x2a, CmdSN 0x800e0071 from world 8191 to dev "t10.ATA_____WDC_WD20EFRX2D68AX9N0_________________________WD2DWCC1T0407884" failed H:0x8 D:0x0 P:0x0
2013-06-05T12:06:06.501Z cpu1:4170)VSCSI: 2653: handle 8220(vscsi0:2):Reset [Retries: 0/0] from (vmm0:magnet3backup)
2013-06-05T12:06:06.501Z cpu1:4170)VSCSI: 2450: handle 8220(vscsi0:2):Completing reset (0 outstanding commands)
2013-06-05T12:06:06.501Z cpu1:4170)VSCSI: 2653: handle 8221(vscsi0:3):Reset [Retries: 0/0] from (vmm0:magnet3backup)
2013-06-05T12:06:06.501Z cpu1:4170)ScsiDeviceIO: 2300: Cmd(0x4124007e6040) 0x2a, CmdSN 0x800e0011 from world 8191 to dev "t10.ATA_____WDC_WD20EFRX2D68AX9N0_________________________WD2DWCC1T0407884" failed H:0x8 D:0x0 P:0x0
2013-06-05T12:06:06.501Z cpu1:4170)ScsiDeviceIO: 2300: Cmd(0x41240080cfc0) 0x2a, CmdSN 0x800e007f from world 8191 to dev "t10.ATA_____WDC_WD20EFRX2D68AX9N0_________________________WD2DWCC1T0407884" failed H:0x8 D:0x0 P:0x0
2013-06-05T12:06:06.501Z cpu1:4170)ScsiDeviceIO: 2300: Cmd(0x412400808cc0) 0x2a, CmdSN 0x800e0005 from world 8191 to dev "t10.ATA_____WDC_WD20EFRX2D68AX9N0_________________________WD2DWCC1T0407884" failed H:0x8 D:0x0 P:0x0
2013-06-05T12:06:06.501Z cpu1:4170)ScsiDeviceIO: 2300: Cmd(0x4124008085c0) 0x2a, CmdSN 0x800e0039 from world 8191 to dev "t10.ATA_____WDC_WD20EFRX2D68AX9N0_________________________WD2DWCC1T0407884" failed H:0x8 D:0x0 P:0x0
2013-06-05T12:06:06.501Z cpu1:4170)ScsiDeviceIO: 2300: Cmd(0x4124007d5ac0) 0x2a, CmdSN 0x800e0006 from world 8191 to dev "t10.ATA_____WDC_WD20EFRX2D68AX9N0_________________________WD2DWCC1T0407884" failed H:0x8 D:0x0 P:0x0
2013-06-05T12:06:06.501Z cpu1:4170)VSCSI: 2450: handle 8219(vscsi0:0):Completing reset (0 outstanding commands)
2013-06-05T12:06:09.532Z cpu2:8231)WARNING: VSCSI: 3481: handle 8221(vscsi0:3):WaitForCIF: Issuing reset; number of CIF:30
2013-06-05T12:06:09.532Z cpu2:8231)WARNING: VSCSI: 2416: handle 8221(vscsi0:3):Ignoring double reset
2013-06-05T12:06:22.060Z cpu0:4157)HBX: 2441: Waiting for timed out [HB state abcdef02 offset 3883008 gen 49 stampUS 101905355158 uuid 51ad9aff-49ab3864-7ad4-001e67340ea1 jrnl <FB 1858000> drv 14.58] on vol 'wd_red1'
2013-06-05T12:06:32.383Z cpu1:4192)VMW_SATP_LOCAL: satp_local_updatePathStates:454: Failed to update path "vmhba34:C0:T0:L0" state. Status=Transient storage condition, suggest retry
2013-06-05T12:06:36.944Z cpu1:4170)VSCSI: 2653: handle 8221(vscsi0:3):Reset [Retries: 1/0] from (vmm0:magnet3backup)
2013-06-05T12:06:55.907Z cpu0:4113)<6>ata3: ahci_port_reset: FAILED
2013-06-05T12:06:55.907Z cpu0:4113)<6>ahci_scsi_abort: cmd 0x2a (0x412401a41a00), FAILED
2013-06-05T12:06:55.907Z cpu0:4113)WARNING: LinScsi: SCSILinuxAbortCommands:1816:Failed, Driver ahci, for vmhba34
2013-06-05T12:06:55.907Z cpu0:4113)<6>ahci_scsi_abort: cmd 0x2a (0x412401ad1ac0), entering...
2013-06-05T12:06:55.907Z cpu0:4113)<7>ata3: ahci_port_reset, entering...
2013-06-05T12:06:57.047Z cpu3:69489)WARNING: VSCSI: 3481: handle 8222(vscsi4:0):WaitForCIF: Issuing reset; number of CIF:1
2013-06-05T12:06:57.047Z cpu3:69489)VSCSI: 2374: handle 8222(vscsi4:0):Reset request on FSS handle 1011988 (1 outstanding commands) from (vmm0:magnet5xp)
2013-06-05T12:06:57.047Z cpu1:4170)VSCSI: 2653: handle 8222(vscsi4:0):Reset [Retries: 0/0] from (vmm0:magnet5xp)
2013-06-05T12:06:57.047Z cpu1:4170)VSCSI: 2450: handle 8222(vscsi4:0):Completing reset (0 outstanding commands)
2013-06-05T12:06:58.120Z cpu1:4113)<6>ata3: ahci_port_reset, hard reseting port
2013-06-05T12:07:06.978Z cpu0:4170)VSCSI: 2653: handle 8221(vscsi0:3):Reset [Retries: 2/0] from (vmm0:magnet3backup)
2013-06-05T12:07:13.386Z cpu2:74325)VMW_SATP_LOCAL: satp_local_updatePathStates:454: Failed to update path "vmhba34:C0:T0:L0" state. Status=Transient storage condition, suggest retry
2013-06-05T12:07:15.064Z cpu0:4157)HBX: 2441: Waiting for timed out [HB state abcdef02 offset 3883008 gen 49 stampUS 101905355158 uuid 51ad9aff-49ab3864-7ad4-001e67340ea1 jrnl <FB 1858000> drv 14.58] on vol 'wd_red1'
2013-06-05T12:07:37.000Z cpu2:4170)VSCSI: 2653: handle 8221(vscsi0:3):Reset [Retries: 3/0] from (vmm0:magnet3backup)
Did esxi host reinstallation solved your problem?
Same issues here, but vmnics start to go down/up/down/up/etc. just shortly after the ahci reset. I'm running NSX and dvSwitches.
Did you check for the any Harddrive failure? RAID issues ?
if it goes in active check the kernel and Device latency . since it is on Local storage i am sure it should be with disk problems .