VMFS5 local data store input/output error

[virtualserver4]

Hi,

I have 3 local hard-disk on my IBM server. 1st drive of 250 GB I have used to create Linux Data Store (datastore1) and from other 2 drives I have created Windows Data Store (datastore2)

Couple of VMs are deployed on both the data stores. Over week end suddenly Windows Data Store (datastore2) vms got down. When I checked on Monday found that datastore is visible to ESXi host but I am unable to browse to VMs.

If I do cd VM directory I get below error:

~ # cd /vmfs/volumes/Windows\ Data\ Store/Win-2k8-2/

sh: getcwd: Input/output error

(unknown) #

~ # esxcfg-scsidevs -l

mpx.vmhba32:C0:T0:L0

   Device Type: Direct-Access

   Size: 1911 MB

   Display Name: Local USB Direct-Access (mpx.vmhba32:C0:T0:L0)

   Multipath Plugin: NMP

   Console Device: /vmfs/devices/disks/mpx.vmhba32:C0:T0:L0

   Devfs Path: /vmfs/devices/disks/mpx.vmhba32:C0:T0:L0

   Vendor: SMART     Model: USB-IBM           Revis: 0000

   SCSI Level: 2  Is Pseudo: false Status: on

   Is RDM Capable: false Is Removable: true

   Is Local: true  Is SSD: false

   Other Names:

      vml.0000000000766d68626133323a303a30

   VAAI Status: unsupported

mpx.vmhba33:C0:T0:L0

   Device Type: CD-ROM

   Size: 0 MB

   Display Name: Local USB CD-ROM (mpx.vmhba33:C0:T0:L0)

   Multipath Plugin: NMP

   Console Device: /vmfs/devices/cdrom/mpx.vmhba33:C0:T0:L0

   Devfs Path: /vmfs/devices/cdrom/mpx.vmhba33:C0:T0:L0

   Vendor: KVM       Model: vmDisk-CD         Revis: 0.01

   SCSI Level: 2  Is Pseudo: false Status: on

   Is RDM Capable: false Is Removable: true

   Is Local: true  Is SSD: false

   Other Names:

      vml.0005000000766d68626133333a303a30

   VAAI Status: unsupported

naa.5000c500478aee93

   Device Type: Direct-Access

   Size: 286102 MB

   Display Name: IBM-ESXS Serial Attached SCSI Disk (naa.5000c500478aee93)

   Multipath Plugin: NMP

   Console Device: /vmfs/devices/disks/naa.5000c500478aee93

   Devfs Path: /vmfs/devices/disks/naa.5000c500478aee93

   Vendor: IBM-ESXS  Model: ST9300605SS       Revis: B556

   SCSI Level: 6  Is Pseudo: false Status: degraded

   Is RDM Capable: true  Is Removable: false

   Is Local: false Is SSD: false

   Other Names:

      vml.02000000005000c500478aee93535439333030

   VAAI Status: unknown

naa.5000c500478b8d3b

   Device Type: Direct-Access

   Size: 286102 MB

   Display Name: IBM-ESXS Serial Attached SCSI Disk (naa.5000c500478b8d3b)

   Multipath Plugin: NMP

   Console Device: /vmfs/devices/disks/naa.5000c500478b8d3b

   Devfs Path: /vmfs/devices/disks/naa.5000c500478b8d3b

   Vendor: IBM-ESXS  Model: ST9300605SS       Revis: B556

   SCSI Level: 6  Is Pseudo: false Status: degraded

   Is RDM Capable: true  Is Removable: false

   Is Local: false Is SSD: false

   Other Names:

      vml.02000000005000c500478b8d3b535439333030

   VAAI Status: unknown

naa.5000c500478d724f

   Device Type: Direct-Access

   Size: 286102 MB

   Display Name: IBM-ESXS Serial Attached SCSI Disk (naa.5000c500478d724f)

   Multipath Plugin: NMP

   Console Device: /vmfs/devices/disks/naa.5000c500478d724f

   Devfs Path: /vmfs/devices/disks/naa.5000c500478d724f

   Vendor: IBM-ESXS  Model: ST9300605SS       Revis: B556

   SCSI Level: 6  Is Pseudo: false Status: degraded

   Is RDM Capable: true  Is Removable: false

   Is Local: false Is SSD: false

   Other Names:

      vml.02000000005000c500478d724f535439333030

   VAAI Status: unknown

~ # esxcfg-scsidevs -m

naa.5000c500478d724f:1                                           /vmfs/devices/disks/naa.5000c500478d724f:1 4f83af28-52abeaf2-6475-5cf3fcba46f2  0  Linux Data Store

naa.5000c500478b8d3b:3                                           /vmfs/devices/disks/naa.5000c500478b8d3b:3 4f7cfc39-4d17993a-a08c-5cf3fcba46f2  0  Windows Data Store

naa.5000c500478aee93:1                                           /vmfs/devices/disks/naa.5000c500478aee93:1 4f7cfc39-4d17993a-a08c-5cf3fcba46f2  1  Windows Data Store

Below is output from dmesg: I suspect to be a hard-disk issue

2014-06-16T10:46:25.385Z cpu5:382346)WARNING: UserObj: 675: Failed to crossdup fd 8, /dev/ptyp0 type CHAR: Busy

2014-06-16T10:46:25.385Z cpu5:382346)WARNING: UserObj: 675: Failed to crossdup fd 10, /dev/ptyp0 type CHAR: Busy

2014-06-16T10:46:25.385Z cpu2:382348)WARNING: UserLinux: 1340: unsupported: (void)

2014-06-16T10:46:52.510Z cpu16:4112)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x28 (0x412440db9c40) to dev "naa.5000c500478b8d3b" on path "vmhba2:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x4 0x32 0x0.Act:NONE

2014-06-16T10:46:52.510Z cpu16:4112)ScsiDeviceIO: 2316: Cmd(0x412440db9c40) 0x28, CmdSN 0x2a to dev "naa.5000c500478b8d3b" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x4 0x32 0x0.

2014-06-16T10:47:34.631Z cpu18:4114)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x28 (0x412441635000) to dev "naa.5000c500478b8d3b" on path "vmhba2:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x4 0x32 0x0.Act:NONE

2014-06-16T10:47:34.631Z cpu18:4114)ScsiDeviceIO: 2316: Cmd(0x412441635000) 0x28, CmdSN 0x1c to dev "naa.5000c500478b8d3b" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x4 0x32 0x0.

2014-06-16T10:47:41.476Z cpu18:4114)ScsiDeviceIO: 2316: Cmd(0x412440da3500) 0x28, CmdSN 0x21 to dev "naa.5000c500478b8d3b" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x4 0x32 0x0.

Please guide me to fix this issue. Thank you all in advance

/virtu4

[virtualserver4]

Just to add... I am able to create files on Windows data store, but when I browse it through GUI it wont list the files inside VM folder

[virtualserver4 · Accepted solution]

This issue is resolved.

Thanks to communities for not replying, which made me to solve this problem myself

/virt4

[GinoLF]

virtualserver4‌ may i know how you solve at the end?

[continuum]

The original post is from 2014 - at that time it was not a popular problem.
Today it is different and happens way more often.
I dont know how OP worked aound it ... I nowadays have 3 quite obscure procedures that make sense.
Nasty thing with this particular problem : typically everybody will consider this issue as something hardware related.
So VMware support usually gets away with showing you some vmkernel-logs. So the support-case you may have opened gets relabelled as "Recovery-case and per definition VMware supports can run away again and so once the problem "flat.vmdks becomes unreadable with I/O error" can be blamed to an undefined hardware issue - the chance that the enduser gets help with the recovery goes down the drain.
My personal statistic for what I call the ESXi-Diva-mode problem is not95% hardware - 5% ESXi-VMFS-problem but more like 60% ESXi-VMFS-issue and only 40% hw related.

Anyway - all cases that I managed to resolve in the last few weeks were done by using a Linux-LiveCD-VM and carefully avoiding everything that could make ESXi think that a normal host from the cluster tries to read the files with I/O errors.
When possible read the files with the I/O-errors not from /vmfs/volumes but rather from Linux in ro only mode in /mnt/esxi/dev/disks/<device>.
Instead of copying a flat.vmdk I recommend to copy a set of pieces you specify as a long list with dd-commands.
When that also fails use a vmfs-fuse build - but not the outdated one from Ubuntu and other repositories.
Sorry - I see that this does not really help you -  but at the moment I am not aware of a straightforward procudure that works in the majority of cases.
When I found one I will write instructions ....

Questions - see my signature
Ulli