Re: ESXi 4 - Datastore Missing

cimgroup · ‎07-23-2012

we currently have a LUN with 2 datastores. 1 is being used for the VM and another was creeated to be mounted as additional storage (E$.) Users complained of not having access to the fileshare that we have on the datastore (E$) so i loooked at it and it was showing 3,000,000TB with -1,500,00 free space. The datastore was originally created with 608GB of free space. how could that be?

Looked at articles about the issue and some recommneded reatrting the VM, didthat and now the whoel datastore is missing. It is still mounted to the VM but the datastore is no where to be found. Also, running "esxcfg-scsidevs -c" only shows 1 datastore and the LUN. if i am correct, it should show 3 one for each of the datastores and one for the LUN. it seems like the missign datastoree went back to the LUN as the free space is showing the combinded space for both.

Is it possible to retrieve the datastore and all the data? please let me know.

sa2057 · ‎07-24-2012

Hi,

Check this

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=101138...

Thanks

SA

cimgroup · ‎07-24-2012

Thank you for the suggestion. Unfortunately, the

esxcfg-volume -l

returns no output for us. We've also found many articles describing how to recreate the partition table to find the datastorage that has disappeared, but when we run the fdisk command, we get the following:

fdisk: cannot read from /vmfs/devices/disks/mpx.vmhba1:C0:T1:L0

We thought it could be a hardware error, but the device still appears (our datastore should be in mpx.vmhba1:C0:T1:L0)...

~ # esxcfg-scsidevs -c
Device UID           Device Type      Console Device                           Size      Multipath PluginDisplay Name
mpx.vmhba0:C0:T0:L0 CD-ROM           /vmfs/devices/cdrom/mpx.vmhba0:C0:T0:L0 0MB       NMP     Local TEAC CD-ROM (mpx.vmhba0:C0:T0:L0)
mpx.vmhba1:C0:T0:L0 Direct-Access    /vmfs/devices/disks/mpx.vmhba1:C0:T0:L0 429215MB NMP     Local VMware Disk (mpx.vmhba1:C0:T0:L0)
mpx.vmhba1:C0:T1:L0 Direct-Access    /vmfs/devices/disks/mpx.vmhba1:C0:T1:L0 1716860MB NMP     Local VMware Disk (mpx.vmhba1:C0:T1:L0)

when we run

fdisk -lu mpx.vmhba1:C0:T1:L0

we get blank output.

If we try to create a brand new store in the GUI, we get the following error:

Call "HostDatastoreSystem.CreateVmfsDatastore" for object "ha-datastoresystem" on ESXi "192.168.xxx.xxx" failed.

Any thoughts?

sa2057 · ‎07-25-2012

Hi,

Check this..

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=103792...

Thanks

SA

sa2057 · ‎07-25-2012

Hi,

Check this KB..

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=100888...

Hope this helps

Thanks

SA

cimgroup · ‎07-25-2012

Thank you for the links/suggestions. In our research we've come across both with no success.

1. Our storage array is a built in Smart Array controller in the DL380 G5 box. The datastore/partition had been up and running for months, so I don't think it is related to the Dell article. There is another partition on the same controller that is active and working properly. That said, can I add a claim rule for the HP controller? What would the format be? It is a Smart Array P400 controller.

2. If we run an fdisk -l on the partition there's no response. If we run just an fdisk, we get an error that it cannot read from the device.

3. If I try to run 'parted' from an SSH session i get an error that 'parted: not found' We're running ESXi 4.1

To recap:

If I run esxcfg-scsidevs -l I get output which includes the partition I'm trying to create a volume/data store on. This is the partition that HAD a datastore on it but that dropped suddenly.

mpx.vmhba1:C0:T1:L0
   Device Type: Direct-Access
   Size: 1716860 MB
   Display Name: Local VMware Disk (mpx.vmhba1:C0:T1:L0)
   Multipath Plugin: NMP
   Console Device: /vmfs/devices/disks/mpx.vmhba1:C0:T1:L0
   Devfs Path: /vmfs/devices/disks/mpx.vmhba1:C0:T1:L0
   Vendor: VMware    Model: Block device      Revis: 1.0
   SCSI Level: 2 Is Pseudo: false Status: on
   Is RDM Capable: false Is Removable: false
   Is Local: true
   Other Names:
      vml.0000000000766d686261313a313a30
   VAAI Status: unknown

If I do an fdisk -l on the Host I do not get any thing that uses that disk.

If I run ~ # fdisk /dev/disks/mpx.vmhba1:C0:T1:L0

I get : fdisk: cannot read from /dev/disks/mpx.vmhba1:C0:T1:L0

My /var/log/messages has:

Jul 25 12:06:59 vmkernel: 0:11:32:57.485 cpu1:93047)<4>cciss: cmd 0x4100b5403000 has CHECK CONDITION byte 2 = 0x3
Jul 25 12:06:59 vmkernel: 0:11:32:57.485 cpu1:93047)NMP: nmp_CompleteCommandForPath: Command 0x28 (0x41027f39e940) to NMP device "mpx.vmhba1:C0:T1:L0" failed on physical path "vmhba1:C0:T1:L0" H:0x3 D:0x0 P:0x0 Possible sense data: 0x5 0x20 0x0.
Jul 25 12:06:59 vmkernel: 0:11:32:57.485 cpu1:93047)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe: NMP device "mpx.vmhba1:C0:T1:L0" state in doubt; requested fast path state update...
Jul 25 12:06:59 vmkernel: 0:11:32:57.485 cpu1:93047)ScsiDeviceIO: 1688: Command 0x28 to device "mpx.vmhba1:C0:T1:L0" failed H:0x3 D:0x0 P:0x0 Possible sense data: 0x5 0x20 0x0.
Jul 25 12:06:59 vmkernel: 0:11:32:57.485 cpu1:93047)<4>cciss: cmd 0x4100b5403000 has CHECK CONDITION byte 2 = 0x3
Jul 25 12:06:59 vmkernel: 0:11:32:57.485 cpu1:93047)NMP: nmp_CompleteCommandForPath: Command 0x28 (0x41027f39e940) to NMP device "mpx.vmhba1:C0:T1:L0" failed on physical path "vmhba1:C0:T1:L0" H:0x3 D:0x0 P:0x0 Possible sense data: 0x5 0x20 0x0.
Jul 25 12:06:59 vmkernel: 0:11:32:57.485 cpu1:93047)ScsiDeviceIO: 1688: Command 0x28 to device "mpx.vmhba1:C0:T1:L0" failed H:0x3 D:0x0 P:0x0 Possible sense data: 0x5 0x20 0x0.
Jul 25 12:06:59 vmkernel: 0:11:32:57.485 cpu1:93047)WARNING: Partition: 801: Partition table read from device mpx.vmhba1:C0:T1:L0 failed: I/O error
Jul 25 12:06:59 vmkernel: 0:11:32:57.485 cpu1:93047)<4>cciss: cmd 0x4100b5403000 has CHECK CONDITION byte 2 = 0x3
Jul 25 12:06:59 vmkernel: 0:11:32:57.485 cpu1:93047)NMP: nmp_CompleteCommandForPath: Command 0x28 (0x41027f39e940) to NMP device "mpx.vmhba1:C0:T1:L0" failed on physical path "vmhba1:C0:T1:L0" H:0x3 D:0x0 P:0x0 Possible sense data: 0x5 0x20 0x0.
Jul 25 12:06:59 vmkernel: 0:11:32:57.485 cpu1:93047)ScsiDeviceIO: 1688: Command 0x28 to device "mpx.vmhba1:C0:T1:L0" failed H:0x3 D:0x0 P:0x0 Possible sense data: 0x5 0x20 0x0.

Thanks!

cimgroup · ‎07-25-2012

We've made some progress, but are still not there yet. Part of our issues are compounded by the fact the server is remote and we do not have an onsite IT person. We were able to get into the ACU and see the partition had a bad RAID array. The array was reconfigured and VMWare is now seeing the datastore if we do an Fdisk -l. The datastore is not showing up under 'Storage' in the GUI. If we run a VMKFSTools -V we see the datastore pop in and out of Storage with a variety of sizes and then ultimatley leave.

Under /vmfs/volumes there is a Symbolic link for the friendly name of the datastore to the GUID storage. The GUID folder does not exist.

Ultimatley we ahve clean backups and can recreate the datastore and move on, but if there's a way to recover, that would be significantly better.

Any thoughts?

sa2057 · ‎07-26-2012

Hi,

Check the attachment for vmfs recovery and also check the below link ..

http://www.virtualizationteam.com/virtualization-vmware/vmware-vi3-virtualization-vmware/vmware-esx-...

Thanks

SA

cimgroup · ‎07-26-2012

I'm not sure this will remedy the issue... when using fdisk, our missing datastore's partition displays correctly:

Device Start End Blocks Id System
/dev/disks/mpx.vmhba1:C0:T1:L0p1 128 1279255949 639627911 fb VMFS

There doesn't appear to be a need to recreate the partition

cimgroup · ‎07-26-2012

Thanks for all of your help. Unfortunately, it did not get us where we needed to be, but it was a good learning process. We ended up just creating a new datastore as we had backups of all of the data.

Thanks.

All

ESXi 4 - Datastore Missing