The dissappearing datastore

[gsmythe]

Sunday evening I noticed that a few of my VMs rebooted and couldn't find their boot disk, or were having serious issues. In ESXI 6 I noticed that they were going "Inaccessible" and the datastore was missing!

I have been trying various things around the internet and I'm running out of options. I hope someone can point me in the right direction.

I have storage attached to the server via FC. I have already rebooted everything, and even replaced the FC adapters, fiber, and rebooted the switch.

Under "Storage Adapters" I can mount the disk, but it can't find the VMFS.

Here is some information that I hope helps:

naa.600144f0bb49a06400005777aea60001

   Display Name: NEXENTA Fibre Channel Disk (naa.600144f0bb49a06400005777aea60001)

   Has Settable Display Name: true

   Size: 6291456

   Device Type: Direct-Access

   Multipath Plugin: NMP

   Devfs Path: /vmfs/devices/disks/naa.600144f0bb49a06400005777aea60001

   Vendor: NEXENTA

   Model: COMSTAR        

   Revision: 1.0

   SCSI Level: 5

   Is Pseudo: false

   Status: degraded

   Is RDM Capable: true

   Is Local: false

   Is Removable: false

   Is SSD: false

   Is VVOL PE: false

   Is Offline: false

   Is Perennially Reserved: false

   Queue Full Sample Size: 0

   Queue Full Threshold: 0

   Thin Provisioning Status: yes

   Attached Filters:

   VAAI Status: unknown

   Other UIDs: vml.0200000000600144f0bb49a06400005777aea60001434f4d535441

   Is Shared Clusterwide: true

   Is Local SAS Device: false

   Is SAS: false

   Is USB: false

   Is Boot USB Device: false

   Is Boot Device: false

   Device Max Queue Depth: 64

   No of outstanding IOs with competing worlds: 32

   Drive Type: unknown

   RAID Level: unknown

   Number of Physical Drives: unknown

   Protection Enabled: false

   PI Activated: false

   PI Type: 0

   PI Protection Mask: NO PROTECTION

   Supported Guard Types: NO GUARD SUPPORT

   DIX Enabled: false

   DIX Guard Type: NO GUARD SUPPORT

   Emulated DIX/DIF Enabled: false

What I don't get is why it shows status as "degraded"...

[root@localhost:/var/log] partedUtil getptbl /vmfs/devices/disks/naa.600144f0bb49a06400005777aea60001

gpt
802048 255 63 12884901888
1 2048 12884899839 EBD0A0A2B9E5443387C068B6B72699C7 linuxNative 0

Why is the disk showing up as linuxNative?? That's concerning to me, shouldn't it be VMFS??

Any ideas on how to get it mounted and get my VMs back?

Thanks,

Greg

[Vdex_ie]

You may have to recreate the partition table.

Please do the following -

1. get the usable sectors. i have used sample values, please replace the values you get

partedUtil getUsableSectors /vmfs/devices/disks/naa.600144f0bb49a06400005777aea60001

34 2147483614

---------------------------

2. get the offset value (either 128 or 2048)

run this command on the esx host

offset="128 2048";for dev in `esxcfg-scsidevs -l | grep -v mpx | grep "Console Device:" | awk {'print $3'}`; do disk=$dev; echo $disk; partedUtil getptbl $disk; { for i in `echo $offset`;do echo "Checking offset found at $i:"; hexdump -n4 -s $((0x100000+(512*$i))) $disk; hexdump -n4 -s $((0x1300000+(512*$i))) $disk; hexdump -C -n 128 -s $((0x130001d + (512*$i))) $disk; done; } | grep -B 1 -A 5 d00d; echo "---------------------"; done

the value you get is the offset. I am using 2048 for this example

---------------------------

3. set the partition table

partedUtil setptbl /vmfs/devices/disks/naa.600144f0bb49a06400005777aea60001"1 2048 2147483614 4AA31E02A400F11DB9590000C2911D1B8 0"

where =

2147483614 = we got this value when we ran the getusablesectors command

2048 = offset

AA31E02A400F11DB9590000C2911D1B8 = vmfs partition GUID

---------------------------

run the command

partedUtil setptbl /vmfs/devices/disks/naa.600144f0bb49a06400005777aea60001"1 2048 214748361 4AA31E02A400F11DB9590000C2911D1B8 0"

and then run vmkfstools -V to look for new datastore

[gsmythe]

Thank you for the response. After running through that, I get this in my vmkernel.log:

2016-08-18T00:06:28.713Z cpu16:226245)FSS: 5334: No FS driver claimed device 'naa.600144f0bb49a06400005777aea60001:1': No filesystem on the device

2016-08-18T00:06:28.713Z cpu16:226245)VC: 3551: Device rescan time 69 msec (total number of devices 5)

2016-08-18T00:06:28.713Z cpu16:226245)VC: 3554: Filesystem probe time 107 msec (devices probed 5 of 5)

2016-08-18T00:06:28.713Z cpu16:226245)VC: 3556: Refresh open volume time 0 msec

.. Did I loose everything?

[continuum]

.. Did I loose everything?
WAIT ... until we two had a chat.
Get latest MOA iso from vm-sickbay.com - and maybe already create a header-dump
dd if="/dev/disks/naa.600144f0bb49a06400005777aea60001:1" bs=1M count=1536 of="tmp-or-another-datastore/grepsmythe.1536"
download to your host,compress it and provide a link.
Please also forget the idea to fix or repair the affected datastore.In the long run its a waste of time.
Once I saw the dump I normally can give you a list of vmdks and have a procedure in mind that will work.
I call it a day for now  as its 3 again ...
If you can get me the data  ....
Ulli

[gsmythe]

I uploaded the file and shared it with you (I think). Let me know how it goes.

TYVM

[continuum]

‌where did you upload it ?

[gsmythe]

I thought I had shared it via these forums, but now I just put it here: https://www.dropbox.com/s/78hi389ugmo43j1/gregsmythe.zip?dl=0

thanks

[continuum · Accepted solution]

Wow - this VMFS-partition looks like it has been quick formatted with NTFS.
The only references to virtual disks I found were inside  vmx-files

scsi0:0.fileName = "pfSense.vmdk"

scsi0:0.fileName = "dsp-oldschool64.vmdk"

scsi0:0.fileName = "2008R2.vmdk"

scsi0:1.fileName = "2008R2_1.vmdk"

scsi0:2.fileName = "2008R2_2.vmdk"

scsi0:3.fileName = "2008R2_3.vmdk"

scsi0:4.fileName = "2008R2_4.vmdk"

scsi0:0.fileName = "Win7 General box.vmdk"

scsi0:0.fileName = "w7.vmdk"

scsi0:0.fileName = "2012_PXE_SRVR.vmdk"

ide0:0.fileName = "Lucky.vmdk"

ide0:0.fileName = "Alicia.vmdk"

If you did not accidentaly formatted it yourself I would recommend to ask your coworkers and eventually even consider sabotage.
As it was a quick format you may be able to recover thick provisioned vmdks.

Ulli

[gsmythe]

I'm a one-man shop and I wouldn't have tried to format my datastore.. I only use the UI for making VMs too..

I think they are all thin-provisioned. Those were the only vmdk files you found?

So am I SOL?

[continuum]

I did not found any vmdk. Just a couple of vmx-files.
I dont think that this damage can be caused by a user mistake in the UI.

[gsmythe]

ok

I wasn't even in the VMware client when it happened, I noticed it when my Internet stopped working (the pfSense VM is/was my firewall) and then I saw that the VMs were becoming "inaccessible" and that the datastore had disappeared from Configuration/Storage. I wouldn't even know how to format a datastore with NTFS or if that's even possible.

Thanks for looking into it though. Looks like I get to rebuild. Lesson learned

[a_p_]

Just a quick note: Issues like this may be caused by an incorrect LUN presentation, i.e. the VMFS LUN being (accidentally) presented to a Windows host with "Automount" enabled. Please double-check that there's no such presentation on the storage system!

André

[gsmythe]

Thanks for the reply. No VMs are running right now (none are configured, lol)..

[gsmythe]

This got me thinking... That day I did create that 2012_PXE VM and I booted a physical machine via PXE and installed 2012 server. That's the only way I can think of that would have caused it to be formatted as NTFS. I don't know how it could have seen the datastore during the Windows install process though..

[continuum]

I looked at the dump again and can verify that the format created a NTFS partition that looks like it was made by Win2012.
> I don't know how it could have seen the datastore during the Windows install process though..
I agree - this is really strange. If you ever find out how you did this please keep us updated.
Ulli

[gsmythe]

How hard would it be to recover deleted files from VMFS? I have an old server where I moved the hard drive files from it (changed the datastore). I'm not up on how VMFS handles deletes... Could that be a possibility to recover them?

[continuum]

Sorry - missed your reply.
If we assume that the VMFS metadata is not available it is still possible to recover data.
Recovering small files is possible with tools like photorec and others that search files based on their signature.
This will produce loads of files without their correct names - often the results are more or less worthless.
Recovering large files depends on the fragmentation rate inside the virtual  disks they were stored in.
So it is of course desired to recover full vmdks. Chances vary:

- unfragmented thick provisioned vmdks - very good if they are eagerzeroed thick - good if lazyzeroed
- fragmented thick provisioned vmdks - depends on number and size of fragments
- thin provisioned vmdks - unlikely