VMware Cloud Community
Gan_Buri_Gan
Contributor
Contributor

Missing .vmdk file from datastore on iscsi (NFS) ZFS pool

We recently suffered a power outage that lasted for longer than our UPS was able to supply.

This caused an unexpected power-off at an ESX host that houses two virtual machines - a FreeBSD that has a ZFS pool for storage purposes over NFS

And a second virtual machine - a Dev Windows Server, whose VMWare files (including the .vmdk) were located on a datastore, which was in fact part of the ZFS pool, and was accessible over iscsi/NFS.


After the outage, the Dev server remained inaccessible until the FreeBDS was brought up, which was to be expected.


However, once the NFS datastore was accessible, and I tried powering on the second machine, I got an error message that the .vmdk file was missing!


I've tried several things thus far - running diagnostics on the ZFS pool seemed to show no problems with it, but I'm currently running a scrub on it, just in case...


The file is simply "not there" - i.e. not accessible either through NFS, or locally, within the shared mount on the ZFS - and the "Used space" does not account for a file of that size, either...I am at a total loss, and would greatly appreciate any help regarding a repair/recovery procedure anyone may suggest.


Thanks in advance!

3 Replies
virtualg_uk
Leadership
Leadership

SSH to the host that has access to the datastore where the VMDK was located and "ls" the VM folder ie:

ls- alh /vmfs/volumes/datastore-name/vm-name

Does that output show a vmName-flat.vmdk file?

If you see vmName-flat.vmdk but are missing a vmName.vmdk then your VMDK descriptor file is lost, however your data is still there.


You can recreate the missing descriptor file with this KB Recreating a missing virtual machine disk descriptor file (1002511) | VMware KB


Let us know how you get on.


Graham | User Moderator | https://virtualg.uk
Gan_Buri_Gan
Contributor
Contributor

Unfortunately, no such files. As a matter of fact, no large files at all.

Some piece of additional info, which I omitted previously, but may or may not be, of importance.

After the FreeBSD ZFS machine was brought up, the host kept on showing the Win Dev VM as "inaccessible". So I proceeded to remove it from inventory, I browsed the datastore to the .vmx file and select "Add to Inventory". It was successfully added, and when I tried to power it on, upon reaching 35%, the host brought up an error message, asking if I moved, copied, or did smth else with the machine...I selected that I moved it, and from that point on, the only error I've been getting, is that the .vmdk file is missing...

Also, there are several vmware-xxx.log files, one of which was huge, prior to my failed attempt...now they are all showing a size of 46.3K.

Is there a way, with a VMWare or linux tool, to scan the folder on a lower-level, and check if the filesystem still has the data on it? I am certain that the power outage did something with the NFS/iSCSI connection that "broke" the file, as it was being accessed at that point...

Btw, the scrubbing of the ZFS passed, and reported:


scrub repaired 0 in 5h36m with 0 errors on Thu Nov 24 01:24:38 2016

So, the ZFS is intact.

Reply
0 Kudos
continuum
Immortal
Immortal

Finding lost vmdks is  completely different with iSCSI and NFS.
Please tell us which of both you actually use.


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

Reply
0 Kudos