Unable to add existing VMDK to new VM

[tk42]

I lost a previous installation of esxi 6.5 due to boot drive failure.  This installation had 2 data stores: one on the boot drive ("VM_POOL") and another 7TB one on a RAID6 configuration ("DELL_RAID6").  This 7TB data store mainly contained one large virtual disk which was used as a secondary drive on a Windows VM.  I lost that VM but I still have that VMDK.

I replaced the boot drive and installed esxi again, but this time with v6.7 U1 A03 (based on a Dell customized ISO). The new system booted up without issues and automatically found the 7TB data store.  I created a new datastore "VM_POOL" for a new Windows VM.  Then I tried to add the VMDK to this new VM as a secondary drive but this is where the problem is. I am getting an error when I try to do so:

• Failed to add disk 'scsi0:1'.
• Failed to power on 'scsi0:1'.
• Cannot open the disk '/vmfs/volumes/59349365-08186f18-21e2-1866da64be1d/Server/Server_1.vmdk' or one of the snapshot disks it depends on.
• Failed to lock the file

There are actually 2 VMDK files in the data store

server_1.vmdk

server_1-000002.vmdk

I should note, in case it matters, that the new VM_POOL is based on VMFS6 whereas the old Dell_RAID6 is on VMFS5.  I hope somebody has encountered the same problem and knows how to resolve it.  Thanks for your feedback in advance.

[tk42]

Does anybody have any ideas about how to resolve this problem?

[a_p_]

First of all, "server_1-000002.vmdk" indicates that the virtual disk has a snapshot. In order to avoid data loss, it is important to mount the snapshot .vmdk file to the VM's configuration rather than the base virtual disk.

To get an overview of the virtual disk's configuration, please enable SSH on the host and connect to is using e.g. putty.

Then go the the virtual disks folder (cd /vmfs/volumes/DELL_RAID6), run ls -lisa, and provide the command's text output (please resize the putty window to avoid unneccessary line breaks).

The output should show four .vmdk files, two small ones, and two large ones with flat/delta/sesparse in their names.

For the small ones, run cat <filename> and paste the commands text output into a reply post.

André

[tk42]

Hi Andre,

Thank you for your response.  Here is the output of the ls command, listing the four files you were referring to:

[root@localhost:/vmfs/volumes/59349365-08186f18-21e2-1866da64be1d/Server] ls -lisa

total 7432774664

   1732      8 drwxr-xr-x    1 root     root           840 Sep  9 02:32 .

      4   1024 drwxr-xr-t    1 root     root          1540 Sep  4 15:19 ..

41944772 3822243840 -rw-------    1 root     root     3929755353088 Oct  8 23:19 Server_1-000002-sesparse.vmdk

46139076      0 -rw-------    1 root     root           337 Mar  6  2019 Server_1-000002.vmdk

4196036 3610529792 -rw-------    1 root     root     7993449533440 Oct 10 19:02 Server_1-flat.vmdk

29361860      0 -rw-------    1 root     root           479 Sep  9 02:32 Server_1.vmdk

The cat command on the bottom file - Server_1.vmdk - reveals the following information:

[root@localhost:/vmfs/volumes/59349365-08186f18-21e2-1866da64be1d/Server] cat  Server_1.vmdk
# Disk DescriptorFile
version=1
encoding="UTF-8"
CID=bc9e3af1
parentCID=ffffffff
createType="vmfs"

# Extent description
RW 15612206120 VMFS "Server_1-flat.vmdk"

# The Disk Data Base
#DDB

ddb.adapterType = "lsilogic"
ddb.geometry.cylinders = "971814"
ddb.geometry.heads = "255"
ddb.geometry.sectors = "63"
ddb.longContentID = "ccfbd85fd4f5dfc9947db8e6bc9e3af1"
ddb.thinProvisioned = "1"
ddb.uuid = "60 00 C2 98 a7 ff 67 02-28 71 89 9d 19 49 74 76"
ddb.virtualHWVersion = "11"

The same command on the other file - Server_1-000002.vmdk shows this:

[root@localhost:/vmfs/volumes/59349365-08186f18-21e2-1866da64be1d/Server] cat  Server_1-000002.vmdk
# Disk DescriptorFile
version=1
encoding="UTF-8"
CID=b7078c89
parentCID=ed6b8dd8
isNativeSnapshot="no"
createType="seSparse"
parentFileNameHint="Server_1.vmdk"
# Extent description
RW 15612206120 SESPARSE "Server_1-000002-sesparse.vmdk"

# The Disk Data Base
#DDB

ddb.grain = "8"
ddb.longContentID = "e7eecaac0b13bc78491855e5b7078c89"

[a_p_]

Some things that I think cause serious issues:

You've started the VM using "Server_1.vmdk" instead of "Server_1-000002.vmdk" which broke the snapshot chain. To fix this, set the "parentCID" in "Server_1-000002.vmdk" to the "CID" value from "Server_1.vmdk", then manually edit the VM's configuration ".vmx" file, and replace "Server_1.vmdk" with "Server_1-000002.vmdk". Finally you have to reload the VM (see https://kb.vmware.com/s/article/1026043​)

The above will fix the snapshot chain as well as the VM's configuration, but there's another issue you likely ran into. According to the file list that you've provided, the 7TB datastore is full. In addition to the thin provisioned 7TB virtual disk, there's a snapshot, and both files together consume ~7TB disk space. Please check how much free disk space ESXi reports. Even if the datastore is not already out of disk space, it may be soon, and there's not enough free disk space to delete/consolidate the snapshot.

To delete/consolidate the snapshot you either need another datastore with ~ 7TB free disk sapce, or backup the VM, and restore it, which should also consolidate the snapshot data to the base disk.

Do you have a current backup? If not, you should take one now to avoid data loss!

André

[tk42]

I just bought a large TB drive to back up those files.  Once I have that in place, I will perform the operation you mention and report back.

[a_p_]

Is this an internal drive which you are going to attach to the RAID controller, and present it to ESXi as another LUN on which a datastore can be created? In this case we may be able to clone the current virtual disk (including the snapshot) to a new virtual disk on the second datastore using e.g. the vmkfstools command.

In case this is an external drive, and you want to download the virtual disks from the datastore, be aware that thin provisioned virtual disks will inflate to their full provisioned size!

André

[tk42]

The new disk is an internal 3.5" Seagate Exos drive but I can't install it because the chassis is configured for 2.5" SAS drives.  It is meant for other purposes after this ordeal is over.  For now I am using it as an external drive via USB dock.

I am not able to download the VMDK files to my PC.  When Windows prompts to save the file, it only shows it with 426 bytes and the web page gives me a HTTP404 error, page cannot be found:

My plan at this point is to attach the external drive to the server via USB and copy the files locally.

[a_p_]

What should work, is to set the ESXi host back into Evaluation Mode (if you are using a free Hypervisor key), and use e.g. VeeamZIP to backup the VM.

This approach however requires that you fix the snapshot chain prior to doing the backup.

Once successfully backed up, you may wipe the VM from the host, and restore it from the backup.

If you want to download the files manually, enable SSH on the host and use WinSCP to do the job.

André

[tk42]

Andre, that's what I'm doing now, using WinSCP to back up the files.  For some reason I'm only getting 22MB/s over the network; this is going to take a while....