Can't start VM... missing vmdk?

[LockonComputer]

Hi,

I'm trying to start a VM that was previously working fine... I was trying to delete some old snapshot but now it won't start with the following error:

Failed - File /vmfs/volumes/5a5947b7-8ba9b686-01a9-4ccc6a44e023/Server3/Server3.vmdk was not found

Errors

VMware ESX cannot find the virtual disk "/vmfs/volumes/5a5947b7-8ba9b686-01a9-4ccc6a44e023/Server3/Server3.vmdk". Verify the path is valid and try again.

[deleted some messages]

The system cannot find the file specified

Cannot open the disk '/vmfs/volumes/5a5947b7-8ba9b686-01a9-4ccc6a44e023/Server3/Server3-000001.vmdk' or one of the snapshot disks it depends on.

Module 'Disk' power on failed.

Failed to start the virtual machine.

The file Server3-000001.vmdk is indeed there but Server3.vmdk is not :

-rw-------    1 root     root     26055430144 Mar 10 22:56 Server3-000001-delta.vmdk

-rw-------    1 root     root           372 Aug  6  2018 Server3-000001.vmdk

-rw-------    1 root     root     214748364800 Aug  6  2018 Server3-flat.vmdk

-rw-r--r--    1 root     root           379 Aug  6  2018 Server3.vmsd

-rwxr-xr-x    1 root     root          3502 Mar 10 22:56 Server3.vmx

drwxr-xr-x    1 root     root           700 Mar 10 23:00 tmp

-rw-r--r--    1 root     root         58801 Mar 10 22:57 vmware-1.log

-rw-r--r--    1 root     root         60046 Mar 10 23:00 vmware-2.log

-rw-r--r--    1 root     root         58716 Mar 10 23:02 vmware-3.log

-rw-r--r--    1 root     root         58801 Mar 11 01:13 vmware.log

but Server3.vmdk is not referenced in the config file:

.encoding = "UTF-8"

config.version = "8"

virtualHW.version = "13"

vmci0.present = "TRUE"

floppy0.present = "FALSE"

numvcpus = "4"

memSize = "8192"

bios.bootRetry.delay = "10"

sched.cpu.units = "mhz"

sched.cpu.affinity = "all"

powerType.suspend = "soft"

tools.upgrade.policy = "manual"

scsi0.virtualDev = "lsisas1068"

scsi0.present = "TRUE"

sata0.present = "TRUE"

usb.present = "TRUE"

ehci.present = "TRUE"

scsi0:0.deviceType = "scsi-hardDisk"

scsi0:0.fileName = "Server3-000001.vmdk"

sched.scsi0:0.shares = "normal"

sched.scsi0:0.throughputCap = "off"

scsi0:0.present = "TRUE"

ethernet0.virtualDev = "e1000e"

ethernet0.networkName = "VM Network"

ethernet0.addressType = "generated"

ethernet0.wakeOnPcktRcv = "FALSE"

ethernet0.present = "TRUE"

sata0:0.deviceType = "cdrom-image"

sata0:0.present = "TRUE"

displayName = "Server3"

guestOS = "windows8srv-64"

toolScripts.afterPowerOn = "TRUE"

toolScripts.afterResume = "TRUE"

toolScripts.beforeSuspend = "TRUE"

toolScripts.beforePowerOff = "TRUE"

tools.syncTime = "FALSE"

uuid.bios = "56 4d 71 e3 4a d5 89 47-c2 b9 9c 37 f6 5a 9b c4"

uuid.location = "56 4d 71 e3 4a d5 89 47-c2 b9 9c 37 f6 5a 9b c4"

vc.uuid = "52 47 eb f8 e9 29 45 55-19 d5 ee cf 13 c0 28 90"

sched.cpu.min = "0"

sched.cpu.shares = "normal"

sched.mem.min = "0"

sched.mem.minSize = "0"

sched.mem.shares = "normal"

sata0:0.fileName = "/vmfs/volumes/5a5947b7-8ba9b686-01a9-4ccc6a44e023/ISO/SW_DVD9_Windows_Svr_Std_and_DataCtr_2012_R2_64Bit_English_-4_MLF_X19-82891.ISO"

ethernet0.generatedAddress = "00:0c:29:5a:9b:c4"

vmci0.id = "-161834044"

cleanShutdown = "TRUE"

tools.guest.desktop.autolock = "FALSE"

nvram = "Server3.nvram"

pciBridge0.present = "TRUE"

svga.present = "TRUE"

pciBridge4.present = "TRUE"

pciBridge4.virtualDev = "pcieRootPort"

pciBridge4.functions = "8"

pciBridge5.present = "TRUE"

pciBridge5.virtualDev = "pcieRootPort"

pciBridge5.functions = "8"

pciBridge6.present = "TRUE"

pciBridge6.virtualDev = "pcieRootPort"

pciBridge6.functions = "8"

pciBridge7.present = "TRUE"

pciBridge7.virtualDev = "pcieRootPort"

pciBridge7.functions = "8"

hpet0.present = "TRUE"

RemoteDisplay.maxConnections = "-1"

sched.cpu.latencySensitivity = "normal"

svga.autodetect = "TRUE"

sata0:0.autodetect = "TRUE"

disk.EnableUUID = "TRUE"

numa.autosize.vcpu.maxPerVirtualNode = "4"

numa.autosize.cookie = "40001"

sched.swap.derivedName = "/vmfs/volumes/5a5947b7-8ba9b686-01a9-4ccc6a44e023/Server3/Server3-56cb274c.vswp"

pciBridge0.pciSlotNumber = "17"

pciBridge4.pciSlotNumber = "21"

pciBridge5.pciSlotNumber = "22"

pciBridge6.pciSlotNumber = "23"

pciBridge7.pciSlotNumber = "24"

scsi0.pciSlotNumber = "160"

usb.pciSlotNumber = "32"

ethernet0.pciSlotNumber = "192"

ehci.pciSlotNumber = "33"

vmci0.pciSlotNumber = "34"

sata0.pciSlotNumber = "35"

scsi0.sasWWID = "50 05 05 63 4a d5 89 40"

ethernet0.generatedAddressOffset = "0"

vm.genid = "-7420548292440970496"

vm.genidX = "2594830963590646655"

monitor.phys_bits_used = "43"

vmotion.checkpointFBSize = "16777216"

vmotion.checkpointSVGAPrimarySize = "16777216"

softPowerOff = "TRUE"

usb:1.speed = "2"

usb:1.present = "TRUE"

usb:1.deviceType = "hub"

usb:1.port = "1"

usb:1.parent = "-1"

migrate.hostLog = "./Server3-56cb274c.hlog"

scsi0:0.redo = ""

usb:0.present = "TRUE"

usb:0.deviceType = "hid"

usb:0.port = "0"

usb:0.parent = "-1"

sata0:0.startConnected = "TRUE"

toolsInstallManager.lastInstallError = "0"

svga.guestBackedPrimaryAware = "TRUE"

tools.remindInstall = "FALSE"

toolsInstallManager.updateCounter = "1"

extendedConfigFile = "Server3.vmxf"

The file may have been accidentally deleted and due to some other technical issues we don't have a good backup.

Is it possible that the delta file can be use?

Is it possible to undelete the missing file from the file system?

Any help is greatly appreciated.

[Alex_Romeo]

Hi,

https://buildvirtual.net/recreating-a-missing-vmdk-descriptor-file/

ARomeo

[IRIX201110141]

Stop!

From my point of view ONLY THE TINY descriptor of the base disk is missing.  Its a simple ascii text file and can recreate.  The reference youre looking for is:

*.vmx -> Server3-000001.vmdk -> Server3.vmdk

So take a look into Server3-000001.vmdk and you will see the parent (which is still missing right now. The important data is in the flat and the delta of course!)  There is a KB about how to recreate a missing *.vmdk descriptor.  Iam a lazy guy and always create a Dummy VM with the same vDisk settings as my missing vDisk and than grep the content of the fresh created descriptor and use it to create the missing one. There is one ID which have to match to all of your descriptors (when having a snapshot chain) and as always the reference to the *-flat.vmdk have to match.

Regards,
Joerg

[TheBobkin · Accepted solution]

AlessandroRomeo68, sorry but what you said is missing the point entirely and potentially could cause undue panic - they look to be just missing the descriptor file not the data.

LockonComputer, welcome to Communities.

"but Server3.vmdk is not referenced in the config file:"

It is not referenced as the last vmdk in the snapshot chain is what is referenced (e.g. Server3-000001.vmdk), this in turn points the Server3.vmdk as the parent file - this can be confirmed using:

# cat /vmfs/volumes/5a5947b7-8ba9b686-01a9-4ccc6a44e023/Server3/Server3-000001.vmdk

I would advise attempting to recreate the missing vmdk descriptor file using the steps in this article:

VMware Knowledge Base

Bob

[Alex_Romeo]

Hi TheBobkin,

Yes you are right.

ARomeo

[a_p_]

How exactly did you delete the snapshots? Using the Snapshot Manager, or straight from the datastore?

All that seems to be missing - as already mentioned - is the base/flat disk's descriptor file "Server3.vmdk". If you still have this file (e.g. in the "tmp" directory), copy it back to the original location.

If you don't have it anymore, it needs to be recreated, and - that's important - the snapshot chain needs to be fixed. If you need help with this, then please run ls -lisa in the VM's folder, and post the output in your next reply. In addition to this, attach the "Server3-000001.vmdk" file to the reply.

André

[LockonComputer]

First, thank you to all of you! The community is amazing.

> How exactly did you delete the snapshots? Using the Snapshot Manager, or straight from the datastore?

I actually never got that far because when I tried to delete the snapshots I got another error (I can see if I can pull it from the log if it is important).  The chain of events was basically that the VH host ran out of disk space, I cleaned up some snapshots from the GUI on other VMs, tried to clean up this one and got the mentioned error, rebooted the VM host "as good measure".  Upon restart, all VMs started except this one.  How the missing file went missing is a mystery to me.

> All that seems to be missing - as already mentioned - is the base/flat disk's descriptor file "Server3.vmdk". If you still have this file (e.g. in the "tmp" directory), copy it back to the original location.

I created the tmp directory after the fact when I was trying to set the VM to clean boot.  It contains:

432040196      8 drwxr-xr-x    1 root     root           700 Mar 10 23:00 .

528509188      8 drwxr-xr-x    1 root     root          1960 Mar 11 01:47 ..

692087044   1024 -rw-------    1 root     root         20307 Aug  6  2018 Server3-Snapshot1.vmsn

578840836   1024 -rw-------    1 root     root          8684 Mar 10 22:56 Server3.nvram

704669956      8 -rw-------    1 root     root          3158 Aug  6  2018 Server3.vmxf

> run ls -lisa in the VM's folder, and post the output in your next reply

528509188      8 drwxr-xr-x    1 root     root          1960 Mar 11 01:47 .

      4   1024 drwxr-xr-t    1 root     root          2100 Oct 22 06:03 ..

696281348 25445376 -rw-------    1 root     root     26055430144 Mar 10 22:56 Server3-000001-delta.vmdk

700475652      0 -rw-------    1 root     root           372 Aug  6  2018 Server3-000001.vmdk

536897796 209715200 -rw-------    1 root     root     214748364800 Aug  6  2018 Server3-flat.vmdk

545286404      0 -rw-r--r--    1 root     root           379 Aug  6  2018 Server3.vmsd

532703492      8 -rwxr-xr-x    1 root     root          3502 Mar 10 22:56 Server3.vmx

432040196      8 drwxr-xr-x    1 root     root           700 Mar 10 23:00 tmp

427845892   1024 -rw-r--r--    1 root     root         58801 Mar 10 22:57 vmware-1.log

448817412   1024 -rw-r--r--    1 root     root         60046 Mar 10 23:00 vmware-2.log

465594628   1024 -rw-r--r--    1 root     root         58716 Mar 10 23:02 vmware-3.log

595618052   1024 -rw-r--r--    1 root     root         58801 Mar 11 01:13 vmware-4.log

612395268   1024 -rw-r--r--    1 root     root         58716 Mar 11 01:47 vmware-5.log

629172484   1024 -rw-r--r--    1 root     root         58716 Mar 11 01:47 vmware.log

> attach the "Server3-000001.vmdk" file to the reply.

# Disk DescriptorFile

version=1

encoding="UTF-8"

CID=130e3d82

parentCID=37f94e62

isNativeSnapshot="no"

createType="vmfsSparse"

parentFileNameHint="Server3.vmdk"

# Extent description

RW 419430400 VMFSSPARSE "Server3-000001-delta.vmdk"

# The Disk Data Base

#DDB

ddb.longContentID = "81ff697654c84c4666932563130e3d82"

ddb.toolsInstallType = "1"

ddb.toolsVersion = "10279"

[LockonComputer]

I believe I successfully created the missing file using the procedure described.  Now when I power on I get the following error:

Task

Power On VM

Key

haTask-4-vim.VirtualMachine.powerOn-111788850

Description

Power On this virtual machine

Virtual machine

Server3

State

Failed - File system specific implementation of LookupAndOpen[file] failed

Errors

File system specific implementation of LookupAndOpen[file] failed

File system specific implementation of LookupAndOpen[file] failed

File system specific implementation of LookupAndOpen[file] failed

File system specific implementation of LookupAndOpen[file] failed

File system specific implementation of LookupAndOpen[file] failed

File system specific implementation of LookupAndOpen[file] failed

File system specific implementation of LookupAndOpen[file] failed

File system specific implementation of LookupAndOpen[file] failed

File system specific implementation of LookupAndOpen[file] failed

File system specific implementation of LookupAndOpen[file] failed

The parent virtual disk has been modified since the child was created. The content ID of the parent virtual disk does not match the corresponding parent content ID in the child

Cannot open the disk '/vmfs/volumes/5a5947b7-8ba9b686-01a9-4ccc6a44e023/Server3/Server3-000001.vmdk' or one of the snapshot disks it depends on.

Module 'Disk' power on failed.

Failed to start the virtual machine.

[TheBobkin]

Hello LockonComputer​

Can you attach the vmdk you created? If you didn't modify the CID so that it matches the Parent CID of the snapshot, it would explain this:

The content ID of the parent virtual disk does not match the corresponding parent content ID in the child

Bob

[LockonComputer]

Here you go, and can't thank you enough!

# Disk DescriptorFile

version=1

encoding="UTF-8"

CID=fffffffe

parentCID=ffffffff

isNativeSnapshot="no"

createType="vmfs"

# Extent description

RW 419430400 VMFS "Server3-flat.vmdk"

# The Disk Data Base

#DDB

ddb.adapterType = "lsilogic"

ddb.geometry.cylinders = "26108"

ddb.geometry.heads = "255"

ddb.geometry.sectors = "63"

ddb.longContentID = "0ed700097adc1dd83f441124fffffffe"

ddb.thinProvisioned = "1"

ddb.uuid = "60 00 C2 9b 8d 9e af 58-c1 b4 c3 d2 ae 61 f5 80"

ddb.virtualHWVersion = "13"

[TheBobkin]

LockonComputer​ , the CID should be CID=37f94e62

Can't check the other aspects right now as not near a lab to check.

Bob

[LockonComputer]

IT'S ALIVE!!!!!  OMG!!!!  I owe you a beer or a new car :smileygrin:   Can't thank you enough!!!!!!!

I'm taking a backup of the server now and then I'll delete the snapshots to recover more space.

I have a related question:  seeing as how there is a 'flat' file and a 'delta' file that consume a lot of space, what is the optimal way to manage the space?  I saw somewhere there was an option to flatten or combine disk images (can't find it now).  Sorry for the newbie questions but still learning VMWare.

Thanks!

[a_p_]

Good to see that the issue has been fixed.

Regarding your question, I'm not sure what exactly you mean with "flatten or combine disk images".

The flat file represents the VMs virtual disk, and the delta file  (sesparse with newer versions) stores store modified blocks if the VM has an active snapshot. According to the files' time stamps, the snapshot was created in August 2018, and grew up to the current size (~25GB). I assume that the one who created that snapshot simply forgot to delete it again, after finishing the work.

Depending on the size of your environment, it may be helpful to either monitor snapshots, and disk space usage, or at least check for this from time to time. A great free tool that I like to use is RVTools, which provides a great overview, and hints on what may be worth looking at.

André

[scott28tt]

Delete the snapshot. That will merge the content of the delta file into the flat file and then the delta file will be removed.