Cannot start VM due to missing file after consolidating snapshots

[kershek]

I'm looking for help in a snapshot issue I'm having with my VM. I try to be thorough in my description of the problem to encourage help without someone having to ask a lot of follow-up questions. I appreciate any assistance you can give me!

I'm using ESXi 3.5 update 2 with all local SATA storage. Due to throughput issues, I want to move all the VMs off the server, rebuild it as Windows Server 2008, install VMWare Server 2, and move the VMs there. I have installed Server 2008 on a different hard drive so I can go back to ESXi if any problems arise.

While it was booted to ESXi, I removed all snapshots to my VMs (or at least I thought I had), so that the vmdk was the most current copy. Then, I used VMWare Converter to copy the VMs from ESXi to my workstation as VMWare workstation files. I booted to Server 2008 and created a datastore via CIFS share to verify I could start up all the VMs before removing ESXi permanently and reclaiming the disk space.

From VMWare Server 2, all the VMs worked fine except one - the Zocalo VM booted up as the initial build, which was the state it was in before the first snapshot. I booted back to ESXi and noticed that I could no longer start Zocalo. In looking at the datastore directory, I saw three snapshots, but the snapshot manager didn't see them at all. Here are the files found:

-rw----- 1 root root 15820978176 Oct 12 04:17 Zocalo-000001-delta.vmdk
-rw--- 1 root root 254 Sep 22 04:38 Zocalo-000001.vmdk
-rw--- 1 root root 536934400 Sep 16 20:48 Zocalo-000002-delta.vmdk
-rw--- 1 root root 247 Sep 16 20:42 Zocalo-000002.vmdk
-rw--- 1 root root 2868967424 Jan 24 23:03 Zocalo-000003-delta.vmdk
-rw--- 1 root root 254 Dec 10 01:51 Zocalo-000003.vmdk
-rw--- 1 root root 2152842854 Sep 15 05:03 Zocalo-Snapshot1.vmsn
-rw--- 1 root root 2152793391 Sep 16 20:49 Zocalo-Snapshot2.vmsn
-rw--- 1 root root 19975 Oct 12 04:17 Zocalo-Snapshot3.vmsn
-rw--- 1 root root 32212254720 Sep 15 05:02 Zocalo-flat.vmdk
-rw--- 1 root root 8684 Jan 24 23:03 Zocalo.nvram
-rw--- 1 root root 399 Sep 15 03:31 Zocalo.vmdk
-rw--- 1 root root 1617 Oct 12 04:17 Zocalo.vmsd
-rw--- 1 root root 1617 Jan 26 09:04 Zocalo.vmsd.bak
-rwxr-xr-x 1 root root 2976 Jan 26 08:34 Zocalo.vmx
-rw----- 1 root root 261 Oct 16 06:06 Zocalo.vmxf

The Zocalo.vmsd files shows the three snapshots. However, the third snapshot shows files that don't exist (Zocalo_1.vmdk, Zocalo_2.vmdk, and Zocalo_3.vmdk). Here's that part of the vmsd file:

snapshot2.disk0.fileName = "Zocalo-000001.vmdk"
snapshot2.disk0.node = "scsi0:0"
snapshot2.disk1.fileName = "/vmfs/volumes/48d708a7-3a9519e6-b432-0022157a057e/Zocalo/Zocalo_1.vmdk"
snapshot2.disk1.node = "scsi0:1"
snapshot2.disk1.mode = "1"
snapshot2.disk2.fileName = "/vmfs/volumes/48d708a7-3a9519e6-b432-0022157a057e/Zocalo/Zocalo_2.vmdk"
snapshot2.disk2.node = "scsi0:2"
snapshot2.disk2.mode = "1"
snapshot2.disk3.fileName = "/vmfs/volumes/48d708a7-3a9519e6-b432-0022157a057e/Zocalo/Zocalo_3.vmdk"
snapshot2.disk3.node = "scsi0:3"
snapshot2.disk3.mode = "1"

I've done extensive searching to find the solution to this problem. One solution I found was to rename the vmsd, create another snapshot and delete it so it would automatically incorporate the older snapshots. However, that didn't work.

Did vmware converter do something destructive to this VM? How can I incorporate these snapshots into a final vmdk? I really don't want to lose this data, as the backup I have is old.

[kjb007]

Looking at the output you have provided, I see all snapshots in place for disk 1. The remaining output suggests that you had 4 drives on the vm's. Is this the case? Does your vmx file reference 4 different disk files? Do they still exist? What filename does scsi0:0 reference in your vmx file?

-KjB

[kershek]

Thanks for your reply.

To answer your question, scsi0:0.fileName = "Zocalo-000003.vmdk" is the file and it exists. In looking at the output of the vmx file, I looked at the configuration of the VM and found a second hard drive (independent to snapshots) attached to a datastore that is no longer available. This is because I wiped that datastore when I installed Windows Server 2008. It doesn't surprise me that I missed this revelation at 2am last night, but as that drive had no data on it, I didn't lose anything. I remember creating that drive multiple times because I kept recreating the datastore, so that might be why it looks like 3 different drives.

So I found my root cause to the missing hard drive. What's the best way to continue? Should I follow the original instructions I found that stated to create a new snapshot with vmware-cmd and delete it, expecting the dangling snapshots to consolidate back into the main vmdk? I have the command ready to go if that's true. What is the risk of losing data by going through with that course of action? Any better ideas?

[Odurasler2]

kershek,

i've experienced similar problem. based on your info, your VM is not starting because it's missing the "3" vmdk files.

first, we need to locate zocalo_1, 2, and 3 vmdks. do they exists? if not, do you see the "flat" vmdks?

this vmfs volume should contain your files, //vmfs/volumes/48d708a7-3a9519e6-b432-0022157a057e/Zocalo

after finding these files, we would need to modify the snapshot chain. so let me know if those vmdks still exist

[kershek]

No, they don't exist any longer (take a look at my previous reply) and I have since removed that hard drive from the VM through the VI Client.

I would like to start the VM, but I wanted to do a backup of the files first. I created a subdirectory and attempted to copy the files, but I get an "Input/output error' after copying 67mb of the first file (Zocalo-000001-delta.vmdk). I'm not sure why that is happening, so there's another mystery to solve. There is 165gb free in the datastore and the entire VM with snapshots is about 55gb.

[kershek]

Update - I decided to try to start the VM anyways. It started fine and it has the most recent data in it, which is great! Now the question remains as to my next step to get this VM off ESXi: Should I create and delete a snapshot or should I try using VMWare Converter to copy it off and hope it gets all the delta changes?

[kjb007 · Accepted solution]

If you use vmware converter, then you don't have to worry about the snapshots. It will in essence create a new drive that is a consolidation of all of your snapshots.

-KjB

[kershek]

It worked! VMWare converter consolidated the snapshots and I have all my data. Thank you very much for the help!