After delete some snapshots can't boot

[nandelbosc]

Hi to all,

I'm new here and I try to do as well as I can...

I have an ESX 3.5.0 with a virtual machine called "Inventari", this machine have and Ubuntu 9.04 inside and 4 snapshots:

1. after clean install
2. after install apache and mysql
3. after dev a little bit
4. day to day use

This morning I tried to remove the 4 snapshots using "Delete All" option on Snapshot Manager, but after a while, (when I returns to VI) i see the VM stopped with some errors messages. It don't start again with the error:

Cannot open the disk '/vmfs/volumes/xxxxxxxx/Inventari/Inventari-000001.vmdk' or one of the snapshot disks it depends on.

Reason: The parent virtual disk has been modified since the child was created

I read (and re-read): http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=100423... and http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&externalId=1007969

But nothing works, I can only start the VM in the initial state, losing all changes made the last 3 months (stored in the snapshots).

If you look to the vmdk files, there is something that do not fit...

# ls -l

total 20991040

-rw-------    1 root     root        43008 Dec 15 10:41 Inventari-000001-delta.vmdk

-rw-------    1 root     root          234 Dec 15 10:42 Inventari-000001.vmdk

-rw-------    1 root     root     16820224 Dec 15 10:42 Inventari-000004-delta.vmdk

-rw-------    1 root     root          227 Dec 15 10:41 Inventari-000004.vmdk

-rw-r--r--    1 root     root     21474836480 Dec 15 11:24 Inventari-flat.vmdk

-rw-------    1 root     root         8684 Dec 15 10:41 Inventari.nvram

-rw-------    1 root     root        19121 Dec 15 11:24 Inventari-Snapshot84.vmsn

-rw-------    1 root     root          402 Dec 15 10:41 Inventari.vmdk

-rw-------    1 root     root         1854 Dec 15 10:41 Inventari.vmsd

-rwxr-xr-x    1 root     root         2120 Dec 15 10:41 Inventari.vmx

-rw-------    1 root     root          264 Dec 15 10:41 Inventari.vmxf

-rw-r--r--    1 root     root        15875 Dec 15 10:41 vmware-13.log

-rw-r--r--    1 root     root        27508 Dec 15 10:41 vmware-14.log

-rw-r--r--    1 root     root        39027 Dec 15 10:41 vmware-15.log

-rw-r--r--    1 root     root        25630 Dec 15 10:41 vmware-16.log

-rw-r--r--    1 root     root        28863 Dec 15 10:41 vmware-17.log

-rw-r--r--    1 root     root        65971 Dec 15 10:41 vmware-18.log

-rw-r--r--    1 root     root        20730 Dec 15 11:24 vmware.log

-r--------    1 root     root        61440 Dec 15 10:41 vmware-vmx-zdump.000

-r--------    1 root     root        61440 Dec 15 10:41 vmware-vmx-zdump.001

The Snapshot 2 and 3 disappeared????!!!!

The parent vmdk file looks ok...

[root@esx4 Inventari]# cat Inventari.vmdk

# Disk DescriptorFile

version=1

CID=0c919b74

parentCID=ffffffff

createType="vmfs"

# Extent description

RW 41943040 VMFS "Inventari-flat.vmdk"

# The Disk Data Base

#DDB

ddb.virtualHWVersion = "4"

ddb.uuid = "60 00 C2 9d e0 69 6b c4-c4 d2 aa 8a 09 80 d1 75"

ddb.geometry.cylinders = "2610"

ddb.geometry.heads = "255"

ddb.geometry.sectors = "63"

ddb.adapterType = "lsilogic"

ddb.toolsVersion = "7302"

But no the childs...

[root@esx4 Inventari]# cat Inventari-000001.vmdk

# Disk DescriptorFile

version=1

CID=61909e84

parentCID=61909e84

createType="vmfsSparse"

parentFileNameHint="Inventari-000004.vmdk"

# Extent description

RW 41943040 VMFSSPARSE "Inventari-000001-delta.vmdk"

# The Disk Data Base

#DDB

The 01 vmdk snapshot file parentCID (61909e84) is the same as own CID (61909e84) ?!?!?!? Why parentFileNameHint point to Inventari-000004.vmdk instead of Inventari-flat.vmdk ?!?!?!

[root@esx4 Inventari]# cat Inventari-000004.vmdk

# Disk DescriptorFile

version=1

CID=61909e84

parentCID=2d769fc4

createType="vmfsSparse"

parentFileNameHint="Inventari.vmdk"

# Extent description

RW 41943040 VMFSSPARSE "Inventari-000004-delta.vmdk"

# The Disk Data Base

#DDB

Why parentFileNameHint points to Inventari.vmdk instead of Inventari-000001.vmdk?!?!?

I tried the following (O.S. says filesystem is corrupt, but I can access it and see that is in initial state, just after finish O.S. installation):

[root@esx4 Inventari]# cat Inventari.vmdk

# Disk DescriptorFile

version=1

CID=0c919b74

parentCID=ffffffff

createType="vmfs"

# Extent description

RW 41943040 VMFS "Inventari-flat.vmdk"

# The Disk Data Base

#DDB

ddb.virtualHWVersion = "4"

ddb.uuid = "60 00 C2 9d e0 69 6b c4-c4 d2 aa 8a 09 80 d1 75"

ddb.geometry.cylinders = "2610"

ddb.geometry.heads = "255"

ddb.geometry.sectors = "63"

ddb.adapterType = "lsilogic"

ddb.toolsVersion = "7302"

[root@esx4 Inventari]# cat Inventari-000001.vmdk

# Disk DescriptorFile

version=1

CID=61909e84

parentCID=0c919b74

createType="vmfsSparse"

parentFileNameHint="Inventari.vmdk"

# Extent description

RW 41943040 VMFSSPARSE "Inventari-000001-delta.vmdk"

# The Disk Data Base

#DDB

[root@esx4 Inventari]# cat Inventari-000004.vmdk

# Disk DescriptorFile

version=1

CID=61909e84

parentCID=2d769fc4

createType="vmfsSparse"

parentFileNameHint="Inventari.vmdk"

# Extent description

RW 41943040 VMFSSPARSE "Inventari-000004-delta.vmdk"

# The Disk Data Base

#DDB

And that (O.S. boots well, but in initial state, O.S. Installation):

[root@esx4 Inventari]# cat Inventari.vmdk

# Disk DescriptorFile

version=1

CID=0c919b74

parentCID=ffffffff

createType="vmfs"

# Extent description

RW 41943040 VMFS "Inventari-flat.vmdk"

# The Disk Data Base

#DDB

ddb.virtualHWVersion = "4"

ddb.uuid = "60 00 C2 9d e0 69 6b c4-c4 d2 aa 8a 09 80 d1 75"

ddb.geometry.cylinders = "2610"

ddb.geometry.heads = "255"

ddb.geometry.sectors = "63"

ddb.adapterType = "lsilogic"

ddb.toolsVersion = "7302"

[root@esx4 Inventari]# cat Inventari-000001.vmdk

# Disk DescriptorFile

version=1

CID=61909e84

parentCID=0c919b74

createType="vmfsSparse"

parentFileNameHint="Inventari.vmdk"

# Extent description

RW 41943040 VMFSSPARSE "Inventari-000001-delta.vmdk"

# The Disk Data Base

#DDB

[root@esx4 Inventari]# cat Inventari-000004.vmdk

# Disk DescriptorFile

version=1

CID=2d769fc4

parentCID=61909e84

createType="vmfsSparse"

parentFileNameHint="Inventari-000001.vmdk"

# Extent description

RW 41943040 VMFSSPARSE "Inventari-000004-delta.vmdk"

# The Disk Data Base

#DDB

No luck!!!

Anyone have ideas???

Thank's in advance!

And no... I don't have backups before try to delete snapshot (yes, CONGRATULATIONS MY SON!)...

[cjscol]

Did your VMFS volume run out of space while deleting the snapshots?

I had the same problem with ESX 3.5.0 as it commits the delta files in the wrong order, i.e.

it merges delta-000004 to delta-000003

then merges delta-000003 to delta-000002

then merges delta-000002 to delta-000001

then finially merges delta-000001 to the flat file

So you need extra spcae in your datatsore for this.

I had the problem when the datastore ran out of space and some of the delta files "disappeared".  Only option I had after contacting VMware support was to boot the virtual machine up from the original flat files and then try to recover the data from backups/replicas I had taken prior to attempting to delete the snapshots.

[nandelbosc]

Thank's for your answer cjscol, but I have more than 30GB of free space...

As i said before, I have no backups :smileycry:

[a_p_]

Please compress (zip) all the VM's files (except the flat, delta and dump files) and attach them to your next post (or try the following by your own).

Here's what I think, how it should work, with the "Inventari-000001.vmdk" in the VM's vmx file:

[root@esx4 Inventari]# cat Inventari.vmdk

# Disk DescriptorFile

version=1

CID=0c919b74

parentCID=ffffffff

createType="vmfs"

# Extent description

RW 41943040 VMFS "Inventari-flat.vmdk"

# The Disk Data Base

#DDB

ddb.virtualHWVersion = "4"

ddb.uuid = "60 00 C2 9d e0 69 6b c4-c4 d2 aa 8a 09 80 d1 75"

ddb.geometry.cylinders = "2610"

ddb.geometry.heads = "255"

ddb.geometry.sectors = "63"

ddb.adapterType = "lsilogic"

ddb.toolsVersion = "7302"

[root@esx4 Inventari]# cat Inventari-000001.vmdk

# Disk DescriptorFile

version=1

CID=61909e84

parentCID=61909e84

createType="vmfsSparse"

parentFileNameHint="Inventari-000004.vmdk"

# Extent description

RW 41943040 VMFSSPARSE "Inventari-000001-delta.vmdk"

# The Disk Data Base

#DDB

[root@esx4 Inventari]# cat Inventari-000004.vmdk

# Disk DescriptorFile

version=1

CID=61909e84

parentCID=0c919b74

createType="vmfsSparse"

parentFileNameHint="Inventari.vmdk"

# Extent description

RW 41943040 VMFSSPARSE "Inventari-000004-delta.vmdk"

# The Disk Data Base

#DDB

After making the changes, remove and re-add the VM from/to the inventory!

I also recommend that - before powering on the modified VM - you take another snapshot!

The 01 vmdk snapshot file parentCID (61909e84) is the same as own CID (61909e84) ?!?!?!? Why parentFileNameHint point to Inventari-000004.vmdk instead of Inventari-flat.vmdk ?!?!?!

This usually indicates that the snapshot has not been modified before the next one was created.

Why parentFileNameHint points to Inventari.vmdk instead of Inventari-000001.vmdk?!?!?

The numbering of the deltas does not necessarily have to be in an ascending order.

André

[nandelbosc]

Thank's André,

attached you have the ZIP file.

Now I'm preparing my infrastructure to try what you comment... In a few minutes I post the results.

[nandelbosc]

Ok, after made your changes:

I can take snapshot succefully, in the snapshot manage I see two snapshots:

- Inventari

  - Consolidate Helper- 1

    - My Snapshot

      - You are here

The guest boots with a file inconsistency error. After check (and correct inode errors), boots ok, but with the O.S. recently installed...

Anyway, thank's for your time!

[a_p_]

Please attach the latest vmware.log file as well as the latest list of files to verify there's no other issue with the snapshot chain.

André

[nandelbosc]

Thank's for your time again Andre,

atteched the last log file and an screenshot of all files

[a_p_ · Accepted solution]

Sorry but I can't tell you what happened. With the .vmdk files you have, I'm afraid the data which has not been committed during the initial snapshot deletion is lost. The snapshot chain looks ok now and all delta files are used, but according to the size of those snapshot files they were just created and don't contain a lot of changes.

André

[nandelbosc]

I feared that. I'm developing the work done again.
thanks anyway a.p.!

I close the post.