Solved: Re: Merge delta.vmdk files to its parent(s)

bao__zhe · ‎07-12-2012

After many days of researching/googling, i can't find out the solution to my problem:smileyplain:...

Here is my situation, i have a flat base disk, a delta1 disk based on base disk, and a delta2 disk based on delta1 disk. How do i merge the delta2 disk to delta1 disk but not base disk?

More backgound: i'm using SSD as datastore and is very constrained by space. All my VMs on the ESXi host are linked clones from a flat base disk using the method decsribed here:

http://sanbarrow.com/linkedcloneswithesxi.html

the VMs are working great, except that snapshots and its associated delta disks are never deleted. I've tried Disk Consolidation without success. It always gives error like "unable to access file since it is locked". This is kinda making sense since my other running VMs are using that base disk.

Thus what i'm looking for is a way to consolidate/merge the delta disks up to a specified level, not all the way back to the base disk. I can do this using VM Workstation but can't find a way to do this in ESXi.

a_p_ · ‎07-15-2012

I took another look at your files and thought about options to remove all snapshots easily by merging them into the linked clones first delta .vmdk file. I did some tests in my lab environment and found a solution that worked for me. If you want to try it, make sure you backup the 2 folders (base VM and clone VM) before you do anything, in order to be able to revert to the current state in case it doesn't work for you!

Here are the steps:

power off the linked clone
backup the linked clone's folder as well as the base VM's folder!
remove the linked clone from the inventory
replace the linked clone's .vmsd file with the one I attached
add the linked clone to the inventory
create a new snapshot
open the Snapshot Manager and click "Delete All"

What the lines in the .vmsd file are supposed to do, is to make ESXi aware that this is a linked clone with it's own "base" disk. Once everything worked as expected, the .vmx file should point to "DXP3Download.vmdk" as its virtual disk and all snapshot files should be gone. As I mentioned before, there are some snapshot files which don't show up in the log files, so these files (most likely 000032-...) may need to be deleted manually.

Remember, you do this at your own risk. However, "No risk, no fun!".

André

View solution in original post

sparrowangelste · ‎07-12-2012

you could always converter the vm as a v2v

then remove the old one..

--------------------- Sparrowangelstechnology : Vmware lover http://sparrowangelstechnology.blogspot.com

bao__zhe · ‎07-13-2012

emm...i didn't quite get what does v2v do and how can it help here?

a_p_ · ‎07-13-2012

Not sure, but from what you explained I assume this could be an issue with the "linked clone's" .vmsd file!? Can you please post some files to see whether they contain any hints? The VM's .vmsd, .vmx and latest vmware.log file (which contains the error message).

André

sparrowangelste · ‎07-13-2012

v2v, basically clone your virutal machine so you can just have 1 vmdk, instead of worrying about merging them

ah just re-read you are constrianed by sapce and need to wkr off your base disk...

that is not going to help then

--------------------- Sparrowangelstechnology : Vmware lover http://sparrowangelstechnology.blogspot.com

bao__zhe · ‎07-13-2012

ok here is more information. I've attached logs, configs and it's very first delta disk.

As you can see the first delta disk

DXP3/DXP3Download/DXP3Download.vmdk

is based on

DXP3/DXP3/DXP3.vmdk

and the current delta disk is

DXP3/DXP3Download/DXP3Download-000054.vmdk

but i only have 2 snapshots with 1st one based on

DXP3/DXP3Download/DXP3Download.vmdk

and the 2nd one based on

DXP3/DXP3Download/DXP3Download-000002.vmdk

a_p_ · ‎07-13-2012

What am I missing? Which .vmdk files do you see in the VM's folder?

According to the log files there are ~30 active snapshots on "DXP3Download.vmdk"? The .vmsd file does not contain information about these snapshots, so they don't show up in the Snapshot Manager and therefore cannot be deleted from there.

What might be necessary is to create a .vmsd file with the correct snapshot chain to be able to delete the snapshots.

André

bao__zhe · ‎07-13-2012

well the problem is, when a snapshot is deleted, it's associated delta disk is not merged up to its parent but are left intact. the snapshot information is removed from the vmsd so can't be seen in the snapshot manager but the delta disk is still part of the chain for all future snapshots. this is taking a lot of spaces.

here is the directory content:

1024 drwxrwxrwx    1 root     root        19.3k Jul 14 01:48 .
   8 drwxr-xr-x    1 root     root         1.1k Jul 13 04:13 ..
1024 -rw-------    1 root     root       640.5k Jul 12 23:05 DXP3Download-000001-ctk.vmdk
148480 -rw-------    1 root     root       144.0M Jul 12 23:05 DXP3Download-000001-delta.vmdk
   0 -rw-------    1 root     root          466 Jul 12 23:05 DXP3Download-000001.vmdk
1024 -rw-------    1 root     root       640.5k Jul 12 23:05 DXP3Download-000002-ctk.vmdk
33792 -rw-------    1 root     root        32.0M Jul 12 23:05 DXP3Download-000002-delta.vmdk
   0 -rw-------    1 root     root          473 Jul 12 23:05 DXP3Download-000002.vmdk
1024 -rw-------    1 root     root       640.5k Jul 12 23:09 DXP3Download-000003-ctk.vmdk
33792 -rw-------    1 root     root        32.0M Jul 12 23:09 DXP3Download-000003-delta.vmdk
   0 -rw-------    1 root     root          473 Jul 12 23:09 DXP3Download-000003.vmdk
1024 -rw-------    1 root     root       640.5k Jul 12 23:05 DXP3Download-000004-ctk.vmdk
33792 -rw-------    1 root     root        32.0M Jul 12 23:05 DXP3Download-000004-delta.vmdk
   0 -rw-------    1 root     root          473 Jul 12 23:05 DXP3Download-000004.vmdk
17408 -rw-------    1 root     root        16.0M Jul 12 23:05 DXP3Download-000005-delta.vmdk
   0 -rw-------    1 root     root          402 Jul 12 23:05 DXP3Download-000005.vmdk
17408 -rw-------    1 root     root        16.0M Jul 12 23:05 DXP3Download-000006-delta.vmdk
   0 -rw-------    1 root     root          402 Jul 12 23:05 DXP3Download-000006.vmdk
17408 -rw-------    1 root     root        16.0M Jul 12 23:06 DXP3Download-000007-delta.vmdk
   0 -rw-------    1 root     root          402 Jul 12 23:06 DXP3Download-000007.vmdk
17408 -rw-------    1 root     root        16.0M Jul 12 23:06 DXP3Download-000008-delta.vmdk
   0 -rw-------    1 root     root          402 Jul 12 23:06 DXP3Download-000008.vmdk
17408 -rw-------    1 root     root        16.0M Jul 12 23:05 DXP3Download-000009-delta.vmdk
   0 -rw-------    1 root     root          402 Jul 12 23:05 DXP3Download-000009.vmdk
17408 -rw-------    1 root     root        16.0M Jul 12 23:06 DXP3Download-000010-delta.vmdk
   0 -rw-------    1 root     root          402 Jul 12 23:06 DXP3Download-000010.vmdk
17408 -rw-------    1 root     root        16.0M Jul 12 23:06 DXP3Download-000011-delta.vmdk
   0 -rw-------    1 root     root          402 Jul 12 23:06 DXP3Download-000011.vmdk
17408 -rw-------    1 root     root        16.0M Jul 12 23:06 DXP3Download-000012-delta.vmdk
   0 -rw-------    1 root     root          402 Jul 12 23:06 DXP3Download-000012.vmdk
17408 -rw-------    1 root     root        16.0M Jul 12 23:05 DXP3Download-000013-delta.vmdk
   0 -rw-------    1 root     root          402 Jul 12 23:06 DXP3Download-000013.vmdk
17408 -rw-------    1 root     root        16.0M Jul 12 23:06 DXP3Download-000014-delta.vmdk
   0 -rw-------    1 root     root          402 Jul 12 23:06 DXP3Download-000014.vmdk
17408 -rw-------    1 root     root        16.0M Jul 12 23:05 DXP3Download-000015-delta.vmdk
   0 -rw-------    1 root     root          402 Jul 12 23:06 DXP3Download-000015.vmdk
17408 -rw-------    1 root     root        16.0M Jul 12 23:06 DXP3Download-000016-delta.vmdk
   0 -rw-------    1 root     root          402 Jul 12 23:06 DXP3Download-000016.vmdk
17408 -rw-------    1 root     root        16.0M Jul 12 23:06 DXP3Download-000017-delta.vmdk
   0 -rw-------    1 root     root          402 Jul 12 23:06 DXP3Download-000017.vmdk
17408 -rw-------    1 root     root        16.0M Jul 12 23:05 DXP3Download-000018-delta.vmdk
   0 -rw-------    1 root     root          402 Jul 12 23:06 DXP3Download-000018.vmdk
17408 -rw-------    1 root     root        16.0M Jul 12 23:05 DXP3Download-000019-delta.vmdk
   0 -rw-------    1 root     root          402 Jul 12 23:06 DXP3Download-000019.vmdk
17408 -rw-------    1 root     root        16.0M Jul 12 23:06 DXP3Download-000020-delta.vmdk
   0 -rw-------    1 root     root          402 Jul 12 23:06 DXP3Download-000020.vmdk
17408 -rw-------    1 root     root        16.0M Jul 12 23:06 DXP3Download-000021-delta.vmdk
   0 -rw-------    1 root     root          402 Jul 12 23:06 DXP3Download-000021.vmdk
33792 -rw-------    1 root     root        32.0M Jul 12 23:06 DXP3Download-000022-delta.vmdk
   0 -rw-------    1 root     root          402 Jul 12 23:06 DXP3Download-000022.vmdk
17408 -rw-------    1 root     root        16.0M Jul 12 23:05 DXP3Download-000023-delta.vmdk
   0 -rw-------    1 root     root          402 Jul 12 23:06 DXP3Download-000023.vmdk
33792 -rw-------    1 root     root        32.0M Jul 12 23:06 DXP3Download-000024-delta.vmdk
   0 -rw-------    1 root     root          402 Jul 12 23:06 DXP3Download-000024.vmdk
33792 -rw-------    1 root     root        32.0M Jul 12 23:05 DXP3Download-000025-delta.vmdk
   0 -rw-------    1 root     root          402 Jul 12 23:05 DXP3Download-000025.vmdk
1024 -rw-------    1 root     root       640.5k Jul 12 23:06 DXP3Download-000026-ctk.vmdk
33792 -rw-------    1 root     root        32.0M Jul 12 23:06 DXP3Download-000026-delta.vmdk
   0 -rw-------    1 root     root          473 Jul 12 23:06 DXP3Download-000026.vmdk
1024 -rw-------    1 root     root       640.5k Jul 12 23:06 DXP3Download-000027-ctk.vmdk
33792 -rw-------    1 root     root        32.0M Jul 12 23:06 DXP3Download-000027-delta.vmdk
   0 -rw-------    1 root     root          473 Jul 12 23:06 DXP3Download-000027.vmdk
1024 -rw-------    1 root     root       640.5k Jul 12 23:06 DXP3Download-000028-ctk.vmdk
50176 -rw-------    1 root     root        48.0M Jul 12 23:06 DXP3Download-000028-delta.vmdk
   0 -rw-------    1 root     root          473 Jul 12 23:06 DXP3Download-000028.vmdk
17408 -rw-------    1 root     root        16.0M Jul 12 23:06 DXP3Download-000029-delta.vmdk
   0 -rw-------    1 root     root          402 Jul 12 23:06 DXP3Download-000029.vmdk
33792 -rw-------    1 root     root        32.0M Jul 12 23:06 DXP3Download-000030-delta.vmdk
   0 -rw-------    1 root     root          402 Jul 12 23:06 DXP3Download-000030.vmdk
1024 -rw-------    1 root     root       640.5k Jul 12 23:07 DXP3Download-000031-ctk.vmdk
33792 -rw-------    1 root     root        32.0M Jul 12 23:07 DXP3Download-000031-delta.vmdk
   0 -rw-------    1 root     root          473 Jul 12 23:07 DXP3Download-000031.vmdk
33792 -rw-------    1 root     root        32.0M Jul 12 23:07 DXP3Download-000032-delta.vmdk
   0 -rw-------    1 root     root          402 Jul 12 23:07 DXP3Download-000032.vmdk
33792 -rw-------    1 root     root        32.0M Jul 12 23:07 DXP3Download-000033-delta.vmdk
   0 -rw-------    1 root     root          402 Jul 12 23:07 DXP3Download-000033.vmdk
33792 -rw-------    1 root     root        32.0M Jul 12 23:07 DXP3Download-000034-delta.vmdk
   0 -rw-------    1 root     root          402 Jul 12 23:07 DXP3Download-000034.vmdk
33792 -rw-------    1 root     root        32.0M Jul 12 23:07 DXP3Download-000035-delta.vmdk
   0 -rw-------    1 root     root          402 Jul 12 23:07 DXP3Download-000035.vmdk
82944 -rw-------    1 root     root        80.0M Jul 12 23:07 DXP3Download-000036-delta.vmdk
   0 -rw-------    1 root     root          402 Jul 12 23:07 DXP3Download-000036.vmdk
33792 -rw-------    1 root     root        32.0M Jul 12 23:07 DXP3Download-000037-delta.vmdk
   0 -rw-------    1 root     root          402 Jul 12 23:07 DXP3Download-000037.vmdk
33792 -rw-------    1 root     root        32.0M Jul 12 23:07 DXP3Download-000038-delta.vmdk
   0 -rw-------    1 root     root          402 Jul 12 23:07 DXP3Download-000038.vmdk
17408 -rw-------    1 root     root        16.0M Jul 12 23:07 DXP3Download-000039-delta.vmdk
   0 -rw-------    1 root     root          402 Jul 12 23:07 DXP3Download-000039.vmdk
17408 -rw-------    1 root     root        16.0M Jul 12 23:07 DXP3Download-000040-delta.vmdk
   0 -rw-------    1 root     root          402 Jul 12 23:07 DXP3Download-000040.vmdk
17408 -rw-------    1 root     root        16.0M Jul 12 23:07 DXP3Download-000041-delta.vmdk
   0 -rw-------    1 root     root          402 Jul 12 23:07 DXP3Download-000041.vmdk
17408 -rw-------    1 root     root        16.0M Jul 12 23:07 DXP3Download-000042-delta.vmdk
   0 -rw-------    1 root     root          402 Jul 12 23:07 DXP3Download-000042.vmdk
65536 -rw-------    1 root     root        64.0M Jul 12 23:07 DXP3Download-000043-delta.vmdk
   0 -rw-------    1 root     root          402 Jul 12 23:07 DXP3Download-000043.vmdk
33792 -rw-------    1 root     root        32.0M Jul 12 23:07 DXP3Download-000044-delta.vmdk
   0 -rw-------    1 root     root          402 Jul 12 23:07 DXP3Download-000044.vmdk
33792 -rw-------    1 root     root        32.0M Jul 12 23:07 DXP3Download-000045-delta.vmdk
   0 -rw-------    1 root     root          402 Jul 12 23:07 DXP3Download-000045.vmdk
17408 -rw-------    1 root     root        16.0M Jul 12 23:07 DXP3Download-000046-delta.vmdk
   0 -rw-------    1 root     root          402 Jul 12 23:07 DXP3Download-000046.vmdk
17408 -rw-------    1 root     root        16.0M Jul 12 23:07 DXP3Download-000047-delta.vmdk
   0 -rw-------    1 root     root          402 Jul 12 23:07 DXP3Download-000047.vmdk
17408 -rw-------    1 root     root        16.0M Jul 12 23:07 DXP3Download-000048-delta.vmdk
   0 -rw-------    1 root     root          402 Jul 12 23:07 DXP3Download-000048.vmdk
17408 -rw-------    1 root     root        16.0M Jul 12 23:07 DXP3Download-000049-delta.vmdk
   0 -rw-------    1 root     root          402 Jul 12 23:07 DXP3Download-000049.vmdk
17408 -rw-------    1 root     root        16.0M Jul 12 23:07 DXP3Download-000050-delta.vmdk
   0 -rw-------    1 root     root          402 Jul 12 23:07 DXP3Download-000050.vmdk
17408 -rw-------    1 root     root        16.0M Jul 12 23:07 DXP3Download-000051-delta.vmdk
   0 -rw-------    1 root     root          402 Jul 12 23:07 DXP3Download-000051.vmdk
1024 -rw-------    1 root     root       640.5k Jul 12 23:07 DXP3Download-000052-ctk.vmdk
17408 -rw-------    1 root     root        16.0M Jul 12 23:07 DXP3Download-000052-delta.vmdk
   0 -rw-------    1 root     root          473 Jul 12 23:07 DXP3Download-000052.vmdk
1024 -rw-------    1 root     root       640.5k Jul 12 23:08 DXP3Download-000053-ctk.vmdk
1024 -rw-------    1 root     root        24.0k Jul 12 23:08 DXP3Download-000053-delta.vmdk
   0 -rw-------    1 root     root          473 Jul 12 23:08 DXP3Download-000053.vmdk
1024 -rw-------    1 root     root       640.5k Jul 13 04:58 DXP3Download-000054-ctk.vmdk
148480 -rw-------    1 root     root       144.0M Jul 14 00:33 DXP3Download-000054-delta.vmdk
   0 -rw-------    1 root     root          473 Jul 13 04:58 DXP3Download-000054.vmdk
   0 -rw-------    1 root     root           23 Jul 13 02:11 DXP3Download-0a8cb101.hlog
1048576 -rw-------    1 root     root         1.0G Jul 13 04:58 DXP3Download-0a8cb101.vswp
1024 -rw-------    1 root     root        17.7k Jul 12 23:08 DXP3Download-Snapshot1.vmsn
1024 -rw-------    1 root     root        27.5k Jul 12 23:08 DXP3Download-Snapshot3.vmsn
1024 -rw-------    1 root     root        24.0k Jul 12 23:05 DXP3Download-delta.vmdk
1024 -rw-------    1 root     root         8.5k Jul 13 05:03 DXP3Download.nvram
   0 -rw-------    1 root     root          372 Jul 12 23:05 DXP3Download.vmdk
   0 -rw-------    1 root     root          911 Jul 13 03:57 DXP3Download.vmsd
   8 -rw-------    1 root     root         3.2k Jul 13 05:04 DXP3Download.vmx
   0 -rw-------    1 root     root          267 Jul 13 02:42 DXP3Download.vmxf
   0 -rw-r--r--    1 root     root            0 Jul 14 01:48 dir.txt
3072 -rw-------    1 root     root         2.2M Jul 12 23:08 vmmcores-1.gz
5120 -rw-------    1 root     root         4.1M Jul 12 23:08 vmware-10.log
1024 -rw-------    1 root     root       236.7k Jul 12 23:08 vmware-11.log
1024 -rw-r--r--    1 root     root       134.8k Jul 13 02:11 vmware-12.log
1024 -rw-r--r--    1 root     root       188.4k Jul 13 03:57 vmware-13.log
1024 -rw-------    1 root     root       277.3k Jul 12 23:08 vmware-8.log
1024 -rw-------    1 root     root       794.0k Jul 12 23:08 vmware-9.log
1024 -rw-r--r--    1 root     root       229.8k Jul 13 05:31 vmware.log
47104 -rw-------    1 root     root        46.0M Jul 13 04:58 vmx-DXP3Download-176992513-1.vswp

will merging the delta disk require the help of a vmsd file? i feel that there is enough information in the vmdk and -delta.vmdk files.

krishna_v78 · ‎07-13-2012

Hi,

Please follow below link for deleting snapshots from command line.

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=100231...

Good Luck!!

Balu.

Gkeerthy · ‎07-13-2012

Hi

The best and easy method for your issue is to use virtual to virtual conversion (V2V) you can use vmware converter standalone verision of you can use the one which comes with the vcenter bundle and install it in the vcenter.

manully deleting and manging this much snapshot files is risky, there is a chance to get crroupted.

http://www.vmware.com/products/converter/

Please don't forget to award point for 'Correct' or 'Helpful', if you found the comment useful. (vExpert, VCP-Cloud. VCAP5-DCD, VCP4, VCP5, MCSE, MCITP)

a_p_ · ‎07-14-2012

This somehow looks like being kind of messed up. Most of the snapshot .vmdk files appear in the vmware.log file but some do not. Without a proper .vmsd file it's not possible to delete individual snapshots (at least not without a lot manual steps). Anyway, in this case with the virtual disk only being 10GB, I'd recommend you either follow the other posters recommendation to do a V2V or - as an alternative - create a clone of the virtual disk with all its snapshots. To create a cloned virtual disk, power off the VM and run the following command in the VM's folder:

vmkfstools -i DXP3Download-000054.vmdk DXP3Download-Clone.vmdk

where "DXP3Download-000054.vmdk" is the current virtual disk file mentioned in the .vmx file. Once cloned replace the VM's virtual disk, test whether everything looks ok and then delete the obsolete .vmdk files. Unless you are sure which files to delete, I recommend you first move them to a temporary folder and delete them later on.

André

bao__zhe · ‎07-14-2012

krishna_v78's suggestion of using command line seems to follow the same principle of regular disk consolidation: create one more snapshot and delete all, except the mentioning of "Quiesce guest file system". I can give it a try if i can understand what it means.

sparrowangels…' and Gkeerthy's suggestion of v2v looks the same as virtual disk cloning? it looks to me that it will creat a independent base disk which is not exactly what i am looking for here. this particular VM is 10GB but my total SSD space is 120GB only...plus, i have other VMs with the same problem but with much larger disk space (thin-provisioned of course). so i still need to find a solution for this problem. (in fact, i only have 7.44GB space left now...which is not good by any standard:(

a.p. seems to be implying that if i can reconstruct a proper .vmsd file then i can delete the snapshots one by one and the delta disks will be merged up one by one and not all at once. so the question remaining will be, how to reconstruct a proper .vmsd file?

krishna_v78 · ‎07-15-2012

Quiesce guest file system - In short, making disk suitable for backup by clearing dirty buffers and also from memory. it will pause all the processes running on the VM for a while. Pl refer below URL for more info.

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=101518...

B.K

a_p_ · ‎07-15-2012

I took another look at your files and thought about options to remove all snapshots easily by merging them into the linked clones first delta .vmdk file. I did some tests in my lab environment and found a solution that worked for me. If you want to try it, make sure you backup the 2 folders (base VM and clone VM) before you do anything, in order to be able to revert to the current state in case it doesn't work for you!

Here are the steps:

power off the linked clone
backup the linked clone's folder as well as the base VM's folder!
remove the linked clone from the inventory
replace the linked clone's .vmsd file with the one I attached
add the linked clone to the inventory
create a new snapshot
open the Snapshot Manager and click "Delete All"

What the lines in the .vmsd file are supposed to do, is to make ESXi aware that this is a linked clone with it's own "base" disk. Once everything worked as expected, the .vmx file should point to "DXP3Download.vmdk" as its virtual disk and all snapshot files should be gone. As I mentioned before, there are some snapshot files which don't show up in the log files, so these files (most likely 000032-...) may need to be deleted manually.

Remember, you do this at your own risk. However, "No risk, no fun!".

André

bao__zhe · ‎07-15-2012

that worked like a magic! i'll try it on other VMs too!

seems the key is the "sentinel0" thing. thanks so much for pointing this out~