VMware Cloud Community
SRCPTony
Contributor
Contributor
Jump to solution

Virtual Disk / Snapshot Consolidation Questions

Hi All,

From the guidelines in Knowledge Base KB Article 1007849 (Link), I have some questions regarding VMDK/Snapshot Consolidation.

First, there are NO SNAPSHOTS (or SS files) in the Snapshot Manager or Browser (however, I believe this is because I never fully committed the snapshots prior).

As can be seen in the image, I have 3 flat files:

FILE1.vmdk (10GB)

FILE1_1.vmdk (35GB)

FILE1_2.vmdk (60GB)

From running the commands (and in the browser), I show that I have 4 files with the extentions: vm_x-x-00000x.vmdk (where x = 1 or 2) along with three original flat files.

As can be seen in the image, they are:

FILE1-000001.vmdk (1.4GB)

FILE1_1-000001.vmdk (7GB)

FILE1_2-000001.vmdk (28GB)

FILE1_2-000002.vmdk (132MB)

The commands show that the files in use are: FILE1-000001.vmdk, FILE1_1-000001.vmdk, FILE1_2-000002.vmdk

My aim now is to consolidate these four x-00000x.vmdk files with the original flat files (I am running out of space on my LUN and badly need to reclaim the space being used by the deltas).

Following the KB Article, I can do this by cloning the LATEST delta file (vm_x-x-00000x.vmdk) to a temporary LUN and the result will be a FLAT file with all the delta changes AND the original structure. I can then do this for each delta disk and the result would be 3 nicely consolidated flat, thick VMDKs with the delta changes and original contents at the size of my original flats.

Am I correct? Or am I missing something?

Any advice/help/tips/suggestions on VMDK/Snapshot Consolidation would be greatly appreciated!

Tony

Reply
0 Kudos
1 Solution

Accepted Solutions
ThompsG
Virtuoso
Virtuoso
Jump to solution

Morning Tony,

I agree 3GB is cutting it a little fine.

Both should achieve the same outcome, but I guess the question that should be asked and answered is: are you able to shut the VM down? If the answer to this is yes, then either option is fine, but I would probably go with the vmkfstools -i option first. If the requirement is to leave the machine running then I would use the VMware Converter to achieve this given the free space you have.

I have just had another thought. If you are using vCenter then you could also clone the virtual machine. This would be almost the same as using the vmkfstools -i option. Doing a clone should be faster than the VMware Converter as all the copying will happen across the disks rather than going across the network. There are assumptions made in this statement howerver generally it should be true. The only issue with doing it this way is the virtual machine shell will need another name, whereas the vmkfstools option you can just reattached the new disks to the same shell. Depends on how comfortable you are with the console.

.

Let us know how you get on.

Kind regards,

Glen

Message was edited by: ThompsG

Message was edited by: ThompsG

View solution in original post

Reply
0 Kudos
12 Replies
ThompsG
Virtuoso
Virtuoso
Jump to solution

Morning Tony,

From reading the knowledge base article and your problem, it does appear that you are on the correct path. This article does of course assume that the machine is powered off in order to complete the steps.

Another option that I have used in a similar situation is as follows:

1. Login to service console as root

2. Run the following command to create a new snapshot:

vmware-cmd <vm-cfg-path> createsnapshot <name> <description> 1 1

3. Run the following command to remove all snapshots:

vmware-cmd <vm-cfg-path> removesnapshots

This will consolidate all the snapshots that are associated with your virtual machine while it is still running. This of course assumes you have some space left for the new snapshot. In your case it will probably take a long time to apply all the snapshots as they must be quite large.

Please accept that this solution is provided as is. All care but no responsibility.

Kind regards,

Glen

SRCPTony
Contributor
Contributor
Jump to solution

Glen,

That would be a fantastic solution however, I only have about 3GB free on the LUN right now, not nearly enough to create a new snapshot. Great idea, though Smiley Happy

One other potential solution brought to my attention (credit: za_mkh on ee) is the possibility of using VM Converter to perform a V2V conversion. That would, in effect, give me a perfect clone of the VM with the deltas consolidated into the flat files.

What do you think? Clone the vmdks or clone the entire vm?

Tony

Reply
0 Kudos
ThompsG
Virtuoso
Virtuoso
Jump to solution

Morning Tony,

I agree 3GB is cutting it a little fine.

Both should achieve the same outcome, but I guess the question that should be asked and answered is: are you able to shut the VM down? If the answer to this is yes, then either option is fine, but I would probably go with the vmkfstools -i option first. If the requirement is to leave the machine running then I would use the VMware Converter to achieve this given the free space you have.

I have just had another thought. If you are using vCenter then you could also clone the virtual machine. This would be almost the same as using the vmkfstools -i option. Doing a clone should be faster than the VMware Converter as all the copying will happen across the disks rather than going across the network. There are assumptions made in this statement howerver generally it should be true. The only issue with doing it this way is the virtual machine shell will need another name, whereas the vmkfstools option you can just reattached the new disks to the same shell. Depends on how comfortable you are with the console.

.

Let us know how you get on.

Kind regards,

Glen

Message was edited by: ThompsG

Message was edited by: ThompsG

Reply
0 Kudos
SRCPTony
Contributor
Contributor
Jump to solution

Glen,

Performed a VM clone last night using VII Client and it worked like a charm!! Not only did the delta disks consolidate back into the flat files at the original size, but I actually gained free space inside the flat virtual disks (inside Windows, not on the actual storage) resulting from some cleaning I had done which had been writing to the appended vmdks!!

Steps:

Shut down VM.

Initiate Clone to fresh LUN.

Select "Do Not Customize"

After finished, booted new VM (no need to rename or re-IP since the shell is identical and the original is offline).

Test for integrity.

Shut down Clone.

Destroy original VM.

Perform Storage VMotion Migration to transfer Clone to original LUN.

Destroy temporary LUN.

Start Clone and rename to original server name in VI Client.

Worked like an absolute charm. Round trip: 3 1/2 hours.

Thanks for all your help!!

Tony

Reply
0 Kudos
ThompsG
Virtuoso
Virtuoso
Jump to solution

Afternoon Tony,

Excellent, glad I could help is some small way.

Kind regards,

Glen

Reply
0 Kudos
slickshoes
Enthusiast
Enthusiast
Jump to solution

Tony,

I have a similar issue where I have a ton of snapshot vmdk's on my datastore but the GUI and CLI say there are no snapshots. I know your issue has been resolved, but did the VM see the flat file as the main disk source, or one of the snapshot disks (in edit settings)?

I'm trying to consolidate all the snapshot vmdk's to the flat file, but ESX thinks the snapshot file (Exch2k7-000013.vmdk) is the disk, and not Exch2k7.vmdk. So when I tried to conslidate them using the CLI and removesnapshots it crashed the VM because all of a sudden one of the "disks" was gone"

Will cloning/converting the VM consolidate these files for me?

Thanks

-DJ

Reply
0 Kudos
SRCPTony
Contributor
Contributor
Jump to solution

@DJ,

My VM was reading/writing to the Snapshot vmdks not the flat files. The .vmx config file looked like this:

scsi0:0.present = "true"

scsi0:0.fileName = "xxxxxxFILE1-000001.vmdk"

scsi0:0.deviceType = "scsi-hardDisk"

scsi0:1.present = "true"

scsi0:1.fileName = "xxxxxxFILE1_1-000001.vmdk"

scsi0:1.deviceType = "scsi-hardDisk"

scsi0:2.present = "true"

scsi0:2.fileName = "xxxxxxFILE1_2-000002.vmdk"

scsi0:2.deviceType = "scsi-hardDisk"

Inside the actual Directory the flat files were there as well causing the directory to balloon to almost twice the size (hence, the root of the problem).

When I performed the clone (either cloning the entire VM in VI Client or doing a disk-by-disk clone from the Console) the disks actually being read from were the Snapshots. But the disks being written to were the flats. So, the clone consoldiated all the disks by comitting any delta changes back up into the original flat files. After I was finished the .vmx config looked like this:

scsi0:0.present = "true"

scsi0:0.fileName = "xxxxxxFILE1.vmdk"

scsi0:0.deviceType = "scsi-hardDisk"

scsi0:1.present = "true"

scsi0:1.fileName = "xxxxxxFILE1_1.vmdk"

scsi0:1.deviceType = "scsi-hardDisk"

scsi0:2.present = "true"

scsi0:2.fileName = "xxxxxxFILE1_2.vmdk"

scsi0:2.deviceType = "scsi-hardDisk"

The sizes were at the original flat file sizes.

So, the question for you I would ask is: Can you boot your VM or is it throwing errors when you try to start it (about missing disks, etc.)? If there are errors a clone isn't going to work correctly.

So, if you can't boot it, take a look at the files in the directory and make a note of the second to last snapshot file after the one that was deleted. Then, change your vmx config to point to that vmdk. Save and try to start the VM. If you can successfully boot it, turn it off and perform a straight VM clone from inside VI Client. This will consolidate the disks perfectly following the steps listed above (reading from the snapshots and committing the deltas back up into the flat, leaving you with the original flats and all your changes).

Note: Any deltas that were stored on your "deleted" snapshot disk are probably lost forever.

Post back some results or further info/questions.

Tony

That is,

Reply
0 Kudos
slickshoes
Enthusiast
Enthusiast
Jump to solution

Thanks for the quick and detailed reply.

My .vmx file also points to the snapshot deltas, not the flat files. Is this correct ESX behavior? Seems strange to me.

The VM can boot now, I had to change the VM to point to the flat file and not the snapshot that had been consolidated. I think I will have to take it down and clone it so they get all flattened out.

How long do you think it will take? I have a pretty good environment. The VM has about 140GB of disks

Thanks again

-DJ

Reply
0 Kudos
SRCPTony
Contributor
Contributor
Jump to solution

DJ

It's the way ESX is designed. Once a snapshot or delta disk is created, it becomes a "child" of the flat (or "parent") and from that point on the vmx config points to and uses the new delta/ss vmdk (while keeping the reference back to the parent). Unlimited "Childs" can be made, each referencing the last all the way back to the parent (which is why VM Snapshots shouldn't be saved permenantly as it can degrade performance and cause other problems).

The flat will remain in the directory but ALL changes are made to the "active" delta/ss (child) vmdk disk.

When you perform a "remove snapshot" (either from Console of VI Client) the delta vmdk actually "commits" the changes back into the flat file. So, it's not just removing the snapshop file but also saving all the changes from the delta vmdks back into the flat (or previous child in this case).

Now, you mention that you've altered your config to point to the flats. Likely then, you are not seeing any of the changes that were made to the system since the first snapshots were taken. So, if you perform a clone at this point, the only thing that will get cloned are the flats and the delta/ss vmdks will be ignored.

You have two options:

1) If you're happy with the VM using the original flats and don't care about anything on the deltas, you can probably just delete all the delta files and go forward using the flats. (Just to be safe, I would shut down the VM, download a delta to local disk from Datastore Browser first, then delete it from the VM directory, boot the VM, and see if any errors crop up. If so, you can always upload it back : ))

OR

2) To achieve a true "Disk consolidation," change the vmx config to point to the most recent ss/delta vmdks. This will require some analyzation of the disk structure in the VM directory (vmdk naming sequence) to determine which are the original flats (parents) and who their deltas (childs) are, and then figuring out the correct order. Once you've determined what the latest deltas (childs) of each flat (parent) are, change your config to point to those and boot up the VM. If it comes up clean, you know you are referencing the most recent vmdks. At that point, perform the clone and you will consolidate all the child disks back into the flat, saving any changes as well, and be good to go into prod with the new clone.

In my case, I had a similar VM (120GB flat disk size, spread over three virtual disks. Plus, 30GB of snapshot/delta/child disks sitting on a 150GB LUN. Not good. Smiley Happy) The VM clone (from VI Client) took about 45 mins-1 hour running over a 1GIG network from one iSCSI LUN to another on the same SAN.

Good luck and let us know how it goes!

Tony

Reply
0 Kudos
SRCPTony
Contributor
Contributor
Jump to solution

DJ,

Upon reading your post again, it definitely sounds like you have corrupt snapshots. Since you executed a "remove all snapshots" command, it's likely that they're all corrupted as well. Taking this into consideration, it sounds like your best bet is to just run with the original flat files and chalk up any changes in the deltas as a loss. You can either just delete the snapshot vmdks and see what happens or perform a clone which will only reference your flats and result in a fresh VM with the original structure and zero references to any snapshots.

Tony

Reply
0 Kudos
slickshoes
Enthusiast
Enthusiast
Jump to solution

Tony,

The server has 3 disks. 0:0 is the C (system) drive, 0:1 contains my Exchange info store and 0:2 is used as a temporary dump for our online vaulting software. I ran the "deletesnapshots" command and it started consolidating the contents of 0:2 to a single disk. I walked away and when I came back a while later, the VM was off and the deletesnapshots was finished and I had the # prompt again. I don't think it ever got to the other 2 disks.

I tried to fire up the VM, but it wouldn't start because 0:2 was pointing to one of the delta files that had been consolidated. So I pointed that single disk (0:2, the one for temp files) to the original flat file. All the deltas for that disk are gone except one that is 1MB. It started up fine and no data was lost on 0:0 or 0:1. I don't care about 0:2 because as i said it is used a location for temporary dumping.

The other 2 disks (C and info store drive) are still pointing to the delta snapshot files.I believe if i turn the VM off and clone it everthing will be back to normal

Thanks !

Reply
0 Kudos
SRCPTony
Contributor
Contributor
Jump to solution

DJ,

I believe you're absolutely correct. Smiley Happy

NOte, when you perform the clone, you will have to give it a new name. Something like SAMESVRNAME-NEW or SAMESVRNAME-1. Select "Do Not Customize." Atferwards, ESX will reference the disks as SAMENAME-NEW (or whatever) but it won't matter because the shell and data will be identical. Leave the original off, boot the new, test for integrity (check out the Datastore to make sure there are no snapshots, although there shouldn't be.) Then perform a "Delete from Disk" on the original. After its gone, simply rename the Clone in VI Client to the original name so it "feels the same again" and all will be well. Smiley Happy

Tony

Reply
0 Kudos