VM with consolidated disk

[sandroalvesbras]

Hi,

I have a VM that had a 100GB disk with 90% occupied.

We created another 150GB disk and migrated the data to this new one.

As this VM had a snapshot, the time to remove the snapshot took a long time even with the VM turned off and was canceled.

Now the disk is about to be consolidated and when we try it has an error.

I need to fix this problem and think of some alternatives:

1 - A backup of the agent using Veeam for example and restore the disk in VMDK format and attach to the VM after removing the bad disk.

But I wondered if it would be applicable to use vSphere Replication to have the shortest possible downtime. But I believe that it will present some type of error because the disks need to be requesting consolidation. Will it really show an error or will it be able to replicate using the disk with the current information (-flat of the snapshot)?

Or would moving the VM from one dastastore to another also work?

Thank you.

[a_p_]

Now the disk is about to be consolidated and when we try it has an error.

Please provide details about the error you see, and how the VM's files currently look like, i.e. post the output of ls -lisa from the VM's folder.

André

[ashilkrishnan]

Hi

If the VM snapshots are not large and a downtime is possible for this VM, I would suggest cloning the disk to consolidate them

1. Go to edit settings and make a note of the hard disk pointer.

VM disk pointing to a snapshot will look like: VMname-00001.vmddk

Regular VMDK without snapshots: VMname.vmdk

2. Shutdown VM and take an SSH to ESXi host

3. Clone the VM snapshot disk using vmkfstools utility. Please refer following as an example:

vmkfstools -i "/vmfs/volumes/Datastore/examplevm/examplevm-000001.vmdk" "/vmfs/volumes/Datastore 2/newexamplevm/newexamplevm.vmdk"

Add '-d thin' at the end to create a thin provisioned disk.

KB reference --> VMware Knowledge Base

Cloning process will consolidate the snapshots chain and create a new disk. Existing snapshot VMDK can be detached from the VM and you can attach this cloned VMDK

Hope that helps

[sandroalvesbras]

Hi ashilkrishnan,

grateful for the guidance.

This VM is very critical and cannot be out of production for a long time.

Imagine this:

- A VM with only one 150GB disk, where 100GB is application data. To avoid expanding the VM disk and separating the OS from the application we created a new 200GB disk and moved the 100GB to this new 200GB disk. In other words, the snapshot disk that was not removed successfully is great for our environment, which has a very stressed storage.

My fear of taking any action on that VM as a snapshot, removing snaphost or even consolidating can freeze the VM, so all actions when I run it I prefer to do with it off.

But as the snaphsot is very large, the system will be off the air for a long time.

So I thought about using the vMotion Storage feature or even a vSphere Replication to keep the VM on for as long as possible.

Thank you.

[IRIX201110141]

Click that dammed "delete all" button within Snapshotmanager and move on.

[sandroalvesbras]

Hi ashilkrishnan,

I no longer have a snapshot, so it is pending to consolidate.

My fear is when asking to consolidate the VM to freeze during the process.

But from experience it will show error.

Thank you.

[ashilkrishnan]

Hi

Please go to edit settings and share a screenshot displaying the hard disks and it's VMDKs.

To consolidate the snapshot, yes we have to run the delete all/consolidate option. Screenshot does not list any snapshots, so consolidation will not work until we create a new snapshot and initiate consolidation.

Do you recall what error was reported when you tried consolidation last time ? We have following options:

1. Create a test snapshot and initiate consolidation. Review the errors, if any

2. Shutdown VM and clone the VMDK using vmkfstools

3. As far as vSphere replication is concerned, it can also be used to create a copy of the disk. You would still require a downtime to detach existing VMDK and attach the replicated disk

[sandroalvesbras]

Hi, ashilkrishnan​

using vSphere Replication I would have a much smaller RPO than cloning, because I would replicate the difference, turn off the production one and connect to another host. In this case, here is a cluster with two hosts.

It's been a long time, I don't remember the mistake.

I believe that the snapshot will not work, because the backup routine started to fail shortly after the consolidation state.

Thank you.

[ashilkrishnan]

Hard disk 1 is running on the base disk and hard disk 2 is the one which is running on snapshot disk

Hard disk 2 requires consolidation

Yes, you can try replicating using vSphere replication from datastore to another

[scott28tt]

As you're using the old Windows client for vSphere, you may also be running what is now an unsupported vSphere version - you should consider an upgrade to at least 6.5 (if you're not already at that version) - particularly as this is clearly a production environment.

[a_p_]

You didn't answer my question, regarding the error message yet.

Anyway, according to your screenshots you seem to have ~200GB free disk space on the datastore, and the snapshot size is ~130GB. With the thin provisioned base disk being ~98GB, snapshot deletion may require up to ~100GB of temporary disk space on the datastore.

So what I would do is to follow  IRIX201110141's recommendation to create another snapshot and then run "Delete All" from the Snapshot Manager. Due to the snapshot size, this may take some time, so please remain patient, and ensure that no image based backup etc. kicks in during that time.

Regarding replication. IIRC, a VM must not have any snapshots when a new replication is started.

André

[sandroalvesbras]

Hi a.p.,

sorry for not responding the error is because this VM is very critical and after we cancel the removal of the snapshot we do not run a new snapshot to see the error.

Even more that we moved it as a host so all history was lost.

My fear is to run a new snapshot and the VM will freeze.

[IRIX201110141]

Press the consolidate button now. There is no other solution.

Regards,
Joerg

[Sukanyad]

Most of the Time the snapshot manager would not show snapshots and the only way we can resolve the snapshots is take a test snapshot and do a consolidate .There might be latency issue while the vm is powered on .But incase you have downtime you can go ahead and do delete all ,that might be faster .But again you need to wait until the consolidation is completed .

[sandroalvesbras]

Hi a.p.Sukanyad​​,

this is an example of a 300G VM with a lot of transaction (Exchange).

I ran a snapshot out of memory that is much faster, see ping.

Imagine a VM with ~ 100GB of snapshot to consolidate.

God help me!!!

Thank you.

[continuum]

When you have to promise "do it like this and nothing can go wrong" I would recommend to do this:

power off VM asap for a downtime of a minute

- delete or remove vmsd-file

- create new snapshot

start VM - which now should use name-000002.vmdk

then use the vmkfstools -i name-000001.vmdk consolidated.vmdk

command from cli to consolidate to a new vmdk

Advantage: if the consolidation fails or slows down to snail speed you can kill the process

Advantage: calculation of required space is way easier

When vmkfstools finished you then take another minutes downtime to switch over to the new location.

Then you optionally add the snapshot2 or go with the state from the first downtime.

Worst case scenario where the VM crashes after the first downtime is not critical as you already started a consolidation from last working state ...

So maybe not the most convenient approach but the safest.