Snapshot Consolidation filled the datastore up.

[reecefroggatt]

Hi there,

I'm looking for a bit of advice. We're a SMB with 2 ESXi hosts and 5 virtual machines. The last few days, we've had loads of issues with our entire network coming to a standstill. We eventually found this to be excessive disk usage by one of the VM's, it seemed as if ESXi was trying to consolidate snapshots automatically - the issue being, there is no snapshots. Anyway, we kept cancelling the task as it was bringing our network to a halt, and so I decided to force consolidate it Saturday morning and let it continue over the weekend.

I've checked on it today, and it has stopped as the data-store is at 100%, ESXi has also shut down some VM's. Now, before we started the consolidation, there was no snapshots in snapshot manager and datastore was at 42%, It's a 1TB disk.

The VMDK file for the VM has jumped from around 200GB to 600GB+.

I'm really stuck as I don't know how to reduce this and get the VM back online, it's our SQL machine and we cant run business without it, unfortunately our last Veeam backup was last week so can't really restore that to another host without loosing a week's data.

Any ideas?

Kindest regards,
Reece.

[reecefroggatt]

Sorry about that.

See amended below. Thanks.

[reecefroggatt]

BD-VOSTRO-001-000001.vmdk - MISSING

BD-VOSTRO-001-000001-sesparse.vmdk  

BD-VOSTRO-001-flat.vmdk             

BD-VOSTRO-001.vmdk 

BD-VOSTRO-001_1-000001.vmdk - MISSING

BD-VOSTRO-001_1-000001-sesparse.vmdk

BD-VOSTRO-001_1.vmdk - MISSING

BD-VOSTRO-001_1-flat.vmdk

BD-VOSTRO-001_2-000001.vmdk - MISSING

BD-VOSTRO-001_2-000001-sesparse.vmdk  

BD-VOSTRO-001_2-flat.vmdk  

BD-VOSTRO-001_2.vmdk - MISSING

[continuum]

Ok - then I think I should visit you  - do you have skype and Anydesk ?

I get a coffee and will be available in 15 minutes

Ulli

[reecefroggatt]

Sure.

Any desk ID is 825 965 396 and Skype is live:reecefroggatt

Cheers

[a_p_]

Hi Ulli,

in case you need them, please find the recreated files in the attached archive.

André

[reecefroggatt]

Hi Andre,

I just uploaded the recreated files into the datastore and just got the following error:

haTask-5-vim.VirtualMachine.powerOn-2775414770

Description

Power On this virtual machine

Virtual machine

bd-vostro-001

State

Failed - File system specific implementation of Ioctl[file] failed

Errors

0" class="" style="box-sizing: border-box; padding-left: 0px; margin-bottom: 10px; margin-left: 25px;">
• File system specific implementation of Ioctl[file] failed
• The parent virtual disk has been modified since the child was created. The content ID of the parent virtual disk does not match the corresponding parent content ID in the child
• Cannot open the disk '/vmfs/volumes/5bb76f75-2f24304d-b7ef-509a4c829407/BD-VOSTRO-001/BD-VOSTRO-001-000001.vmdk' or one of the snapshot disks it depends on.
• Module 'Disk' power on failed.
• Failed to start the virtual machine.

Any ideas?

Kind regards,

Reece.

[a_p_]

Did you upload all the files, i.e. did you also overwrite the existing "BD-VOSTRO-001.vmdk"?

This is necessary so that the snapshot .vmdk file's parentCID matches the base file's CID.

If you've uploaded all the files, and it still shows this error, try reloading the VM (see https://kb.vmware.com/s/article/1026043)

André

[continuum]

Hi Andre

thanks for the descriptorfiles.

We were able to launch the VM now.

We decided to remove disk_1 - it was not used at all.

Next we will add another physical disk to ESXi so that we can make sure another consolidation attempt will not exhaust free space again.

As the C:drive is way larger than necessary - 1.8TB with less than 200GB used we consider to run Converter against this vmdk next weekend to resize the vmdk to a reasonable size.

Right now Reece is looking for an additional disk so that we can put one snapshot for the C-drive on it to survive the next week.

Delete_all is too risky for my taste so we will not use that.

Anyway - we are almost back in safe waters.

The Windows survived the running out of space scenario without requiring a checkdisk ....

Thanks Andre

Ulli

[a_p_]

Hey Ulli,

thanks for the update.

I actually don't see a risk in deleting the snapshots if the VM runs as expected, and there's sufficient free disk space for the thin provisioned base disks to grow.

Anyway, great to hear that the VM is back up.

André

[continuum]

Final plan:

copy out the database to USB

rebuild the ESXi

create new VM and re import the database.
Not a bad decision to get rid of that oversized VM ( which was a p2v import originally)

Thanks for the 5 bucks for the coffee

Ulli