Hi there,
I'm having an issue with one of our VMs (thin provisioned) which is creating a lot of .vmdk files for some reason. There is a warning displayed asking me to consolidate the disks but it doesn't work.
Furthermore, the snapshot manager does not show any snapshots taken (same in CLI) and i'm at a loss of what to do now, the machine will not boot anymore.
Is anyone able to point me in the right direction? Attached you can find some screenshots of the datastore files.
Thanks!
Pieter
Due to the base virtual disk being thin provisioned with a provisioned size of 800GB, the current file size of the flat.vmdk, and snapshots which sizes sum up to ~400 GB, deleting/merging the snapshots could require anything between a few GB, and 400 GB depending on the changed data blocks.
Unless you have another datastore with ~800GB free disk space, your options are limited.
Since you are using Veeam B&R, I'd suggest that you temporarily free up some disk space (a few GB should do) on the datastore by e.g. shutting down all the VMs on that datastore (this removes the memory swap files), and deleting unnecessary ISO images from the ISO folder. Then try to backup the VM using Veeam B&R. If that works, you may then delete the VM, and recover it again from the backup.
André
The first thing I would do is move the other machine off that Datastore and also move anything in the ISO folder off as you need some space free on the datastore. I believe the "CTK" disk are related to change block tracking backups from Veeam so my guess is that you've got an incomplete backup because the disk is full and you don't have enough space to complete the backup.
Hi Battybishop,
Thank you for the fast response. I had the same idea yesterday and moved another machine to another datastore, after which I had some 50GB of free space left. Hoping veeam would be able to complete the backup i left it running but instead it created a new 00005.vmdk file. I can confirm the machine is now running on that 0005 disk.
Thanks!
Pieter
Veeam's not a product I've used so i'll leave to others to help but have you reached out to Veeam support?
Thanks for the suggestion. I've now created a veeam ticket as well.
Kind regards,
Pieter
Unless Veeam can/will help you with this, please run ls -lisa > filelist.txt from the command line to create a complete file listing of the VM's files/folder, and attach the filelist.txt to your next reply.
André
Due to the base virtual disk being thin provisioned with a provisioned size of 800GB, the current file size of the flat.vmdk, and snapshots which sizes sum up to ~400 GB, deleting/merging the snapshots could require anything between a few GB, and 400 GB depending on the changed data blocks.
Unless you have another datastore with ~800GB free disk space, your options are limited.
Since you are using Veeam B&R, I'd suggest that you temporarily free up some disk space (a few GB should do) on the datastore by e.g. shutting down all the VMs on that datastore (this removes the memory swap files), and deleting unnecessary ISO images from the ISO folder. Then try to backup the VM using Veeam B&R. If that works, you may then delete the VM, and recover it again from the backup.
André
This is more or less what I ended up doing.
I've restored the machine from a previous backup and now I have a clean datastore again. I still have no idea what caused the 0000x.vmdk files being created but after the new backup this night there are no extra files being created so I think I'm good for now.
Thank you for your responses!
Pieter
The snapshots have likely (and will still be) created during the backup job. After a successful backup, the snapshot should be deleted.
However, it looks like that did not work at some point in time, so that a snapshot grew that much in size, that there was not enough disk space left to delete it. Due to this, subsequent snapshots could not be deleted either.
André
Have you figured out what is going on ? I am having the same issue. I ended up moving some of my gold images off the full drive but after the migration, I now have folders on the old drive with multiple OLD vmdk's and then the vmdks that migrated to the new storage. Moving them did trigger a consolidate snapshots, and currently am doing that, but its taking forever. Hoping the consolidate fixes it.
