Failing to consolidate disks on on esxi 6

[Thysdeb]

HI All,

Ok something went seriously wrong here and i am very new to this, have no clue how to fix. and i do not have other esxi hosts to move this to etc, and VeeAM does not want to back up this anymore due to these issues...

How do i go about trying to fix this..it looks really dangerous to me ...here is how the folder for this vm looks like... please see the screenshot..

[a_p_]

That should indeed be cleaned up.

As a first step, please run

df -h > filelist.txt
ls -lisa >> filelist.txt

in the VM's folder to find about about the free disk space, and for complete file listing (including the used disk space).
Download filelist.txt, and attach this file to your next reply.

>>> VeeAM does not want to back up this anymore ...
Does Veeam B&R show any error messages related to this? Which ones?

What you may also verify, is that - in case Veeam uses "Hot-Add" for VM backups - none of the VM's virtual disk files are still mapped to the Veeam Backup server, or one of its proxies.

André

[Thysdeb]

HI Andre,

Apologies for the delay were out of office last week..

Please find attached the listings

Error from VeeAM is just a generic error ..

2023/05/29 13:13:32 :: Processing lws7a Error: VDDK error: 1 (Unknown error). Value: 0x0000000000000001
Failed to upload disk. Skipped arguments: [vddkConnSpec>];
Agent failed to process method {DataTransfer.SyncDisk}.
Exception from server: VDDK error: 1 (Unknown error). Value: 0x0000000000000001
Unable to retrieve next block transmission command. Number of already processed blocks: [12044].
Failed to download disk 'Device '\\.\PhysicalDrive1''.

NO Hot -Add as far as i can tell..

Kind Regards,

Thys

[a_p_]

I love unknown errors. That's so helpful 😉
According to the files, something happened on May 9th 04:24, which may be the root cause of this issue (dump files).

In this case I see tow options:

1. Create another snapshot, then run "Delete All" from the Snapshot Manager
2. Clone the virtual disk from the command line (it looks like there's sufficient free disk space to do this)

Option 1 is like surgery on the open heart. Nothing should go wrong, but you never know. This option can be done online, but with no recent backups, you should consider option 2.

Option 2 is an offline option (VM powered off). With this option we'd clone the virtual disk manually, i.e. the original .vmdk files remain untouched. However, it may take some time to clone the 1.6TB. If you want to go with this option, let me know, so that I can provide you with the required steps.

For both option, ensure that the Veeam backup job for this VM is stopped/disabled, so that it will not kick in while the consolidation is running! I'd also suggest that you reboot the Veeam B&R server if possible to ensure that e.g. no stale processes will access/lock the VM's .vmdk files.

André

[Thysdeb]

HI NAdre, yes those unknown errors are just the best!!!🙄

....

think i must try option 2 please ... i have another on-site vmware host i can copy/clone this to?...will it be possible, or do i need to do this first on the same host ... a question. so if I clone the server, the new folder will then ONLY have the needed disks and all those other will be gone?.. and your day this is the safest?..

the backup has be disabled/stopped (veeam) when i saw there were issues...it was restarted in the meantime as well..

Again thanks so much for your help....

KInd Regards, Thys

[a_p_]

With 2.8 TB free disk space, I'd suggest that you clone the virtual disk on the same datastore (in the VM's folder) itself.
Btw. the virtual disks size that need to be cloned is just 180GB. I got confused with the VM's second virtual disk (1.6TB) which does not have active snapshots. In this case the cloning shouldn't take that long.

To do this:

1. make sure then backup for this VM is turned off
2. shut down the VM
3. connect to the host via e.g. putty and cd to the VM's folder
4. find out the current snapshot's .vmdk file name in the VM's configuration file (e.g. grep vmdk *.vmx)
5. clone the virtual disk from the current snapshot using the following command (replace ### with what you found in the .vmx file)
   vmkfstools -i lws7a-000###.vmdk lws7a_0.vmdk
6. edit the .vmx file, and replace the snapshot file name lws7a-000###.vmdk with lws7a_0.vmdk
7. reload the VM by following the steps in https://kb.vmware.com/s/article/1026043 (this step is important!)

Next power on the VM. If it works as expected, you may then run a backup, and finally delete all the now obsolete .vmdk files.
These are the those that do NOT start with "lws7a_0" or "lws7a_1".

If you have any questions, please ask before you start the steps!

Note: I picked "lws7a_0.vmdk" as the clone's file name, so that it is easier to delete the obsolete files later. You can of course use any other file name, except for the ones that already exist in the VM's folder

André

[Thysdeb]

HI Andre,

Thanks a million..going to try this this weekend when I hope i can schedule some down time..

KInd Regards,

THys

[Thysdeb]

Hi Andre,

Have some time on my hands, not going to do it right now but looking around solong.... some questions, and apologies for my stupidity again ....this is so very new to me...

1. grep output this

[root@roamer2:/vmfs/volumes/5f9ff131-ca7bc822-061e-782bcb788437/lws7a] grep vmdk *.vmx
scsi0:0.fileName = "lws7a-000168.vmdk"
scsi0:1.fileName = "lws7a_1.vmdk"

2. The bold disk entry is the one to clone right? .. BUT here is my worry i see this is only like 1KB big.. or will the command go and fetch all the other disks and "combine"? to get to the 180GB..

Kind Regards,

Thys

[a_p_]

Don't worry, that looks as expected.

1. The bold .vmdk file (scsi0:0.fileName) is the one that needs to be consolidated.
2. The size is correct, because it's just a descriptor file (basically a plain text file), which points to its associated data file. In this case to "lws7a-000168-delta.vmdk". The file also contains a reference to its direct parent file, so that the snapshots can be processed/used in the correct order.

André