Can't merge split VM hard disk

[f1alan]

Hi,

We have recently setup quite a comprehensive server using VMware Workstation Pro (running on Windows Server 2012). However, it has got to the stage where VMware is (understandably) slow at times because our disks have grown substantially and are split over multiple files (there are also multiple snapshots that we took as we were building the machine).

The problem is that VMware can stop responding for a while when we try to interact with it. We have done some reading (VMware Workstation Speed-Up, How to Fix Slow Performance | BC Blog) and have come to the conclusion that the problem may be that we are using a hard drive that is split over multiple files. With hindsight we wouldn't have selected this option when we started but it was the default.

Having tested vmware-vdiskmanager.exe with the original vmdk file successfully, I thought that it would just be a case of running the following command on the vmdk file corresponding with the most recent snapshot:

"C:\Program Files (x86)\VMware\VMware Workstation\vmware-vdiskmanager.exe" -r ubuntu-000030.vmdk -t 0 C:\virtual_machines_merged\ubuntu\ubuntu.vmdk

It seemed to be working and got to "Convert: 100% done." but after a while it gave a "vmware virtual disk development kit error" error message box but unfortunately the message box was completely blank. However, in the console it gave the following error:

VERIFY D:\build\ob\bora-3770994\bora\lib\public\memaligned.h:241

CoreDump: Can't find DbgHelp.dll!

Win32 object usage: GDI 4, USER 1

At the point of failure the vmdk file had grown to around 16GB in size.

Has anyone else encountered this problem or can provide advice on an alternative strategy?

Thanks,

Alan

[a_p_]

Welcome to the Community,

which version of VMware Workstation do you use? What's the provisioned size of the virtual disk, and how many active snapshots does the VM currently have?

IMO the reason for slowness is not really related to the file format itself, but definitely to running on multiple active snapshots. The sparse file format has several advantages, especially when it comes to cloning, clean-up, ..., so I woulnld't necessarily recommend to change it.

I can't tell you what the error message means, but I'd suggest (after a server backup) that you run chkdsk to rule out logical errors on the host, also make sure that *.vm* files are exclused from being scanned by you A/V application. After this is done try to see whether

"C:\Program Files (x86)\VMware\VMware Workstation\vmware-vdiskmanager.exe" -r ubuntu-000030.vmdk -t 1 C:\virtual_machines_merged\ubuntu\ubuntu.vmdk

causes the same error message.

André

[f1alan]

Thanks for your response.

Current Size: 1.2GB

Maximum Size: 500GB

Is that what you meant by provisioned size?

Version: 12.1.1 build-3770994

Thanks for the AV suggestion, that is definitely worth looking at.

Alan

[f1alan]

Sorry, forgot to say that we currently have 30 snapshots.

[a_p_]

This definitely explains the slowness. So please go ahead with the above mentioned steps.

André

[f1alan]

Today I tried copying all of the VM files to the equivalent location on a different machine (fresh install of Windows) and with the antivirus disabled.

First I tried using the same command as I had already tried (-t 0) and was able to replicate the error. I then tried -t 1 as you suggested and got the same error.

If you have any other suggestions, I'll gladly try them.

Thanks,

Alan

[a_p_]

With the 500GB provisioned virtual disk size (i.e. the size in the VMs settings), the virtual disk consists of 32 chunks. With 30 active snapshots the number of data .vmdk files is 32 x 31 (base+snapshots) = 992. Maybe the command has an issue with the huge number of files (still assuming/hoping that the files are not corrupt).

What you may try at this point is to create a full clone of the VM via the GUI.

If this also stops with an error, we may need to split the snapshot consolidation into multiple runs, which requires manual changes to the files. For this I'd like to ask you to compress/zip all of the VM's descriptor/header .vmdk files (the small text files without -s###.vmdk in their names) along with the VM's .vmx and .vmsd file, and attach the .zip archive to a reply post.

André

[f1alan]

I tried cloning the VM but that did indeed fail with the same error.

Please find attached a zip file containing the requested files. I have had to obfuscate some of the content but functionally they are unchanged. I can do a diff with any modifications that you make.

[a_p_ · Accepted solution]

To ensure I ddn't miss something, did you try to run a VM with the cloned 16GB virtual disk (even with the error message at the end)? Is the 16GB about the space that's used by the guest OS?

Anyway, what you may try now is to run a clone in several steps. With the ~30 snapshots, I'd suggest we try this with 10 snapshots at a time.

So the first run would be(using the target directory you posted earlier)

"C:\Program Files (x86)\VMware\VMware Workstation\vmware-vdiskmanager.exe" -r ubuntu-000010.vmdk -t 1 C:\virtual_machines_merged\ubuntu\ubuntu-run1.vmdk

If this succeeds edit "ubuntu-000011.vmdk" (after backing up the original file), to chain this anpshot to the newly created clone.

• set the value of the "parentCID" to the value of the "CID" in "ubuntu-run1.vmdk"
• set parentFileNameHint="C:\virtual_machines_merged\ubuntu\ubuntu-run1.vmdk"

Now we are ready for a second run:

"C:\Program Files (x86)\VMware\VMware Workstation\vmware-vdiskmanager.exe" -r ubuntu-000020.vmdk -t 1 C:\virtual_machines_merged\ubuntu\ubuntu-run2.vmdk

If this succeeds edit "ubuntu-000021.vmdk"

• set the value of the "parentCID" to the value of the "CID" in "ubuntu-run2.vmdk"
• set parentFileNameHint="C:\virtual_machines_merged\ubuntu\ubuntu-run2.vmdk"

Now we are ready for the final run:

"C:\Program Files (x86)\VMware\VMware Workstation\vmware-vdiskmanager.exe" -r ubuntu-000029.vmdk -t 1 C:\virtual_machines_merged\ubuntu\ubuntu-run3.vmdk

Assuming the target folder was empty prior to starting these runs, you can certainly use another target .vmdk name at this point. Otherwise you may use vmware-vdiskmanager with the "-n" option to rename the final clone later.

The above doen't modify the original files (except for the manual edits I mentioned). If one of the runs fails, we will be able to modify the clone command (i.e. walk down the snapshot chain) to see whether we can find out which snapshot causes the issue (if any).

If everything worked without issues, move all the .vmdk files as well as the .vmsd file from the VM's original folder to a temporary sub-directory, and move the latest cloned .vmdk files from the 3rd run into the VM's folder. Then edit the .vmx file, and replace the virtual disk's file name with the one of the clone.

André

[f1alan]

To ensure I ddn't miss something, did you try to run a VM with the cloned 16GB virtual disk (even with the error message at the end)? Is the 16GB about the space that's used by the guest OS?

Yes, and it was clearly corrupt when I tried to run it.

Thanks for the instructions. I will give this a try soon.

Alan

[f1alan]

Thanks again for your help a.p. I just wanted to follow up to say that I've been able to successfully merge our snapshots using your technique.

In the end it wasn't quite as simple as doing 3 runs of 10 snapshots. It worked on my local test machine but I still got that error on our production server. I ended up merging the files in one by one and gradually building up the eventual vmdk file. I also took the opportunity to defragment the vmdk file and so far it is a noticeable improvement in performance.

Alan

[a_p_]

Glad to her you were able to resolve the issue, and performance increased again.

Anyway, just one quick question: When you merged the snapshots on-by-one, did you follow the snapshot chain? I'm asking because I noticed that the not all of the file names were in an ascending order (compared to the snapshot chain, i.e. the information within the files)?

André

[f1alan]

Yes, I noticed that too and made sure that I followed the chain.

Alan