VM with lots of VMDK

[tractng]

Trying to chase this issue down with a coworker.  We adopted this environment which is on 6.5.  There is a VM with over 100 VMDK files.  The machine is a Windows  Server which boots up fine but its very slow.  In the setting, hard disk is assigned 100 GB.

Any ideas how we can chase this issue down?

Thanks,
TT

[StephenMoll]

Sounds like it might have a load of snapshots.

Can you check to see if it has any snapshots, and if so how many?

Keeping snapshots for too long can be detrimental to performance. If using snapshots try not to have more than 2 or 3 at a time, and try not to use them for more than 72 hours. See https://kb.vmware.com/s/article/1025279

Find who if anyone is creating them and why? Might be possible to remove them all.

I think some back-up tools use snapshots, if you have such a process is it working correctly and removing snapshots after completing its backup tasks.

[a_p_]

Please run the following commends from the command line (in the VM's folder), and attach the resulting filelist.txt to your next reply.

df -h > filelist.txt
ls -lisa >> filelist.txt

In case that you are using a VM based backup application which uses the Hot-Add mode, check whether either the backup server itself, or one of its proxies (if used) still has one of the VM's virtual disks attached. I've seen this a couple of times, mainly caused by failed, or disrupted backup processes.

André

[tractng]

We took a manual snapshot and tried to delete all (failed).  Also tried to consolidate and failed.  There are two total snapshots.

[markey165]

Hi TT,

You mention the VM has "100 VMDK files" then mention "Hard disk is assigned 100GB" which implies the VM has a single disk attached?   Can you elaborate a bit more on whether these disks are directly attached or are you just seeing the VMDK files in the Datastore Browser?   Can you add a screenshot or produce a filelist as suggested by @a_p_ ?

One other thought at the moment, is these could be snapshot files leftover by backup software which, for whatever reason aren't being exposed in the UI.  A key indicator will be the format of the filenames. If you're seeing anything like *-000001.vmdk, *-000002.vmdk and so on, these would suggest snapshot files are present.

Also, when looking at the VM settings, examine the hard disk setting (assuming there is only 1 disk attached) and determine which VMDK the VM is running from. If it is one of the above named vmdk's it would suggest the VM is running from snapshots. If you had a chain of 100 snapshots, that would definitely impact performance.

What backup product are you using (if any) to backup this VM?

[tractng]

The commands A_P suggested error out (weird both my other 6.5 & 7 environments were fine in running those commands).  The vmdk are inside the datastore browser.  Originally when we posted this issue, we only had on vmdk disk attached but now we have 2 vmdk disks.

We are using Unitrends backup.  We removed the VM from the backup from the application.  We noticed backup was also failing for many days.

We are downloading the VM into our NAS and will try to upload it to another host that is on version 7.

At this point, can we point to the original VMDK  (we actually tried but it errors out, didn't allow us)?  We don't have any valuable data.  Its just a WSUS server.

[compdigit44]

Would it be possible to upload a screen shot of what you are setting in the datastore for the VM, along with settings in the vmx file.

[tractng]

See attached.

[a_p_]

Something doesn't match with the information that you've provided.

The screenshot shows a VM with ~15,7GB RAM, and a single 100GB disk (NS2-000111.vmdk).
However, the .vmx file shows a VM with 32 GB RAM, and 2 virtual disks.

What's even worse, is that it looks like 2 different snapshots from the same virtual disk are attached to that VM!?
scsi0:0.fileName = "NS2-000114.vmdk"
scsi0:1.fileName = "NS2-000115.vmdk"

André

[StephenMoll]

Yeah that looks a bit messed up. 

I think you're lucky the VM runs at all.

I think I would be doing an export and reimporting to see if that cleans things up, or if it has data you need, copy it off whilst you can, and consider building a new, replacement VM.

[markey165]

You could try one of the following that might or might not fix it. The last one is the most likely to resolve this.

1. Try storage vMotioning the VM to a new Datastore
2. Try removing the VM from the Inventory, then re-importing it via the VMX file

The above probably won't work, but the next steps might:

• Remove the VM from the inventory. Create a new VM with the exact same CPU/RAM spec, when you get to the step where you create new or choose existing disk, choose the existing disk (ie the same file as the current VM is using) and spin up the new VM. At this point it should hopefully pickup the snapshot chain, and create a new clean VMX file in the process.
• I can't guarantee this will work, but its worth a shot. Just make sure you have a backup (or take a copy of the VM folder on the same datastore) beforehand, so you can roll back.

Let us know how you get on 👍

[tractng]

We downloaded all the files and uploaded to a standalone host.  Registered as a new VM and got a failed message.  See attached. 

Then we followed the bottom suggestion by creating a new VM and pointing it the existing vmdk file.  Tried to boot up but it kept on going into the boot manager.  We disabled UEFI (chosen BIOS instead).  Now we are the Windows Option to repair, etc. :(.

[AnaghB]

Hello @tractng ,

From the discussion it looks like VM has 114 snapshot chain and the maximum snapshots supported on Vmware is 32. This will cause corruption is OS. The best possible way to go ahead with this is below.

Proposed Solutions:

1. If VM has 114 snapshots then I believe the backup is taking those snapshots but the deletion isnt working. So the 1st solution here would be to restore the VM from last known successful backup. The restored VM will not have any snapshots and will have all the data until the last successful backup.

2. If there is no backup then we need to Power OFF the VM and clone both the disks from command line. This will require free space on datastore as we will create copy of 2 disks. This will also require down time on the VM until the cloning is completed. Once the cloning completes we will attach the cloned disks to the same VM and delete the old disk post verification of data.