NewNow
Contributor
Contributor

"virtual machine disk consolidation is needed" - what to do ?

Hi,

I came to the company where was installed ESXi host 5.1 , and five VM inside. Using also VM Sphere client 5.1.0.

Trying to figure what's the best, and send some question about backup/replication. Tnx for the answers.

Now, i'm facing this situation. Two virtual machines, in Summary tab in Sphere client shows  "virtual machine disk consolidation is needed".


At this moment everything working good, but i did the backup today from one virtual machine. Power off one virtual machines, export OVF template from Sphere, folder of files on my external hdd. After that power on this virtual machine, everything OK.


As i can assume, size of the file is close what is populated on C parition. Partition E is almost free, second partition.


A)  Is it safe to do consolidation from sphere? VM power off or on?


B) When i check HDD from sphere, Edit Settings, it's 250Gb HDD, and it's file is 00004.vmdk.   // visible on picture i sent

    

C) Summary tab in Sphere, shows provisioned storage, and used storage // visible on picture i sent


D) Snaphot manager from Sphere - // visible on picture i sent


E) Picture, where we can see backup which i made today, , and on the right panel content of this VM. 


Is this a valid backup file from the actual state just before the backup, because i can't relate this backup file from the content of this VM ?


What to do to prevent something bad, and run out of freee space?


Tnx for help and aswers.


Mladen




0 Kudos
10 Replies
a_p_
Leadership
Leadership

Welcome to the Community,

according to the files in the VM's folder the VM has 4 snapshots, with the last one created in August 2013. As you can see from the .vmdk file sizes you are effectively wasting more than 200GB.

Since the virtual disk seems to be thin provisioned, deleting/consolidating the snapshots may temporarily require a huge amount of disk space. How much free disk space do you currently have on the datastore? If you have more than 250GB free disk space then you shouldn't have issues with deleting/consolidating the snapshots. Keep in mind though that due to the huge sizes, this may take a couple of hours to finish, so please be patient!

André

NewNow
Contributor
Contributor

Hi Andre,

This VM which i send is 250Gb HDD,  70GB is occupied, and the 180GB is free. (Windows server)

        VM2  (also have issue for consolidation in Sumarry Tab)  and VM3    //   pictures of both VM, with df -h in CLI

So, for the Windows server, when consolidation took place, 180Gb free + free from another two will be used?

Mladen     

0 Kudos
NewNow
Contributor
Contributor

Andre, did you mean free space on general datatore, or free space inside of each VM disk?


Picture of datastore complete 1Tb, populated for fiveVM.

Because, here we can see, that there is no free space on datastore, everything is populated for virtual machines,

0 Kudos
a_p_
Leadership
Leadership

Yes, what's important is the free disk space on the datastore itself. To find out which virtual disks are thin provisioned and what's their physical size, please provide the output of ls -lisa for each of the five VM. Instead of posting screenshots, please consider to run ls -lisa > vmname-files.txt (where vmname is the VM's name), download and compress/zip these files, and attach the .zip file to a reply post.

André

NewNow
Contributor
Contributor

Ok, here it is.


Mladen

0 Kudos
a_p_
Leadership
Leadership

What you need to consolidate the snapshots is approximately 150GB free disk space. Since the datastore is full, and there's no other datastore available, you don't have many options. What I would do is to temporarily export "BackupServer" (~250GB) to free up disk space on the datastore, then consolidate "CentOSServer" and "WindowsServer" (one after the other) and finally upload "BackupServer" again. To do this you may use the vSphere Client's option to export the BackupServer as OVF to your local disk (assuming you have sufficient free disk space), and once done delete it from the disk on the ESXi host (right click the VM in the inventory).

Hint: If the "BackupServer" requires a specific UUID and/or MAC address (due to e.g. licensing), download/backup its .vmx and .vmxf files before deleting it. After deploying the OVF again, remove the VM from the inventory, upload the two files to the VM's datastore, and then add the VM to the inventory again (right clicking the .vmx file). Make sure you answer the question about whether you moved or copied the VM on first start with "I moved it" to preserve the UUID and MAC address.

As motioned earlier, the consolidation may take hours, so please be very patient!

André

0 Kudos
NewNow
Contributor
Contributor

Yes, i see.

Yesterday i took the backup from these three VM : "WindowsServer", "BackupServer" and "CentOOSServer".

I have this three OVF files exported on my external storage.

So, what i have now, is :


To copy these files .vmx and .vmxf files from "Backup Server".


Delete  VM "BackupServer" from sphere client. 


Do the consolidation of "CentOOSserver", and then consolidation of "WindowsServer."

After all that, deploy again from Sphere client "BackupServer" from external storage.

Mladen

0 Kudos
a_p_
Leadership
Leadership

Sounds good to me. Make sure you select "Delete from disk" for the BackupServer and verify the free disk space after deleting the VM.

There may also be other options available to solve the issue, but with this approach - and assuming that you can afford downtime for the BackupServer - the other VMs can be consolidated while still powered on.

André

0 Kudos
thulase
Contributor
Contributor

Hi,

We are also facing same issue with avamar backups. It is taking the snapshot every day for backup but the snapshots are not deleting due to issue. and HDD is attached to snapshot disk(1_0002.vmdk)

No snapshot in manger but the large number of snapshot files are in VM datastore

Try the below 3 options to resolve the issue.

First option;

take one new snapshot. After snapshot created wait 10 min and use Delete all option to delete the snapshot. Then do disk consolidated

If you have unsuccessfully attempted to consolidate using the Take a Snapshot then Delete all Snapshots technique, attempt the technique again, but this time select the Quiesce guest file system checkbox to enable quiesced snapshots when you Take a Snapshot.

if it is not working

if it is critical or IMP server take full clone of VM and follow the below steps.

Second option

Committing snapshots when there are no snapshot entries in the Snapshot Manager

Using the command prompt Delete the snapshot and delete all snapshot and then try to do consolidate.

See the below vmware KB 1002310 article for more information

http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&externalId=1002310&sl...

On an ESXi host, to commit all snapshots using the command line:

  1. 1. Log into the ESXi host as root via the console or an SSH session. For more information, see Tech Support Mode for Emergency Support (1003677) or Using Tech Support Mode in ESXi 4.1 and ESXi 5.x (1017910).

    Note: The commands in this procedure can also be executed remotely using the vSphere Command Line for both ESX and ESXi hosts. For more information, see the vSphere Command-Line Interface Documentation.
  2. 2. Run this command to get a list of virtual machines and the VMID for each virtual machine:

    # vim-cmd vmsvc/getallvms

    The output appears similar to:

    Vmid Name File Guest OS Version Annotation
    1 vm1 [datastore1] vm1/vm1.vmx windows7Server64Guest vmx-08
    3 testvm [iscsi1] testvm/testvm.vmx winNetDatacenterGuest vmx-08


    Make a note of the VMID for the specific virtual machine.
  3. 3. To verify if the snapshot exists, run this command and check the Snapshot Name, Snapshot Created On, and Snapshot State:

    # vim-cmd vmsvc/snapshot.get [VMID]

    The output appears similar to:

    Get Snapshot:
    |-ROOT
    --Snapshot Name : Test
    --Snapshot Desciption :
    --Snapshot Created On : 7/27/2011 13:49:55
    --Snapshot State : powered on
  4. 4. Run this command to create a new snapshot:

    # vim-cmd vmsvc/snapshot.create [VmId] [snapshotName] [snapshotDescription] [includeMemory] [quiesced]

    For example, to create a snapshot on the virtual machine named testvm:

    # vim-cmd vmsvc/snapshot.create 3 snapshot1 snapshot 0 0

    Notes:

  1. 5. Run this command to remove all snapshots:

    # vim-cmd vmsvc/snapshot.removeall [VMID]

    Notes:
    • This command consolidates the snapshot information and deletes the snapshot -delta files from the relevant virtual machine folder.
    • If you have unsuccessfully attempted to consolidate using the Take a Snapshot then Delete all Snapshots technique, attempt the technique again, but this time select the Quiesce guest file system checkbox to enable quiesced snapshots when youTake a Snapshot.


If it is not working use the below option

Third  option


Cloning individual virtual machine disks via the ESX/ESXi host terminal

See the below vmware KB 1027876 article for more information

http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&externalId=1027876&sl...

Cloning a virtual machine disk with delta files or snapshots

To clone a virtual machine disk using the ESX/ESXi host terminal:

Use the esxcli vm process list to find out the vmx file location

Output like:

Examplevm

   World ID: 16551234

   Process ID: 0

VMX Cartel ID: 16551233

UUID: 42 21 e6 e3 57 45 f4 94-bf 07 42 2c 35 4e e5 8a

Display Name: examplevm

Config File: /vmfs/volumes/4f506dc0-b06c13b1-7212-0026b976cfdc/examplevm/examplevm.vmx

  1. 1. Log into the ESX/ESXi host's terminal.
  2. 2. Navigate to the virtual machine's directory using the cd command. It is located at:

    /vmfs/volumes/source_datastore/virtual_machine/
  3. 3. Confirm the destination directory where the clone will be copied to. Create this directory, if required.

    For example, if this destination directory does not exist:

    /vmfs/volumes/destination_datastore/virtual_machine/

    Create the directory using this command:

    mkdir /vmfs/volumes/destination_datastore/virtual_machine/


  4. 4. Clone the virtual hard disk from its current snapshot delta point using the vmkfstools -i command.

    # vmkfstools -i /vmfs/volumes/Storage1/examplevm/examplevm-000003.vmdk /vmfs/volumes/Storage2/examplevm_clone.vmdk

Notes:

  • Encapsulate objects with quotes where appropriate to ensure spaces and other special characters are interpreted correctly.
  • By default, the vmkfstools -i command will create a destination thick provisioned disk. If you would like the destination disk to be thin provisioned, add -d thin to the end of the above command.

    Example:

    # vmkfstools -i /vmfs/volumes/Storage1/examplevm/examplevm.vmdk /vmfs/volumes/Storage2/examplevm_clone.vmdk -d thin


The output appears similar to:

Destination disk format: VMFS thick
Cloning disk '/vmfs/volumes/Storage1 (3)/examplevm/examplevm-000003.vmdk'...
Clone: 100% done.


Note: If the process fails, try selecting the next snapshot delta point in the snapshot tree. The selected point or one of its parents may be corrupt.


Thanks&regards,

Thulasee

0 Kudos
a_p_
Leadership
Leadership

@


you are welcome to reply to any discussion, but please take the time to read through the replies to fully understand the issue before replying. The pasted KB article may work, but without any free disk space as in this case, thing could even become worse with this approach.


André

0 Kudos