VMware Cloud Community
reecefroggatt
Contributor
Contributor

Snapshot Consolidation filled the datastore up.

Hi there,

I'm looking for a bit of advice. We're a SMB with 2 ESXi hosts and 5 virtual machines. The last few days, we've had loads of issues with our entire network coming to a standstill. We eventually found this to be excessive disk usage by one of the VM's, it seemed as if ESXi was trying to consolidate snapshots automatically - the issue being, there is no snapshots. Anyway, we kept cancelling the task as it was bringing our network to a halt, and so I decided to force consolidate it Saturday morning and let it continue over the weekend.

I've checked on it today, and it has stopped as the data-store is at 100%, ESXi has also shut down some VM's. Now, before we started the consolidation, there was no snapshots in snapshot manager and datastore was at 42%, It's a 1TB disk.

The VMDK file for the VM has jumped from around 200GB to 600GB+.

I'm really stuck as I don't know how to reduce this and get the VM back online, it's our SQL machine and we cant run business without it, unfortunately our last Veeam backup was last week so can't really restore that to another host without loosing a week's data.

Any ideas?

Kindest regards,
Reece.

Reply
0 Kudos
29 Replies
a_p_
Leadership
Leadership

To see what can be done, please provide some information:

  • run ls -lisa > filelist.txt in the VM's folder (from the command line), download filelist.txt, and attach it to your next reply
  • are there other datastores? If yes what's their free disk space?

André

Reply
0 Kudos
continuum
Immortal
Immortal

To find out wether a VM uses one or more snapshots looking up snapshotmanager is not the appropriate procedure.

Instead look up which vmdks are used by reading the vmx-file.

If you want advice provide a copy of the vmx-file or the latest vmware.log and a list of all vmdk-files in the VM-directory.


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

Reply
0 Kudos
reecefroggatt
Contributor
Contributor

Hi,

Thanks for the reply. Which files do you require?

Cheers.

1.PNG2.PNG

Reply
0 Kudos
reecefroggatt
Contributor
Contributor

Hi,

Unfortunately, on this host there is only one data-store.

How do I find the folder on the datastore through SSH?

[root@hyp01-che-dc:/etc] ls

SHAC_Config.ini                  krb5.conf                        protocols

X11                              krb5.keytab                      random-seed

banner                           krb5.mkey                        rc.local

chkconfig.db                     krb5.realms                      rc.local.d

config                           likewise                         resolv.conf

dhclient-vmk0.conf               localtime                        security

dhclient-vmk0.leases             motd                             services

dhclient6-vmk0.conf              nscd.conf                        sfcb

dhclient6-vmk0.leases            nsswitch.conf                    shadow

eToken.conf                      ntp.conf                         shells

environment                      ntp.drift                        shutdown.d

group                            ntp.keys                         slp.reg

host.conf                        openwsman                        ssh

hosts                            opt                              vSphereProvisioning-resolv.conf

init.d                           pam.d                            vmotion-resolv.conf

inittab                          passwd                           vmsyslog.conf

issue                            passwdqc.conf                    vmsyslog.conf.d

keymap                           profile                          vmware

krb5.acl                         profile.local                    vmware-tools

Reply
0 Kudos
a_p_
Leadership
Leadership

The VM's files are located at /vmfs/volumes/<datastore-name>/<vm-name>

André

Reply
0 Kudos
continuum
Immortal
Immortal

We need to know the size of all vmdks.

Next we need to know how much free space is left on that datastore.

Do you have any other datastores ? - if yes - how much free space do you have there ?

Attach the vmx-file and the vmware.log to your next reply


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

Reply
0 Kudos
reecefroggatt
Contributor
Contributor

Hi,


Thanks for the reply.

I have attached the VMX and log file below.

There is only 1 datastore on this host, all the other VM's have been moved to the second host. This VM is only 250gb ish and yet the datastore is still full. I can't quite figure it out.

3.PNG

Reply
0 Kudos
reecefroggatt
Contributor
Contributor

Hi,

I can't see a filelist.txt in that directory?

[root@hyp01-che-dc:~] cd /vmfs/volumes

[root@hyp01-che-dc:/vmfs/volumes] ls

5bb76f75-2f24304d-b7ef-509a4c829407  5d715663-f207b463-27e4-509a4c829407  datastore1

5d71565c-f336d546-c181-509a4c829407  6d1d5c76-144b694a-645e-e203c19054d2  faa6557f-21694609-68d9-86d274a9def2

[root@hyp01-che-dc:/vmfs/volumes] cd datastore1

[root@hyp01-che-dc:/vmfs/volumes/5bb76f75-2f24304d-b7ef-509a4c829407] ls

BD-VOSTRO-001     Installation ISO  bd-dc-svr         dc1-che-dc        vmkdump

[root@hyp01-che-dc:/vmfs/volumes/5bb76f75-2f24304d-b7ef-509a4c829407] cd bd-vostro-001

-sh: cd: can't cd to bd-vostro-001

[root@hyp01-che-dc:/vmfs/volumes/5bb76f75-2f24304d-b7ef-509a4c829407] cd BD-VOSTRO-001

[root@hyp01-che-dc:/vmfs/volumes/5bb76f75-2f24304d-b7ef-509a4c829407/BD-VOSTRO-001] ls

BD-VOSTRO-001-000001-ctk.vmdk         BD-VOSTRO-001.vmxf                    vmware-10.log

BD-VOSTRO-001-000001-sesparse.vmdk    BD-VOSTRO-001_1-000001-ctk.vmdk       vmware-11.log

BD-VOSTRO-001-ctk.vmdk                BD-VOSTRO-001_1-000001-sesparse.vmdk  vmware-12.log

BD-VOSTRO-001-eb67ff80.hlog           BD-VOSTRO-001_1-ctk.vmdk              vmware-13.log

BD-VOSTRO-001-flat.vmdk               BD-VOSTRO-001_1-flat.vmdk             vmware-14.log

BD-VOSTRO-001.nvram                   BD-VOSTRO-001_2-000001-ctk.vmdk       vmware-9.log

BD-VOSTRO-001.vmdk                    BD-VOSTRO-001_2-000001-sesparse.vmdk  vmware.log

BD-VOSTRO-001.vmsd                    BD-VOSTRO-001_2-ctk.vmdk              vmx-BD-VOSTRO-001-2642021814-1.vswp

[root@hyp01-che-dc:/vmfs/volumes/5bb76f75-2f24304d-b7ef-509a4c829407/BD-VOSTRO-001]

Kind regards,

Reece.

Reply
0 Kudos
a_p_
Leadership
Leadership

The output doesn't contain the required information.

Please run ls -lisa, which also lists sizes, time stamps, ...

If possible redirect the output into a file as mentioned in my previous post, and attach (don't paste) the file to a reply.

André

Reply
0 Kudos
reecefroggatt
Contributor
Contributor

Hi there,

Please see below.

[root@hyp01-che-dc:/vmfs/volumes/5bb76f75-2f24304d-b7ef-509a4c829407/BD-VOSTRO-001] ls -lisa

total 850851264

   3268    128 drwxr-xr-x    1 root     root         81920 Sep 29 16:51 .

      4   1024 drwxr-xr-t    1 root     root         77824 Sep 29 17:00 ..

92276612   8192 -rw-------    1 root     root       7631424 Sep 29 15:26 BD-VOSTRO-001-000001-ctk.vmdk

71305092 14133248 -rw-------    1 root     root     18557153280 Sep 29 15:26 BD-VOSTRO-001-000001-sesparse.vmdk

58722180   8192 -rw-------    1 root     root       7631424 Sep 28 22:59 BD-VOSTRO-001-ctk.vmdk

8390532      0 -rw-r--r--    1 root     root            53 Sep 24 12:56 BD-VOSTRO-001-eb67ff80.hlog

16779140 718387200 -rw-------    1 root     root     2000397795328 Sep 29 15:26 BD-VOSTRO-001-flat.vmdk

79693700     64 -rw-------    1 root     root          8684 Sep 28 22:58 BD-VOSTRO-001.nvram

20973444      0 -rw-------    1 root     root           602 Sep 28 22:58 BD-VOSTRO-001.vmdk

12584836      0 -rw-------    1 root     root            78 Sep 25 14:30 BD-VOSTRO-001.vmsd

88082308     64 -rwx------    1 root     root          4328 Sep 28 22:58 BD-VOSTRO-001.vmx

83888004     64 -rw-------    1 root     root          3184 Oct  9  2018 BD-VOSTRO-001.vmxf

104859524   2048 -rw-------    1 root     root       1954688 Sep 28 22:58 BD-VOSTRO-001_1-000001-ctk.vmdk

96470916 304128 -rw-------    1 root     root     375676928 Sep 28 22:58 BD-VOSTRO-001_1-000001-sesparse.vmdk

54527876   2048 -rw-------    1 root     root       1954688 Sep 24 21:09 BD-VOSTRO-001_1-ctk.vmdk

29362052  83968 -rw-------    1 root     root     32017219584 Sep 21 12:32 BD-VOSTRO-001_1-flat.vmdk

117442436   8192 -rw-------    1 root     root       7631744 Sep 28 22:58 BD-VOSTRO-001_2-000001-ctk.vmdk

109053828 10382336 -rw-------    1 root     root     11301359616 Sep 28 22:58 BD-VOSTRO-001_2-000001-sesparse.vmdk

25167748   7168 -rw-------    1 root     root       7631744 Sep 24 21:09 BD-VOSTRO-001_2-ctk.vmdk

41944964 107402240 -rw-------    1 root     root     250060210176 Sep 24 13:12 BD-VOSTRO-001_2-flat.vmdk

50333572   2048 -rw-------    1 root     root       1239848 Sep 27 17:37 vmware-10.log

121636740   1024 -rw-------    1 root     root        392174 Sep 28 14:21 vmware-11.log

125831044   1024 -rw-------    1 root     root        657101 Sep 28 22:49 vmware-12.log

130025348   1024 -rw-------    1 root     root        316463 Sep 28 22:58 vmware-13.log

62916484     64 -rw-------    1 root     root         43870 Sep 29 16:24 vmware-14.log

264243076   3072 -rw-------    1 root     root       2159308 Sep 24 12:56 vmware-9.log

67110788     64 -rw-------    1 root     root         43786 Sep 29 16:51 vmware.log

218105732 112640 -rw-------    1 root     root     115343360 May 23 12:26 vmx-BD-VOSTRO-001-2642021814-1.vswp

Reply
0 Kudos
reecefroggatt
Contributor
Contributor

Output as attachment.

Reply
0 Kudos
continuum
Immortal
Immortal

In the current state there is not much you can do.

Can you add a SATA disk as additional storage ?

Do you have the time to move one part of the snapshot chain to another host ?


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

Reply
0 Kudos
a_p_
Leadership
Leadership

In order to be able to delete the snapshots (i.e. consolidate the virtual disks) we need up to ~25GB (+ some overhead) free disk space.

Is it possible to cleanly shutdown the other VMs on the host? This would free up disk space, that's currently used by swap files.

Once powered off, refresh the storage usage, to ensure that you have at least 35GB free disk space.

If this is the case, open the Snapshot Manager for this VM, and run "Delete All". If this option is grayed out, create another temporary snapshot to enable it.

Questions:

  • Do you have other VM's with snapshots?
  • Why did you provision the VM with ~2.5TB disk space, if you only have a 1 TB physical disk? That's really dangerous.

André

Reply
0 Kudos
continuum
Immortal
Immortal

Andre - do you think it is safe to run delete_all while the VM has all 3 disks attached ?


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

Reply
0 Kudos
reecefroggatt
Contributor
Contributor

Hi both,

Thank you for your advice.

I need to get the VM started so I can make an emergency backup of the data before continuing.

I have moved the other VM's from the host and now have 58GB available, however I can't start the VM.

I am getting the following error.

Power On VM

Key

haTask-5-vim.VirtualMachine.powerOn-2775414741

Description

Power On this virtual machine

Virtual machine

bd-vostro-001

State

Failed - File BD-VOSTRO-001_2-000001.vmdk was not found

Errors

    0" class="" style="box-sizing: border-box; padding-left: 0px; margin-bottom: 10px; margin-left: 25px;">
  • VMware ESX cannot find the virtual disk "BD-VOSTRO-001_2-000001.vmdk". Verify the path is valid and try again.
  • File system specific implementation of MakeOID[file] failed
  • File system specific implementation of MakeOID[file] failed
  • The system cannot find the file specified
  • Cannot open the disk 'BD-VOSTRO-001_2-000001.vmdk' or one of the snapshot disks it depends on.
  • Module 'Disk' power on failed.
  • VMware ESX cannot find the virtual disk "BD-VOSTRO-001_1-000001.vmdk". Verify the path is valid and try again.
  • File system specific implementation of MakeOID[file] failed
  • File system specific implementation of MakeOID[file] failed
  • The system cannot find the file specified
  • Cannot open the disk 'BD-VOSTRO-001_1-000001.vmdk' or one of the snapshot disks it depends on.
  • VMware ESX cannot find the virtual disk "BD-VOSTRO-001-000001.vmdk". Verify the path is valid and try again.
  • File system specific implementation of MakeOID[file] failed
  • File system specific implementation of MakeOID[file] failed
  • The system cannot find the file specified
  • Cannot open the disk 'BD-VOSTRO-001-000001.vmdk' or one of the snapshot disks it depends on.
  • Failed to start the virtual machine.

Do you have any advice please? The contents of the datastore is:

BD-VOSTRO-001-000001-ctk.vmdk         BD-VOSTRO-001.vmsd                    BD-VOSTRO-001_2-000001-ctk.vmdk       vmware-18.log

BD-VOSTRO-001-000001-sesparse.vmdk    BD-VOSTRO-001.vmx                     BD-VOSTRO-001_2-000001-sesparse.vmdk  vmware-19.log

BD-VOSTRO-001-ctk.vmdk                BD-VOSTRO-001.vmxf                    BD-VOSTRO-001_2-ctk.vmdk              vmware-20.log

BD-VOSTRO-001-eb67ff80.hlog           BD-VOSTRO-001_1-000001-ctk.vmdk       BD-VOSTRO-001_2-flat.vmdk             vmware.log

BD-VOSTRO-001-flat.vmdk               BD-VOSTRO-001_1-000001-sesparse.vmdk  vmware-15.log                         vmx-BD-VOSTRO-001-2642021814-1.vswp

BD-VOSTRO-001.nvram                   BD-VOSTRO-001_1-ctk.vmdk              vmware-16.log

BD-VOSTRO-001.vmdk                    BD-VOSTRO-001_1-flat.vmdk             vmware-17.log

Kind regards,
Reece.

Reply
0 Kudos
continuum
Immortal
Immortal

Install WinSCP and show a screenshot of the directory.

Looks like the descriptor vmdks are gone.


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

Reply
0 Kudos
reecefroggatt
Contributor
Contributor

Hi,

Thanks for the reply. I really appreciate it.

Please see the below screenshot.

4.PNG

Kind regards.

Reply
0 Kudos
continuum
Immortal
Immortal

We want to see a screenshot because we want to see the full file names - now look at your screenshot ....

Try again


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

Reply
0 Kudos
continuum
Immortal
Immortal

Ignore the ctk files for now.

Which of this files is missing ?

BD-VOSTRO-001-000001.vmdk        

BD-VOSTRO-001-000001-sesparse.vmdk   

BD-VOSTRO-001-flat.vmdk              

BD-VOSTRO-001.vmdk  

               

BD-VOSTRO-001_1-000001.vmdk

BD-VOSTRO-001_1-000001-sesparse.vmdk

BD-VOSTRO-001_1.vmdk

BD-VOSTRO-001_1-flat.vmdk

BD-VOSTRO-001_2-000001.vmdk      

BD-VOSTRO-001_2-000001-sesparse.vmdk   

BD-VOSTRO-001_2-flat.vmdk   

BD-VOSTRO-001_2.vmdk 


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

Reply
0 Kudos