Hi there,
I'm looking for a bit of advice. We're a SMB with 2 ESXi hosts and 5 virtual machines. The last few days, we've had loads of issues with our entire network coming to a standstill. We eventually found this to be excessive disk usage by one of the VM's, it seemed as if ESXi was trying to consolidate snapshots automatically - the issue being, there is no snapshots. Anyway, we kept cancelling the task as it was bringing our network to a halt, and so I decided to force consolidate it Saturday morning and let it continue over the weekend.
I've checked on it today, and it has stopped as the data-store is at 100%, ESXi has also shut down some VM's. Now, before we started the consolidation, there was no snapshots in snapshot manager and datastore was at 42%, It's a 1TB disk.
The VMDK file for the VM has jumped from around 200GB to 600GB+.
I'm really stuck as I don't know how to reduce this and get the VM back online, it's our SQL machine and we cant run business without it, unfortunately our last Veeam backup was last week so can't really restore that to another host without loosing a week's data.
Any ideas?
Kindest regards,
Reece.
To see what can be done, please provide some information:
André
To find out wether a VM uses one or more snapshots looking up snapshotmanager is not the appropriate procedure.
Instead look up which vmdks are used by reading the vmx-file.
If you want advice provide a copy of the vmx-file or the latest vmware.log and a list of all vmdk-files in the VM-directory.
Hi,
Thanks for the reply. Which files do you require?
Cheers.
Hi,
Unfortunately, on this host there is only one data-store.
How do I find the folder on the datastore through SSH?
[root@hyp01-che-dc:/etc] ls
SHAC_Config.ini krb5.conf protocols
X11 krb5.keytab random-seed
banner krb5.mkey rc.local
chkconfig.db krb5.realms rc.local.d
config likewise resolv.conf
dhclient-vmk0.conf localtime security
dhclient-vmk0.leases motd services
dhclient6-vmk0.conf nscd.conf sfcb
dhclient6-vmk0.leases nsswitch.conf shadow
eToken.conf ntp.conf shells
environment ntp.drift shutdown.d
group ntp.keys slp.reg
host.conf openwsman ssh
hosts opt vSphereProvisioning-resolv.conf
init.d pam.d vmotion-resolv.conf
inittab passwd vmsyslog.conf
issue passwdqc.conf vmsyslog.conf.d
keymap profile vmware
krb5.acl profile.local vmware-tools
The VM's files are located at /vmfs/volumes/<datastore-name>/<vm-name>
André
We need to know the size of all vmdks.
Next we need to know how much free space is left on that datastore.
Do you have any other datastores ? - if yes - how much free space do you have there ?
Attach the vmx-file and the vmware.log to your next reply
Hi,
I can't see a filelist.txt in that directory?
[root@hyp01-che-dc:~] cd /vmfs/volumes
[root@hyp01-che-dc:/vmfs/volumes] ls
5bb76f75-2f24304d-b7ef-509a4c829407 5d715663-f207b463-27e4-509a4c829407 datastore1
5d71565c-f336d546-c181-509a4c829407 6d1d5c76-144b694a-645e-e203c19054d2 faa6557f-21694609-68d9-86d274a9def2
[root@hyp01-che-dc:/vmfs/volumes] cd datastore1
[root@hyp01-che-dc:/vmfs/volumes/5bb76f75-2f24304d-b7ef-509a4c829407] ls
BD-VOSTRO-001 Installation ISO bd-dc-svr dc1-che-dc vmkdump
[root@hyp01-che-dc:/vmfs/volumes/5bb76f75-2f24304d-b7ef-509a4c829407] cd bd-vostro-001
-sh: cd: can't cd to bd-vostro-001
[root@hyp01-che-dc:/vmfs/volumes/5bb76f75-2f24304d-b7ef-509a4c829407] cd BD-VOSTRO-001
[root@hyp01-che-dc:/vmfs/volumes/5bb76f75-2f24304d-b7ef-509a4c829407/BD-VOSTRO-001] ls
BD-VOSTRO-001-000001-ctk.vmdk BD-VOSTRO-001.vmxf vmware-10.log
BD-VOSTRO-001-000001-sesparse.vmdk BD-VOSTRO-001_1-000001-ctk.vmdk vmware-11.log
BD-VOSTRO-001-ctk.vmdk BD-VOSTRO-001_1-000001-sesparse.vmdk vmware-12.log
BD-VOSTRO-001-eb67ff80.hlog BD-VOSTRO-001_1-ctk.vmdk vmware-13.log
BD-VOSTRO-001-flat.vmdk BD-VOSTRO-001_1-flat.vmdk vmware-14.log
BD-VOSTRO-001.nvram BD-VOSTRO-001_2-000001-ctk.vmdk vmware-9.log
BD-VOSTRO-001.vmdk BD-VOSTRO-001_2-000001-sesparse.vmdk vmware.log
BD-VOSTRO-001.vmsd BD-VOSTRO-001_2-ctk.vmdk vmx-BD-VOSTRO-001-2642021814-1.vswp
[root@hyp01-che-dc:/vmfs/volumes/5bb76f75-2f24304d-b7ef-509a4c829407/BD-VOSTRO-001]
Kind regards,
Reece.
The output doesn't contain the required information.
Please run ls -lisa, which also lists sizes, time stamps, ...
If possible redirect the output into a file as mentioned in my previous post, and attach (don't paste) the file to a reply.
André
Hi there,
Please see below.
[root@hyp01-che-dc:/vmfs/volumes/5bb76f75-2f24304d-b7ef-509a4c829407/BD-VOSTRO-001] ls -lisa
total 850851264
3268 128 drwxr-xr-x 1 root root 81920 Sep 29 16:51 .
4 1024 drwxr-xr-t 1 root root 77824 Sep 29 17:00 ..
92276612 8192 -rw------- 1 root root 7631424 Sep 29 15:26 BD-VOSTRO-001-000001-ctk.vmdk
71305092 14133248 -rw------- 1 root root 18557153280 Sep 29 15:26 BD-VOSTRO-001-000001-sesparse.vmdk
58722180 8192 -rw------- 1 root root 7631424 Sep 28 22:59 BD-VOSTRO-001-ctk.vmdk
8390532 0 -rw-r--r-- 1 root root 53 Sep 24 12:56 BD-VOSTRO-001-eb67ff80.hlog
16779140 718387200 -rw------- 1 root root 2000397795328 Sep 29 15:26 BD-VOSTRO-001-flat.vmdk
79693700 64 -rw------- 1 root root 8684 Sep 28 22:58 BD-VOSTRO-001.nvram
20973444 0 -rw------- 1 root root 602 Sep 28 22:58 BD-VOSTRO-001.vmdk
12584836 0 -rw------- 1 root root 78 Sep 25 14:30 BD-VOSTRO-001.vmsd
88082308 64 -rwx------ 1 root root 4328 Sep 28 22:58 BD-VOSTRO-001.vmx
83888004 64 -rw------- 1 root root 3184 Oct 9 2018 BD-VOSTRO-001.vmxf
104859524 2048 -rw------- 1 root root 1954688 Sep 28 22:58 BD-VOSTRO-001_1-000001-ctk.vmdk
96470916 304128 -rw------- 1 root root 375676928 Sep 28 22:58 BD-VOSTRO-001_1-000001-sesparse.vmdk
54527876 2048 -rw------- 1 root root 1954688 Sep 24 21:09 BD-VOSTRO-001_1-ctk.vmdk
29362052 83968 -rw------- 1 root root 32017219584 Sep 21 12:32 BD-VOSTRO-001_1-flat.vmdk
117442436 8192 -rw------- 1 root root 7631744 Sep 28 22:58 BD-VOSTRO-001_2-000001-ctk.vmdk
109053828 10382336 -rw------- 1 root root 11301359616 Sep 28 22:58 BD-VOSTRO-001_2-000001-sesparse.vmdk
25167748 7168 -rw------- 1 root root 7631744 Sep 24 21:09 BD-VOSTRO-001_2-ctk.vmdk
41944964 107402240 -rw------- 1 root root 250060210176 Sep 24 13:12 BD-VOSTRO-001_2-flat.vmdk
50333572 2048 -rw------- 1 root root 1239848 Sep 27 17:37 vmware-10.log
121636740 1024 -rw------- 1 root root 392174 Sep 28 14:21 vmware-11.log
125831044 1024 -rw------- 1 root root 657101 Sep 28 22:49 vmware-12.log
130025348 1024 -rw------- 1 root root 316463 Sep 28 22:58 vmware-13.log
62916484 64 -rw------- 1 root root 43870 Sep 29 16:24 vmware-14.log
264243076 3072 -rw------- 1 root root 2159308 Sep 24 12:56 vmware-9.log
67110788 64 -rw------- 1 root root 43786 Sep 29 16:51 vmware.log
218105732 112640 -rw------- 1 root root 115343360 May 23 12:26 vmx-BD-VOSTRO-001-2642021814-1.vswp
In the current state there is not much you can do.
Can you add a SATA disk as additional storage ?
Do you have the time to move one part of the snapshot chain to another host ?
In order to be able to delete the snapshots (i.e. consolidate the virtual disks) we need up to ~25GB (+ some overhead) free disk space.
Is it possible to cleanly shutdown the other VMs on the host? This would free up disk space, that's currently used by swap files.
Once powered off, refresh the storage usage, to ensure that you have at least 35GB free disk space.
If this is the case, open the Snapshot Manager for this VM, and run "Delete All". If this option is grayed out, create another temporary snapshot to enable it.
Questions:
André
Andre - do you think it is safe to run delete_all while the VM has all 3 disks attached ?
Hi both,
Thank you for your advice.
I need to get the VM started so I can make an emergency backup of the data before continuing.
I have moved the other VM's from the host and now have 58GB available, however I can't start the VM.
I am getting the following error.
Key
haTask-5-vim.VirtualMachine.powerOn-2775414741
Description
Power On this virtual machine
Virtual machine
State
Failed - File BD-VOSTRO-001_2-000001.vmdk was not found
Errors
Do you have any advice please? The contents of the datastore is:
BD-VOSTRO-001-000001-ctk.vmdk BD-VOSTRO-001.vmsd BD-VOSTRO-001_2-000001-ctk.vmdk vmware-18.log
BD-VOSTRO-001-000001-sesparse.vmdk BD-VOSTRO-001.vmx BD-VOSTRO-001_2-000001-sesparse.vmdk vmware-19.log
BD-VOSTRO-001-ctk.vmdk BD-VOSTRO-001.vmxf BD-VOSTRO-001_2-ctk.vmdk vmware-20.log
BD-VOSTRO-001-eb67ff80.hlog BD-VOSTRO-001_1-000001-ctk.vmdk BD-VOSTRO-001_2-flat.vmdk vmware.log
BD-VOSTRO-001-flat.vmdk BD-VOSTRO-001_1-000001-sesparse.vmdk vmware-15.log vmx-BD-VOSTRO-001-2642021814-1.vswp
BD-VOSTRO-001.nvram BD-VOSTRO-001_1-ctk.vmdk vmware-16.log
BD-VOSTRO-001.vmdk BD-VOSTRO-001_1-flat.vmdk vmware-17.log
Kind regards,
Reece.
Install WinSCP and show a screenshot of the directory.
Looks like the descriptor vmdks are gone.
Hi,
Thanks for the reply. I really appreciate it.
Please see the below screenshot.
Kind regards.
We want to see a screenshot because we want to see the full file names - now look at your screenshot ....
Try again
Ignore the ctk files for now.
Which of this files is missing ?
BD-VOSTRO-001-000001.vmdk
BD-VOSTRO-001-000001-sesparse.vmdk
BD-VOSTRO-001-flat.vmdk
BD-VOSTRO-001.vmdk
BD-VOSTRO-001_1-000001.vmdk
BD-VOSTRO-001_1-000001-sesparse.vmdk
BD-VOSTRO-001_1.vmdk
BD-VOSTRO-001_1-flat.vmdk
BD-VOSTRO-001_2-000001.vmdk
BD-VOSTRO-001_2-000001-sesparse.vmdk
BD-VOSTRO-001_2-flat.vmdk
BD-VOSTRO-001_2.vmdk