VMware Cloud Community
alienAK
Contributor
Contributor
Jump to solution

'msg.hbacommon.outofspace:There is no more space for virtual disk 'XXXX.vmdk'.

Hey,

I'm running ESXI 6.5 and every few hours i get the error 'msg.hbacommon.outofspace:There is no more space for virtual disk 'XXXX.vmdk' and the VM stops. I can get the VM back up and running again by deleting a snapshot but the issue just happens again a few hours later.

Looking at the data store it shows nearly full but only about 700GB of the 1TB is provisioned. If i check the disk uage on the VM it is taking up no where near the amount ESXI says it is. about 200GB. I only have a couple of snapshots left so dont think I will be able to get the VM back running.

The disks allocated to the VM are Thin provisioned and this host only has 2 VMs the one with the issue and another that is running pfsense.

Info about the VM.

2 vmfs connected, one for OS(50GB) and another for data(600GB). the data one is the one that is filling up.

The VM is running rclone and has a couple of gdrives connected but I do not see how the data drive is filling up as anytime I have checked it is over 50% unused.

Anyone have any idea how to resolve

1 Solution

Accepted Solutions
a_p_
Leadership
Leadership
Jump to solution

It's possible to mark a virtual disk as "independent-persistent" to exclude it from snapshots.

However, in order to do this you need to first consolidate/delete all the existing snapshots, which can't be done due to the low free disk space, and the thin provisioned virtual disks.

Do you have another disk/LUN to which the current virtual disks can be cloned (manually from the command line)?

Other than this you can backup/restore the VM (requires appropriate licensing, and a backup application), or export the VM to OVF/OVA, and import it again. Both options (backup/restore, and export/import) will consolidate the snapshots. However, these two options require that you delete the VM from disk between the backup/export, and the restore/import, so make sure that you select a reliable target.

In any case, downtime will be required.

Btw. what's the purpose of all the snapshots? Please remember that snapshots are no backups. If the disk/datastore goes south, you may loose everything.

André

View solution in original post

Reply
0 Kudos
4 Replies
a_p_
Leadership
Leadership
Jump to solution

Each snapshot can grow up to its base disk's provisioned size, so it's not very surprising that disk space gets exhausted with multiple snapshots, and a lot of changes.

To find out what can/should be done, please provide some information. To get the required information, temporarily enable SSH on the host to allow putty and WinSCP to connect to it.

The VM's folder is "/vmfs/volumes/datastore1/vu18"

  • connect to the host using putty, and run ls -lisa > filelist.txt in the VM's folder
  • use WinSCP, and download filelist.txt as well as all the descriptor .vmdk files (the ones without "ctk", "flat", "delta", or "sesparse" in their names)
  • compress/zip the downloaded files, and attach the .zip archive to a reply post

André

alienAK
Contributor
Contributor
Jump to solution

Thank you for your response.

I have attached the files you requested.

Is there some way to only take snapshots of the OS vmfs. Data on the data disk changes regularly so does not need snapshots. It sounds like it would also resolve the issue I am seeing.

Reply
0 Kudos
a_p_
Leadership
Leadership
Jump to solution

It's possible to mark a virtual disk as "independent-persistent" to exclude it from snapshots.

However, in order to do this you need to first consolidate/delete all the existing snapshots, which can't be done due to the low free disk space, and the thin provisioned virtual disks.

Do you have another disk/LUN to which the current virtual disks can be cloned (manually from the command line)?

Other than this you can backup/restore the VM (requires appropriate licensing, and a backup application), or export the VM to OVF/OVA, and import it again. Both options (backup/restore, and export/import) will consolidate the snapshots. However, these two options require that you delete the VM from disk between the backup/export, and the restore/import, so make sure that you select a reliable target.

In any case, downtime will be required.

Btw. what's the purpose of all the snapshots? Please remember that snapshots are no backups. If the disk/datastore goes south, you may loose everything.

André

Reply
0 Kudos
alienAK
Contributor
Contributor
Jump to solution

thank you

all sorted now.

Reply
0 Kudos