VMware Cloud Community
malefik
Enthusiast
Enthusiast

vSAN 6.6 free space gone! Why? Where?

Hello everybody!

- vSphere VCSA 6.5.0.20000 Build 9451637

- VMware ESXi, 6.5.0, 8935087

- vSAN 6.6

Strange situation. We had 9Tb of free space, but suddenly it was instantly reduced to 2.4TB.

At the same time, the "other" elements take up a lot of space (screenshot).

This happened at a time when backup operations (Veeam Backup & Replication) and virtual machine pool were updated on VMware View 7.4 (linked clones) at the same time.

- I checked the snapshots (in the properties of the virtual machines vSphere) - they are not

- Removed View virtual machines and created them anew

But there is still no free space.

What to do? Any ideas?

pastedImage_0.png

0 Kudos
4 Replies
TheBobkin
Champion
Champion

Hello malefik​,

Potentially your back-up/update/whatever was done created/leftover a load of un-needed/duplicate/failed state Objects (either snapshots or clones) - start looking at what this 'Other' data is composed of - pretty much anything not associated with a VM presently will show as Unassociated/Other.

RVC is a good start for this e.g. vsan.obj_status_report -t and then vsan.object_info <UUIDofUnassociatedObject>.

https://www.virten.net/2017/06/vsan-6-6-rvc-guide-part-3-object-management/

Bob

0 Kudos
malefik
Enthusiast
Enthusiast

Hello, Bob. Thanks for the help!

Yes, I see a lot of objects. Too much to work with each separately. Can I somehow automate the process of deleting these objects?

And also: how can I completely clear the vSAN datastore from the data (format)? its possible easy and fast?

0 Kudos
TheBobkin
Champion
Champion

Hello malefik​,

"Yes, I see a lot of objects."

What do they consist of? e.g. snapshots, clones, VMs/vmdks that were scheduled for removal but not removed? Do they reside in healthy accessible namespace folders (and if so, are they the only things in these folders)? If they are then deletion via the GUI (Datastore browser) or PowerCLI could be a cleanup option that shouldn't take a huge amount of time. Are they all VMs that View has recreated? I vaguely recall at some point seeing some method of helping View to clean them up from that side.

"Can I somehow automate the process of deleting these objects?"

If they are healthy and accessible then no, not really - if you can make a (formatted) list it is fairly trivial to make a script to do this but scripting deletion of anything (on vSAN or otherwise) is not a great idea (as there is no 'undo' button). If one were to go this path then do it explicitly (e.g. not using a 'for' loop) with a line for each deletion and obviously one should only be working against this list once they are 100% certain all Objects are safe to delete.

"And also: how can I completely clear the vSAN datastore from the data (format)? its possible easy and fast?"

This depends on the data and your external storage availability/capabilities (another vSAN or SAN/NAS); e.g. a cluster full of non-static applications, file-servers and databases would likely need to be fully moved off and back or backed-up + restored, while on the opposite end of the spectrum, a VDI cluster with temporary instant-clones with no/little persistent data may just be a case of moving off the Golden-images, any persistent customer data, wiping down the cluster and re-provisioning the pools.

Bob

0 Kudos
GreatWhiteTec
VMware Employee
VMware Employee

Hi Malefik,

One other thing I want to point out is your swap space. You are currently using close to 2TB of space. If you are not overcommitting memory in the cluster, you will most likely not use this space. For versions prior to vSAN 6.7, the swap space was provisioned as thick. Starting in 6.7 this space is now thin by default. There are many ways to disable this and get that space back. Refer to my blog post below for further info and steps.

As you may know, the swap file is created when VMs boot up, and destrioyed when they are shut down. If you disable the thickProvision option, those VMs will need to reboot for the file to be deleted, and not recreated.

https://greatwhitetec.com/2017/03/20/vsan-sparse-swap/  

0 Kudos