4 Replies Latest reply on Oct 15, 2018 5:56 AM by Great_White_Tec

    vSAN 6.6 free space gone! Why? Where?

    malefik Enthusiast

      Hello everybody!

      - vSphere VCSA 6.5.0.20000 Build 9451637

      - VMware ESXi, 6.5.0, 8935087

      - vSAN 6.6

       

      Strange situation. We had 9Tb of free space, but suddenly it was instantly reduced to 2.4TB.

      At the same time, the "other" elements take up a lot of space (screenshot).

      This happened at a time when backup operations (Veeam Backup & Replication) and virtual machine pool were updated on VMware View 7.4 (linked clones) at the same time.

      - I checked the snapshots (in the properties of the virtual machines vSphere) - they are not

      - Removed View virtual machines and created them anew

      But there is still no free space.

       

      What to do? Any ideas?

       

        • 1. Re: vSAN 6.6 free space gone! Why? Where?
          TheBobkin Virtuoso
          VMware EmployeesvExpert

          Hello malefik,

           

           

          Potentially your back-up/update/whatever was done created/leftover a load of un-needed/duplicate/failed state Objects (either snapshots or clones) - start looking at what this 'Other' data is composed of - pretty much anything not associated with a VM presently will show as Unassociated/Other.

          RVC is a good start for this e.g. vsan.obj_status_report -t and then vsan.object_info <UUIDofUnassociatedObject>.

          https://www.virten.net/2017/06/vsan-6-6-rvc-guide-part-3-object-management/

           

           

          Bob

          • 2. Re: vSAN 6.6 free space gone! Why? Where?
            malefik Enthusiast

            Hello, Bob. Thanks for the help!

            Yes, I see a lot of objects. Too much to work with each separately. Can I somehow automate the process of deleting these objects?

            And also: how can I completely clear the vSAN datastore from the data (format)? its possible easy and fast?

            • 3. Re: vSAN 6.6 free space gone! Why? Where?
              TheBobkin Virtuoso
              VMware EmployeesvExpert

              Hello malefik,

               

               

              "Yes, I see a lot of objects."

              What do they consist of? e.g. snapshots, clones, VMs/vmdks that were scheduled for removal but not removed? Do they reside in healthy accessible namespace folders (and if so, are they the only things in these folders)? If they are then deletion via the GUI (Datastore browser) or PowerCLI could be a cleanup option that shouldn't take a huge amount of time. Are they all VMs that View has recreated? I vaguely recall at some point seeing some method of helping View to clean them up from that side.

               

              "Can I somehow automate the process of deleting these objects?"

              If they are healthy and accessible then no, not really - if you can make a (formatted) list it is fairly trivial to make a script to do this but scripting deletion of anything (on vSAN or otherwise) is not a great idea (as there is no 'undo' button). If one were to go this path then do it explicitly (e.g. not using a 'for' loop) with a line for each deletion and obviously one should only be working against this list once they are 100% certain all Objects are safe to delete.

               

              "And also: how can I completely clear the vSAN datastore from the data (format)? its possible easy and fast?"

              This depends on the data and your external storage availability/capabilities (another vSAN or SAN/NAS); e.g. a cluster full of non-static applications, file-servers and databases would likely need to be fully moved off and back or backed-up + restored, while on the opposite end of the spectrum, a VDI cluster with temporary instant-clones with no/little persistent data may just be a case of moving off the Golden-images, any persistent customer data, wiping down the cluster and re-provisioning the pools.

               

               

              Bob

              • 4. Re: vSAN 6.6 free space gone! Why? Where?
                Great_White_Tec Expert
                VMware EmployeesvExpert

                Hi Malefik,

                 

                One other thing I want to point out is your swap space. You are currently using close to 2TB of space. If you are not overcommitting memory in the cluster, you will most likely not use this space. For versions prior to vSAN 6.7, the swap space was provisioned as thick. Starting in 6.7 this space is now thin by default. There are many ways to disable this and get that space back. Refer to my blog post below for further info and steps.

                 

                As you may know, the swap file is created when VMs boot up, and destrioyed when they are shut down. If you disable the thickProvision option, those VMs will need to reboot for the file to be deleted, and not recreated.

                 

                https://greatwhitetec.com/2017/03/20/vsan-sparse-swap/