Clean a VSAN Datastore

[klinden]

I am running VSAN in a lab, and have noticed that the folders and files of VMs sometimes do not get deleted properly. Is there a VSAN command to check for unused objects on the datastore similar to View Composer replicas?

[jonretting]

If you are seeing inconsistencies in the file system  versus what you know has been deleted, that would be indicative of a greater issue with your VSAN. The only problems i experience like you describe in lab/production is with the Content Library being stored on the VSAN, or objects associated with VM Templates. Not being able to quickly bring templates under storage profiles is a separate issue. Thanks, -Jon

[klinden]

I have 144 objects that are unassociated in VSAN. I can see that the files with the vsan.object_status_report command, but how do you delete them?

localhost/lab/computers/cluster01/hosts> vsan.obj_status_report ~cluster -t

2015-08-26 19:08:11 +0000: Querying all VMs on VSAN ...

2015-08-26 19:08:11 +0000: Querying all objects in the system from esxi01.test.lab ...

2015-08-26 19:08:12 +0000: Querying all disks in the system from esxi01.test.lab ...

2015-08-26 19:08:13 +0000: Querying all components in the system from esxi01.test.lab ...

2015-08-26 19:08:14 +0000: Querying all object versions in the system ...

2015-08-26 19:08:16 +0000: Got all the info, computing table ...

Histogram of component health for non-orphaned objects

+-------------------------------------+------------------------------+

| Num Healthy Comps / Total Num Comps | Num objects with such status |

+-------------------------------------+------------------------------+

| 1/1 (OK)                            |  113                         |

| 3/3 (OK)                            |  31                          |

+-------------------------------------+------------------------------+

Total non-orphans: 144

Histogram of component health for possibly orphaned objects

+-------------------------------------+------------------------------+

| Num Healthy Comps / Total Num Comps | Num objects with such status |

+-------------------------------------+------------------------------+

+-------------------------------------+------------------------------+

Total orphans: 0

Total v1 objects: 0

Total v2 objects: 144

+-----------------------------------------+---------+---------------------------+

| VM/Object                               | objects | num healthy / total comps |

+-----------------------------------------+---------+---------------------------+

| Unassociated objects                    |         |                           |

|    4e149455-bf33-0002-3390-ecf4bbc3a0d8 |         | 1/1                       |

|    d1c91e55-31d8-9605-35d3-ecf4bbc3a0d8 |         | 1/1                       |

|    e1a55b55-c3e3-7406-52b6-ecf4bbc3a240 |         | 3/3                       |

|    bf6f8355-6ca3-0107-0ede-ecf4bbc3a038 |         | 1/1                       |

|    89c91e55-fb29-2808-6e8e-ecf4bbc3a0d8 |         | 1/1                       |

|    133b8855-bfde-4109-942e-ecf4bbc3a0d8 |         | 1/1                       |

|    e7c89f55-53fb-ea0b-01e8-ecf4bbc3a240 |         | 1/1                       |

|    82c91e55-bbeb-1c14-c7f0-ecf4bbc3a0d8 |         | 1/1                       |

|    0aca1e55-43cc-1415-9d0c-ecf4bbc3a0d8 |         | 1/1                       |

|    b44e8855-f42e-7c17-e8e6-ecf4bbc3a038 |         | 1/1                       |

|    688e9e55-5f97-2218-afed-ecf4bbc3a0d8 |         | 1/1                       |

|    b3c91e55-3367-5318-15d6-ecf4bbc3a0d8 |         | 1/1                       |

|    f26e8455-dcdb-6218-f316-ecf4bbc3a038 |         | 1/1                       |

|    38d92755-eb33-111a-ba1a-ecf4bbc3a240 |         | 1/1                       |

|    b7e72655-477f-121a-2907-ecf4bbc3a240 |         | 1/1                       |

|    0cca1e55-43f2-8121-0391-ecf4bbc3a0d8 |         | 1/1                       |

|    c1c91e55-1ba0-4c23-0ffc-ecf4bbc3a0d8 |         | 1/1                       |

|    0eca1e55-6326-c129-6cb2-ecf4bbc3a0d8 |         | 1/1                       |

|    0fcf9f55-d71d-4b2a-483d-ecf4bbc3a158 |         | 1/1                       |

|    b44e8855-1432-792a-a678-ecf4bbc3a038 |         | 1/1                       |

|    ae1e2d55-3353-912a-6901-ecf4bbc3a240 |         | 1/1                       |

|    a6608355-6c16-0a2b-1415-ecf4bbc3a038 |         | 1/1                       |

|    b4e72655-40a1-9f2e-88f7-ecf4bbc3a240 |         | 1/1                       |

|    cec91e55-db55-712f-08e1-ecf4bbc3a0d8 |         | 1/1                       |

|    80013955-3208-0230-eed6-ecf4bbc3a158 |         | 1/1                       |

|    b7628455-b43c-c130-8608-ecf4bbc3a038 |         | 1/1                       |

|    f37c4355-441f-0c31-5417-ecf4bbc3a038 |         | 3/3                       |

|    b4389555-94fb-9335-6508-ecf4bbc3a158 |         | 3/3                       |

|    15d92755-e334-0939-2109-ecf4bbc3a240 |         | 1/1                       |

|    0cca1e55-bb18-df39-3e35-ecf4bbc3a0d8 |         | 1/1                       |

|    7cb55355-9305-6140-222b-ecf4bbc3a240 |         | 3/3                       |

|    d1023955-9299-bd40-309a-ecf4bbc3a158 |         | 1/1                       |

|    15d92755-13fd-054a-c50e-ecf4bbc3a240 |         | 1/1                       |

|    ae11a055-7f23-3a4c-6031-ecf4bbc3a158 |         | 3/3                       |

|    14ca1e55-cba4-574d-9544-ecf4bbc3a0d8 |         | 1/1                       |

|    d0c91e55-bbfe-0b4e-625e-ecf4bbc3a0d8 |         | 1/1                       |

|    ebc91e55-63c6-984e-3d9f-ecf4bbc3a0d8 |         | 1/1                       |

|    63b55355-9341-1f50-61f9-ecf4bbc3a240 |         | 3/3                       |

|    15d92755-2b22-0052-f558-ecf4bbc3a240 |         | 1/1                       |

|    d2628455-a49b-6554-68ee-ecf4bbc3a038 |         | 1/1                       |

|    62ea2755-f336-0058-5be8-ecf4bbc3a240 |         | 1/1                       |

|    11d92755-e3c1-4159-7475-ecf4bbc3a240 |         | 1/1                       |

|    b4979255-5c45-d05f-32ee-ecf4bbc3a038 |         | 1/1                       |

|    b54e8855-ac5e-7460-61d7-ecf4bbc3a038 |         | 1/1                       |

|    ff749e55-57ad-3f61-8204-ecf4bbc3a0d8 |         | 1/1                       |

|    fb1b3055-37c9-f062-4404-ecf4bbc3a158 |         | 3/3                       |

|    37d92755-63ec-8e63-1696-ecf4bbc3a240 |         | 1/1                       |

|    916e8455-4a31-6164-0fc0-ecf4bbc3a158 |         | 1/1                       |

|    e21e2d55-b34b-5c67-a2eb-ecf4bbc3a240 |         | 1/1                       |

|    da409155-c496-b567-a9e2-ecf4bbc3a240 |         | 3/3                       |

|    e86e8455-a432-0368-cd67-ecf4bbc3a038 |         | 1/1                       |

|    13d92755-8b40-7568-ddde-ecf4bbc3a240 |         | 1/1                       |

|    a8458455-9ce8-6e69-8766-ecf4bbc3a038 |         | 1/1                       |

|    d2c91e55-ebdb-c56a-108a-ecf4bbc3a0d8 |         | 1/1                       |

|    35d92755-437a-aa6b-d671-ecf4bbc3a240 |         | 1/1                       |

|    b9979255-847e-ed6b-5bf9-ecf4bbc3a038 |         | 1/1                       |

|    f5d82755-6a9b-5a6c-2c35-ecf4bbc3a0d8 |         | 3/3                       |

|    60b55355-5100-966c-94ad-ecf4bbc3a240 |         | 3/3                       |

|    c56b3a55-135b-436d-f987-ecf4bbc3a240 |         | 3/3                       |

|    d4c91e55-0bbe-456d-6502-ecf4bbc3a0d8 |         | 1/1                       |

|    c56b3a55-7362-706d-79f9-ecf4bbc3a240 |         | 3/3                       |

|    d2f1b855-9c9e-896d-030e-ecf4bbc3a240 |         | 3/3                       |

|    06ca1e55-9351-376e-0cd6-ecf4bbc3a0d8 |         | 1/1                       |

|    b3c91e55-a30d-636e-1612-ecf4bbc3a0d8 |         | 1/1                       |

|    7b013955-2a6f-596f-6524-ecf4bbc3a158 |         | 1/1                       |

|    38ca1e55-331b-2570-2573-ecf4bbc3a0d8 |         | 1/1                       |

|    8e183055-974a-dd70-cac6-ecf4bbc3a158 |         | 3/3                       |

|    2f173055-ef3d-3a71-b64a-ecf4bbc3a158 |         | 3/3                       |

|    c0c91e55-cb8d-6972-2387-ecf4bbc3a0d8 |         | 1/1                       |

|    33d02b55-d34b-dd72-dd03-ecf4bbc3a240 |         | 1/1                       |

|    173b8855-17b9-b577-8b0f-ecf4bbc3a0d8 |         | 1/1                       |

|    c56b3a55-4ba1-8c78-57f5-ecf4bbc3a240 |         | 3/3                       |

|    fbcf9f55-dfa9-9b78-00a6-ecf4bbc3a0d8 |         | 3/3                       |

|    c56b3a55-2396-a378-0026-ecf4bbc3a240 |         | 3/3                       |

|    09ca1e55-f34a-f17c-2849-ecf4bbc3a0d8 |         | 1/1                       |

|    33b79355-34bb-b57d-3e1a-ecf4bbc3a240 |         | 1/1                       |

|    ac308355-7c25-dd7d-14bd-ecf4bbc3a038 |         | 1/1                       |

|    0bca1e55-e3b6-8d7e-4809-ecf4bbc3a0d8 |         | 1/1                       |

|    d2c91e55-6bc5-7981-99c6-ecf4bbc3a0d8 |         | 1/1                       |

|    1bd92755-ab05-fd81-8459-ecf4bbc3a240 |         | 1/1                       |

|    a63a8355-843e-ea85-6afa-ecf4bbc3a038 |         | 1/1                       |

|    665ead55-43ea-8586-66aa-ecf4bbc3a240 |         | 3/3                       |

|    b4e72655-d78f-0687-cd30-ecf4bbc3a240 |         | 1/1                       |

|    73628455-9444-9388-e7bd-ecf4bbc3a038 |         | 1/1                       |

|    36ca1e55-5371-a989-24a5-ecf4bbc3a0d8 |         | 1/1                       |

|    36ca1e55-fb8a-df8b-20ec-ecf4bbc3a0d8 |         | 1/1                       |

|    b6e72655-6cdd-f68e-ddf8-ecf4bbc3a240 |         | 1/1                       |

|    171b8455-a51a-1191-2e0d-ecf4bbc3a038 |         | 1/1                       |

|    0fca1e55-2347-dc91-a2d9-ecf4bbc3a0d8 |         | 1/1                       |

|    38ca1e55-9372-6d92-76c0-ecf4bbc3a0d8 |         | 1/1                       |

|    d9c91e55-d37d-cb92-a711-ecf4bbc3a0d8 |         | 1/1                       |

|    0aca1e55-83a5-ae96-4a5f-ecf4bbc3a0d8 |         | 1/1                       |

|    80c91e55-bbab-3b99-e291-ecf4bbc3a0d8 |         | 1/1                       |

|    b24e8855-34d7-6199-f6a7-ecf4bbc3a038 |         | 1/1                       |

|    e9c91e55-ab33-089b-e064-ecf4bbc3a0d8 |         | 1/1                       |

|    38ca1e55-4b29-729d-d01b-ecf4bbc3a0d8 |         | 1/1                       |

|    c71a3055-af27-48a0-a865-ecf4bbc3a158 |         | 3/3                       |

|    767e4355-b45c-28a2-0e58-ecf4bbc3a038 |         | 3/3                       |

|    bde72655-7c4c-60a2-357f-ecf4bbc3a240 |         | 1/1                       |

|    4fd82755-9b35-b0a6-c2b5-ecf4bbc3a240 |         | 1/1                       |

|    bce72655-6baa-7ea8-51fc-ecf4bbc3a240 |         | 1/1                       |

|    0dca1e55-e3a3-e1a9-afe4-ecf4bbc3a0d8 |         | 1/1                       |

|    04ca1e55-e7db-1fac-1c20-ecf4bbc3a0d8 |         | 1/1                       |

|    25575f55-ffd4-cbad-9b36-ecf4bbc3a0d8 |         | 3/3                       |

|    36d92755-fcf3-79b4-a0dc-ecf4bbc3a240 |         | 1/1                       |

|    c0c91e55-38ec-a3b6-4087-ecf4bbc3a0d8 |         | 1/1                       |

|    0dca1e55-8b1f-8bb7-870c-ecf4bbc3a0d8 |         | 1/1                       |

|    905adb55-6886-1fb9-cf73-ecf4bbc3a158 |         | 3/3                       |

|    e2e72755-0374-07ba-7a82-ecf4bbc3a240 |         | 1/1                       |

|    4f149455-275e-40ba-9c10-ecf4bbc3a0d8 |         | 1/1                       |

|    b4e72655-ba25-e8bc-b70d-ecf4bbc3a240 |         | 1/1                       |

|    16d92755-3bbe-e5bf-8ee4-ecf4bbc3a240 |         | 1/1                       |

|    d8a65b55-1379-6bc4-4017-ecf4bbc3a240 |         | 3/3                       |

|    02708455-8ca3-d8c6-d3c3-ecf4bbc3a038 |         | 1/1                       |

|    4fd82755-3320-3ec7-bae4-ecf4bbc3a240 |         | 1/1                       |

|    d1c91e55-d564-56c7-9eb6-ecf4bbc3a0d8 |         | 1/1                       |

|    a9c91e55-73e1-a7c7-7ecf-ecf4bbc3a0d8 |         | 1/1                       |

|    163b8855-27ad-0cc8-5f75-ecf4bbc3a0d8 |         | 1/1                       |

|    b8979255-4c89-e8c9-e0a7-ecf4bbc3a038 |         | 1/1                       |

|    93b55355-a31d-98cf-4136-ecf4bbc3a240 |         | 3/3                       |

|    a9c91e55-1b56-e4d0-3d56-ecf4bbc3a0d8 |         | 1/1                       |

|    2b085e55-eadc-edd1-083a-ecf4bbc3a158 |         | 3/3                       |

|    13ca1e55-fb48-05d3-ffca-ecf4bbc3a0d8 |         | 1/1                       |

|    b99bae55-4ae1-f7d3-8ff1-ecf4bbc3a038 |         | 1/1                       |

|    b24e8855-b4c8-9ad6-2519-ecf4bbc3a038 |         | 1/1                       |

|    ab6f3e55-ea22-c2d6-a91b-ecf4bbc3a158 |         | 3/3                       |

|    0eca1e55-7b8b-10d8-af26-ecf4bbc3a0d8 |         | 1/1                       |

|    37228255-849c-a0d8-6b1a-ecf4bbc3a038 |         | 1/1                       |

|    72628455-cc83-74db-4a56-ecf4bbc3a038 |         | 1/1                       |

|    80c91e55-ab21-26dd-9fe7-ecf4bbc3a0d8 |         | 1/1                       |

|    88153055-0f45-b1e2-672b-ecf4bbc3a158 |         | 3/3                       |

|    d4d72755-0838-fce4-b513-ecf4bbc3a0d8 |         | 3/3                       |

|    d1869a55-7ab7-d9e5-1f61-ecf4bbc3a158 |         | 3/3                       |

|    ad872d55-4bb8-d7e6-a8bf-ecf4bbc3a240 |         | 1/1                       |

|    51d82755-33fb-ceea-9d7f-ecf4bbc3a240 |         | 1/1                       |

|    19153055-9fef-1aeb-6d1b-ecf4bbc3a158 |         | 3/3                       |

|    14d92755-539c-13ef-25d5-ecf4bbc3a240 |         | 1/1                       |

|    0aca1e55-d348-b7f4-7c3e-ecf4bbc3a0d8 |         | 1/1                       |

|    11ca1e55-30cd-e9f4-1ec4-ecf4bbc3a0d8 |         | 1/1                       |

|    51d82755-d31e-82f7-f56e-ecf4bbc3a240 |         | 1/1                       |

|    37ca1e55-4976-cdf7-675b-ecf4bbc3a0d8 |         | 1/1                       |

|    18d92755-2b08-98fa-8d44-ecf4bbc3a240 |         | 1/1                       |

|    eac91e55-b35a-a0fb-84b4-ecf4bbc3a0d8 |         | 1/1                       |

|    a5608355-bc7f-16ff-e380-ecf4bbc3a038 |         | 1/1                       |

+-----------------------------------------+---------+---------------------------+

+------------------------------------------------------------------+

| Legend: * = all unhealthy comps were deleted (disks present)     |

|         - = some unhealthy comps deleted, some not or can't tell |

|         no symbol = We cannot conclude any comps were deleted    |

+------------------------------------------------------------------++

/localhost/lab/computers/cluster01/hosts> vsan.object_info ~cluster 4e149455-bf33-0002-3390-ecf4bbc3a0d8

2015-08-26 19:09:52 +0000: Fetching VSAN disk info from esxi01.test.lab (may take a moment) ...

2015-08-26 19:09:52 +0000: Fetching VSAN disk info from esxi03.test.lab (may take a moment) ...

2015-08-26 19:09:52 +0000: Fetching VSAN disk info from esxi04.test.lab (may take a moment) ...

2015-08-26 19:09:52 +0000: Fetching VSAN disk info from esxi02.test.lab (may take a moment) ...

2015-08-26 19:09:55 +0000: Done fetching VSAN disk infos

DOM Object: 4e149455-bf33-0002-3390-ecf4bbc3a0d8 (v2, owner: esxi01.test.lab, policy: spbmProfileGenerationNumber = 0, hostFailuresToTolerate = 0, spbmProfileId = 1f22fcd2-0b19-4448-a50d-7d9b6e906e05, proportionalCapacity = 0, cacheReservation = 0, stripeWidth = 1)

  Component: 7e00de55-7e81-2059-61c4-ecf4bbc3a038 (state: ACTIVE (5), host: esxi01.test.lab, md: naa.6b82a720d711cf001c7602b10425ffb6, ssd: t10.ATA_____DELL_P320h2DMTFDGAL350SAH________________00000000135002069912,

                                                   votes: 1, usage: 0.0 GB)

  Extended attributes:

    Address space: 4294967296B (4.00 GB)

    Object class: vdisk

    Object path: /vmfs/volumes/vsan:524a3c6c832ad012-14c6833baa212919/47149455-e776-322d-4f68-ecf4bbc3a0d8/view-w81-09007-vdm-disposable-1d4d54ab-efc4-4d72-b832-b8e05c5d8fb5.vmdk

[zdickinson]

This is most likely due to the way the files were deleted.  If you right click a machine from the VIC or web client and delete, all files get removed.  If you remove a machine from inventory and then delete them through the datastore browser or from a host command line or through WinSCP, etc.. hidden files do not get deleted.  I have also found that some solutions will leverage an old API and can leave things behind.  The explanation I got from a solutions vendor is that regular data stores store hidden files at the top level of the file system, vSAN does so one below.

I'm waiting for the vendor to fix, in the mean time these tiny files are just being left behind.  My guess is there will not be a clean up tool.  Thank you, Zach.

[elerium · Accepted solution]

Delete at your own risk using /usr/lib/vmware/osfs/bin/objtool (run this in esxcli for help), you may want to check the object using vsan.object_info (in RVC) to get info on the object before you destroy it.

I recently had to do this when a storage vmotion for a 10TB VM failed and I found i had lost 10TB of space on the target VSAN.

[jonretting]

Since this is lab you might want to try upgrading to VSAN 6, install the Health Plugin, and run the data object health process. However, i would normally subscribe to scripted RVC method, like Elerium describes above. There are also new PowerCLI tools for VSAN6, which could be easier to systematically check/delete.  -Retting

[jonretting]

Just out of curiosity were you within a terabyte or so of free space on your VSAN datastore before experiencing issues?

Last night I deliberately ballooned the storage on my VSAN, attaching a 2TB VMDK to a VM and filled it up. The point was to simulate something a client might do, and freshen up some automation tools in the process of fixing it. However the event did produce a very similar outcome to yours, and thought i should share. Maybe some of the events i describe you experienced before/after your problem.

At the time VSAN had 4TB reported free capacity. The point of failure came when my VSAN hit around 1.2TB free space, and each disk per group was at 800-900GB (1TB each). The final VMDK to see mass data writes, become completely unstable. After forcing a VM power down that virtual disk and in turn the VM got marked "inaccessible" (i/o errors). Further inspection revealed a laundry list of absent/failed disk components.

In order to locate the UUID for the problematic object I opened up RVC and ran "vsan.obj_status_report ~<host> --print-table --filter-table 18/32". This printed out a table just showing me objects which are inaccessible.

"vsan.object_info ~0 <UUID>" reported 1.4TB of addressed space, VSAN Re-sync was frozen attempting to sync one 82GB component; however that didn't prevent it from carrying out other sync/policy related events. Since I was using the default storage policy for that disk, 1.4TB logical and 2.8TB physical.

Wanting to salvage the file server, I removed the VM from inventory, then added it back in. Also removed the affected disk, but did not delete the disk (doing so would obviously result in an i/o error, and VM becoming marked "inaccessible" again). However I was still left with 90% usage on my VSAN Datastore, and the health check plugin was unable to repair the related object. So I jumped on one of the hosts and ran "/usr/lib/vmware/osfs/bin/objtool delete -f -u <UUID>". There was also the side affect of some other objects on other disks losing some of their redundancy. Probably a result of a scramble to re-arrange objects due to the ballooning use. The Health plugin quickly remedied that problem. I conducted the test with a max com size of 255GB, and plan on reducing it to 180GB for the next run.

Best, -Retting

[klinden]

When I worked with support, this ended up being the command we used to delete all of the objects. This doesn't seem to be documented anywhere on official VMware material.