Seeking answers and discussions on Unassociated Objects

[duffer1]

I am looking for a good explanation of what Unassociated Objects are, how they come about, how does vSAN deal with them, is this something that I should be monitoring, etc.

Here is the reason I am bringing this up.  Last week, I had an issue with a vmware host.  While fixing the issue, I stumbled across a list of unassociated objects on that vSAN Cluster.  So we took a look at all 5 of our vSAN clusters and discovered approx. 15 TB of storage tied up in unassociated objects.  After tracking down the location of the UID's and the vm's they were tied to, I deleted several of the unassociated objects Tuesday morning of this week.  I looked at the clusters late in the day Tuesday and noticed that a few of the unassociated objects were still listed even the though the removal command appeared to work successfully earlier in the day.  On Wednesday, I checked and the objects were still listed.  However, I discovered something else, several of the unassociated objects listed on Tuesday were gone on Wednesday without manually removing them.  In our vSAN-2 Cluster, there were 11 objects showing as unassociated on Tuesday and on Wednesday, there was only 1 item listed and this item was not listed on Tuesday.  We never manually deleted these objects. 

I have attempted to look for documentation and posts on Unassociated Objects and how to manage them but I haven't found anything.  Based on the fact there was 14+ TB of storage used by the unassociated objects, it seems this is something I need to keep an eye on.  Or understand what it is, why it occurs, how it is avoided, how it is managed, etc.  Please provide insight.

[jameseydoyle]

Hi duffer1,

Unassociated (sometimes called 'orphaned') objects are any objects that exist on the vSAN datastore that are not associated with a registered VM in the vCenter inventory.

Some objects would ordinarily be unassociated, like user created folders (namespace objects), the .vsan-stats namespace object used for storing the performance data etc.

However, if you have significant numbers of unassociated objects, they may indicative of failed cleanups of VMs or other issues and may be consuming on your datastore. Examples would be swap objects that were not cleaned up properly, discarded snapshot chains etc.

If you have a large number of unassociated objects, open a ticket with VMware Support. They can assist you in identifying which unassociated objects are expected and which ones are not. Then you can work with them in deciding which ones to clean up from your datastore.

[TheBobkin]

Hello duffer,

First off, I would like to say to anyone reading this - proceed with caution when deleting anything (on vSAN or otherwise).

Certain Objects will show as 'Unassociated' for a number of reasons:

- All vswap Objects (.vswp) of powered-on VMs and/or unassociated vswaps (e.g. after an ungraceful shutdown or power outage).

- vswap Objects that are not in use can be purged using RVC with vsan.purge_inaccessible_vswap_objects  (Again - proceed with caution and verify these are not in use as prompted)

- vswap Objects that are in use should be left alone (as VMs may crash if you remove their .vswp).

- VMDK Objects of VMs that have been 'removed from inventory'. These are still on disk and if you do not need them anymore then remove as necessary - you don't have to do anything from CLI, datastore browser via the Web Client should be sufficient for cleaning up, Namespace UUIDs that don't have a friendly-name symlink will be VMs that are not currently registered in inventory.

- Disk (.vmdk) Objects used in vSphere Replication (apologies but don't quote me on this, it is either this or another disk-creating VMware product, I don't currently have notes access to check).

- Objects that are not in a healthy enough state to be identified (this may change as data is brought back online).

It should be possible to verify what a healthy unassociated Object is by using vsan.object_info <pathToCluster> <UUIDofObject>

If you output to file/copy+paste and sanitize the output of vsan.obj_status_report <pathToCluster> -t  you can use a short bit of script to compare this list with objtool getAttr to list friendly names via CLI which speeds up the identification process (I *may* be able to PM you something to this end if you are not too scripty but no promises).

Bob

[duffer1]

Just to reiterate Bob's caution.  Please don't take my actions as a general recommendation.  When I deleted the files, they were not production related vm's and I attempted to triple and quadruple check what we were touching.  I also worked with our VMware consultant.  I would recommend as a general rule to open a case with VMware Support before deleting files.  Contacting Support will greatly reduce the chance of a RGE occuring.  RGE = Resume Generating Event.  😛

[duffer1]

So in my instance, would someone speculate why 1 day I have 11 objects showing orphaned and the next there is only 1 object listed and it wasn't in the list the previous day?  The only out of ordinary process, was I storage vmotioned a vm to another vSAN cluster. 

Another question:  Should vSAN be regularly monitored for orphaned objects?   In my instance, I had a VM that was jacked up back in April and had to have disks deleted to fix.  I found orphaned objects relating to that VM that amounted to 4 TB of storage.   While this is not a regular occurance, I am thinking vSAN doesn't self patrol for orphaned files or if it does, I am not seeing the alerts. 

[TheBobkin]

Hello duffer,

Thanks for advising others to contact GSS before taking potentially dangerous actions - you would likely be horrified (but not surprised :D) at some of the situations that have arisen from others following 'guides' on the internet before calling us.

Regarding deleted Objects 'coming back' in your environment, might these Objects have had a single healthy/unhealthy component from a host that was not active at the time that was brought back online/into the cluster?

Regarding change in number of Unassociated Objects without deleting anything - not really possible to tell anything conclusive without knowing what they were, if they were vswaps and the VMs were moved/powered-off this is fairly self-explanatory, could have also been unhealthy Objects that couldn't be determined what they were at the time but more components came back (e.g. after bringing a node back or running vsan.check_state <cluster> -r ).

vSAN doesn't 'patrol' nor delete any Objects by design for data-safety reasons.

Should Admins keep track of what is on disk and in their inventory? - Sure of course, whether on vSAN or otherwise.

Bob

[duffer1]

I have attached a screen shot showing the items we discovered on Tuesday.  The item highlighted in green was deleted (btw, this server was no longer in use).  All of the other items no longer appear when we run the following command: vsan.obj_status_report ./ -t -u.   This is the current output we get

+-----------------------------------------+---------+---------------------------+

| Unassociated objects                                       |         |                 |

|    b5dd8159-da98-b833-cd05-0025b5001a5e |         | 3/3           |

+-----------------------------------------+---------+---------------------------+

We did not perform any other maintenance on the vSAN-2 cluster.  Any ideas why the vmdk's are no longer listed?  I browsed the vSAN2 datastore and found the epsi16rport_3.vmdk still there.  Why would this file show up on the unassociated list one day and not be there the very next day?  Please keep in mind, I am not wanting to know whether I can delete the file.  I am trying to understand why items appear on the unassociated list.  In my black & white mind, I would think, once a vmdk is on the list, it doesn't  come off.  I would think the only way a vmdk would come off the list would be by it getting re-associated by manual intervention.  I understand how the other items you mentioned above would come on and off the list.

[TheBobkin]

Hello,

There can be a lot of other factors at play - e.g. Objects being reconfigured/moved(rebalance), running on snapshots, Object compliance with Storage Policy (spbm.check_compliance can be useful here).

Do you have the vsan.object_info and cmmds-tool data of epsi16rport_3.vmdk before and after?

Also, what is the info for both of these on b5dd8159-da98-b833-cd05-0025b5001a5e ?

Bob

[duffer1]

I have attached the a screenshot of the folder for the current unassociated object.  I think this unassociated object probably needs to be deleted.  I will be opening a case with  support tomorrow to get assistance with it. 

As for the EPSI... vmdk file, we do not have any before or after information.  I did notice the icon next to this VMDK is not the same as the other VMDK files.  I am wondering if this disk could have been removed from the vm and no longer in use.  When I look at the settings for the EPSI16Report VM and specifically the disks, there are 5 drives assigned to this VM.  Non of the 5 drives reference the disk, epsi16report_3.vmdk.   So according to the setting for the VM, this disk, epsi16report_3.vmdk, does not appear to be in use.

[TheBobkin]

Hello duffer,

Yes, I don't think that was attached (same for  epsi16rport_2.vmdk), the timestamp on them is from months ago (while others are current) and neither have associated .ctk files.

Are these still not attached to the VM and not showing as Unassociated?

That Object (b5dd8159-da98-b833-cd05-0025b5001a5e) is a stats-DB Object used to store Performance service stats.

It can be checked if it is the current one in use on the Performance Service page, also check if it is compliant with its Storage Policy here.

You can also query this via RVC using vsan.perf.stats_object_info

kb.vmware.com/kb/2144403

Bob

[duffer1]

Are these still not attached to the VM and not showing as unassociated?

This is what is weird, the epsi16report_2.vmdk file is not attached to the vm and it is not showing up as unassociated.  Why would this occur?

[TheBobkin]

Hello duffer,

Well it is still located in an active namespace folder thus perhaps these won't appear as Unassociated (when healthy, not being data-moved and compliant with their Storage Policy), this should be easy enough to test. As I said though - really not possible to look at this deeper without the data from when this was occurring.

Bob