- VMware Technology Network
- :
- Cloud & SDDC
- :
- vSAN
- :
- VMware vSAN Discussions
- :
- Need to identify a VM with the Inaccessible VSAN O...
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Need to identify a VM with the Inaccessible VSAN Objects
Hi Guys.
For a lot of reasons that I’m not going to discuss here, we have right now 13 inaccessible VSAN objects in our 6 node VSAN cluster, we had some issues in the cluster that make this happen.
So when we execute a vsan.check_state ~cluster command, they are listed in the step one, when we run the command again with the -r modifier, they keep listing in the step one, here are the output of the both executions.
That I want to know is how we can identify those objects to which virtual machine(s) depends or owns and also how can I get rid of this orphaned objects.
Hope you can help us.
Have a nice day.
> vsan.check_state ~cluster
2015-01-21 14:16:31 +0000: Step 1: Check for inaccessible VSAN objects
Detected 13 objects to be inaccessible
Detected 42fe3754-208b-4217-e562-002590f94756 on XXXXXXXX.XXXXXXX.XXXXX to be inaccessible
Detected 44bf6f54-fc65-4421-a8cf-002590f94756 on XXXXXXXX.XXXXXXX.XXXXX to be inaccessible
Detected 0f537054-8351-9a32-1f63-002590f9c36c on XXXXXXXX.XXXXXXX.XXXXX to be inaccessible
Detected 807e2054-7cff-cd44-ce6b-002590f94a56 on XXXXXXXX.XXXXXXX.XXXXX to be inaccessible
Detected f3032a54-7abc-6272-f9a9-002590f961ec on XXXXXXXX.XXXXXXX.XXXXX to be inaccessible
Detected d3f35254-3562-0684-28d2-002590f961ec on XXXXXXXX.XXXXXXX.XXXXX to be inaccessible
Detected a0664254-735d-1191-7f2e-002590f94a56 on XXXXXXXX.XXXXXXX.XXXXX to be inaccessible
Detected e6372054-93b2-3894-eb91-002590f961ec on XXXXXXXX.XXXXXXX.XXXXX to be inaccessible
Detected 99c35c54-8cf9-2faa-752d-002590f94a56 on XXXXXXXX.XXXXXXX.XXXXX to be inaccessible
Detected 4a8c2054-f486-75b5-b5f7-002590f94756 on XXXXXXXX.XXXXXXX.XXXXX to be inaccessible
Detected a7392f54-c40b-51c3-ec69-002590f94756 on XXXXXXXX.XXXXXXX.XXXXX to be inaccessible
Detected e7527054-c1a8-4ac9-0466-002590f9c36c on XXXXXXXX.XXXXXXX.XXXXX to be inaccessible
Detected 7bde6454-c642-96f6-b13f-002590f961ec on XXXXXXXX.XXXXXXX.XXXXX to be inaccessible
2015-01-21 14:16:32 +0000: Step 2: Check for invalid/inaccessible VMs
2015-01-21 14:16:32 +0000: Step 3: Check for VMs for which VC/hostd/vmx are out of sync
>
> vsan.check_state ~cluster -r
2015-01-21 14:15:42 +0000: Step 1: Check for inaccessible VSAN objects
Detected 42fe3754-208b-4217-e562-002590f94756 to not be inaccessible, refreshing state
Detected 0f537054-8351-9a32-1f63-002590f9c36c to not be inaccessible, refreshing state
Detected 807e2054-7cff-cd44-ce6b-002590f94a56 to not be inaccessible, refreshing state
Detected f3032a54-7abc-6272-f9a9-002590f961ec to not be inaccessible, refreshing state
Detected a0664254-735d-1191-7f2e-002590f94a56 to not be inaccessible, refreshing state
Detected e6372054-93b2-3894-eb91-002590f961ec to not be inaccessible, refreshing state
Detected 99c35c54-8cf9-2faa-752d-002590f94a56 to not be inaccessible, refreshing state
Detected a7392f54-c40b-51c3-ec69-002590f94756 to not be inaccessible, refreshing state
Detected 7bde6454-c642-96f6-b13f-002590f961ec to not be inaccessible, refreshing state
Detected 44bf6f54-fc65-4421-a8cf-002590f94756 to not be inaccessible, refreshing state
Detected d3f35254-3562-0684-28d2-002590f961ec to not be inaccessible, refreshing state
Detected 4a8c2054-f486-75b5-b5f7-002590f94756 to not be inaccessible, refreshing state
Detected e7527054-c1a8-4ac9-0466-002590f9c36c to not be inaccessible, refreshing state
2015-01-21 14:15:44 +0000: Step 1b: Check for inaccessible VSAN objects, again
Detected 42fe3754-208b-4217-e562-002590f94756 is still inaccessible
Detected 44bf6f54-fc65-4421-a8cf-002590f94756 is still inaccessible
Detected 0f537054-8351-9a32-1f63-002590f9c36c is still inaccessible
Detected 807e2054-7cff-cd44-ce6b-002590f94a56 is still inaccessible
Detected f3032a54-7abc-6272-f9a9-002590f961ec is still inaccessible
Detected d3f35254-3562-0684-28d2-002590f961ec is still inaccessible
Detected a0664254-735d-1191-7f2e-002590f94a56 is still inaccessible
Detected e6372054-93b2-3894-eb91-002590f961ec is still inaccessible
Detected 99c35c54-8cf9-2faa-752d-002590f94a56 is still inaccessible
Detected 4a8c2054-f486-75b5-b5f7-002590f94756 is still inaccessible
Detected a7392f54-c40b-51c3-ec69-002590f94756 is still inaccessible
Detected e7527054-c1a8-4ac9-0466-002590f9c36c is still inaccessible
Detected 7bde6454-c642-96f6-b13f-002590f961ec is still inaccessible
2015-01-21 14:15:44 +0000: Step 2: Check for invalid/inaccessible VMs
2015-01-21 14:15:44 +0000: Step 2b: Check for invalid/inaccessible VMs again
2015-01-21 14:15:45 +0000: Step 3: Check for VMs for which VC/hostd/vmx are out of sync
>
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi,
If you're absolutely sure about removing the objects (I advice you to make a support request for that) you can remove them from one off the esx servers with the objecttool (/usr/lib/vmware/osfs/bin/objtool).
But first make sure you don't need the objects anymore.
I can't find the RVC command to find to which vm a object belongs.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Use the following command's to find the object:
> vsan.cmmds_find -u 3dce6554-3ad1-e139-cfa2-2c768a540dcc -t LSOM_OBJECT localhost/XXX/computers/XXXXXXXXXX
+---+-------------+--------------------------------------+------------------------------+---------+-----------------------------------------------------------+
| # | Type | UUID | Owner | Health | Content |
+---+-------------+--------------------------------------+------------------------------+---------+-----------------------------------------------------------+
| 1 | LSOM_OBJECT | 3dce6554-3ad1-e139-cfa2-2c768a540dcc | XXXXXXXXXXXXXXXXXXXXXXXXXXXX | Healthy | {"diskUuid"=>"52fb3d33-b17f-37eb-4c1f-0044d362e517", |
| | | | | | "compositeUuid"=>"be2e5b54-a291-cc63-c58b-a0d3c1f4c5ec", |
| | | | | | "capacityUsed"=>107376279552, |
| | | | | | "physCapacityUsed"=>105308487680} |
+---+-------------+--------------------------------------+------------------------------+---------+-----------------------------------------------------------+
And then to find the originating disk use the compositeUuid:
> vsan.object_info localhost/XXX/computers/XXXXXXXXXX be2e5b54-a291-cc63-c58b-a0d3c1f4c5ec
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi crosdorff, thanks for your reply, here are the output for the objects that we have missing, for those ID's there's not LSOM Objects.
> vsan.cmmds_find -u 42fe3754-208b-4217-e562-002590f94756 -t LSOM_OBJECT ~cluster
+---+------+------+-------+--------+---------+
| # | Type | UUID | Owner | Health | Content |
+---+------+------+-------+--------+---------+
+---+------+------+-------+--------+---------+
> vsan.cmmds_find -u 44bf6f54-fc65-4421-a8cf-002590f94756 -t LSOM_OBJECT ~cluster
+---+------+------+-------+--------+---------+
| # | Type | UUID | Owner | Health | Content |
+---+------+------+-------+--------+---------+
+---+------+------+-------+--------+---------+
> vsan.cmmds_find -u 0f537054-8351-9a32-1f63-002590f9c36c -t LSOM_OBJECT ~cluster
+---+------+------+-------+--------+---------+
| # | Type | UUID | Owner | Health | Content |
+---+------+------+-------+--------+---------+
+---+------+------+-------+--------+---------+
> vsan.cmmds_find -u 807e2054-7cff-cd44-ce6b-002590f94a56 -t LSOM_OBJECT ~cluster
+---+------+------+-------+--------+---------+
| # | Type | UUID | Owner | Health | Content |
+---+------+------+-------+--------+---------+
+---+------+------+-------+--------+---------+
> vsan.cmmds_find -u f3032a54-7abc-6272-f9a9-002590f961ec -t LSOM_OBJECT ~cluster
+---+------+------+-------+--------+---------+
| # | Type | UUID | Owner | Health | Content |
+---+------+------+-------+--------+---------+
+---+------+------+-------+--------+---------+
> vsan.cmmds_find -u d3f35254-3562-0684-28d2-002590f961ec -t LSOM_OBJECT ~cluster
+---+------+------+-------+--------+---------+
| # | Type | UUID | Owner | Health | Content |
+---+------+------+-------+--------+---------+
+---+------+------+-------+--------+---------+
> vsan.cmmds_find -u a0664254-735d-1191-7f2e-002590f94a56 -t LSOM_OBJECT ~cluster
+---+------+------+-------+--------+---------+
| # | Type | UUID | Owner | Health | Content |
+---+------+------+-------+--------+---------+
+---+------+------+-------+--------+---------+
> vsan.cmmds_find -u e6372054-93b2-3894-eb91-002590f961ec -t LSOM_OBJECT ~cluster
+---+------+------+-------+--------+---------+
| # | Type | UUID | Owner | Health | Content |
+---+------+------+-------+--------+---------+
+---+------+------+-------+--------+---------+
> vsan.cmmds_find -u 99c35c54-8cf9-2faa-752d-002590f94a56 -t LSOM_OBJECT ~cluster
+---+------+------+-------+--------+---------+
| # | Type | UUID | Owner | Health | Content |
+---+------+------+-------+--------+---------+
+---+------+------+-------+--------+---------+
> vsan.cmmds_find -u 4a8c2054-f486-75b5-b5f7-002590f94756 -t LSOM_OBJECT ~cluster
+---+------+------+-------+--------+---------+
| # | Type | UUID | Owner | Health | Content |
+---+------+------+-------+--------+---------+
+---+------+------+-------+--------+---------+
> vsan.cmmds_find -u a7392f54-c40b-51c3-ec69-002590f94756 -t LSOM_OBJECT ~cluster
+---+------+------+-------+--------+---------+
| # | Type | UUID | Owner | Health | Content |
+---+------+------+-------+--------+---------+
+---+------+------+-------+--------+---------+
> vsan.cmmds_find -u e7527054-c1a8-4ac9-0466-002590f9c36c -t LSOM_OBJECT ~cluster
+---+------+------+-------+--------+---------+
| # | Type | UUID | Owner | Health | Content |
+---+------+------+-------+--------+---------+
+---+------+------+-------+--------+---------+
> vsan.cmmds_find -u 7bde6454-c642-96f6-b13f-002590f961ec -t LSOM_OBJECT ~cluster
+---+------+------+-------+--------+---------+
| # | Type | UUID | Owner | Health | Content |
+---+------+------+-------+--------+---------+
+---+------+------+-------+--------+---------+
>
But if you run the command with no filtering of types here are the example of one of the objects missing.
+
> vsan.cmmds_find -u 7bde6454-c642-96f6-b13f-002590f961ec ~cluster
+---+---------------+--------------------------------------+--------------------------+---------+------------------------------------------------------------------+
| # | Type | UUID | Owner | Health | Content |
+---+---------------+--------------------------------------+--------------------------+---------+------------------------------------------------------------------+
| 1 | DOM_OBJECT | 7bde6454-c642-96f6-b13f-002590f961ec | xxxxxxxxxxxx.xxxxxxx.xxx | Healthy | {"type"=>"Configuration", |
| | | | | | "attributes"=> |
| | | | | | {"CSN"=>33, |
| | | | | | "addressSpace"=>118111600640, |
| | | | | | "compositeUuid"=>"7bde6454-c642-96f6-b13f-002590f961ec"}, |
| | | | | | "child-1"=> |
| | | | | | {"type"=>"RAID_1", |
| | | | | | "attributes"=>{}, |
| | | | | | "child-1"=> |
| | | | | | {"type"=>"Component", |
| | | | | | "attributes"=> |
| | | | | | {"addressSpace"=>118111600640, |
| | | | | | "componentState"=>9, |
| | | | | | "componentStateTS"=>1416390453, |
| | | | | | "staleLsn"=>478070, |
| | | | | | "staleCsn"=>25, |
| | | | | | "transient"=>1, |
| | | | | | "faultDomainId"=>"541f2fc4-1499-97a6-5cbc-002590f961ec"}, |
| | | | | | "componentUuid"=>"7cde6454-b68e-9512-f379-002590f961ec", |
| | | | | | "diskUuid"=>"52d7602a-ff4e-4b36-e72e-2e4f6bc730ef"}, |
| | | | | | "child-2"=> |
| | | | | | {"type"=>"Component", |
| | | | | | "attributes"=> |
| | | | | | {"addressSpace"=>118111600640, |
| | | | | | "componentState"=>5, |
| | | | | | "componentStateTS"=>1416003384, |
| | | | | | "staleLsn"=>0, |
| | | | | | "staleCsn"=>0, |
| | | | | | "bytesToSync"=>0, |
| | | | | | "recoveryETA"=>0, |
| | | | | | "faultDomainId"=>"541f2153-4083-35af-901a-002590f94a56"}, |
| | | | | | "componentUuid"=>"7e7e6654-a2b2-376e-f071-002590f961ec", |
| | | | | | "diskUuid"=>"5286e68f-8566-e60e-203a-6e12f00d7e1f"}, |
| | | | | | "child-3"=> |
| | | | | | {"type"=>"Component", |
| | | | | | "attributes"=> |
| | | | | | {"addressSpace"=>118111600640, |
| | | | | | "componentState"=>9, |
| | | | | | "componentStateTS"=>1416456532, |
| | | | | | "staleLsn"=>18446744073709551615, |
| | | | | | "staleCsn"=>0, |
| | | | | | "bytesToSync"=>0, |
| | | | | | "recoveryETA"=>0, |
| | | | | | "transient"=>1, |
| | | | | | "faultDomainId"=>"5458d0e0-8a0a-0695-1fed-002590f9c36c"}, |
| | | | | | "componentUuid"=>"1d326d54-90a2-1ca5-232a-002590f94a56", |
| | | | | | "diskUuid"=>"52f8c0cb-426d-cfec-9e32-2f6f215394b6"}, |
| | | | | | "child-4"=> |
| | | | | | {"type"=>"Component", |
| | | | | | "attributes"=> |
| | | | | | {"addressSpace"=>118111600640, |
| | | | | | "componentState"=>6, |
| | | | | | "componentStateTS"=>1421849635, |
| | | | | | "staleLsn"=>18446744073709551615, |
| | | | | | "staleCsn"=>0, |
| | | | | | "bytesToSync"=>0, |
| | | | | | "recoveryETA"=>0, |
| | | | | | "faultDomainId"=>"541f2fc4-1499-97a6-5cbc-002590f961ec"}, |
| | | | | | "componentUuid"=>"7f696d54-2026-43eb-ef73-002590f94a56", |
| | | | | | "diskUuid"=>"529700f8-260e-c199-7460-60565f3017fc"}}, |
| | | | | | "child-2"=> |
| | | | | | {"type"=>"Witness", |
| | | | | | "attributes"=> |
| | | | | | {"componentState"=>6, |
| | | | | | "componentStateTS"=>1421849635, |
| | | | | | "isWitness"=>1, |
| | | | | | "faultDomainId"=>"5458d0e0-8a0a-0695-1fed-002590f9c36c"}, |
| | | | | | "componentUuid"=>"7f696d54-2c87-47eb-7982-002590f94a56", |
| | | | | | "diskUuid"=>"520cab5b-0566-1200-9402-e7b3d01fb4b2"}} |
| 2 | CONFIG_STATUS | 7bde6454-c642-96f6-b13f-002590f961ec | xxxxxxxxxxxx.xxxxxxx.xxx | Healthy | {"state"=>13, "csn"=>33} |
+---+---------------+--------------------------------------+--------------------------+---------+------------------------------------------------------------------+
>
So everithing appears to be healty but I does not fully understand this output.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Ok, last night i had the same situation with crashed Adaptec ASR-6405.
After its crash - I've got stale VSAN with 12-13 inaccessible components with no way for Maintenance mode (especially Full).
So the cause was small 2-8 Gb file chunks (considered CLOMD as components) on vmfs partition of some disks in Disk Group.
1. You need to know where is this inaccessible components resides on HDD via rvc command vsan.object_info ~cluster UUID <of inaccessible component>
Example (couldn't find real one):
/localhost/DC> vsan.object_info ~cluster 7e62c152-7dfb-c6e5-07b8-001b2193b9a4
2014-01-01 18:35:16 +0000: Fetching VSAN disk info from esx1.virten.lab (may take a moment) ...
2014-01-01 18:35:16 +0000: Fetching VSAN disk info from esx4.virten.lab (may take a moment) ...
2014-01-01 18:35:16 +0000: Fetching VSAN disk info from esx2.virten.lab (may take a moment) ...
2014-01-01 18:35:16 +0000: Fetching VSAN disk info from esx3.virten.lab (may take a moment) ...
2014-01-01 18:35:18 +0000: Done fetching VSAN disk infos
DOM Object: 7e62c152-7dfb-c6e5-07b8-001b2193b9a4 (owner: esx1.virten.lab, policy: NO POLICY)
Witness: c135c452-cd77-0733-1708-001b2193b9a4 (state: ACTIVE (5), host: esx4.virten.lab, md: t10.ATA_____WDC_WD300HLFSD2D03DFC1, ssd: t10.ATA_____SanDisk_SDSSDP064G)
RAID_1
Component: c135c452-2f04-0533-dbbc-001b2193b9a4 (state: ABSENT (0), host: esx1.virten.lab, md: t10.ATA_____WDC_WD300HLFSD2D00NLR1, ssd: t10.ATA_____SanDisk_SDSSDP064G)
Component: 7e62c152-763d-1400-2b06-001b2193b9a4 (state: ACTIVE (5), host: esx2.virten.lab, md: t10.ATA_____WDC_WD3000HLFS2D01G6U1, ssd: t10.ATA_____SanDisk_SDSSDP064G)
All components of this DOM Object (UUID) would be zero in size, but some "spoiled" may be not. That is those chunks with corrupted data, considered CLOMD as components.
You could see attribute NO GROUP POLICY and strange states of zeroed components as ACTIVE.
2. Locate this "spoiled" HDDs uuid and remove them from Disk Groups or recreate Disk Groups with HDD and SSD formatting.
Put the host into Maintenance mode - Ensure accessibility (if possible).
Remove Disk Group or "spoiled" HDD from it, if some.
Remove partitions from all HDD and SSD (IT IS VERY IMPORTANT) via partedUtil or by creating and removing Storage (vmfs).
3. After removing HDD or SSD or Disk Group wait for Component Sync.
View it status by vsan.resync_dashboard
/localhost/DC> vsan.resync_dashboard ~cluster -r 10
2013-12-27 20:05:56 +0000: Querying all VMs on VSAN ...
2013-12-27 20:05:56 +0000: Querying all objects in the system from esx1.virten.lab ...
2013-12-27 20:05:56 +0000: Got all the info, computing table ...
+-----------+-----------------+---------------+
| VM/Object | Syncing objects | Bytes to sync |
+-----------+-----------------+---------------+
+-----------+-----------------+---------------+
| Total | 0 | 0.00 GB |
+-----------+-----------------+---------------+
You can contact me in PM or by mail philzy<eta>mail.ru for further assistance.