I have a challenge for the vSAN experts. - Unassoc...

Pkiertzner · ‎05-15-2017

Hi

Long story short. vSphere 6.5.0d, vSAN 6.6

I broke the vSAN in my homelab..... lesson learned. Done run your backup appliance on the same datastore that it need to protect

3 node setup. VCSA running on the vSAN. host 3 ran out of space during a snapshot operation. All froze to hell.

I reinstalled host 3 and the VCSA.

Host 1 + 2 left the old vSAN cluster and joined the newly created vSAN cluster with host 3.

New vSAN UUID 59181efb-e6a3-c91a-4585-b8aeedec5878

Old vSAN UUID 522c4555-0f08-253c-da57-5dbd0b111b51

esxcli vsan cluster get

Cluster Information

Enabled: true

Current Local Time: 2017-05-15T12:02:32Z

Local Node UUID: 57a5723b-0faa-b572-7bf7-b8aeedec43bf

Local Node Type: NORMAL

Local Node State: BACKUP

Local Node Health State: HEALTHY

Sub-Cluster Master UUID: 57a5701e-edc5-0732-3db1-b8aeedec579d

Sub-Cluster Backup UUID: 57a5723b-0faa-b572-7bf7-b8aeedec43bf

Sub-Cluster UUID: 59181efb-e6a3-c91a-4585-b8aeedec5878

Sub-Cluster Membership Entry Revision: 7

Sub-Cluster Member Count: 3

Sub-Cluster Member UUIDs: 57a5701e-edc5-0732-3db1-b8aeedec579d, 57a5723b-0faa-b572-7bf7-b8aeedec43bf, 59181efb-e6a3-c91a-4585-b8aeedec5878

Sub-Cluster Membership UUID: d6971859-4c32-efd6-5cd8-00249b1aced2

Unicast Mode Enabled: true

Maintenance Mode State: OFF

When executing esxcli vsan debug vmdk list

I get some vdisk objects that exists i a none existing path. (Path to the old vSAN UUID)

Object: f0706b58-3898-8ac9-0130-00249b1aced0

Health: healthy

Type: vdisk

Path: /vmfs/volumes/vsan:522c45550f08253c-da575dbd0b111b51/f0706b58-3898-8ac9-0130-00249b1aced0/DC02.vmdk

Directory Name: N/A

How can I change the path for the vdisk object to i.e. /vmfs/volumes/vsan:59181efbe6a3c91a-4585b8aeedec5878/lost-found or move the vmdk file to another datastore?

- TheBobkin helped me to change the path.

From RVC I get the following output:

/localhost/Datacenter/computers> vsan.obj_status_report 0 -t

2017-05-15 11:22:53 +0000: Querying all VMs on vSAN ...

2017-05-15 11:22:53 +0000: Querying all objects in the system from 192.168.254.203 ...

2017-05-15 11:22:54 +0000: Querying all disks in the system from 192.168.254.203 ...

2017-05-15 11:22:54 +0000: Querying all components in the system from 192.168.254.203 ...

2017-05-15 11:22:55 +0000: Querying all object versions in the system ...

2017-05-15 11:22:58 +0000: Got all the info, computing table ...

Histogram of component health for non-orphaned objects

+-------------------------------------+------------------------------+

| Num Healthy Comps / Total Num Comps | Num objects with such status |

+-------------------------------------+------------------------------+

| 3/3 (OK) | 10 |

| 1/1 (OK) | 1 |

+-------------------------------------+------------------------------+

Total non-orphans: 11

Histogram of component health for possibly orphaned objects

+-------------------------------------+------------------------------+

| Num Healthy Comps / Total Num Comps | Num objects with such status |

+-------------------------------------+------------------------------+

Total orphans: 0

Total v1 objects: 0

Total v2 objects: 0

Total v2.5 objects: 0

Total v3 objects: 0

Total v5 objects: 11

+-----------------------------------------+---------+---------------------------+

| VM/Object | objects | num healthy / total comps |

+-----------------------------------------+---------+---------------------------+

| Unassociated objects | | |

| 4a59ad58-8859-bd11-31cc-00249b1aced2 | | 3/3 |

| 41d7ac58-5aa6-0413-c7fb-00249b1aced2 | | 3/3 |

| 4696a557-0611-0028-fd8a-b8aeedec579d | | 1/1 |

| 8dcac257-fcf9-5429-6bc5-b8aeedec5878 | | 3/3 |

| fc111759-04cc-9c61-ba99-00249b1aced1 | | 3/3 |

| bdfd5758-f60c-3c7e-3ddf-b8aeedec5878 | | 3/3 |

| 218b6658-52bb-1f9d-4b2e-b8aeedec43bf | | 3/3 |

| fd111759-a22e-43a9-e637-00249b1aced1 | | 3/3 |

| f0706b58-3898-8ac9-0130-00249b1aced0 | | 3/3 |

| e4611859-e2de-38cd-8c58-00249b1aced2 | | 3/3 |

| 73927658-f45f-f7d6-de14-00249b1aced0 | | 3/3 |

+-----------------------------------------+---------+---------------------------+

+------------------------------------------------------------------+

| Legend: * = all unhealthy comps were deleted (disks present) |

| - = some unhealthy comps deleted, some not or can't tell |

| no symbol = We cannot conclude any comps were deleted |

+------------------------------------------------------------------+

Here are listed the Unassociated objects of all the vdisk object that exists i the none existing path.

How can they become associated objects?

Updates:

Let us focus on the f0706b58-3898-8ac9-0130-00249b1aced0 vmdk object

There is NO vmnamespace for the vmdk object f0706b58-3898-8ac9-0130-00249b1aced0.

I then created a VM - DC02 from vCenter and the vmnamespace 72d51959-3ae4-4cf4-f21f-b8aeedec5878 was created

From RVC

vmdk object

/localhost/Datacenter/computers> vsan.object_info 0 f0706b58-3898-8ac9-0130-00249b1aced0

DOM Object: f0706b58-3898-8ac9-0130-00249b1aced0 (v5, owner: 192.168.254.202, proxy owner: None, policy: stripeWidth = 1, proportionalCapacity = 0, cacheReservation = 0, CSN = 516, spbmProfileGenerationNumber = 4, spbmProfileId = aa6d5a82-1c88-45da-85d3-3d74b91a5bad, hostFailuresToTolerate = 1, objectVersion = 5, SCSN = 517, forceProvisioning = 0, spbmProfileName = Virtual SAN Default Storage Policy)

RAID_1

Component: cc071059-e2cd-fed3-82e1-b8aeedec43bf (state: ACTIVE (5), host: 192.168.254.202, md: t10.ATA_____Samsung_SSD_850_EVO_250GB_______________S2R6NX0H712689T_____, ssd: t10.ATA_____Samsung_SSD_850_EVO_M.2_250GB___________S33CNX0H705224E_____,

votes: 1, usage: 19.1 GB, proxy component: false)

Component: 615b1859-c27f-2ea3-0a9f-00249b1aced0 (state: ACTIVE (5), host: 192.168.254.203, md: t10.ATA_____Samsung_SSD_850_EVO_250GB_______________S2R6NX0H701014Y_____, ssd: t10.ATA_____Samsung_SSD_850_EVO_M.2_250GB___________S33CNX0H536089Z_____,

votes: 1, usage: 19.1 GB, proxy component: false)

Witness: d9631859-7e2c-e993-725a-00249b1aced0 (state: ACTIVE (5), host: 192.168.254.201, md: t10.ATA_____Samsung_SSD_850_EVO_250GB_______________S2R6NX0H708025M_____, ssd: t10.ATA_____Samsung_SSD_850_EVO_M.2_250GB___________S33CNX0H536116T_____,

votes: 1, usage: 0.0 GB, proxy component: false)

Extended attributes:

Address space: 42949672960B (40.00 GB)

Object class: vdisk

Object path: /vmfs/volumes/vsan:59181efbe6a3c91a-4585b8aeedec5878/72d51959-3ae4-4cf4-f21f-b8aeedec5878/DC02.vmdk

Object capabilities: NONE

The directory /vmfs/volumes/vsan:59181efbe6a3c91a-4585b8aeedec5878/72d51959-3ae4-4cf4-f21f-b8aeedec5878 does exist

The vmnamespace

/localhost/Datacenter/computers> vsan.object_info 0 72d51959-3ae4-4cf4-f21f-b8aeedec5878

DOM Object: 72d51959-3ae4-4cf4-f21f-b8aeedec5878 (v5, owner: 192.168.254.201, proxy owner: None, policy: spbmProfileId = aa6d5a82-1c88-45da-85d3-3d74b91a5bad, spbmProfileName = Virtual SAN Default Storage Policy, forceProvisioning = 0, cacheReservation = 0, CSN = 2, hostFailuresToTolerate = 1, spbmProfileGenerationNumber = 0, stripeWidth = 1, proportionalCapacity = [0, 100])

RAID_1

Component: 72d51959-a2ca-98f4-03e0-b8aeedec5878 (state: ACTIVE (5), host: 192.168.254.203, md: t10.ATA_____Samsung_SSD_850_EVO_250GB_______________S2R6NX0H701014Y_____, ssd: t10.ATA_____Samsung_SSD_850_EVO_M.2_250GB___________S33CNX0H536089Z_____,

votes: 1, usage: 0.3 GB, proxy component: false)

Component: 72d51959-9613-9af4-bafa-b8aeedec5878 (state: ACTIVE (5), host: 192.168.254.201, md: t10.ATA_____Samsung_SSD_850_EVO_250GB_______________S2R6NX0H708025M_____, ssd: t10.ATA_____Samsung_SSD_850_EVO_M.2_250GB___________S33CNX0H536116T_____,

votes: 1, usage: 0.3 GB, proxy component: false)

Witness: 72d51959-0244-9bf4-1cf8-b8aeedec5878 (state: ACTIVE (5), host: 192.168.254.202, md: t10.ATA_____Samsung_SSD_850_EVO_250GB_______________S2R6NX0H712689T_____, ssd: t10.ATA_____Samsung_SSD_850_EVO_M.2_250GB___________S33CNX0H705224E_____,

votes: 1, usage: 0.0 GB, proxy component: false)

Extended attributes:

Address space: 273804165120B (255.00 GB)

Object class: vmnamespace

Object path: /vmfs/volumes/vsan:59181efbe6a3c91a-4585b8aeedec5878/

Object capabilities: NONE

From esxcli

esxcli vsan debug vmdk list

Object: 72d51959-3ae4-4cf4-f21f-b8aeedec5878

Health: healthy

Type: vmnamespace

Path: /vmfs/volumes/vsan:59181efbe6a3c91a-4585b8aeedec5878/

Directory Name: DC02

Object: f0706b58-3898-8ac9-0130-00249b1aced0

Health: healthy

Type: vdisk

Path: /vmfs/volumes/vsan:59181efbe6a3c91a-4585b8aeedec5878/72d51959-3ae4-4cf4-f21f-b8aeedec5878/DC02.vmdk

Directory Name: N/A

/usr/lib/vmware/osfs/bin/objtool getAttr -u 72d51959-3ae4-4cf4-f21f-b8aeedec5878

Object Attributes --

UUID:72d51959-3ae4-4cf4-f21f-b8aeedec5878

Object type:vsan

Object size:273804165120

User friendly name:DC02

HA metadata:(null)

Allocation type:Thick

Policy:((\"stripeWidth\" i1) (\"cacheReservation\" i0) (\"proportionalCapacity\" i0) (\"hostFailuresToTolerate\" i1) (\"forceProvisioning\" i0) (\"spbmProfileId\" \"aa6d5a82-1c88-45da-85d3-3d74b91a5bad\") (\"spbmProfileGenerationNumber\" l+0) (\"spbmProfileName\" \"Virtual SAN Default Storage Policy\"))

Object class: vmnamespace

Object capabilities: NONE

Object path: /vmfs/volumes/vsan:59181efbe6a3c91a-4585b8aeedec5878/

Group uuid: 00000000-0000-0000-0000-000000000000

/usr/lib/vmware/osfs/bin/objtool getAttr -u f0706b58-3898-8ac9-0130-00249b1aced0

Object Attributes --

UUID:f0706b58-3898-8ac9-0130-00249b1aced0

Object type:vsan

Object size:42949672960

User friendly name:(null)

HA metadata:(null)

Allocation type:Thin

Policy:((\"stripeWidth\" i1) (\"cacheReservation\" i0) (\"proportionalCapacity\" i0) (\"hostFailuresToTolerate\" i1) (\"forceProvisioning\" i0) (\"spbmProfileId\" \"aa6d5a82-1c88-45da-85d3-3d74b91a5bad\") (\"spbmProfileGenerationNumber\" l+4) (\"objectVersion\" i5) (\"CSN\" l129) (\"SCSN\" l129) (\"spbmProfileName\" \"Virtual SAN Default Storage Policy\"))

Object class: vdisk

Object capabilities: NONE

Object path: /vmfs/volumes/vsan:59181efbe6a3c91a-4585b8aeedec5878/72d51959-3ae4-4cf4-f21f-b8aeedec5878/DC02.vmdk

Group uuid: ec706b58-98e7-a27e-2079-00249b1aced0 -> Original vmnamespace - that vmnamespace does NOT exist anymore

----- So how can I "dump" / "copy" the vmdk data from the vdisk object and attach it as a vmdk to a new VM?

TheBobkin · ‎05-15-2017

Hello,

You can use objtool to set the path of the Objects to the present location - the new vsanDatastore created when you created a new cluster.

For example to change a namespace Object:

#/usr/lib/vmware/osfs/bin/objtool setAttr -d /vmfs/volumes/vsan:currentUUID/ -u <UUID-of-namespace-Object>

And similarly for a vmdk Object:

#/usr/lib/vmware/osfs/bin/objtool setAttr -d /vmfs/volumes/vsan:currentUUID/NamespaceUUID/VMName.vmdk -u <UUID-of-vmdk-Object>

For the vmdk you listed:

#/usr/lib/vmware/osfs/bin/objtool setAttr -d /vmfs/volumes/vsan:59181efbe6a3c91a-4585b8aeedec5878/8ccac257-3c86-9781-f351-b8aeedec5878/MGMT.vmdk -u 8dcac257-fcf9-5429-6bc5-b8aeedec5878

You can get the UUID of a vmdk object multiple ways:

- Using the debug tool in 6.6

- Using cat on the descriptor file it is shown under Extent Description "vsan://UUID-of-vmdk-object"

- From RVC vsan.obj_status_report . -t (but this will not tell you wheter they are disks or namespace Objects.)

- From RVC using vsan.vm_object_info ./<PathToVM>

Using the Object UUID you can also check the current path and other info using objtool:

#/usr/lib/vmware/osfs/bin/objtool getAttr -u 8dcac257-fcf9-5429-6bc5-b8aeedec5878

I tested the above in a lab and it works but obviously exercise caution when changing things like these manually.

Power-off and unregister from inventory any VMs before doing this and test the procedure with a least-important VM to start.

Bob

-o- If you found this comment useful please click the 'Helpful' button and/or select as 'Answer' if you consider it so, please ask follow-up questions if you have any -o-

Pkiertzner · ‎05-15-2017

Hi Bob

From a test vmdk object: (DC02)

/usr/lib/vmware/osfs/bin/objtool getAttr -u f0706b58-3898-8ac9-0130-00249b1aced0

Object Attributes --

UUID:f0706b58-3898-8ac9-0130-00249b1aced0

Object type:vsan

Object size:42949672960

User friendly name:(null)

HA metadata:(null)

Allocation type:Thin

Policy:((\"stripeWidth\" i1) (\"cacheReservation\" i0) (\"proportionalCapacity\" i0) (\"hostFailuresToTolerate\" i1) (\"forceProvisioning\" i0) (\"spbmProfileId\" \"aa6d5a82-1c88-45da-85d3-3d74b91a5bad\") (\"spbmProfileGenerationNumber\" l+4) (\"objectVersion\" i5) (\"CSN\" l129) (\"SCSN\" l129) (\"spbmProfileName\" \"Virtual SAN Default Storage Policy\"))

Object class: vdisk

Object capabilities: NONE

Object path: /vmfs/volumes/vsan:522c45550f08253c-da575dbd0b111b51/ec706b58-98e7-a27e-2079-00249b1aced0/DC02.vmdk

Group uuid: ec706b58-98e7-a27e-2079-00249b1aced0

I then executed:

/usr/lib/vmware/osfs/bin/objtool setAttr -d /vmfs/volumes/vsan:59181efbe6a3c91a-4585b8aeedec5878/ec706b58-98e7-a27e-2079-00249b1aced0/DC02.vmdk -u f0706b58-3898-8ac9-0130-00249b1aced0

The Object path has now been changed, but I can't browse to it.

This is all there is:

/vmfs/volumes/vsan:59181efbe6a3c91a-4585b8aeedec5878] ls -l

total 2048

lrwxr-xr-x 1 root root 36 May 15 15:22 FTP -> b5bc1959-2a0d-751e-4d3d-b8aeedec43bf

drwxr-xr-t 1 root root 3360 May 15 14:45 b5bc1959-2a0d-751e-4d3d-b8aeedec43bf

drwxr-xr-t 1 root root 1960 May 14 14:00 e4611859-e2de-38cd-8c58-00249b1aced2

"ec706b58-98e7-a27e-2079-00249b1aced0" does not exist. Shall I create the directory first with /usr/lib/vmware/osfs/bin/osf-mkdir ?

I Don't have a namespace for DC02.

TheBobkin · ‎05-15-2017

Hello,

Are you positive there is no namespace Object?

Check using:

#/usr/lib/vmware/osfs/bin/objtool getAttr -u ec706b58-98e7-a27e-2079-00249b1aced0

If there is then change the path of this to the current (new) vsanDatastore:

#/usr/lib/vmware/osfs/bin/objtool setAttr -d /vmfs/volumes/vsan:59181efbe6a3c91a-4585b8aeedec5878/ -u ec706b58-98e7-a27e-2079-00249b1aced0

If it does not exist then sure try create a new one (and change the vmdk-Object path again accordingly).

Bob

-o- If you found this comment useful please click the 'Helpful' button and/or select as 'Answer' if you consider it so, please ask follow-up questions if you have any -o-

Pkiertzner · ‎05-15-2017

Hi Bob

There is no ec706b58-98e7-a27e-2079-00249b1aced0 object.

I have created a VM named DC02 with no hdd from vCenter

Content of vsan:

/vmfs/volumes/vsan:59181efbe6a3c91a-4585b8aeedec5878] ls -l

total 3072

drwxr-xr-t 1 root root 1680 May 15 16:21 72d51959-3ae4-4cf4-f21f-b8aeedec5878

lrwxr-xr-x 1 root root 36 May 15 16:34 DC02 -> 72d51959-3ae4-4cf4-f21f-b8aeedec5878

I then ran - /usr/lib/vmware/osfs/bin/objtool setAttr -d /vmfs/volumes/vsan:59181efbe6a3c91a-4585b8aeedec5878/72d51959-3ae4-4cf4-f21f-b8aeedec5878/DC02.vmdk -u f0706b58-3898-8ac9-0130-00249b1aced0

/usr/lib/vmware/osfs/bin/objtool getAttr -u f0706b58-3898-8ac9-0130-00249b1aced0

Object Attributes --

UUID:f0706b58-3898-8ac9-0130-00249b1aced0

Object type:vsan

Object size:42949672960

User friendly name:(null)

HA metadata:(null)

Allocation type:Thin

Policy:((\"stripeWidth\" i1) (\"cacheReservation\" i0) (\"proportionalCapacity\" i0) (\"hostFailuresToTolerate\" i1) (\"forceProvisioning\" i0) (\"spbmProfileId\" \"aa6d5a82-1c88-45da-85d3-3d74b91a5bad\") (\"spbmProfileGenerationNumber\" l+4) (\"objectVersion\" i5) (\"CSN\" l129) (\"SCSN\" l129) (\"spbmProfileName\" \"Virtual SAN Default Storage Policy\"))

Object class: vdisk

Object capabilities: NONE

Object path: /vmfs/volumes/vsan:59181efbe6a3c91a-4585b8aeedec5878/72d51959-3ae4-4cf4-f21f-b8aeedec5878/DC02.vmdk

Group uuid: ec706b58-98e7-a27e-2079-00249b1aced0

/usr/lib/vmware/osfs/bin/objtool getAttr -u 72d51959-3ae4-4cf4-f21f-b8aeedec5878

Object Attributes --

UUID:72d51959-3ae4-4cf4-f21f-b8aeedec5878

Object type:vsan

Object size:273804165120

User friendly name:DC02

HA metadata:(null)

Allocation type:Thick

Policy:((\"stripeWidth\" i1) (\"cacheReservation\" i0) (\"proportionalCapacity\" i0) (\"hostFailuresToTolerate\" i1) (\"forceProvisioning\" i0) (\"spbmProfileId\" \"aa6d5a82-1c88-45da-85d3-3d74b91a5bad\") (\"spbmProfileGenerationNumber\" l+0) (\"spbmProfileName\" \"Virtual SAN Default Storage Policy\"))

Object class: vmnamespace

Object capabilities: NONE

Object path: /vmfs/volumes/vsan:59181efbe6a3c91a-4585b8aeedec5878/

Group uuid: 00000000-0000-0000-0000-000000000000

Content of /vmfs/volumes/vsan:59181efbe6a3c91a-4585b8aeedec5878/72d51959-3ae4-4cf4-f21f-b8aeedec5878:

-rw-r--r-- 1 root root 291 May 15 16:21 DC02-324e9c42.hlog

-rw-r--r-- 1 root root 0 May 15 16:21 DC02.vmsd

-rwxr-xr-x 1 root root 1701 May 15 16:21 DC02.vmx

No DC02.vmdk file

Pkiertzner · ‎05-15-2017

From RVC vsan.obj_status_report 0 -t

f0706b58-3898-8ac9-0130-00249b1aced0 is still "Unassociated objects"

VM/Object | objects | num healthy / total comps |

+------------------------------------------------------------------+---------+---------------------------+

|| DC02 | 1 | |

| [vsan] 72d51959-3ae4-4cf4-f21f-b8aeedec5878/DC02.vmx | | 3/3 |

+------------------------------------------------------------------+---------+---------------------------+

| Unassociated objects | | |

| 4a59ad58-8859-bd11-31cc-00249b1aced2 | | 3/3 |

| 41d7ac58-5aa6-0413-c7fb-00249b1aced2 | | 3/3 |

| 4696a557-0611-0028-fd8a-b8aeedec579d | | 1/1 |

| 8dcac257-fcf9-5429-6bc5-b8aeedec5878 | | 3/3 |

| fc111759-04cc-9c61-ba99-00249b1aced1 | | 3/3 |

| bdfd5758-f60c-3c7e-3ddf-b8aeedec5878 | | 3/3 |

| 218b6658-52bb-1f9d-4b2e-b8aeedec43bf | | 3/3 |

| fd111759-a22e-43a9-e637-00249b1aced1 | | 3/3 |

| f0706b58-3898-8ac9-0130-00249b1aced0 | | 3/3 |

| e4611859-e2de-38cd-8c58-00249b1aced2 | | 3/3 |

| 73927658-f45f-f7d6-de14-00249b1aced0 | | 3/3 |

+------------------------------------------------------------------+---------+---------------------------+

so how to reassociate the object?

Pkiertzner · ‎05-15-2017

Hi Bob

I think the error is the group uuid (ec706b58-98e7-a27e-2079-00249b1aced0) for the DC02.vmdk object.

TheBobkin · ‎05-15-2017

No, ec706b58-98e7-a27e-2079-00249b1aced0 is/was the original namespace object/path for that vmdk.

Have you attached that vmdk to that test VM?

Is the old vsanDatastore still browsable?

Pkiertzner · ‎05-15-2017

Hi Bob

I can't attach the vmdk since it does "not" exist.

I can't browse the old vsanDatastore.

I'm 99% sure that the error is the group uuid "ec706b58-98e7-a27e-2079-00249b1aced0" for the DC02.vmdk object.

Is there a way to dump the vmdk data from the vsan object to a new vdisk object?

TheBobkin · ‎05-16-2017

Hello,

Okay, I tried to recreate this scenario in HOL labs for 3rd time with 3 different results (as nested-labs do not act normal sometimes!).

Interestingly, the last time I did this (removed host from cluster, created new cluster, left cluster on other two nodes and joined this new cluster) resulted in all the VMs home folders and data sitting in the new vsanDatastore ( "vsanDatastore (1)" name and new UUID) so I am not sure how much what you did differs from this scenario and or if this is expected behaviour.

What I tried after this (and you might also) is to leave this new cluster and rejoin the OLD cluster using it's UUID - after doing this on all hosts the vsanDatastore was the 'original' one with old UUID and all VM data was present and located here.

Unsure where you can get the old Sub-Cluster UUID as cmmds-tool find -t SUB_CLUSTER only returns current cluster, try clomd/vmkernel.log or notes if you documented this.

Bob

-o- If you found this comment useful please click the 'Helpful' button and/or select as 'Answer' if you consider it so, please ask follow-up questions if you have any -o-

Pkiertzner · ‎05-17-2017

Hi Bob

Thanks for your input.

I managed to recreate the old original vSAN cluster UUID 522c4555-0f08-253c-da57-5dbd0b111b51

esxcli vsan cluster get

Cluster Information

Enabled: true

Current Local Time: 2017-05-17T16:27:12Z

Local Node UUID: 57a5723b-0faa-b572-7bf7-b8aeedec43bf

Local Node Type: NORMAL

Local Node State: AGENT

Local Node Health State: HEALTHY

Sub-Cluster Master UUID: 57a5701e-edc5-0732-3db1-b8aeedec579d

Sub-Cluster Backup UUID: 59181efb-e6a3-c91a-4585-b8aeedec5878

Sub-Cluster UUID: 522c4555-0f08-253c-da57-5dbd0b111b51

Sub-Cluster Membership Entry Revision: 5

Sub-Cluster Member Count: 3

Sub-Cluster Member UUIDs: 57a5701e-edc5-0732-3db1-b8aeedec579d, 59181efb-e6a3-c91a-4585-b8aeedec5878, 57a5723b-0faa-b572-7bf7-b8aeedec43bf

Sub-Cluster Membership UUID: dd761c59-0258-3b91-87e3-00249b1aced2

Unicast Mode Enabled: true

Maintenance Mode State: OFF

RVC

/localhost/Datacenter/computers> vsan.obj_status_report 0 -t

2017-05-17 16:20:08 +0000: Querying all VMs on vSAN ...

2017-05-17 16:20:08 +0000: Querying all objects in the system from 192.168.254.203 ...

2017-05-17 16:20:08 +0000: Querying all disks in the system from 192.168.254.203 ...

2017-05-17 16:20:09 +0000: Querying all components in the system from 192.168.254.203 ...

2017-05-17 16:20:09 +0000: Querying all object versions in the system ...

2017-05-17 16:20:12 +0000: Got all the info, computing table ...

Histogram of component health for non-orphaned objects

+-------------------------------------+------------------------------+

| Num Healthy Comps / Total Num Comps | Num objects with such status |

+-------------------------------------+------------------------------+

| 3/3 (OK) | 10 |

| 1/1 (OK) | 1 |

+-------------------------------------+------------------------------+

Total non-orphans: 11

Histogram of component health for possibly orphaned objects

+-------------------------------------+------------------------------+

| Num Healthy Comps / Total Num Comps | Num objects with such status |

+-------------------------------------+------------------------------+

Total orphans: 0

Total v1 objects: 0

Total v2 objects: 0

Total v2.5 objects: 0

Total v3 objects: 0

Total v5 objects: 11

+-----------------------------------------+---------+---------------------------+

| VM/Object | objects | num healthy / total comps |

+-----------------------------------------+---------+---------------------------+

| Unassociated objects | | |

| 4a59ad58-8859-bd11-31cc-00249b1aced2 | | 3/3 |

| 41d7ac58-5aa6-0413-c7fb-00249b1aced2 | | 3/3 |

| 4696a557-0611-0028-fd8a-b8aeedec579d | | 1/1 |

| 8dcac257-fcf9-5429-6bc5-b8aeedec5878 | | 3/3 |

| fc111759-04cc-9c61-ba99-00249b1aced1 | | 3/3 |

| bdfd5758-f60c-3c7e-3ddf-b8aeedec5878 | | 3/3 |

| 218b6658-52bb-1f9d-4b2e-b8aeedec43bf | | 3/3 |

| fd111759-a22e-43a9-e637-00249b1aced1 | | 3/3 |

| f0706b58-3898-8ac9-0130-00249b1aced0 | | 3/3 |

| e4611859-e2de-38cd-8c58-00249b1aced2 | | 3/3 |

| 73927658-f45f-f7d6-de14-00249b1aced0 | | 3/3 |

+-----------------------------------------+---------+---------------------------+

But the vmdk object/files does still not show in the vsanDatastore....

I still believe that is it because the missing vmnamespace for vdisk vmdk object.. So still no luck.

TheBobkin · ‎05-17-2017

Hello,

Where/how are you checking this?

SSH to each of the hosts:

# cd vmfs/volumes/

# ls -lah

What is the UUID of what 'vsanDatastore' points to?

Bob

-o- If you found this comment useful please click the 'Helpful' button and/or select as 'Answer' if you consider it so, please ask follow-up questions if you have any -o-

Pkiertzner · ‎05-17-2017

Hi

Yes SSH to each host.

/vmfs/volumes/vsan:522c45550f08253c-da575dbd0b111b51

vsanDatastore -> vsan:522c45550f08253c-da575dbd0b111b51

I have tried both with and without ec706b58-98e7-a27e-2079-00249b1aced0 directory created with /vmfs/volumes] /usr/lib/vmware/osfs/bin/osfs-mkdir in the vsanDatatore. The VMDK does not show. I think it is because the vmnamespace with uuid ec706b58-98e7-a27e-2079-00249b1aced0 is missing.

From RVC:

+-----------------------------------------+---------+---------------------------+

| VM/Object | objects | num healthy / total comps |

+-----------------------------------------+---------+---------------------------+

| Unassociated objects | | |

| 4a59ad58-8859-bd11-31cc-00249b1aced2 | | 3/3 |

| 41d7ac58-5aa6-0413-c7fb-00249b1aced2 | | 3/3 |

| 4696a557-0611-0028-fd8a-b8aeedec579d | | 1/1 |

| 8dcac257-fcf9-5429-6bc5-b8aeedec5878 | | 3/3 |

| fc111759-04cc-9c61-ba99-00249b1aced1 | | 3/3 |

| bdfd5758-f60c-3c7e-3ddf-b8aeedec5878 | | 3/3 |

| 218b6658-52bb-1f9d-4b2e-b8aeedec43bf | | 3/3 |

| fd111759-a22e-43a9-e637-00249b1aced1 | | 3/3 |

| f0706b58-3898-8ac9-0130-00249b1aced0 | | 3/3 |

| e4611859-e2de-38cd-8c58-00249b1aced2 | | 3/3 |

| 73927658-f45f-f7d6-de14-00249b1aced0 | | 3/3 |

+-----------------------------------------+---------+---------------------------+

All vdisk / vmdk object are healthy but listed as Unassociated objects.

One specific object. All components are healthy:

/localhost/Datacenter/computers/K-Street/hosts> vsan.object_info 0 f0706b58-3898-8ac9-0130-00249b1aced0

DOM Object: f0706b58-3898-8ac9-0130-00249b1aced0 (v5, owner: unknown, proxy owner: None, policy: cacheReservation = 0, hostFailuresToTolerate = 1, spbmProfilt Storage Policy, SCSN = 535, forceProvisioning = 0, spbmProfileGenerationNumber = 4, proportionalCapacity = 0, CSN = 528, objectVersion = 5, stripeWidth = 1

RAID_1

Component: cc071059-e2cd-fed3-82e1-b8aeedec43bf (state: ACTIVE (5), host: 57a5723b-0faa-b572-7bf7-b8aeedec43bf, md: 5287a626-2038-acdc-0314-ed299e7521ef,

votes: 1, usage: 19.1 GB, proxy component: false)

Component: 615b1859-c27f-2ea3-0a9f-00249b1aced0 (state: ACTIVE (5), host: 192.168.254.203, md: t10.ATA_____Samsung_SSD_850_EVO_250GB_______________S2R6NXH536089Z_____,

votes: 1, usage: 19.1 GB, proxy component: false)

Witness: d9631859-7e2c-e993-725a-00249b1aced0 (state: ACTIVE (5), host: 57a5701e-edc5-0732-3db1-b8aeedec579d, md: 52fca517-c33b-af7b-babb-6ce82bea783b, ssd

votes: 1, usage: 0.0 GB, proxy component: false)

Extended attributes:

Address space: 42949672960B (40.00 GB)

Object class: vdisk

Object path: /vmfs/volumes/vsan:522c45550f08253c-da575dbd0b111b51/ec706b58-98e7-a27e-2079-00249b1aced0/DC02.vmdk

Object capabilities: NONE

Could you as a test try to create a VM with 1 GB vmdk? Then list the vsanDatastore directory for this VM and then run:

/usr/lib/vmware/osfs/bin/objtool -u UUID for the vmnamespace.

That UUID would be listed in the vdisk /vmdk object as group uuid.

Then delete the vmnamespace object /usr/lib/vmware/osfs/bin/objtool -d UUID for the vmnamespace. and verify that you can browse to the vmdk file.

All

I have a challenge for the vSAN experts. - Unassociated objects - Update