VMware Cloud Community
DanMc85
Enthusiast
Enthusiast
Jump to solution

Help w/vSAN - VM Disk Objects shown as Unknown Object

The following issue is in a ROBO Stretched Cluster connected by 10Gb Crossover Cable in a 2 Node vSAN with Witness.

So a vSphere 7 ESXi w/vSAN upgrade went south.

- ESXi 7 was reinstalled on two host servers with a new ESXi 7 vSAN witness appliance.

- vCenter was still intact and a new Cluster was made. New vmkernel ports. Added to existing vDS.

The problem I am having, is the vsandatastore was recovered. However when I add any VM back to inventory the virtual disks are 0\blank\missing. However, the VSANDATASTORE used space appears that all the data is still there.

VMs added will say errors like:

"Some of the disks of the virtual machine ___ failed to load. The information present for them in the virtual machine configuration may be incomplete"

Power On - Operation Failed - "Unable to enumerate all disks."

When I browse a host in WinSCP, under volumes I see:

/vmfs/volumes/vsan:52e8a7a1b473de38-2dc9078322a3e571 - This is active datastore and has all the VM's but no big disk files.

vsanDatastore - Appears to be a shortcut that does not work which I think is pointing to the original old vsan volume. /vmfs/volumes/vsan:52aceefd58033ced-e6d380dbb3a4e95a

If I open up a VMDK from a VM I see, for example:

# Disk DescriptorFile version=4 encoding="UTF-8" CID=ee6267c9 parentCID=d6a51678 createType="vsanSparse" parentFileNameHint="OPNSense Firewall.vmdk" # Extent description RW 67108864 VSANSPARSE "vsan://a439555e-6e08-833c-e1a0-000e1e7254b0"  # The Disk Data Base #DDB  ddb.longContentID = "e34cf36849d729a254145eb2ee6267c9" ddb.toolsInstallType = "0" ddb.toolsVersion = "2147483647" 

Digging a little deeper and it looks like the disk objects are showing in vSAN as " Unknown object type". There are Unknown object type objects with matching vSAN GUID's mentioned in VM vmdk files. Such as the one mentioned above...

RW 67108864 VSANSPARSE "vsan://a439555e-6e08-833c-e1a0-000e1e7254b0"

Unknown object type Healthy vSAN Default Storage Policy a439555e-6e08-833c-e1a0-000e1e7254b0

esxcli vsan storage list - on Host 1

naa.5000cca258d82813 Device: naa.5000cca258d82813 Display Name: naa.5000cca258d82813 Is SSD: false VSAN UUID: 520645fa-a797-38d4-5e68-22b2171861dc VSAN Disk Group UUID: 520d0603-6e38-a207-1f2b-b6472f57d6f9 VSAN Disk Group Name: naa.55cd2e404c18cb88 Used by this host: true In CMMDS: true On-disk format version: 10 Deduplication: false Compression: false Checksum: 4179814668856239213 Checksum OK: true Is Capacity Tier: true Encryption Metadata Checksum OK: true Encryption: false DiskKeyLoaded: false Is Mounted: true Creation Time: Fri Nov  8 12:45:56 2019  naa.55cd2e404c18cb88 Device: naa.55cd2e404c18cb88 Display Name: naa.55cd2e404c18cb88 Is SSD: true VSAN UUID: 520d0603-6e38-a207-1f2b-b6472f57d6f9 VSAN Disk Group UUID: 520d0603-6e38-a207-1f2b-b6472f57d6f9 VSAN Disk Group Name: naa.55cd2e404c18cb88 Used by this host: true In CMMDS: true On-disk format version: 10 Deduplication: false Compression: false Checksum: 6518366817165333701 Checksum OK: true Is Capacity Tier: false Encryption Metadata Checksum OK: true Encryption: false DiskKeyLoaded: false Is Mounted: true Creation Time: Fri Nov  8 12:45:56 2019  naa.5000cca252cdeadd Device: naa.5000cca252cdeadd Display Name: naa.5000cca252cdeadd Is SSD: false VSAN UUID: 5242f42c-474e-1c1a-9016-927431e9713d VSAN Disk Group UUID: 52782092-2deb-db53-aaa1-cc71b1679b43 VSAN Disk Group Name: naa.55cd2e404c2055aa Used by this host: true In CMMDS: true On-disk format version: 10 Deduplication: false Compression: false Checksum: 12173014117720582296 Checksum OK: true Is Capacity Tier: true Encryption Metadata Checksum OK: true Encryption: false DiskKeyLoaded: false Is Mounted: true Creation Time: Tue Jan 21 12:40:33 2020  naa.55cd2e404c2055aa Device: naa.55cd2e404c2055aa Display Name: naa.55cd2e404c2055aa Is SSD: true VSAN UUID: 52782092-2deb-db53-aaa1-cc71b1679b43 VSAN Disk Group UUID: 52782092-2deb-db53-aaa1-cc71b1679b43 VSAN Disk Group Name: naa.55cd2e404c2055aa Used by this host: true In CMMDS: true On-disk format version: 10 Deduplication: false Compression: false Checksum: 5367773553359825337 Checksum OK: true Is Capacity Tier: false Encryption Metadata Checksum OK: true Encryption: false DiskKeyLoaded: false Is Mounted: true Creation Time: Fri Nov  8 12:45:45 2019  naa.5000cca23b7ee4e4 Device: naa.5000cca23b7ee4e4 Display Name: naa.5000cca23b7ee4e4 Is SSD: false VSAN UUID: 52e46a8d-cdb8-1159-38d5-cfbde318e15f VSAN Disk Group UUID: 520d0603-6e38-a207-1f2b-b6472f57d6f9 VSAN Disk Group Name: naa.55cd2e404c18cb88 Used by this host: true In CMMDS: true On-disk format version: 10 Deduplication: false Compression: false Checksum: 4127245025532564424 Checksum OK: true Is Capacity Tier: true Encryption Metadata Checksum OK: true Encryption: false DiskKeyLoaded: false Is Mounted: true Creation Time: Fri Nov  8 12:45:56 2019  naa.5000cca23b834064 Device: naa.5000cca23b834064 Display Name: naa.5000cca23b834064 Is SSD: false VSAN UUID: 52f4d6fa-0c8f-603d-453c-3da1646496ce VSAN Disk Group UUID: 52782092-2deb-db53-aaa1-cc71b1679b43 VSAN Disk Group Name: naa.55cd2e404c2055aa Used by this host: true In CMMDS: true On-disk format version: 10 Deduplication: false Compression: false Checksum: 6169125189223970042 Checksum OK: true Is Capacity Tier: true Encryption Metadata Checksum OK: true Encryption: false DiskKeyLoaded: false Is Mounted: true Creation Time: Fri Nov  8 12:45:45 2019 

esxcli vsan storage list - on Host 2

naa.5000cca252cc5f25 Device: naa.5000cca252cc5f25 Display Name: naa.5000cca252cc5f25 Is SSD: false VSAN UUID: 5234687a-c379-e17e-a53d-4cf9e9fc8e7c VSAN Disk Group UUID: 52be0b83-8590-9e78-3d27-ad7926eb9ad0 VSAN Disk Group Name: naa.55cd2e404c18cc1f Used by this host: true In CMMDS: true On-disk format version: 10 Deduplication: false Compression: false Checksum: 5370575494238580185 Checksum OK: true Is Capacity Tier: true Encryption Metadata Checksum OK: true Encryption: false DiskKeyLoaded: false Is Mounted: true Creation Time: Wed Jan 22 02:29:14 2020  naa.55cd2e404c2055af Device: naa.55cd2e404c2055af Display Name: naa.55cd2e404c2055af Is SSD: true VSAN UUID: 524971da-1c92-f80e-9199-0fa74a253354 VSAN Disk Group UUID: 524971da-1c92-f80e-9199-0fa74a253354 VSAN Disk Group Name: naa.55cd2e404c2055af Used by this host: true In CMMDS: true On-disk format version: 10 Deduplication: false Compression: false Checksum: 18423783201084117590 Checksum OK: true Is Capacity Tier: false Encryption Metadata Checksum OK: true Encryption: false DiskKeyLoaded: false Is Mounted: true Creation Time: Fri Nov  8 12:46:06 2019  naa.5000cca23b437b3c Device: naa.5000cca23b437b3c Display Name: naa.5000cca23b437b3c Is SSD: false VSAN UUID: 5255b6bb-5de1-49c7-67af-aee865cde07a VSAN Disk Group UUID: 524971da-1c92-f80e-9199-0fa74a253354 VSAN Disk Group Name: naa.55cd2e404c2055af Used by this host: true In CMMDS: true On-disk format version: 10 Deduplication: false Compression: false Checksum: 7220870063380886230 Checksum OK: true Is Capacity Tier: true Encryption Metadata Checksum OK: true Encryption: false DiskKeyLoaded: false Is Mounted: true Creation Time: Thu Jan 16 03:16:23 2020  naa.55cd2e404c18cc1f Device: naa.55cd2e404c18cc1f Display Name: naa.55cd2e404c18cc1f Is SSD: true VSAN UUID: 52be0b83-8590-9e78-3d27-ad7926eb9ad0 VSAN Disk Group UUID: 52be0b83-8590-9e78-3d27-ad7926eb9ad0 VSAN Disk Group Name: naa.55cd2e404c18cc1f Used by this host: true In CMMDS: true On-disk format version: 10 Deduplication: false Compression: false Checksum: 9858777924305020986 Checksum OK: true Is Capacity Tier: false Encryption Metadata Checksum OK: true Encryption: false DiskKeyLoaded: false Is Mounted: true Creation Time: Sun Jan 19 03:42:32 2020  naa.5000cca258d81d2c Device: naa.5000cca258d81d2c Display Name: naa.5000cca258d81d2c Is SSD: false VSAN UUID: 52c41e3e-e943-1a1a-d001-5fcb6de4997b VSAN Disk Group UUID: 524971da-1c92-f80e-9199-0fa74a253354 VSAN Disk Group Name: naa.55cd2e404c2055af Used by this host: true In CMMDS: true On-disk format version: 10 Deduplication: false Compression: false Checksum: 3826993888305396649 Checksum OK: true Is Capacity Tier: true Encryption Metadata Checksum OK: true Encryption: false DiskKeyLoaded: false Is Mounted: true Creation Time: Tue Jan 21 12:42:14 2020  naa.5000cca23b0bce74 Device: naa.5000cca23b0bce74 Display Name: naa.5000cca23b0bce74 Is SSD: false VSAN UUID: 52cb0133-f1c4-b3e1-dd47-fd9f5a786437 VSAN Disk Group UUID: 52be0b83-8590-9e78-3d27-ad7926eb9ad0 VSAN Disk Group Name: naa.55cd2e404c18cc1f Used by this host: true In CMMDS: true On-disk format version: 10 Deduplication: false Compression: false Checksum: 4710347119789749474 Checksum OK: true Is Capacity Tier: true Encryption Metadata Checksum OK: true Encryption: false DiskKeyLoaded: false Is Mounted: true Creation Time: Sun Jan 19 03:42:32 2020 

Is there any way to make these VM's usable again or is all my data lost?

Example: In picture below - this highlighted unknown object is the parent disk vmdk to one of my VMs

vSAN.JPG

- DanMc85
VCP-DCV 2021
1 Solution

Accepted Solutions
DanMc85
Enthusiast
Enthusiast
Jump to solution

I found this article:

VMware Knowledge Base

I wonder if this would work:

Command Syntax: /usr/lib/vmware/osfs/bin/objtool setAttr -u <Object UUID> -d <Path to VMDK>

KB Example: /usr/lib/vmware/osfs/bin/objtool setAttr -u 19b10b5d-48fd-733d-a2f0-005056263753 -d "/vmfs/volumes/vsan:529e00851e70c3b9-215f67dee5e0391b/6fb10b5d-5bd1-f2aa-219a-005056263753/Virtual Machine 1.vmdk"

- DanMc85
VCP-DCV 2021

View solution in original post

Reply
0 Kudos
15 Replies
GreatWhiteTec
VMware Employee
VMware Employee
Jump to solution

A few things going on.... House keeping first.

#1 Sounds to me that the vCenter you are using is still pre version 7, correct? If so, I would start with getting vCenter to same version.

#2 Rebuilding the hosts without cleaning them up or proper decommission preserves DGs/vSAN partitions

#3 New vSAN Witness = lost that info - lost 1/3 of the cluster

#4 Hosts could have still been part of a vSAN cluster (depending on actions during the "reinstall"). Did you verify status of hosts before adding to new cluster?

esxcli vsan cluster get

Reply
0 Kudos
DanMc85
Enthusiast
Enthusiast
Jump to solution

1. This is same vCenter which was 6.7 and was upgraded to version 7.0.  Was on a different host, not in the vSAN cluster so it was not affected by this issue.

2. vSAN partitions were preserved. I can access every file such as the VMs VMX, logs, etc... just the Metadata for the virtual disks are all showing as unknown object type in vSAN.

- Basically on the Dell R730xd's the SD Card's were clean installed with ESXi 7.0 after things went south from the 6.7 to 7.0 upgrade on the same SD cards.

3. I still have the old vSAN witness VM if that matters. Can it be rejoined into the "new" vSAN cluster?  Should I use CLI and have it leave the "old" cluster?

4. If I recall correctly, hosts did not show any cluster info after reinstall.

I started going by this guide that others had luck with when this all happened:

https://www.virtuallyghetto.com/2014/07/does-reinstalling-esxi-with-an-existing-vsan-datastore-wipe-...

This is the current cluster info:

Cluster Information

   Enabled: true

   Current Local Time: 2020-04-09T12:29:52Z

   Local Node UUID: 5e8e60a5-e9ed-dee6-e1d2-000e1e73da60

   Local Node Type: NORMAL

   Local Node State: BACKUP

   Local Node Health State: HEALTHY

   Sub-Cluster Master UUID: 5e8e650e-4186-d7dc-e787-000e1e7254b0

   Sub-Cluster Backup UUID: 5e8e60a5-e9ed-dee6-e1d2-000e1e73da60

   Sub-Cluster UUID: 52e8a7a1-b473-de38-2dc9-078322a3e571

   Sub-Cluster Membership Entry Revision: 1

   Sub-Cluster Member Count: 3

   Sub-Cluster Member UUIDs: 5e8e650e-4186-d7dc-e787-000e1e7254b0, 5e8e60a5-e9ed-dee6-e1d2-000e1e73da60, 5e8df455-1969-7774-13f3-005056866422

   Sub-Cluster Member HostNames: ESXi-Primary, ESXi-Secondary, vSANWitness7

   Sub-Cluster Membership UUID: bd0d8f5e-64bd-c331-eb51-000e1e7254b0

   Unicast Mode Enabled: true

   Maintenance Mode State: OFF

   Config Generation: 19cc3529-2f92-4a60-939b-7e3e66dc695c 4 2020-04-09T10:54:37.910

- DanMc85
VCP-DCV 2021
Reply
0 Kudos
TheBobkin
Champion
Champion
Jump to solution

Hello Dan,

Welcome to Communities.

"/vmfs/volumes/vsan:52e8a7a1b473de38-2dc9078322a3e571 - This is active datastore and has all the VM's but no big disk files."

Please explain exactly what you see (screenshots may help) - there should be no "big disk files" anywhere as we don't store vmdk data as -flat.vmdk but as objects.

If via SSH you can see and cd into the namespaces of the VMs and see the vmx there then if you have a test VM, try unregister and re-register it.

"vsanDatastore - Appears to be a shortcut that does not work which I think is pointing to the original old vsan volume. /vmfs/volumes/vsan:52aceefd58033ced-e6d380dbb3a4e95a"

Yes as you have probably made a new 'vsanDatastore' by making a new cluster - this has probably borked the pathing to all the VMs e.g. it is looking for vmx at /vmfs/volumes/vsan:52e8a7a1b473de38-2dc9078322a3e571/VM-NamespaceUUID/VM-Name.vmx but this is not present.

"Is there any way to make these VM's usable again or is all my data lost?"

There are actually likely many ways to get the data usable again, you could probably make the data usable again using the method I outlined here vSAN Expert Challenge # 2 - Update 1  - however if this is Production or in any way important data then please call GSS and don't attempt doing this yourself. I posted that years ago and know a lot more about breaking/fixing things now and when I look at it again, I think it may actually be a case of just changing the current path of the objects (as opposed to making dummy VMs etc.).

The simpler way of fixing this would be if you have an original cluster member (unsure if Witness is enough) and rejoin the hosts to that cluster.

Bob

Reply
0 Kudos
DanMc85
Enthusiast
Enthusiast
Jump to solution

Thanks Bob,

1. I can register the VMs without issue. It shows in the cluster. All the proper settings and such.

However, the Virtual Hard Drive is missing. Shows 0 for size and the VMs won't start.

Says errors such as:

"Some of the disks of the virtual machine ___ failed to load. The information present for them in the virtual machine configuration may be incomplete"

Power On - Operation Failed - "Unable to enumerate all disks."

2. Okay

3. I will take a look at that provided link. Thanks.  I do have the original Witness member... it was a 6.7 Witness Virtual Appliance running on another server. So the old witness is intact.

OPNData.JPG

- DanMc85
VCP-DCV 2021
Reply
0 Kudos
DanMc85
Enthusiast
Enthusiast
Jump to solution

So should I use this tool you think?

./usr/lib/vmware/osfs/bin/objtool setattr

So if I have a VM that is unimportant to me...

example:

/vmfs/volumes/vsan:52e8a7a1b473de38-2dc9078322a3e571/e657c85d-6400-b084-de63-246e96702128

Which is a OPNSense Firewall (I have a .xml config backup of this if it gets hosed)

OPNSense Firewall.vmdk shows:

# Disk DescriptorFile

version=4

encoding="UTF-8"

CID=d6a51678

parentCID=ffffffff

createType="vmfs"

# Extent description

RW 67108864 VMFS "vsan://e957c85d-7038-8d1c-8e99-246e96702128"

# The Disk Data Base

#DDB

ddb.adapterType = "lsilogic"

ddb.deletable = "true"

ddb.geometry.cylinders = "4177"

ddb.geometry.heads = "255"

ddb.geometry.sectors = "63"

ddb.longContentID = "02bd24337d42372eded6fa44d6a51678"

ddb.thinProvisioned = "1"

ddb.toolsInstallType = "0"

ddb.toolsVersion = "2147483647"

ddb.uuid = "60 00 C2 9e 52 a3 69 fd-58 a4 8b e7 63 49 d8 14"

ddb.virtualHWVersion = "4"

Note this was earlier - it looks like the cluster changed the parentCIDs to all f's and also I noticed the path changed. However, this VM used to show the following:

RW 67108864 VSANSPARSE "vsan://a439555e-6e08-833c-e1a0-000e1e7254b0"

Unknown object type Healthy vSAN Default Storage Policy a439555e-6e08-833c-e1a0-000e1e7254b0

matches RW 67108864 VSANSPARSE "vsan://a439555e-6e08-833c-e1a0-000e1e7254b0"

- DanMc85
VCP-DCV 2021
Reply
0 Kudos
DanMc85
Enthusiast
Enthusiast
Jump to solution

If I do a esxcli vsan debug vmdk list

Object: e957c85d-7038-8d1c-8e99-246e96702128

   Health: healthy

   Type: vdisk

   Path: /vmfs/volumes/vsan:52aceefd58033ced-e6d380dbb3a4e95a/e657c85d-6400-b084-de63-246e96702128/OPNSense Firewall.vmdk

   Directory Name: N/A

Object: e657c85d-6400-b084-de63-246e96702128

   Health: healthy

   Type: vmnamespace

   Path: /vmfs/volumes/vsan:52aceefd58033ced-e6d380dbb3a4e95a/OPNSense Firewall

   Directory Name: OPNSense Firewall

Object: a439555e-6e08-833c-e1a0-000e1e7254b0

   Health: healthy

   Type: vdisk

   Path: /vmfs/volumes/vsan:52aceefd58033ced-e6d380dbb3a4e95a/e657c85d-6400-b084-de63-246e96702128/OPNSense Firewall-000001.vmdk

   Directory Name: N/A

etc...

I see the issue.... they all have the old cluster GUID: vsan:52aceefd58033ced-e6d380dbb3a4e95a

Is there a command to batch change the beginning of all these paths from vsan:52aceefd58033ced-e6d380dbb3a4e95a to vsan:52e8a7a1b473de38-2dc9078322a3e571

Or what is the proper command to change each of these manually?

Also when running proper commands should hosts be placed in maintenance mode or not?

Thanks for all your help!!!! Greatly appreciated.

- DanMc85
VCP-DCV 2021
Reply
0 Kudos
DanMc85
Enthusiast
Enthusiast
Jump to solution

I found this article:

VMware Knowledge Base

I wonder if this would work:

Command Syntax: /usr/lib/vmware/osfs/bin/objtool setAttr -u <Object UUID> -d <Path to VMDK>

KB Example: /usr/lib/vmware/osfs/bin/objtool setAttr -u 19b10b5d-48fd-733d-a2f0-005056263753 -d "/vmfs/volumes/vsan:529e00851e70c3b9-215f67dee5e0391b/6fb10b5d-5bd1-f2aa-219a-005056263753/Virtual Machine 1.vmdk"

- DanMc85
VCP-DCV 2021
Reply
0 Kudos
TheBobkin
Champion
Champion
Jump to solution

Hello Dan,

Glad to hear you figured it out but as I said, it is a case of  "just changing the current path of the objects" , fairly trivial to script it for all DOM-Objects as it is a replace of vsanDatastore ID with the other one - could even do it without sed/awk etc. and just do it against a list with replace in notepad++ after enumerating all Objects and their paths.

That being said, putting original cluster back together (or not getting it into this state in the first place) is the preferable option.

Bob

DanMc85
Enthusiast
Enthusiast
Jump to solution

Indeed Bob! Thanks for pointing me in the right direction, that's for sure!

Everything is back 100% now running on the new ESX7 with vCenter 7.

I did notice vCenter 7 has some visual issues with vSAN menus in Chrome under Skyline Health and picking a stretched cluster witness, but works fine in Firefox. Odd.

But will probably be a future bugfix.

Thanks again,

Dan

- DanMc85
VCP-DCV 2021
Reply
0 Kudos
dbray925
Contributor
Contributor
Jump to solution

Hey Dan,

I know it's 2 years later, but what was the next steps you did to get the VM to stop showing as invalid. I fixed the UUID same as you, and it went from "Missing" to "Exists", but after registering the VM, it still shows 0 disk space, and won't power on with error "Unable to enumerate all disks."

Reply
0 Kudos
TheBobkin
Champion
Champion
Jump to solution

@dbray925 You commented that you had "exact same issue" in another thread where the cluster was network partitioned - this thread was relating to a totally different reason for data objects being inaccessible (new cluster and thus OSFS paths broken) - which is it?

Reply
0 Kudos
dbray925
Contributor
Contributor
Jump to solution

As with most issues in technologies....the issue changes as things progress. This thread is closer to that current state I'm in now. The new cluster is up and running, the "/usr/lib/vmware/osfs/bin/objtool setAttr -u" was able to remove the missing from the objects within the vSAN volume, but the last little bit of confusion is getting the VM (without vCenter) back into a registered state. It registers, I can see everything in the ESXi console, but can't power it on.

Getting closer....

Reply
0 Kudos
TheBobkin
Champion
Champion
Jump to solution

@dbray925 Are you sure whatever was done resulted in a new vsanDatastore? This can be confirmed by just using cd to /vmfs/volumes/ and using ls to get the vsan:12345678 ID then validating that is different than the path shown for the objects in 'esxcli vsan debug object list --all' output (e.g. the vsanDatastore ID will be different).

 

If this is the case then this is waaaaay cleaner to resolve by just finding the original Cluster UUID and rejoining that.

 

But anyway, I think I might know what is the last part of your problem in current state - in vSAN 7.0 U1 with the addition of remote vsanDatastore there was a change to paths as listed in the vmdk descriptors themselves - any Objects created in 7.0 U1 or later now have the vsan:12345678/ObjectUUID in them as path where as older Objects only have ObjectUUID - if this is the case, these can be quite easily fixed by just making a short script that uses sed replace in each vmdk descriptor substituting old vsanDatastore ID for new one.

Reply
0 Kudos
dbray925
Contributor
Contributor
Jump to solution

Well, after 2.5 hours on the phone with Support, they are still confused, so I'm thinking it is not as easy as one would hope. The initial article: https://kb.vmware.com/s/article/70774 only changed the vdisk UUID to the new cluster, not the vmnamespace. So they had me run a variation of the above KB article, using "Group UUID". That updated the vmnamespace, but the VM still does not power up, giving the same "Unable to enumerate all disks" error. I really thought they had it....but alas, not even VMware understands VMware anymore 🤣

Waiting for call back, and fingers crossed. Meanwhile, all my VMs are invalid, and the environment is remains offline.

 

Reply
0 Kudos
TheBobkin
Champion
Champion
Jump to solution

@dbray925, "So they had me run a variation of the above KB article, using "Group UUID". That updated the vmnamespace" - specifically what was run and why?
The GUUID of VMs would have remained the same if you changed vsanDatastore (unless something else was changed as well or these were already incorrect).

 

While I am sure I have mentioned most/all of this in the above comments from 2 years ago and/or the links to other similar threads, here is a summary of this issue in a nutshell and what needs to be done to resolve it:

 

A new vsanDatastore (vsan:52593271481c0fe5-71a03ec490475d42) is created where an existing one already exists (vsan:5237a5452da27a37-feba9aab61282291), resulting in the Objects all looking like this in 'esxcli vsan debug object list --all' output:

 

A namespace object (note, depending on build it may not say the VM name in path and show that elsewhere in the entry):
Object UUID: 21089763-12f2-9ad2-ec33-e4434b15e2b0
Path: /vmfs/volumes/vsan:5237a5452da27a37-feba9aab61282291/VM-Name (Missing)

A vmdk object whose descriptor resides in the above namespace:
Object UUID: c2c0a861-3a2b-46e4-3aba-246e9609f308
Path /vmfs/volumes/vsan:5237a5452da27a37-feba9aab61282291/21089763-12f2-9ad2-ec33-e4434b15e2b0/VM-Name.vmdk (Missing)

 

This path would be fixed by updating the path of the objects to change it to the new vsan:xxxxxxxxxx e.g.:

 

(NOTE: if you can already cd into the namespaces, they may not need path update, I don't think I have done this since maybe 7.0 U1/U2 but historically namespaces didn't need this)
# /usr/lib/vmware/osfs/bin/objtool setAttr -u 21089763-12f2-9ad2-ec33-e4434b15e2b0 -d /vmfs/volumes/vsan:52593271481c0fe5-71a03ec490475d42/VM-Name

# /usr/lib/vmware/osfs/bin/objtool setAttr -u c2c0a861-3a2b-46e4-3aba-246e9609f308 -d /vmfs/volumes/vsan:52593271481c0fe5-71a03ec490475d42/21089763-12f2-9ad2-ec33-e4434b15e2b0/VM-Name.vmdk

 


Secondly, as mentioned above, any vmdk Objects created in 7.0 U1 or later also have reference to the vsanDatastore ID and this also needs to be changed:

cat VM-Name.vmdk
# Disk DescriptorFile
version=4
encoding="UTF-8"
CID=fffffffe
parentCID=ffffffff
createType="vmfs"

# Extent description
RW 20971520 VMFS "vsan://5237a5452da27a37-feba9aab61282291/c2c0a861-3a2b-46e4-3aba-246e9609f308" <<<<<<---------

 

If you navigate to a VMs namespace this can be easily replaced like so (though if want to be careful, do this manually or in some other more careful way)
# sed -i -e 's/wrongUUID/correctUUID/' *.vmdk
here being:
# sed -i -e 's/5237a5452da27a37-feba9aab61282291/52593271481c0fe5-71a03ec490475d42/' *.vmdk

 

After doing the above, I would re-register or reload the VM before attempting power-on.

 

Note regarding to anyone reading this in future (or now), while I do work for VMware and specifically vSAN GS, the above is not being provided in any official capacity and using any of the above information or commands is done at your own risk.

Reply
0 Kudos