DanMc85
Enthusiast
Enthusiast

Help w/vSAN - VM Disk Objects shown as Unknown Object

Jump to solution

The following issue is in a ROBO Stretched Cluster connected by 10Gb Crossover Cable in a 2 Node vSAN with Witness.

So a vSphere 7 ESXi w/vSAN upgrade went south.

- ESXi 7 was reinstalled on two host servers with a new ESXi 7 vSAN witness appliance.

- vCenter was still intact and a new Cluster was made. New vmkernel ports. Added to existing vDS.

The problem I am having, is the vsandatastore was recovered. However when I add any VM back to inventory the virtual disks are 0\blank\missing. However, the VSANDATASTORE used space appears that all the data is still there.

VMs added will say errors like:

"Some of the disks of the virtual machine ___ failed to load. The information present for them in the virtual machine configuration may be incomplete"

Power On - Operation Failed - "Unable to enumerate all disks."

When I browse a host in WinSCP, under volumes I see:

/vmfs/volumes/vsan:52e8a7a1b473de38-2dc9078322a3e571 - This is active datastore and has all the VM's but no big disk files.

vsanDatastore - Appears to be a shortcut that does not work which I think is pointing to the original old vsan volume. /vmfs/volumes/vsan:52aceefd58033ced-e6d380dbb3a4e95a

If I open up a VMDK from a VM I see, for example:

# Disk DescriptorFile version=4 encoding="UTF-8" CID=ee6267c9 parentCID=d6a51678 createType="vsanSparse" parentFileNameHint="OPNSense Firewall.vmdk" # Extent description RW 67108864 VSANSPARSE "vsan://a439555e-6e08-833c-e1a0-000e1e7254b0"  # The Disk Data Base #DDB  ddb.longContentID = "e34cf36849d729a254145eb2ee6267c9" ddb.toolsInstallType = "0" ddb.toolsVersion = "2147483647" 

Digging a little deeper and it looks like the disk objects are showing in vSAN as " Unknown object type". There are Unknown object type objects with matching vSAN GUID's mentioned in VM vmdk files. Such as the one mentioned above...

RW 67108864 VSANSPARSE "vsan://a439555e-6e08-833c-e1a0-000e1e7254b0"

Unknown object type Healthy vSAN Default Storage Policy a439555e-6e08-833c-e1a0-000e1e7254b0

esxcli vsan storage list - on Host 1

naa.5000cca258d82813 Device: naa.5000cca258d82813 Display Name: naa.5000cca258d82813 Is SSD: false VSAN UUID: 520645fa-a797-38d4-5e68-22b2171861dc VSAN Disk Group UUID: 520d0603-6e38-a207-1f2b-b6472f57d6f9 VSAN Disk Group Name: naa.55cd2e404c18cb88 Used by this host: true In CMMDS: true On-disk format version: 10 Deduplication: false Compression: false Checksum: 4179814668856239213 Checksum OK: true Is Capacity Tier: true Encryption Metadata Checksum OK: true Encryption: false DiskKeyLoaded: false Is Mounted: true Creation Time: Fri Nov  8 12:45:56 2019  naa.55cd2e404c18cb88 Device: naa.55cd2e404c18cb88 Display Name: naa.55cd2e404c18cb88 Is SSD: true VSAN UUID: 520d0603-6e38-a207-1f2b-b6472f57d6f9 VSAN Disk Group UUID: 520d0603-6e38-a207-1f2b-b6472f57d6f9 VSAN Disk Group Name: naa.55cd2e404c18cb88 Used by this host: true In CMMDS: true On-disk format version: 10 Deduplication: false Compression: false Checksum: 6518366817165333701 Checksum OK: true Is Capacity Tier: false Encryption Metadata Checksum OK: true Encryption: false DiskKeyLoaded: false Is Mounted: true Creation Time: Fri Nov  8 12:45:56 2019  naa.5000cca252cdeadd Device: naa.5000cca252cdeadd Display Name: naa.5000cca252cdeadd Is SSD: false VSAN UUID: 5242f42c-474e-1c1a-9016-927431e9713d VSAN Disk Group UUID: 52782092-2deb-db53-aaa1-cc71b1679b43 VSAN Disk Group Name: naa.55cd2e404c2055aa Used by this host: true In CMMDS: true On-disk format version: 10 Deduplication: false Compression: false Checksum: 12173014117720582296 Checksum OK: true Is Capacity Tier: true Encryption Metadata Checksum OK: true Encryption: false DiskKeyLoaded: false Is Mounted: true Creation Time: Tue Jan 21 12:40:33 2020  naa.55cd2e404c2055aa Device: naa.55cd2e404c2055aa Display Name: naa.55cd2e404c2055aa Is SSD: true VSAN UUID: 52782092-2deb-db53-aaa1-cc71b1679b43 VSAN Disk Group UUID: 52782092-2deb-db53-aaa1-cc71b1679b43 VSAN Disk Group Name: naa.55cd2e404c2055aa Used by this host: true In CMMDS: true On-disk format version: 10 Deduplication: false Compression: false Checksum: 5367773553359825337 Checksum OK: true Is Capacity Tier: false Encryption Metadata Checksum OK: true Encryption: false DiskKeyLoaded: false Is Mounted: true Creation Time: Fri Nov  8 12:45:45 2019  naa.5000cca23b7ee4e4 Device: naa.5000cca23b7ee4e4 Display Name: naa.5000cca23b7ee4e4 Is SSD: false VSAN UUID: 52e46a8d-cdb8-1159-38d5-cfbde318e15f VSAN Disk Group UUID: 520d0603-6e38-a207-1f2b-b6472f57d6f9 VSAN Disk Group Name: naa.55cd2e404c18cb88 Used by this host: true In CMMDS: true On-disk format version: 10 Deduplication: false Compression: false Checksum: 4127245025532564424 Checksum OK: true Is Capacity Tier: true Encryption Metadata Checksum OK: true Encryption: false DiskKeyLoaded: false Is Mounted: true Creation Time: Fri Nov  8 12:45:56 2019  naa.5000cca23b834064 Device: naa.5000cca23b834064 Display Name: naa.5000cca23b834064 Is SSD: false VSAN UUID: 52f4d6fa-0c8f-603d-453c-3da1646496ce VSAN Disk Group UUID: 52782092-2deb-db53-aaa1-cc71b1679b43 VSAN Disk Group Name: naa.55cd2e404c2055aa Used by this host: true In CMMDS: true On-disk format version: 10 Deduplication: false Compression: false Checksum: 6169125189223970042 Checksum OK: true Is Capacity Tier: true Encryption Metadata Checksum OK: true Encryption: false DiskKeyLoaded: false Is Mounted: true Creation Time: Fri Nov  8 12:45:45 2019 

esxcli vsan storage list - on Host 2

naa.5000cca252cc5f25 Device: naa.5000cca252cc5f25 Display Name: naa.5000cca252cc5f25 Is SSD: false VSAN UUID: 5234687a-c379-e17e-a53d-4cf9e9fc8e7c VSAN Disk Group UUID: 52be0b83-8590-9e78-3d27-ad7926eb9ad0 VSAN Disk Group Name: naa.55cd2e404c18cc1f Used by this host: true In CMMDS: true On-disk format version: 10 Deduplication: false Compression: false Checksum: 5370575494238580185 Checksum OK: true Is Capacity Tier: true Encryption Metadata Checksum OK: true Encryption: false DiskKeyLoaded: false Is Mounted: true Creation Time: Wed Jan 22 02:29:14 2020  naa.55cd2e404c2055af Device: naa.55cd2e404c2055af Display Name: naa.55cd2e404c2055af Is SSD: true VSAN UUID: 524971da-1c92-f80e-9199-0fa74a253354 VSAN Disk Group UUID: 524971da-1c92-f80e-9199-0fa74a253354 VSAN Disk Group Name: naa.55cd2e404c2055af Used by this host: true In CMMDS: true On-disk format version: 10 Deduplication: false Compression: false Checksum: 18423783201084117590 Checksum OK: true Is Capacity Tier: false Encryption Metadata Checksum OK: true Encryption: false DiskKeyLoaded: false Is Mounted: true Creation Time: Fri Nov  8 12:46:06 2019  naa.5000cca23b437b3c Device: naa.5000cca23b437b3c Display Name: naa.5000cca23b437b3c Is SSD: false VSAN UUID: 5255b6bb-5de1-49c7-67af-aee865cde07a VSAN Disk Group UUID: 524971da-1c92-f80e-9199-0fa74a253354 VSAN Disk Group Name: naa.55cd2e404c2055af Used by this host: true In CMMDS: true On-disk format version: 10 Deduplication: false Compression: false Checksum: 7220870063380886230 Checksum OK: true Is Capacity Tier: true Encryption Metadata Checksum OK: true Encryption: false DiskKeyLoaded: false Is Mounted: true Creation Time: Thu Jan 16 03:16:23 2020  naa.55cd2e404c18cc1f Device: naa.55cd2e404c18cc1f Display Name: naa.55cd2e404c18cc1f Is SSD: true VSAN UUID: 52be0b83-8590-9e78-3d27-ad7926eb9ad0 VSAN Disk Group UUID: 52be0b83-8590-9e78-3d27-ad7926eb9ad0 VSAN Disk Group Name: naa.55cd2e404c18cc1f Used by this host: true In CMMDS: true On-disk format version: 10 Deduplication: false Compression: false Checksum: 9858777924305020986 Checksum OK: true Is Capacity Tier: false Encryption Metadata Checksum OK: true Encryption: false DiskKeyLoaded: false Is Mounted: true Creation Time: Sun Jan 19 03:42:32 2020  naa.5000cca258d81d2c Device: naa.5000cca258d81d2c Display Name: naa.5000cca258d81d2c Is SSD: false VSAN UUID: 52c41e3e-e943-1a1a-d001-5fcb6de4997b VSAN Disk Group UUID: 524971da-1c92-f80e-9199-0fa74a253354 VSAN Disk Group Name: naa.55cd2e404c2055af Used by this host: true In CMMDS: true On-disk format version: 10 Deduplication: false Compression: false Checksum: 3826993888305396649 Checksum OK: true Is Capacity Tier: true Encryption Metadata Checksum OK: true Encryption: false DiskKeyLoaded: false Is Mounted: true Creation Time: Tue Jan 21 12:42:14 2020  naa.5000cca23b0bce74 Device: naa.5000cca23b0bce74 Display Name: naa.5000cca23b0bce74 Is SSD: false VSAN UUID: 52cb0133-f1c4-b3e1-dd47-fd9f5a786437 VSAN Disk Group UUID: 52be0b83-8590-9e78-3d27-ad7926eb9ad0 VSAN Disk Group Name: naa.55cd2e404c18cc1f Used by this host: true In CMMDS: true On-disk format version: 10 Deduplication: false Compression: false Checksum: 4710347119789749474 Checksum OK: true Is Capacity Tier: true Encryption Metadata Checksum OK: true Encryption: false DiskKeyLoaded: false Is Mounted: true Creation Time: Sun Jan 19 03:42:32 2020 

Is there any way to make these VM's usable again or is all my data lost?

Example: In picture below - this highlighted unknown object is the parent disk vmdk to one of my VMs

vSAN.JPG

1 Solution

Accepted Solutions
DanMc85
Enthusiast
Enthusiast

I found this article:

VMware Knowledge Base

I wonder if this would work:

Command Syntax: /usr/lib/vmware/osfs/bin/objtool setAttr -u <Object UUID> -d <Path to VMDK>

KB Example: /usr/lib/vmware/osfs/bin/objtool setAttr -u 19b10b5d-48fd-733d-a2f0-005056263753 -d "/vmfs/volumes/vsan:529e00851e70c3b9-215f67dee5e0391b/6fb10b5d-5bd1-f2aa-219a-005056263753/Virtual Machine 1.vmdk"

View solution in original post

0 Kudos
9 Replies
GreatWhiteTec
VMware Employee
VMware Employee

A few things going on.... House keeping first.

#1 Sounds to me that the vCenter you are using is still pre version 7, correct? If so, I would start with getting vCenter to same version.

#2 Rebuilding the hosts without cleaning them up or proper decommission preserves DGs/vSAN partitions

#3 New vSAN Witness = lost that info - lost 1/3 of the cluster

#4 Hosts could have still been part of a vSAN cluster (depending on actions during the "reinstall"). Did you verify status of hosts before adding to new cluster?

esxcli vsan cluster get

A+, DCSE, MCP, MCSA, MCSE, MCTS, MCITP, MCDBA, NCDA, NCIE-SAN, NCIE-BR, VCP4, VCP5, VCP5-DT, VCAP5-DCA _____________________ If you find this or any other answer useful please consider awarding points by marking the answer correct or helpful.
0 Kudos
DanMc85
Enthusiast
Enthusiast

1. This is same vCenter which was 6.7 and was upgraded to version 7.0.  Was on a different host, not in the vSAN cluster so it was not affected by this issue.

2. vSAN partitions were preserved. I can access every file such as the VMs VMX, logs, etc... just the Metadata for the virtual disks are all showing as unknown object type in vSAN.

- Basically on the Dell R730xd's the SD Card's were clean installed with ESXi 7.0 after things went south from the 6.7 to 7.0 upgrade on the same SD cards.

3. I still have the old vSAN witness VM if that matters. Can it be rejoined into the "new" vSAN cluster?  Should I use CLI and have it leave the "old" cluster?

4. If I recall correctly, hosts did not show any cluster info after reinstall.

I started going by this guide that others had luck with when this all happened:

https://www.virtuallyghetto.com/2014/07/does-reinstalling-esxi-with-an-existing-vsan-datastore-wipe-...

This is the current cluster info:

Cluster Information

   Enabled: true

   Current Local Time: 2020-04-09T12:29:52Z

   Local Node UUID: 5e8e60a5-e9ed-dee6-e1d2-000e1e73da60

   Local Node Type: NORMAL

   Local Node State: BACKUP

   Local Node Health State: HEALTHY

   Sub-Cluster Master UUID: 5e8e650e-4186-d7dc-e787-000e1e7254b0

   Sub-Cluster Backup UUID: 5e8e60a5-e9ed-dee6-e1d2-000e1e73da60

   Sub-Cluster UUID: 52e8a7a1-b473-de38-2dc9-078322a3e571

   Sub-Cluster Membership Entry Revision: 1

   Sub-Cluster Member Count: 3

   Sub-Cluster Member UUIDs: 5e8e650e-4186-d7dc-e787-000e1e7254b0, 5e8e60a5-e9ed-dee6-e1d2-000e1e73da60, 5e8df455-1969-7774-13f3-005056866422

   Sub-Cluster Member HostNames: ESXi-Primary, ESXi-Secondary, vSANWitness7

   Sub-Cluster Membership UUID: bd0d8f5e-64bd-c331-eb51-000e1e7254b0

   Unicast Mode Enabled: true

   Maintenance Mode State: OFF

   Config Generation: 19cc3529-2f92-4a60-939b-7e3e66dc695c 4 2020-04-09T10:54:37.910

0 Kudos
TheBobkin
VMware Employee
VMware Employee

Hello Dan,

Welcome to Communities.

"/vmfs/volumes/vsan:52e8a7a1b473de38-2dc9078322a3e571 - This is active datastore and has all the VM's but no big disk files."

Please explain exactly what you see (screenshots may help) - there should be no "big disk files" anywhere as we don't store vmdk data as -flat.vmdk but as objects.

If via SSH you can see and cd into the namespaces of the VMs and see the vmx there then if you have a test VM, try unregister and re-register it.

"vsanDatastore - Appears to be a shortcut that does not work which I think is pointing to the original old vsan volume. /vmfs/volumes/vsan:52aceefd58033ced-e6d380dbb3a4e95a"

Yes as you have probably made a new 'vsanDatastore' by making a new cluster - this has probably borked the pathing to all the VMs e.g. it is looking for vmx at /vmfs/volumes/vsan:52e8a7a1b473de38-2dc9078322a3e571/VM-NamespaceUUID/VM-Name.vmx but this is not present.

"Is there any way to make these VM's usable again or is all my data lost?"

There are actually likely many ways to get the data usable again, you could probably make the data usable again using the method I outlined here vSAN Expert Challenge # 2 - Update 1  - however if this is Production or in any way important data then please call GSS and don't attempt doing this yourself. I posted that years ago and know a lot more about breaking/fixing things now and when I look at it again, I think it may actually be a case of just changing the current path of the objects (as opposed to making dummy VMs etc.).

The simpler way of fixing this would be if you have an original cluster member (unsure if Witness is enough) and rejoin the hosts to that cluster.

Bob

0 Kudos
DanMc85
Enthusiast
Enthusiast

Thanks Bob,

1. I can register the VMs without issue. It shows in the cluster. All the proper settings and such.

However, the Virtual Hard Drive is missing. Shows 0 for size and the VMs won't start.

Says errors such as:

"Some of the disks of the virtual machine ___ failed to load. The information present for them in the virtual machine configuration may be incomplete"

Power On - Operation Failed - "Unable to enumerate all disks."

2. Okay

3. I will take a look at that provided link. Thanks.  I do have the original Witness member... it was a 6.7 Witness Virtual Appliance running on another server. So the old witness is intact.

OPNData.JPG

0 Kudos
DanMc85
Enthusiast
Enthusiast

So should I use this tool you think?

./usr/lib/vmware/osfs/bin/objtool setattr

So if I have a VM that is unimportant to me...

example:

/vmfs/volumes/vsan:52e8a7a1b473de38-2dc9078322a3e571/e657c85d-6400-b084-de63-246e96702128

Which is a OPNSense Firewall (I have a .xml config backup of this if it gets hosed)

OPNSense Firewall.vmdk shows:

# Disk DescriptorFile

version=4

encoding="UTF-8"

CID=d6a51678

parentCID=ffffffff

createType="vmfs"

# Extent description

RW 67108864 VMFS "vsan://e957c85d-7038-8d1c-8e99-246e96702128"

# The Disk Data Base

#DDB

ddb.adapterType = "lsilogic"

ddb.deletable = "true"

ddb.geometry.cylinders = "4177"

ddb.geometry.heads = "255"

ddb.geometry.sectors = "63"

ddb.longContentID = "02bd24337d42372eded6fa44d6a51678"

ddb.thinProvisioned = "1"

ddb.toolsInstallType = "0"

ddb.toolsVersion = "2147483647"

ddb.uuid = "60 00 C2 9e 52 a3 69 fd-58 a4 8b e7 63 49 d8 14"

ddb.virtualHWVersion = "4"

Note this was earlier - it looks like the cluster changed the parentCIDs to all f's and also I noticed the path changed. However, this VM used to show the following:

RW 67108864 VSANSPARSE "vsan://a439555e-6e08-833c-e1a0-000e1e7254b0"

Unknown object type Healthy vSAN Default Storage Policy a439555e-6e08-833c-e1a0-000e1e7254b0

matches RW 67108864 VSANSPARSE "vsan://a439555e-6e08-833c-e1a0-000e1e7254b0"

0 Kudos
DanMc85
Enthusiast
Enthusiast

If I do a esxcli vsan debug vmdk list

Object: e957c85d-7038-8d1c-8e99-246e96702128

   Health: healthy

   Type: vdisk

   Path: /vmfs/volumes/vsan:52aceefd58033ced-e6d380dbb3a4e95a/e657c85d-6400-b084-de63-246e96702128/OPNSense Firewall.vmdk

   Directory Name: N/A

Object: e657c85d-6400-b084-de63-246e96702128

   Health: healthy

   Type: vmnamespace

   Path: /vmfs/volumes/vsan:52aceefd58033ced-e6d380dbb3a4e95a/OPNSense Firewall

   Directory Name: OPNSense Firewall

Object: a439555e-6e08-833c-e1a0-000e1e7254b0

   Health: healthy

   Type: vdisk

   Path: /vmfs/volumes/vsan:52aceefd58033ced-e6d380dbb3a4e95a/e657c85d-6400-b084-de63-246e96702128/OPNSense Firewall-000001.vmdk

   Directory Name: N/A

etc...

I see the issue.... they all have the old cluster GUID: vsan:52aceefd58033ced-e6d380dbb3a4e95a

Is there a command to batch change the beginning of all these paths from vsan:52aceefd58033ced-e6d380dbb3a4e95a to vsan:52e8a7a1b473de38-2dc9078322a3e571

Or what is the proper command to change each of these manually?

Also when running proper commands should hosts be placed in maintenance mode or not?

Thanks for all your help!!!! Greatly appreciated.

0 Kudos
DanMc85
Enthusiast
Enthusiast

I found this article:

VMware Knowledge Base

I wonder if this would work:

Command Syntax: /usr/lib/vmware/osfs/bin/objtool setAttr -u <Object UUID> -d <Path to VMDK>

KB Example: /usr/lib/vmware/osfs/bin/objtool setAttr -u 19b10b5d-48fd-733d-a2f0-005056263753 -d "/vmfs/volumes/vsan:529e00851e70c3b9-215f67dee5e0391b/6fb10b5d-5bd1-f2aa-219a-005056263753/Virtual Machine 1.vmdk"

View solution in original post

0 Kudos
TheBobkin
VMware Employee
VMware Employee

Hello Dan,

Glad to hear you figured it out but as I said, it is a case of  "just changing the current path of the objects" , fairly trivial to script it for all DOM-Objects as it is a replace of vsanDatastore ID with the other one - could even do it without sed/awk etc. and just do it against a list with replace in notepad++ after enumerating all Objects and their paths.

That being said, putting original cluster back together (or not getting it into this state in the first place) is the preferable option.

Bob

DanMc85
Enthusiast
Enthusiast

Indeed Bob! Thanks for pointing me in the right direction, that's for sure!

Everything is back 100% now running on the new ESX7 with vCenter 7.

I did notice vCenter 7 has some visual issues with vSAN menus in Chrome under Skyline Health and picking a stretched cluster witness, but works fine in Firefox. Odd.

But will probably be a future bugfix.

Thanks again,

Dan

0 Kudos