VSAN VMDKs Blank

[Dr_Virt]

Had to do a full shutdown of vSAN cluster. Followed KB to enter maintenance mode after shutting down all vms with no data migration. 

IN restart about 1/3 of the VMs show the VMDKS as 0.66KB and receive error that VM cannot be started because VMDK cannot be enumerated. 

Is there a way to recover the VMDKs? Is there a scan VSAN function to find missing data? Anything?

[TheBobkin]

@Dr_Virt , Specifically which kb article? If it was one with a shutdown script then please validate that you ran the post-shutdown recovery part of the script to reform the cluster etc. .
Is vCenter up? If so then check: Cluster > Monitor > vSAN > Health > Retest, note any failed tests and validate that all nodes in the cluster are returning info for test results, please share details of any non-green health checks.
Potentially some nodes in the cluster are partitioned or still in Maintenance Mode (either at the vSAN-level or this + vSphere-level).
Just so that you are aware - vSAN doesn't store vmdk data as -flat files, it stores them as Objects and thus it is entirely expected behaviour for vmdks to be just tiny descriptor files (as they don't contain data, just pointers and metadata).

[Dr_Virt]

Worked with VMware support. Found that for some reason the VSAN ID changed. 

Had to manually go back and update all of the VSAN object paths to the new ID. 

Successfully recovered all VMDKs.

[TheBobkin]

@Dr_Virt, This typically occurs if for instance one makes a new vSAN cluster on a node and joins the other nodes to it so the path is going to be wrong as it will have a new vsanDatastore ID - this is generally fixable by easier means than changing the path of every Object (but then again this is relatively easy to script) such as just rejoining all nodes back to the original cluster.
This has come up previously here:
https://communities.vmware.com/t5/VMware-vSAN-Discussions/I-have-lost-my-vSAN-Could-you-help/td-p/51...

[Dr_Virt]

I did script it. 

We were rebuilding the cluster. Took Host1 out built new cluster. Moved Host2-Host4 into new cluster. 

Last time I did this (vCenter died), I built new vCenter, new Cluster, imported all hosts, no issues. 

[TheBobkin]

@Dr_Virt, what you did this time around vs creating a new cluster and adding the hosts (basically importing the cluster) are not the same thing though - if you leave cluster on one host and then add it to a new cluster (or create a new one by enabling vSAN) then this is a new cluster with a new vsanDatastore and thus the issue.

In later builds (7.0 only IIRC, don't think was backported to 6.7) a guardrail has been implemented that will prevent adding hosts 2-4 to the cluster in this state with error message stating that the clusters UUIDs don't match (as they are basically 2 clusters).