topic Re: VSAN problem after crash one cache drive in VMware vSAN Discussions

VSAN problem after crash one cache drive

jynx_km — Mon, 30 Oct 2023 00:34:59 GMT

Hi.

I have 4 node ( ftt=1 ) cluster. After damage vsan doesn't rebuild object. I had one of object ABSENT. After delete inaccessible object I can't power on VM. There is problem with vmdk. How to recover it? On power there is file not found

/localhost/dc1/computers/cls01/resourcePool/vms> vsan.vm_object_info Vmxxxxx

2023-10-29 23:59:52 +0000: Fetching vSAN disk info from 192.168.2.152 (may take a moment) ...

2023-10-29 23:59:52 +0000: Fetching vSAN disk info from 192.168.2.153 (may take a moment) ...

2023-10-29 23:59:52 +0000: Fetching vSAN disk info from 192.168.2.151 (may take a moment) ...

2023-10-29 23:59:52 +0000: Fetching vSAN disk info from 192.168.2.176 (may take a moment) ...

2023-10-29 23:59:52 +0000: Fetching vSAN disk info from 192.168.2.160 (may take a moment) ...

2023-10-29 23:59:53 +0000: Done fetching vSAN disk infos

VM VMxxxx:

VM has a non-vSAN datastore

Disk backing:

[cls01-vsan01] 23381365-0851-f719-0c77-e43d1a2ad266/VMxxxx-000001.vmdk

[cls01-vsan01] 23381365-0851-f719-0c77-e43d1a2ad266/VMxxxx.vmdk

[cls01-vsan01] 23381365-0851-f719-0c77-e43d1a2ad266/VMxxxx.vmx

DOM Object: 23381365-0851-f719-0c77-e43d1a2ad266 (v11, owner: 192.168.2.160, proxy owner: None, policy: stripeWidth = 1, cacheReservation = 0, proportionalCapacity = [0, 100], hostFailuresToTolerate = 1, forceProvisioning = 0, spbmProfileId = aa6d5a82-1c88-45da-85d3-3d74b91a5bad, spbmProfileGenerationNumber = 0, CSN = 269, SCSN = 263, spbmProfileName = vSAN Default Storage Policy)

RAID_1

Component: c6673e65-9ab4-a26c-bc5f-b8ca3a694c4c (state: ACTIVE (5), host: 192.168.2.151, capacity: naa.5002538f41a5b9f5, cache: t10.NVMe____Samsung_SSD_980_1TB_____________________2446B111D5382500,

votes: 1, usage: 0.6 GB, proxy component: false)

Component: a6793e65-7c2c-bca7-25aa-e43d1a2ad266 (state: ACTIVE (5), host: 192.168.2.153, capacity: naa.600508b1001cc6658131bdaa6b99e85f, cache: naa.600508b1001cb55d75f95b40596da2ec,

votes: 1, usage: 0.6 GB, proxy component: false)

Witness: e7e03e65-c625-919d-ec18-e43d1a2ad266 (state: ACTIVE (5), host: 192.168.2.160, capacity: naa.6d4ae520a53f52002b76d7671f806885, cache: t10.NVMe____Samsung_SSD_960_EVO_250GB_______________C80EB38153382500,

votes: 1, usage: 0.0 GB, proxy component: false)

[cls01-vsan01] 23381365-0851-f719-0c77-e43d1a2ad266/VMxxxx-000001.vmdk

DOM Object: 2b321b65-f40d-0c96-5313-e43d1a2ad266 (v11, owner: 192.168.2.153, proxy owner: None, policy: stripeWidth = 1, cacheReservation = 0, proportionalCapacity = [0, 100], hostFailuresToTolerate = 1, forceProvisioning = 0, spbmProfileId = aa6d5a82-1c88-45da-85d3-3d74b91a5bad, spbmProfileGenerationNumber = 0, CSN = 243, SCSN = 232, spbmProfileName = vSAN Default Storage Policy)

RAID_1

Component: 7ded3d65-febc-1ad0-b794-b8ca3a694c4c (state: ACTIVE (5), host: 192.168.2.153, capacity: naa.5002538a06cf4d60, cache: naa.600508b1001cb55d75f95b40596da2ec,

votes: 1, usage: 64.8 GB, proxy component: false)

Component: b8793e65-d23c-01bb-4b87-90e2ba16a33c (state: ACTIVE (5), host: 192.168.2.151, capacity: naa.5002538f3364b839, cache: naa.5002538f41a5ba07,

votes: 1, usage: 64.8 GB, proxy component: false)

Witness: fc983d65-f7b6-82b2-41a6-b8ca3a694c4c (state: ACTIVE (5), host: 192.168.2.152, capacity: naa.5002538e90b64be7, cache: t10.NVMe____Corsair_MP400___________________________E61B20C04BA77964,

votes: 1, usage: 0.0 GB, proxy component: false)

[cls01-vsan01] 23381365-0851-f719-0c77-e43d1a2ad266/VMxxxx.vmdk

Couldn't find info about DOM object '24381365-729a-538f-f3cf-e43d1a2ad266'

--> this is doesn't exists: '24381365-729a-538f-f3cf-e43d1a2ad266'

files in folder of vm:

-rw------- 1 root root 0 Oct 29 03:18 .2b321b65-f40d-0c96-5313-e43d1a2ad266.lck

drwxr-xr-x 1 root root 560 Oct 29 23:57 .dvsData

-r-------- 1 root root 1441792 Sep 26 19:59 .fbb.sf

-r-------- 1 root root 267026432 Sep 26 19:59 .fdc.sf

-rw------- 1 root root 0 Oct 2 21:12 .old.2b321b65-f40d-0c96-5313-e43d1a2ad266.lck.old

-r-------- 1 root root 1179648 Sep 26 19:59 .pb2.sf

-r-------- 1 root root 268435456 Sep 26 19:59 .pbc.sf

-r-------- 1 root root 262733824 Sep 26 19:59 .sbc.sf

drwx------ 1 root root 280 Sep 26 19:59 .sdd.sf

-r-------- 1 root root 4194304 Sep 26 19:59 .vh.sf

-rw------- 1 root root 325 Oct 20 11:58 VMxxxxx-000001.vmdk

-rw-r--r-- 1 root root 1625 Oct 29 23:23 VMxxxxx-062a49d7.hlog

-rw------- 1 root root 295336 Oct 2 21:12 VMxxxxx-Snapshot1.vmsn

-rw------- 1 root root 270840 Oct 23 08:05 VMxxxxx.nvram

-rw------- 1 root root 555 Sep 26 20:21 VMxxxxx.vmdk

-rw-r--r-- 1 root root 485 Oct 2 21:12 VMxxxxx.vmsd

-rwxr-xr-x 1 root root 3808 Oct 29 23:57 VMxxxxx.vmx

-rw------- 1 root root 3932 Sep 27 21:27 VMxxxxx.vmxf

-rw------- 1 root root 555 Oct 30 00:03 old.vmdk

-rw-r--r-- 1 root root 37261 Oct 19 08:11 vmware-58.log

-rw-r--r-- 1 root root 37262 Oct 19 08:14 vmware-59.log

-rw-r--r-- 1 root root 37262 Oct 19 08:17 vmware-60.log

-rw-r--r-- 1 root root 37349 Oct 19 08:20 vmware-61.log

-rw-r--r-- 1 root root 37262 Oct 19 08:23 vmware-62.log

-rw-r--r-- 1 root root 328691 Oct 20 11:58 vmware-63.log

-rw-r--r-- 1 root root 242190 Oct 23 08:05 vmware.log

Re: VSAN problem after crash one cache drive

depping — Mon, 30 Oct 2023 09:50:08 GMT

I would recommend not touching the the environment and contacting support ASAP. Either that, or restore from backup.

Re: VSAN problem after crash one cache drive

TheBobkin — Mon, 30 Oct 2023 14:16:09 GMT

@jynx_km, Is this a homelab? If not then why are you using unsupported consumer-grade disks?

Can you cat the file VMxxxxx.vmdk to indicate what object is backing the base-disk?

What was the UUID of the object you deleted and why did you delete that?

Re: VSAN problem after crash one cache drive

jynx_km — Mon, 30 Oct 2023 14:59:18 GMT

# Disk DescriptorFile

version=4

encoding="UTF-8"

CID=8714a425

parentCID=ffffffff

createType="vmfs"

# Extent description

RW 503316480 VMFS "vsan://24381365-729a-538f-f3cf-e43d1a2ad266"

# The Disk Data Base

#DDB

ddb.adapterType = "lsilogic"

ddb.geometry.cylinders = "31330"

ddb.geometry.heads = "255"

ddb.geometry.sectors = "63"

ddb.longContentID = "c7e852a3a673d0f1b3d6d1838714a425"

ddb.thinProvisioned = "1"

ddb.toolsInstallType = "1"

ddb.toolsVersion = "11333"

ddb.uuid = "60 00 C2 94 7e 0c bf ea-63 dc 16 9d 10 f1 42 60"

ddb.virtualHWVersion = "14"

i delete:

This disk was absent ( damaged )

Component State: ABSENT, Address Space(B): 29870784512 (27.82GB), Disk UUID: 52732347-62a1-4604-b2c9-e1d13d7258c2

Re: VSAN problem after crash one cache drive

TheBobkin — Mon, 30 Oct 2023 15:47:31 GMT

@jynx_km What you referenced there is a component, not an object - did you delete object with UUID 24381365-729a-538f-f3cf-e43d1a2ad266?

If yes, then this gone and you need to restore from backups.

Re: VSAN problem after crash one cache drive

jynx_km — Mon, 30 Oct 2023 17:00:50 GMT

Ok. I understand - but this object was inaccessible. I couldn't recover it. Is there any way to get file with snapshot? Copy or move to other place?

I have more that objects which are inaccessible, and I can't do anything for them:

Detected 9 objects to be inaccessible

Detected ce5ee760-8414-7819-071f-e43d1a2ad266 on 192.168..x.x to be inaccessible

Detected 0575e760-b0c2-95c1-5a08-e43d1a2b31be on 192.168..x.x to be inaccessible

Detected a9b21c64-02a3-7701-e0b1-e43d1a2b31be on 192.168.x.x to be inaccessible

Detected b0730162-08ff-5f4d-5bce-f4ce46aa0cb4 on 192.168.x.xto be inaccessible

Detected c59c1e65-5423-7653-8a91-90e2ba16a33c on 192.168.x.x to be inaccessible

Detected c6fb9963-baf7-b10d-dade-f4ce46aa0cb4 on 192.168..x.x to be inaccessible

Detected dbfb9963-a87f-501f-acb6-f4ce46aa0cb4 on 192.168.x.x to be inaccessible

Detected fbb24761-a087-51d8-2c5e-f4ce46aa0cb4 on 192.168.x.x to be inaccessible

Detected 75bcb964-3af5-65e7-7f3e-90e2ba16a33c on 192.168.x.x to be inaccessible