VSAN Issue

[MJohnsnPhx]

Earlier this week our VM infrastructure burped for lack of a better word. As we brought everything back up we noticed that our storage was only a 1/3rd of what it should be. Looking into it we see that all 3 of our hosts are in the VSAN cluster, but only the disks of one are being read. The other two are showing connected but when we do a 'esxcli vsan storage list' we noticed that all of the disks on the two hosts are showing as 'In CMMDS: false'.

So we have 3 hosts, our fourth is ready to be added once we are done figuring this out. 1 has all of its drives being read, the other two are listed as connected, but drives are not being read.

Hoping anyone has a suggestion on what we can look into to bring the drives back in. Thanks in advance.

[admin]

Greetings!

As a quick next step, I would suggest you to go through the steps mentioned in the below article:

VSAN Part 32 - Datastore capacity not adding up - CormacHogan.com‌

Hope this helps you to resolve this issue.

_________________________

Was your question answered correctly? If so, please remember to mark your question as answered when you get the correct answer and award points to the person providing the answer. This helps others searching for a similar issue.

Cheers!

-Shivam

[vpradeep01]

Hello,

There are multiple reasons as why VSAN will unmount a disk from the DG, but it is possible that VSAN has proactively unmounted a disk from the DG in response to environmental conditions.

They are:

-Latency at the SSD/Flash drives. If the latency is at the flash, then the entire Disks (Flash+Capacity tier drives)  in the disk group set will be mounted,

-Latency at the Capacity tier drives

-You may also events such as "congestion" on the affected host in the vmkernel. ( Not always )

- IO failures on the flash drives.

The only way the cmmds can mount the affected drives is by rebooting the affected host if the manual mount fails.

The best mitigate this is to upgrade the firmware on the SSD + MD drives and drivers + controllers on Storage adapters once the host is up.

Note:

If congestion is reported, please report to GSS as they have to increase the certain advanced parameters to allow the the hosts to be return stable,

[MJohnsnPhx]

When looking in Vsphere it shows the drives are mounted as shown below. But when I log into the ESXI shell, it shows that the drives are not ready for VSAN because they are not mounted.

[vpradeep01]

Hello,

Good day !

1. The "Mounted" status which you see in the vSAN Cluster -> Manage -> Disk management -> Disk Group-> "Operational status" is actually the physical status of the disk.

2. The CMMDS: FALSE indicates the vSAN has unclaimed the disk temporarily from the vSAN datastore due to the above mentioned issues. You can also see this in the same screenshot which you have pasted here.

vSAN Cluster -> Manage -> Disk management -> Disk Group-> vSAN health Status

A working/claimed/mounted disk should be listed here as "Healthy"

-Also when you run df -h on the affected host, you will see reduction in the actual size of the vSAN datastore

3. Re-mount or force mount should allow the disks to be mounted back to the hosts else reboot should get them online.

I hope this answers your question.

Cheers,

Pradeep Venkatesh

[MJohnsnPhx]

A reboot doesn't bring em online, and try to mount the disk gives us the error now a VSAN disk.

[darcidinovmw]

I would say that at this point the troubleshooting steps and speed of the communities is not in your best interests. I would recommend opening a ticket with GSS in order to try and get a better idea of what's going on.