High LAT/Rd and LAT/Wr but no latency on disks for vSAN

[alienjoker]

Hi,

I have been running vSAN 5.5U2 for circa 6 months in a lab configuration with no issues, but over the weekend, noticed everything had come to an abrupt halt. I restarted all three nodes (I use the same MacMini configuration as William Lam) and expected things to pick up where they left off - Note there were no changes to the setup at all.

The VMs began to power back on but as soon as I started to try and use them, noticed extremely poor performance.

A quick esxtop showed no signs of problems with the disks (D/AVG etc) but the VM latency was in the high hundreds. I started shutting down the few VMs I have running (about 10) and left nothing other than the VCSA up.

ADAPTR PATH	NPTH   CMDS/s  READS/s WRITES/s MBREAD/s MBWRTN/s DAVG/cmd KAVG/cmd GAVG/cmd QAVG/cmd
vmhba0 -	1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
vmhba32 -	1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
vmhba33 -	1 1.46 0.49 0.97 0.00 0.00 0.62 0.04 0.65 0.01
vmhba34 -	0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
vmhba35 -	0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
vmhba36 -	0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
vmhba37 -	0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
vmhba38 -	0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

GID VMNAME	VDEVNAME NVDISK   CMDS/s  READS/s WRITES/s MBREAD/s MBWRTN/s LAT/rd LAT/wr
25112 LAB-VCA-V001	- 2 3.36 0.00 3.36 0.00 0.02   0.00 1310.38

Eventually the VCSA ground to a complete halt to the point where eventually I couldn't even manage my cluster unless I was at the CLI on each of the hosts. I was still able to SSH to the VCSA to run some Ruby commands where the vsan health status came back "OK".

I checked the Network ports to make sure a crazy Sync wasn't underway as a consequence of the failure and that my network wasn't saturated (GB between each of the hosts) but the network stats were low and vsan health reports suggested there were no outstanding syncs.

Has anyone got any ideas where the Latency in this particular instance is coming from? VMkernel for example? How can I check this?

Update Edit:

Ok, so we're now in a slightly worse place than we were last night.

Having rebooted all three hosts, the vSAN appears to be stable again, HOWEVER, when I list the contents of the vsanDatastore, I'm missing 60% of my VMs (including the VCSA). Any tips for attempting to bring them back - like I say, its only homelab, but the troubleshooting bit is all part of the fun. Capacity wise, the datastore is reporting the same consumption before the failure so I imagine the data is there, I just can't see it!

[alienjoker]

So, further digging (after building a new VCSA so I could access RVS), a vsan.object_info against what are now listed as inaccesible UUIDs, brings back the following (for example):-

DOM Object: 40828855-cd1d-bd28-98c7-0c4de99aaae2 (v1, owner: 192.168.1.50, policy: No POLICY entry found in CMMDS)

  RAID_1

    Component: 40828855-720a-558a-d8f8-0c4de99aaae2 (state: ACTIVE (5), host: 192.168.1.50, md: t10.ATA_____APPLE_HDD_HTS541010A9E662_____________________J580001MK0EL2P, ssd: t10.ATA_____Samsung_SSD_850_PRO_1TB_________________S1SRNWAFB06290N_____,

                                                     votes: 1, usage: 40.0 GB)

    Component: 40828855-16d2-568a-f741-0c4de99aaae2 (state: DEGRADED (9), csn: STALE (26!=71), host: 192.168.1.51, md: t10.ATA_____APPLE_HDD_HTS541010A9E662_____________________J890011VHESYGC, ssd: t10.ATA_____Samsung_SSD_850_PRO_1TB_________________S1SRNWAFB06608X_____,

                                                     votes: 1, usage: 40.0 GB)

  Witness: 40828855-9add-578a-0fec-0c4de99aaae2 (state: ABSENT (6), csn: STALE (0!=71), host: Unknown, md: t10.ATA_____APPLE_HDD_HTS541010A9E662_____________________J58000BTHNEYMF, ssd: Unknown, note: LSOM object not found,

                                                 votes: 1)

With usage in this particular case of 40GB, this would suggest this is indeed one of my missing VMs, I just don't know how to retrieve the contents?

What is interesting, is that this has highlighted the possibly root cause of the problem (a Degraded disk group).

Any help would be greatly appreciated.

[JohnNicholsonVM]

Somewhere buried in the RVC manual I think there might be the command your looking for, although I think 5.5update2 may not support that.

Samsung_SSD_850_PRO are known to perform terribly (highly inconsistent have timeouts that will make VSAN think they are APD).  AHCI controllers are also not supported (and perform poorly).  If you must use them upgrade to 6.0.

[srodenburg]

I had the exact same kind of problems. Read this post and weep...

Buy HCL-Listed SATA or SAS SSD's ? Why consumer grade SATA SSD's make no sense.

In other words, expect your setup to crumble and die during any kind of high I/O actions like resyncs and rebuilds etc.