We just setup a 12 node VSAN cluster where all the storage is used for cache/storage tiers, so no local drives available to use to store the system logs, scratch files, or core dumps should they occur. I was just reading up on VSAN 8 File Services and that seems like a good candidate from the availability perspective. I have another unrelated server that is providing NFS storage, but it's a single unit and if it goes down, all 12 hosts will have that ripped out from under them. At least with VSAN there are multiple agents so loosing a host shouldnt cause a self-licking ice cream cone-style meltdown.
Short of buying a dual-headed NAS or adding additional non-VSAN drives to these servers, is this a crazy solution?
The answer to this is simple, vSAN File Services is not supported to use with ESXI, the purpose of this is to use it with VMs, see here: https://docs.vmware.com/en/VMware-vSphere/8.0/vsan-administration/GUID-4D24850D-0C85-4C5F-8CD7-93EB5...
When the vSAN file service is enabled, you can create one or more file shares on the vSAN datastore. vSAN File Service does not support using NFS file shares on ESXi.