VSAN Battle (2 Node Cluster with witness node) Cannot Write to VSAN

[danielmgrinnell]

I am having issues with my VSAN Enabled Cluster and cant seem to write to the datastore or migrate any VM's or upload anything to it. When i attempt to vMotion a VM over to the VSAN DS i get an error  " Cannot complete file create operation, An unknown error has occurred."

Here is my Layout,

Two Clusters one with Witness Node, one with 2 Nodes

Each Host has 2 Drives with one Cache and one Capacity running as FLASH and HDD as a hybrid Type. Fault domain I have a witness preferred and secondary each host is split.

Here is the Kicker, i have figured out a work around, if i change the perfered fd host and while its updating the fault domain i can for that brief moment write to the datastore. This is how i am migrating VMs at the moment.

Any suggestions?

Thanks

[elerium · Accepted solution]

Is VSAN health check showing that everything is working fine? My guess is your 2 hosts have issues communicating with the witness node or vice versa.

Try changing the VSAN storage policy to enable ForceProvisioning as a test, if this works, problem is probably network config/route issue communicating with the witness.

[danielmgrinnell]

Well done, the force provisioning seem to allow me to deploy VMs and migrate VMs to my VSAN.

I do think you're right with the host communicating with each other considering the witness host keeps complaining about not being able to communicate with vSAN hosts running the service. I have all vmkernels on same vlan and ip subnet should i set those up on a isolated subnet / vlan ? Also my performance health service for the VSAN cluster wont display and doesn't seem to be working properly.

Thanks !

[elerium]

I would recommend getting the health services working, many VSAN issues are difficult to troubleshoot without it. I find it's invaluable to resolving many configuration issues.

The most common scenario for the network design is VSAN traffic with hosts using L2 network, and VSAN traffic with witness using L3 network. Your hosts may have difficulty communicating with your witness on L3 if you have not added static routes on your hosts and witness. Also for VSAN traffic (even between hosts/witness), it will only use the VSAN kernels to communicate. Because the hosts default to using the management vmkernel for routing, you may need to add routes to the routing tables on your hosts/witness. I've had to do this for my 2 node ROBO install so the hosts can communicate with the witness.

This guide explains it in more detail than I have:

see pages 19-28 and 68-69.

http://www.vmware.com/files/pdf/products/vsan/VMware-Virtual-SAN-6.2-Stretched-Cluster-Guide.pdf