Problem creating new VSAN 6.6 cluster (host incompatible due to use of newer on-disk format)

[MikeOD]

I am trying to set up a small VSAN 6.6 cluster using three HP DL360 G8 servers, each with one ssd and three hdd devices.  The hardware is all on the VMWare and VSAN hardware compatibility lists, and all the drivers/firmware are current.

The devices are all running the latest version of ESXI, build 5310538, and the VCenter is 6.5d. These hosts were connected to our fiber channel SAN and have been running fine.

Each host has a dedicated VMKernel port (vmk2), running on a dual 10Gb connection to a Cisco 10Gb switch.  The ports on the switch are in a dedicated VLAN.  I’ve confirmed that they can all PING each other over the vsan kernel port. I’m using standard switches.  The switch has two uplinks, one active the other standby.

When I enabled the VSAN, it found the SSD/HDD’s on each host, when through the various configurations, but then on the general screen under the the “on disk format version”, it says that the pre check completed with issues.  On the Details it has a yellow triangle and says that the "Virtual SAN cluster is network partitioned" and the "disk format version" section has a yellow triangle and says "all 12 disks on version 5.0"

On the Monitor/Virtual San Health tab, it’s saying that there is a software version compatibility failed.  On the details it says

"Host incompatible due to use of newer on-disk format by other hosts in the cluster”, and the recommendation to “Upgrade ESXi software of this host”.

As I mentioned above, all three hosts are at the same, latest version, 6.5.0, build 5310538

Any suggestions would be appreciated

Mike O'Donnell

[GreatWhiteTec]

Can you confirm that your vCenter is 6.5.0.5500 Build Number 5318154.

You may want to run the following commands:

esxcli vsan cluster get - make sure that you see all the hosts in the same cluster

If hosts are not seeing each other, you will need to add the hosts to unicastagent

esxcli vsan cluster unicast agent list  - if list is empty then the host has no neighbors.

You can add hosts to the unicast agent list by using the command: esxcli vsan cluster unicast add

[TheBobkin]

Hello Mike,

The cluster being partitioned is the main issues here.

Connect to each host via SSH and use:

# esxcli vsan cluster get

You can tell that the cluster is partitioned if each host sees "Sub-Cluster Member Count: 1"

Copy the "Local Node UUID" from each host along with the configured vSAN interface IP found using:

#esxcli vsan network list

# esxcfg-vmknic | grep vmk<#>

Then add all hosts in the cluster to each others lists by adding each hosts entry on every host:

#esxcli vsan cluster unicastagent add –a <IP> -u <LOCAL NODE UUID> -U 1 –t node

Check that all the lists have been populated after adding the abov entries, this should show these:

# esxcli unicast agent list

Check that cluster has formed:

# esxcli vsan cluster get

If it has not then manually form by leave then join cluster using the Sub-Cluster UUID from a host.

The other health alerts are likely cosmetic in nature as a result of the cluster not being formed and communicating with one another.

If the disk-groups were created after installing vSAN 6.6 (and there are no host advanced settings stating create legacy version disks) then all of the disks are on version 5.0 which is normal.

Bob

-o- If you found this comment useful please click the 'Helpful' button and/or select as 'Answer' if you consider it so, please ask follow-up questions if you have any -o-

[MikeOD]

I'll try it tomorrow.

If that takes care of it, what puzzles me is that since all the nodes seem like they can see each other, what would have caused it to not join them together?

[MikeOD]

I noticed on the esxcli vsan cluster unicastagent add, there is an option "-i" for the "name of the bound outgoing network interface".  Do I need to specify "-i vmk2" in the "add" command?  That is the kernel interface for the VSAN.

[TheBobkin]

I'm not sure what circumstances that is necessary but the times I have configured it so far it added them to unicast agent list fine without specifying -i (outbound interface).

Bob

[MikeOD]

Looks like it was "user error" (mine)...

I knew VSAN 6.6 was being released with ESXi and VCenter 6.5.0d, so I waited for the update before I started the VSAN install.   I updated the hosts to 6.5.0d, but apparently missed updating this VCenter (it was still at 6.5.0b).  We have two VCenter systems, I had updated one, but that was NOT the one I was working on for VSAN. 

Once I applied the "6.5.0d" update and rebooted, I went back into VCenter and everything is showing green.  The hosts are no longer partitioned, the disk format version is showing a green checkmark, and the VSAN Health is good.

Thank you for the responses.

It's amazing how much better things work when have the right versions....