VMware Cloud Community
srodenburg
Expert
Expert

Tip: How to fix "General vSAN Error" during 6.7 to 6.7 U1 upgrade (Stretched Clusters only)

Hello,

I've seen this a couple of times now when people are upgrading a 2 Node ROBO, or normal Stretched Cluster, from v6.7 to 6.7 U1.  The problems arises when they are done upgrading all the ESXi Servers and they want to upgrade the disk-format to v7.

They run the pre-upgrade checker which says everything is ok.

Then they click on "Upgrade" and the process gets stuck on 3% and after a couple of minutes, barfs with the error "General vSAN Error". On the surface, there is no clue what went wrong. Everything looked ok, the pre-upgrade-checker found no issues. So what it going on?

When one digs a little deeper, the actual error is "Detected host(s) are running in mixed mode, please upgrade ESX servers to the same version."

So people think "say what? I just upgraded all of the ESXi servers. This cannot be!!"

No, but they are forgetting they are running a stretched cluster (either normal or ROBO) and Stretched Clusters have Witness Appliances. So what happens is that they upgraded all hosts to 6.7U1, but forgot to replace the Witness Appliance with a 6.7 U1 version. So indeed there is a mix, from a vSAN Perspective, of 6.7 and 6.7U1 hosts.

A 6.7 Witness Appliance does not understand what on-disk format v7 is.

Solution:  replace the Witness Appliance with a 6.7 U1 variant and retry the disk-format upgrade. It will work now.

Hope this helps others when they get stuck.

I've seen it happen more than once: people forgetting all about the Witness-Appliance's existence after a while. And a 6.7U1 Stretched Cluster or ROBO works with a 6.7 Witness as long as the disk-format is still v6. So there is no real incentive to replace it because "hey it works just fine".

To be clear: this does not affect normal vSAN Clusters.

@ VMware:  a more helpful error message would be uuhhh, more helpful. But even better, the pre-upgrade-checker should tell that the on-disk-format cannot take place, due to the Witness being the wrong version.

1 Reply
ralfthiel
Contributor
Contributor

Maybe this is clear for the most, but just to be sure: You don't even have to replace the witness appliance... just upgrade it to ESXi 6.7 U1 / vSAN 6.7 U1 like you do with your hardware hosts (e.g. with vSphere Update Manager)