update vcenter - esxi host in a vsan environment

[gepit68]

Hi All,

I am new in this community. I am not an expert in VMWare technology so please be passion with me. Currently working in a company running vmWare with VSAN. I have a task to update Vcenter, Esxi Host to a new version, they were running version

ESXI Hosts Version: 6.7.0.11675023

VCSA: 6.7.0.2000

New Version: 6.7.0,15160138

VCSA: 6.7.0.32000

VSANHealth version: 6.7.0-3.89.14840358

So far I have updated vcsa and esxi. The question I have do I need to update VSAN to a newer version same as vcsa and hosts? Currently Cluster\ VSAN Health Check showing a few red question mark as:

Network

* Hosts with connectivity issues ( I have double check connectiviy Host - VCSA and it is fine)

* All hosts have a VSAN vmknic configured

Physical disk

* Physical disk health retrieval issues

Data

* vSAN object health

Cluster

* ESXi vSAN Health service installation

* vSAN Health Service up-to-date

* vSAN CLOMD liveness

* vSAN cluster configuration consistency

* Software version compatibility

* Disk format version

vSAN iSCI target service

* Home Object

Another thing is VSAN\Disk Management has a yellow warning with All 62 disk on version 7.0

I have double check the version of vsanhealth on all of ESXi hosts and are consistent with version 6.7.0-3.89.14840358.

Any help would be greatly appreciated

[TheBobkin]

Hello gepit68​

Welcome to Communities (and vSAN and VMware :smileygrin: ).

"do I need to update VSAN to a newer version same as vcsa and hosts?"

No - vSAN is part of ESXi and thus gets updated when ESXi is updated.

Most of the vSAN Health warnings you see may be explained by one or more hosts with vSAN network mis-configured (or not configured) - please look at the Health check for more information as to which hosts are being marked as having no vSAN vmknic configured and the specifics of 'vSAN cluster configuration consistency' (e.g. which hosts are triggering it and what aspect is varying from the rest of the cluster). Things to look for:

- Hosts that are not part of the vSAN cluster residing in the vSphere cluster.

- Hosts with no vSAN networking configured (when they should have it).

- Hosts with inconsistent vSAN and/or Witness (if Stretched cluster) networking configured (e.g. MTU).

- Hosts set to ignore vCenter cluster membership updates.

"Another thing is VSAN\Disk Management has a yellow warning with All 62 disk on version 7.0"

This is expected as 6.7 U2 and later have a new On-Disk Format (v10) which is not updated automatically (e.g. once the whole cluster has been updated and all is good, you have to push the pre-check and upgrade buttons).

Bob

[gepit68]

Morning Bob

Thanks for the prompt response, the interesting part is vsan health check reports that all the Hosts are experiencing the same issue or behaviour.

- 6 Hosts are part of the Cluster

- All of them are configured with management network, vmotion, and storage vmk0,vmk1,vmk2

I will keep looking, currently log a ticket with HP and see if they can help.

Thanks again for your response

[TheBobkin]

Hello gepit68​

If ALL hosts are reporting issues with the various Health checks you noted then it could be an issue with the vSAN Health service itself which runs on vCenter and talks to vsanmgmtd service on the hosts (or potentially vsanmgmtd on host side).

It is not going to break anything to restart this service by logging into the vCSA as root and using the following:

# service-control --stop vmware-vsan-health

# service-control --start vmware-vsan-health

"I will keep looking, currently log a ticket with HP and see if they can help."

No offence to HPE but this isn't going to be their area of expertise, I would advise opening a Support Request with VMware GSS vSAN if possible.

Bob

[gepit68]

Hi Bob

Done that, restart vmware-vsan-health and also vsphere client. Same thing after it. HP is the vendor and OEM Vmware on all Hosts, so we have HPE support for hardware and Software. I totally agreed what you said but I do not have access to vmware support for those boxes.

Any other thoughts?

And again thanks for your help.

[TheBobkin]

Hello gepit68​

Sorry I missed this in the first place (it has been annoying me since switched to client version number vs vCSA build version in 'about vsphere' in UI Client) but "VCSA: 6.7.0.32000" is not a 6.7 U3 vCenter but actually is 6.7 U2 (U2c, build:14070457) - it is not supported nor expected to be fully functional (and aspects such as vSAN Health where we made so many changes may be more affected) to manage a vSAN cluster of a higher major revision e.g. 6.7 U3 vSAN hosts with 6.7 U2 vCenter, this varies from general ESXi host management in this respect:

vSAN + vcenter:

https://www.vmware.com/resources/compatibility/sim/interop_matrix.php#interop&508=&2=

ESXi + vCenter:

https://www.vmware.com/resources/compatibility/sim/interop_matrix.php#interop&1=&2=

Thus please upgrade your vCenter to a 6.7 U3 build before troubleshooting this further - not just saying this as a throwaway support comment of 'please upgrade to latest', if you look back on vSAN Communities, the number of issues with Health/Disk-Group management that have been as a result of not adhering to the above matrices is quite significant.

Bob

[gepit68]

Hi Bob,

I was in a call with HP\Vmware. One thing that they discovered is vsan-vds is currently running version 6.5.0 and according with this kb Upgrade a vSphere Distributed Switch to a Later Version the current version is not supported "

Version 6.6.0	Compatible with ESXi version 6.7 and later.
Version 6.5.0	Compatible with ESXi version 6.5 and later. Features released with later vSphere Distributed Switch versions are not supported.

So my next step will be to upgrade VDS to version 6.6.0 - ESXI 6.7 or later. Hopefully this will be the root of the problem.

What do you think?

[TheBobkin]

Hello gepit68

No, vDS 6.5 is supported on 6.7 - "Compatible with ESXi version 6.5 and later."

Bob

[gepit68]

Hi Bob

Apologised in advance, I should listen to your straight away rather than waste my time with Support. I have updated vcsa to 6.7.0.4100 and bingo all green and happy cluster. VDS was nothing to do with the issue, anyway I have upgraded to 6.6. The only thing is pending Disk format version, as far as I know it is not necessary to upgraded but do you think I should do it? It is any risk in doing it?

Bob, Thank you so much for your help. You are a champion!!! God Bless you!!

[TheBobkin]

Hello gepit68​

More than happy to help and glad to hear it is looking better now.

Just so you are aware I am also 'Support', the key difference though is that vSAN is my primary SME and has been for nearly 4 years now - I wouldn't really blame anyone for not being able to read between the lines and realise this is pure vC/vSAN version issue (and fairly expected behaviour as vC being on lower major version is unsupported), but I also can't find the logic of coming to the conclusion that vDS version is the cause.

"The only thing is pending Disk format version, as far as I know it is not necessary to upgraded but do you think I should do it? It is any risk in doing it?"

I would advise updating these to v10 as this is a requirement to benefit from most of the improvements made in vSAN 6.7 U3. Provided all hosts have been updated to 6.7 U3 this shouldn't pose any issues but do validate that the upgrade pre-check doesn't highlight anything that needs to be resolved before proceeding with upgrading.

Bob