HA cluster malfunctioning Build mismatch (13638473 != 9291058)

[ALKosmos]

Hi,

We have vCenter vCSA 6.0 and two ESXi 6 host in HA cluster.

Presumably after upgrade vCenter from 6.0.0.30800 to 6.0.0.30900 we finded out  our HA Cluster does not work. When we try to migrate any VM from one ESXi host to other we got "The host is reporting errors in attempts to provide vSphere HA support".

We tried to disable HA Cluster and enable it back.  What we got:

In log /var/log/fdm.log on ESXi host we have:

2019-07-16T19:59:44.248Z warning fdm[FFFB2B70] [Originator@6876 sub=Election opID=SWI-60b7acd9] [120 times] Build mismatch (13638473 != 9291058) from host 10.160.250.135 - dropping message

It looks like we have different HA agents on ESXi hosts.

Any help or ideas would be very appreciated.

[daphnissov]

What are the build numbers of your hosts in this cluster?

[ALKosmos]

Thank you for your response.

We have two ESXi hosts 6.0.0, 5224934

10.160.250.136 host (problematic one)

[root@vsphere2:~] df -h
Filesystem   Size   Used Available Use% Mounted on
VMFS-5       8.6T   7.0T      1.7T  80% /vmfs/volumes/vol0001
VMFS-5       1.8T   1.5T    277.9G  85% /vmfs/volumes/vd01_vol01
VMFS-5       1.8T   1.3T    563.4G  70% /vmfs/volumes/vd02_vol01
vfat       285.8M 203.6M     82.2M  71% /vmfs/volumes/58b722df-7de90578-f1ce-9cdc716ed488
vfat       249.7M 211.0M     38.8M  84% /vmfs/volumes/f0a358e7-c0061847-34b2-06e403d483ae
vfat       249.7M 211.0M     38.8M  84% /vmfs/volumes/7d6330e6-2a698917-8a86-d6ea1936b1a8

10.160.250.135 host

[root@vsphere1:~] df -h
Filesystem   Size   Used Available Use% Mounted on
VMFS-5       1.8T   1.3T    563.4G  70% /vmfs/volumes/vd02_vol01
VMFS-5       1.8T   1.5T    277.9G  85% /vmfs/volumes/vd01_vol01
VMFS-5       8.6T   7.0T      1.7T  80% /vmfs/volumes/vol0001
VMFS-5     911.2G   1.7G    909.5G   0% /vmfs/volumes/iscsi2 (1)
vfat       249.7M 211.0M     38.7M  85% /vmfs/volumes/4106c9e6-7cfcf3ad-4447-dfa6d9c316dd
vfat       249.7M 211.0M     38.7M  85% /vmfs/volumes/4bbedae4-9c4a8d3f-feed-57e7048e6a86
vfat       285.8M 203.6M     82.2M  71% /vmfs/volumes/58b6b9d2-6900046c-0381-9cdc716ed5a8

vCSA  vCenter 6.0,  6.0.0.30900. 4 CPU, 16 Gb

vcenter:~ # df -h
Filesystem                            Size  Used Avail Use% Mounted on
/dev/sda3                              11G  6.3G  4.0G  62% /
udev                                  7.9G  164K  7.9G   1% /dev
tmpfs                                 7.9G   40K  7.9G   1% /dev/shm
/dev/sda1                             128M   39M   83M  32% /boot
/dev/mapper/core_vg-core               25G  188M   24G   1% /storage/core
/dev/mapper/log_vg-log                 19G  9.1G  8.7G  52% /storage/log
/dev/mapper/db_vg-db                  9.9G  1.4G  8.0G  15% /storage/db
/dev/mapper/dblog_vg-dblog            5.0G  2.9G  1.9G  61% /storage/dblog
/dev/mapper/seat_vg-seat               19G  7.1G   11G  40% /storage/seat
/dev/mapper/netdump_vg-netdump       1001M   18M  932M   2% /storage/netdump
/dev/mapper/autodeploy_vg-autodeploy  9.9G  151M  9.2G   2% /storage/autodeploy
/dev/mapper/invsvc_vg-invsvc          5.0G  270M  4.5G   6% /storage/invsvc
tmpfs                                 1.0M     0  1.0M   0% /var/spool/snmp

[daphnissov]

Ok, so you've got really old ESXi hosts there. They're actually pre-U1 while your vCenter is post U3. First thing I'd do is bring those ESXi hosts up to U3 because the problem will most likely go away.

[msripada]

Issue is because of the HA timed out. Restart vCenter and let us know if that helps .. This issue might have started post upgrading the vcenter recently so the fdm is not updated on all hosts

[ALKosmos]

Hi,

Thank you!

I restarted vCenter many times without any luck, even tried to restore old vCenter 6.0.0.30800 from a backup, but that one couldn't connect to our ESXi host due to user/password issue (never changed though). When I tried to disconnect and connect back ESXi hosts to old vCenter 6.0.0.30800, it started reporting about EVC problems and so on, I gave up and started back new vCenter  6.0.0.30900.

[ALKosmos]

Here is the solution:

This is exactly our case.

https://www.vcloudnine.de/vmware-update-manager-reports-error-code-99-during-scan-operation/

+

VMware Knowledge Base

And we do not have an idea why it happened either.