esxi 5 disconnects from vCenter server

[hafeezpashamoha]

One of our esxi 5 Update 2 (build 914586) shows disconnected from vCenter server (ver 5 update 1b). FDM log show, lost master, see logs below.

SSL Cert is valid until 2024 and we are not using any customer SSL certs, has anybody experienced this strange behaviour?

2013-06-11T11:11:08.743Z [2CC01B90 info 'Election'] MasterShutdown

2013-06-11T11:11:09.038Z [FFE52B90 info 'Election' opID=SWI-6f90a274] [ClusterElection::ChangeState] Slave => Startup : Lost master

2013-06-11T11:11:09.038Z [FFE52B90 info 'Cluster' opID=SWI-6f90a274] Change state to Startup:0

2013-06-11T11:11:09.039Z [FFF15B90 verbose 'Cluster' opID=SWI-f6258ec6] [ClusterManagerImpl::CheckElectionState] Transitioned from Slave to Startup

2013-06-11T11:11:09.039Z [FFF15B90 info 'Invt' opID=SWI-f6258ec6] [InventoryManagerImpl::NotifyDatastoreUnlockedLocally] Invoked for datastore (/vmfs/volumes/50e25d37-cffd0503-d212-0017a47708f2).

2013-06-11T11:11:09.039Z [2CC42B90 info 'Message'] Destroying connection

2013-06-11T11:11:09.039Z [FFE52B90 verbose 'Election' opID=SWI-6f90a274] CheckVersion: Version[0] Other host GT : 29 > 28

2013-06-11T11:11:09.039Z [FFF15B90 info 'Invt' opID=SWI-f6258ec6] [InventoryManagerImpl::NotifyDatastoreUnlockedLocally] Invoked for datastore (/vmfs/volumes/3a7a9446-b1422325).

2013-06-11T11:11:09.039Z [FFE52B90 verbose 'Cluster' opID=SWI-6f90a274] VersionChange: version[0] 29 from host-4698,192.168.18.69

2013-06-11T11:11:09.040Z [FFF15B90 info 'Invt' opID=SWI-f6258ec6] [InventoryManagerImpl::NotifyDatastoreUnlockedLocally] Invoked for datastore (/vmfs/volumes/512257fc-ab79de70-5468-d8d385591ca3).

2013-06-11T11:11:09.040Z [FFF15B90 info 'Invt' opID=SWI-f6258ec6] [InventoryManagerImpl::NotifyDatastoreUnlockedLocally] Invoked for datastore (/vmfs/volumes/5113d47c-03d30920-3481-f4ce46884743).

2013-06-11T11:11:09.040Z [FFF15B90 info 'Invt' opID=SWI-f6258ec6] [InventoryManagerImpl::NotifyDatastoreUnlockedLocally] Invoked for datastore (/vmfs/volumes/512257df-65aa526e-7815-d8d385591ca3).

2013-06-11T11:11:09.040Z [FFF15B90 info 'Invt' opID=SWI-f6258ec6] [InventoryManagerImpl::NotifyDatastoreUnlockedLocally] Invoked for datastore (/vmfs/volumes/5122578b-31f1e5e6-4a3c-d8d385591ca3).

2013-06-11T11:11:09.040Z [FFF15B90 info 'Invt' opID=SWI-f6258ec6] [InventoryManagerImpl::NotifyDatastoreUnlockedLocally] Invoked for datastore (/vmfs/volumes/51225693-ae3815ab-01c3-d8d385591ca3).

2013-06-11T11:11:09.040Z [FFF15B90 info 'Invt' opID=SWI-f6258ec6] [InventoryManagerImpl::NotifyDatastoreUnlockedLocally] Invoked for datastore (/vmfs/volumes/5113cd4d-427af962-30b4-f4ce46884743).

2013-06-11T11:11:09.040Z [FFF15B90 info 'Invt' opID=SWI-f6258ec6] [InventoryManagerImpl::NotifyDatastoreUnlockedLocally] Invoked for datastore (/vmfs/volumes/512257b9-ee9f35f5-7e91-d8d385591ca3).

2013-06-11T11:11:09.040Z [FFF15B90 info 'Invt' opID=SWI-f6258ec6] [InventoryManagerImpl::NotifyDatastoreUnlockedLocally] Invoked for datastore (/vmfs/volumes/51225671-dc1029e0-20a7-d8d385591ca3).

2013-06-11T11:11:09.041Z [FFF15B90 info 'Invt' opID=SWI-f6258ec6] [InventoryManagerImpl::NotifyDatastoreUnlockedLocally] Invoked for datastore (/vmfs/volumes/5122564f-4f4ccc99-a715-d8d385591ca3).

2013-06-11T11:11:09.041Z [FFF15B90 info 'Invt' opID=SWI-f6258ec6] [InventoryManagerImpl::NotifyDatastoreUnlockedLocally] Invoked for datastore (/vmfs/volumes/512256bd-66b52452-086e-d8d385591ca3).

2013-06-11T11:11:09.041Z [FFF15B90 info 'Cluster' opID=SWI-f6258ec6] [ClusterManagerImpl::MainLoop] curState 1 lastState 4

2013-06-11T11:11:09.041Z [FFE52B90 info 'Cluster' opID=SWI-6f90a274] VersionChange: fetching version[0] 29 from host-4698,192.168.18.69

2013-06-11T11:11:09.041Z [FFE52B90 verbose 'Election' opID=SWI-6f90a274] CheckVersion: Version[0] Other host GT : 29 > 28

2013-06-11T11:11:09.042Z [FFE52B90 verbose 'Cluster' opID=SWI-6f90a274] VersionChange: version[0] 29 from host-4709,192.168.18.86

2013-06-11T11:11:09.042Z [FFCEB400 info 'Invt' opID=SWI-afe9a8c6] [InventoryManagerImpl::ProcessClusterChange] Cluster state changed to Startup

2013-06-11T11:11:09.042Z [FFE52B90 verbose 'Cluster' opID=SWI-6f90a274] VersionChange: Already fetching newer version 29 > 29

2013-06-11T11:11:09.043Z [FFE52B90 verbose 'Election' opID=SWI-6f90a274] CheckVersion: Version[0] Other host GT : 29 > 28

2013-06-11T11:11:09.043Z [FFCEB400 verbose 'PropertyProvider' opID=SWI-afe9a8c6] RecordOp ASSIGN: clusterState, fdmService

2013-06-11T11:11:09.043Z [FFE52B90 verbose 'Cluster' opID=SWI-6f90a274] VersionChange: version[0] 29 from host-4703,192.168.18.71

2013-06-11T11:11:09.043Z [FFCEB400 verbose 'FDM' opID=SWI-afe9a8c6] [FdmService::Handle::ClusterStateNotification] Cluster state changed: Slave -> Startup

2013-06-11T11:11:09.043Z [FFE52B90 verbose 'Cluster' opID=SWI-6f90a274] VersionChange: Already fetching newer version 29 > 29

2013-06-11T11:11:09.043Z [FFCEB400 verbose 'Placement' opID=SWI-afe9a8c6] [PlacementManagerImpl::ClusterStateListener::Handle] New cluster state is 1

2013-06-11T11:11:09.044Z [FFE52B90 verbose 'Election' opID=SWI-6f90a274] CheckVersion: Version[0] Other host GT : 29 > 28

2013-06-11T11:11:09.044Z [FFE52B90 verbose 'Cluster' opID=SWI-6f90a274] VersionChange: version[0] 29 from host-4705,192.168.18.84

2013-06-11T11:11:09.044Z [FFCEB400 verbose 'Execution' opID=SWI-afe9a8c6] [ExecutionManagerImpl::ClusterStateListener::Handle] New cluster state is 1

2013-06-11T11:11:09.044Z [FFCEB400 verbose 'Policy' opID=SWI-afe9a8c6] [PolicyManager::Handle(ClusterStateNotification)] Transitioning to startup (1). Disabling global policy and enabling local policy.

2013-06-11T11:11:09.044Z [FFCEB400 verbose 'Monitor' opID=SWI-afe9a8c6] [IsoAddressMonitor::Handle::ClusterStateNotification] Cluster state changed to 1

2013-06-11T11:11:09.044Z [FFE52B90 verbose 'Cluster' opID=SWI-6f90a274] VersionChange: Already fetching newer version 29 > 29

2013-06-11T11:11:09.045Z [FFCEB400 verbose 'Monitor' opID=SWI-afe9a8c6] [PingableAddressMonitor::Handle::ClusterStateNotification] Cluster state changed to 1

2013-06-11T11:11:09.045Z [FFCEB400 verbose 'Monitor' opID=SWI-afe9a8c6] [HostAccessMonitor::ClusterStateListener] Cluster state changed to 1

2013-06-11T11:11:09.045Z [FFE52B90 verbose 'Election' opID=SWI-6f90a274] CheckVersion: Version[0] Other host GT : 29 > 28

2013-06-11T11:11:09.045Z [FFDD0B90 warning 'Libs' opID=SWI-60789709] SSL_VerifyX509: Certificate verification is disabled, so connection will proceed despite the error

2013-06-11T11:11:09.045Z [FFE52B90 verbose 'Cluster' opID=SWI-6f90a274] VersionChange: version[0] 29 from host-4707,192.168.18.85

2013-06-11T11:11:09.046Z [FFDD0B90 warning 'Libs' opID=SWI-60789709] SSL_VerifyX509: Certificate verification is disabled, so connection will proceed despite the error

2013-06-11T11:11:09.046Z [FFE52B90 verbose 'Cluster' opID=SWI-6f90a274] VersionChange: Already fetching newer version 29 > 29

2013-06-11T11:11:09.046Z [FFDD0B90 warning 'Libs' opID=SWI-60789709] SSL_VerifyX509: Certificate verification is disabled, so connection will proceed despite the error

[lvaibhavt]

Hi Hafeez,

I guess you would have already tried ... removing the host from VC and then adding it again

thanks

[weinstein5]

Also look at if there were any issues with the physical network -

[hafeezpashamoha]

I have re-connected and removed and added host without luck. There are other hosts in cluster on same subnet and others esxi hosts are fine. No networking issue detected. Anything else i should be looking at?

[Cyberfed27]

We are experiencing similar anomalies with some of our ESXi 5.1 hosts.

They will show as disconnected in vCenter, then, randomly show back online.

Network connectivity (ping) is never lost during this time to the host or guest VM's.

No HA operations take place because the VM's remain available.

Happens every now and then on a few hosts.

[hafeezpashamoha]

In our case, host disconnect and have to be manually re-connected.

[Cyberfed27]

We have only had that happen once with one host. Even while it remained "disconnected/unavailable" the host and VMs were indeed still up.