VMware Cloud Community
Wallace44
Contributor
Contributor
‎08-24-2015 11:41 PM
Jump to solution

ESXi 5.5 update 2 | Hosts disconnecting and reconnecting randomly

We have 3 x HP DL380 G9 servers running ESXi 5.5 u2.

They have been in production for approximately 6-8 months so far with no issues whatsoever, until recently.

Recently, in Vcentre the hosts will go into a 'disconnected' state, and trigger the 'host power and connectivity' alarm. After a couple of minutes the host will reconnect and everything continues as normal. This disconnection doesn't cause any VMs to disconnect or reboot. During the time that Vcentre shows the host as disconnected, we can still SSH and ping the ESX host without problem.

I've checked through the VPXD logs and found that at the time of the outage, this occurs:

2015-08-22T01:45:52.952+08:00 [09988 info 'vpxdvpxdVmomi' opID=SWI-1021c707] [ClientAdapterBase::InvokeOnSoap] Invoke done (192.168.101.171, vpxapi.VpxaService.fetchQuickStats)


2015-08-22T01:45:52.952+08:00 [09988 info 'vpxdvpxdVmomi' opID=SWI-1021c707] [ClientAdapterBase::InvokeOnSoap] Invoke done (192.168.101.173, vpxapi.VpxaService.fetchQuickStats)


2015-08-22T01:45:52.952+08:00 [09988 info 'vpxdvpxdVmomi' opID=SWI-1021c707] [ClientAdapterBase::InvokeOnSoap] Invoke done (192.168.101.172, vpxapi.VpxaService.fetchQuickStats)


2015-08-22T01:46:52.953+08:00 [09988 info 'vpxdvpxdVmomi' opID=SWI-1021c707] [ClientAdapterBase::InvokeOnSoap] Invoke done (192.168.101.171, vpxapi.VpxaService.fetchQuickStats)


2015-08-22T01:46:52.953+08:00 [09988 info 'vpxdvpxdVmomi' opID=SWI-1021c707] [ClientAdapterBase::InvokeOnSoap] Invoke done (192.168.101.173, vpxapi.VpxaService.fetchQuickStats)


2015-08-22T01:46:52.953+08:00 [09988 info 'vpxdvpxdVmomi' opID=SWI-1021c707] [ClientAdapterBase::InvokeOnSoap] Invoke done (192.168.101.172, vpxapi.VpxaService.fetchQuickStats)


2015-08-22T01:47:05.413+08:00 [09892 info 'commonvpxLro' opID=E83DF74C-00001BE1-18] [VpxLRO] -- BEGIN task-internal-35516 --  -- vmodl.query.PropertyCollector.cancelWaitForUpdates -- 4cb606ef-f3da-9ace-e253-d4c9e9b48bfe(529b4a61-85a4-90fe-ce8c-6361ed144841)


2015-08-22T01:47:05.414+08:00 [09028 error 'SoapAdapter.HTTPService'] Failed to read request; stream: <io_obj p:0x000000000cc5fd08, h:4656, <TCP '[::1]:8085'>, <TCP '[::1]:56470'>>, error: class Vmacore::SystemException(An established connection was aborted by the software in your host machine)


2015-08-22T01:47:05.414+08:00 [09892 info 'commonvpxLro' opID=E83DF74C-00001BE1-18] [VpxLRO] -- FINISH task-internal-35516 --  -- vmodl.query.PropertyCollector.cancelWaitForUpdates -- 


2015-08-22T01:47:52.954+08:00 [09988 info 'vpxdvpxdVmomi' opID=SWI-1021c707] [ClientAdapterBase::InvokeOnSoap] Invoke done (192.168.101.172, vpxapi.VpxaService.fetchQuickStats)


2015-08-22T01:47:52.954+08:00 [09988 info 'vpxdvpxdVmomi' opID=SWI-1021c707] [ClientAdapterBase::InvokeOnSoap] Invoke done (192.168.101.173, vpxapi.VpxaService.fetchQuickStats)


2015-08-22T01:47:52.954+08:00 [09988 info 'vpxdvpxdVmomi' opID=SWI-1021c707] [ClientAdapterBase::InvokeOnSoap] Invoke done (192.168.101.171, vpxapi.VpxaService.fetchQuickStats)


2015-08-22T01:48:07.047+08:00 [09916 info 'commonvpxLro' opID=24b2ad7b] [VpxLRO] -- BEGIN task-internal-35517 --  -- vim.LicenseAssignmentManager.queryAssignedLicenses -- ffe39310-e804-b7d6-92d8-61203ec022ce(52008866-be52-bc37-8007-48ec7a383c04)


2015-08-22T01:48:07.047+08:00 [09916 info 'commonvpxLro' opID=24b2ad7b] [VpxLRO] -- FINISH task-internal-35517 --  -- vim.LicenseAssignmentManager.queryAssignedLicenses -- 


2015-08-22T01:48:25.450+08:00 [16416 error 'vpxdvpxdVmomi'] [VpxdClientAdapter] Got vmacore exception: Operation timed out


2015-08-22T01:48:25.450+08:00 [16416 error 'vpxdvpxdVmomi'] [VpxdClientAdapter] Backtrace:


--> backtrace[00] rip 000000018018b7fa 


--> backtrace[01] rip 0000000180104c78 


--> backtrace[02] rip 0000000180105f6e 


--> backtrace[03] rip 000000018008ff08 


--> backtrace[04] rip 00000001800418d6 


--> backtrace[05] rip 00000001800445de 


--> backtrace[06] rip 0000000180048382 


--> backtrace[07] rip 00000000004f0ca9 


--> backtrace[08] rip 00000000004f1399 


--> backtrace[09] rip 00007ff7689280df 


--> backtrace[10] rip 00007ff76892f418 


--> backtrace[11] rip 00007ff768931b9a 


--> backtrace[12] rip 00007ff768928257 


--> backtrace[13] rip 00007ff768930d89 


--> backtrace[14] rip 000000000046157e 


--> backtrace[15] rip 000000000055a15a 


--> backtrace[16] rip 00007ff7682b379d 


--> backtrace[17] rip 00007ff7682b4406 


--> backtrace[18] rip 0000000180152544 


--> backtrace[19] rip 00000001801525d1 


--> backtrace[20] rip 0000000180156eec 


--> backtrace[21] rip 00000001801587ba 


--> backtrace[22] rip 0000000180158cfc 


--> backtrace[23] rip 0000000180158e98 


--> backtrace[24] rip 0000000180153ab3 


--> backtrace[25] rip 0000000180154b03 


--> backtrace[26] rip 0000000180154e43 


--> backtrace[27] rip 000000018019d62a 


--> backtrace[28] rip 0000000067642fdf 


--> backtrace[29] rip 0000000067643080 


--> backtrace[30] rip 00007ffdbd7e13d2 


--> backtrace[31] rip 00007ffdbfab5444 


--> 


2015-08-22T01:48:25.454+08:00 [21316 info 'vpxdvpxdVmomi' opID=HB-host-42@141303-7d8115aa] [ClientAdapterBase::InvokeOnSoap] Invoke done (192.168.101.171, vmodl.query.PropertyCollector.Filter.destroy)


2015-08-22T01:48:25.454+08:00 [21316 warning 'VpxProfiler' opID=HB-host-42@141303-7d8115aa] [VpxdHostSync] GetChanges host:192.168.101.171 (192.168.101.171) [GetChangesTime] took 1800035 ms


2015-08-22T01:48:25.454+08:00 [21316 warning 'VpxProfiler' opID=HB-host-42@141303-7d8115aa] [VpxdHostSync] DoHostSync:0000000009D0F6C0 [DoHostSyncTime] took 1800035 ms


2015-08-22T01:48:25.454+08:00 [21316 warning 'vpxdvpxdInvtHostCnx' opID=HB-host-42@141303-7d8115aa] [VpxdInvtHostSyncHostLRO] DoHostSync failed for host host-42


2015-08-22T01:48:25.454+08:00 [21316 warning 'vpxdvpxdInvtHostCnx' opID=HB-host-42@141303-7d8115aa] [VpxdInvtHostSyncHostLRO] Host sync failed to host-42


2015-08-22T01:48:25.454+08:00 [21316 error 'vpxdvpxdInvtHostCnx' opID=HB-host-42@141303-7d8115aa] [VpxdInvtHostSyncHostLRO] FixNotRespondingHost failed for host host-42, marking host as notResponding


2015-08-22T01:48:25.501+08:00 [21316 warning 'vpxdvpxdMoHost' opID=HB-host-42@141303-7d8115aa] [HostMo] host connection state changed to [NO_RESPONSE] for host-42


2015-08-22T01:48:25.524+08:00 [21316 warning 'VpxProfiler' opID=HB-host-42@141303-7d8115aa] InvtHostSyncLRO::StartWork [HostSyncTime] took 1800105 ms


2015-08-22T01:48:25.524+08:00 [21316 warning 'VpxProfiler' opID=HB-host-42@141303-7d8115aa] VpxLro::LroMain [TotalTime] took 1800105 ms


2015-08-22T01:48:25.524+08:00 [21316 info 'commonvpxLro' opID=HB-host-42@141303-7d8115aa] [VpxLRO] -- FINISH task-internal-35432 -- host-42 -- VpxdInvtHostSyncHostLRO.Synchronize -- 


2015-08-22T01:48:25.869+08:00 [09292 info 'vpxdvpxdHostUpgrader' opID=SWI-10381aa1] [VpxdHostUpgrader] Preinstalled bundle found: not installing


2015-08-22T01:48:25.869+08:00 [09292 info 'vpxdvpxdInvtHostCnx' opID=SWI-10381aa1] [VpxdIntHost] Missed 180 heartbeats for host 192.168.101.171


2015-08-22T01:48:25.869+08:00 [09832 info 'commonvpxLro' opID=HB-host-42@141316-52709f38] [VpxLRO] -- BEGIN task-internal-35518 -- host-42 -- VpxdInvtHostSyncHostLRO.Synchronize -- 


2015-08-22T01:48:25.870+08:00 [09832 info 'vpxdvpxdInvtHostCnx' opID=HB-host-42@141316-52709f38] [VpxdHostSync] Synchronizing host: host-42 (192.168.101.171, 192.168.101.171)


2015-08-22T01:48:25.899+08:00 [09832 info 'vpxdvpxdVmomi' opID=HB-host-42@141316-52709f38] [ClientAdapterBase::InvokeOnSoap] Invoke done (192.168.101.171, vpxapi.VpxaService.retrieveChanges)


2015-08-22T01:48:25.901+08:00 [09832 info 'vpxdvpxdVmomi' opID=HB-host-42@141316-52709f38] [ClientAdapterBase::InvokeOnSoap] Invoke done (192.168.101.171, vmodl.query.PropertyCollector.createFilter)


2015-08-22T01:48:25.902+08:00 [10564 info 'vpxdvpxdVmomi'] [ClientAdapterBase::InvokeOnSoap] Invoke done (192.168.101.171, vmodl.query.PropertyCollector.waitForUpdates)


2015-08-22T01:48:26.402+08:00 [09876 info 'commonvpxLro' opID=task-internal-35519-3cef20c4] [VpxLRO] -- BEGIN task-internal-35519 --  -- ScheduledTaskLRO -- 


2015-08-22T01:48:26.433+08:00 [17708 info 'commonvpxLro' opID=task-internal-35520-36662076] [VpxLRO] -- BEGIN task-internal-35520 --  -- ScheduledTaskLRO -- 


2015-08-22T01:48:26.436+08:00 [17708 info 'commonvpxLro' opID=task-internal-35520-36662076] [VpxLRO] -- FINISH task-internal-35520 --  -- ScheduledTaskLRO -- 


2015-08-22T01:48:31.420+08:00 [09876 info 'commonvpxLro' opID=task-internal-35519-3cef20c4] [VpxLRO] -- FINISH task-internal-35519 --  -- ScheduledTaskLRO -- 


2015-08-22T01:48:32.009+08:00 [10092 info 'commonvpxLro' opID=HB-host-47@184250-7ea813a6] [VpxLRO] -- BEGIN task-internal-35521 -- host-47 -- VpxdInvtHostSyncHostLRO.Synchronize -- 


2015-08-22T01:48:32.010+08:00 [10092 info 'vpxdvpxdInvtHostCnx' opID=HB-host-47@184250-7ea813a6] [VpxdHostSync] Synchronizing host: host-47 (192.168.101.173, 192.168.101.173)


2015-08-22T01:48:32.014+08:00 [10092 info 'vpxdvpxdVmomi' opID=HB-host-47@184250-7ea813a6] [ClientAdapterBase::InvokeOnSoap] Invoke done (192.168.101.173, vpxapi.VpxaService.retrieveChanges)


2015-08-22T01:48:32.016+08:00 [10092 info 'vpxdvpxdVmomi' opID=HB-host-47@184250-7ea813a6] [ClientAdapterBase::InvokeOnSoap] Invoke done (192.168.101.173, vmodl.query.PropertyCollector.createFilter)


2015-08-22T01:48:32.040+08:00 [19544 info 'vpxdvpxdVmomi'] [ClientAdapterBase::InvokeOnSoap] Invoke done (192.168.101.173, vmodl.query.PropertyCollector.waitForUpdates)


2015-08-22T01:48:32.077+08:00 [19544 info 'vpxdvpxdVmomi'] [ClientAdapterBase::InvokeOnSoap] Invoke done (192.168.101.173, vmodl.query.PropertyCollector.waitForUpdates)


2015-08-22T01:48:32.080+08:00 [19544 info 'vpxdvpxdVmomi'] [ClientAdapterBase::InvokeOnSoap] Invoke done (192.168.101.173, vmodl.query.PropertyCollector.waitForUpdates)


2015-08-22T01:48:32.083+08:00 [10092 info 'vpxdvpxdVmomi' opID=HB-host-47@184250-7ea813a6] [ClientAdapterBase::InvokeOnSoap] Invoke done (192.168.101.173, vmodl.query.PropertyCollector.Filter.destroy)


2015-08-22T01:48:32.085+08:00 [10092 info 'vpxdvpxdVmomi' opID=HB-host-47@184250-7ea813a6] [ClientAdapterBase::InvokeOnSoap] Invoke done (192.168.101.173, vmodl.query.PropertyCollector.cancelWaitForUpdates)


2015-08-22T01:48:32.085+08:00 [10092 info 'vpxdvpxdInvtHostCnx' opID=HB-host-47@184250-7ea813a6] [VpxdHostSync] Retrieved host update to 184250 for host-47


2015-08-22T01:48:32.086+08:00 [10092 info 'vpxdvpxdInvtHostDb' opID=HB-host-47@184250-7ea813a6] [VpxdInvtHost::SaveFieldsToDb] IPMI info of host 192.168.101.173 is not set


2015-08-22T01:48:32.091+08:00 [10092 info 'vpxdvpxdInvtHostCnx' opID=HB-host-47@184250-7ea813a6] [VpxdHostSync] Completed host synchronization for host-47

It doesn't appear to be a single host that is affected, it could be any. It also doesn't appear to happen at the same time during the day.

I've checked for known issues and found some SSL handshake timeout KB articles, but don't believe the same thing is occurring here. I have tested increasing the timeout as per that article anyway (KB 2064246) but that has not made any difference.

The devices are all connected to an HP5406ZL2 switch, with management over a vswitch with 2 x 10GBe interfaces in active / active with route based on the originating virtual Port ID set.

The server running vCentre is also the SRM server, and our backup exec server.

Any information that anyone can provide would be greatly appreciated.

1 Solution

Accepted Solutions
Wallace44
Contributor
Contributor
‎10-04-2015 07:21 PM
Jump to solution

This weekend just gone one of the hosts disconnected completely and wouldn't come back online.

To resolve we followed the info in this KB article:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=203779...

This has now been run on all the hosts and the random disconnects have disappeared. I will be looking to apply the updates mentioned in that article ASAP.

Just FYI for anyone else that stumbles across this.

View solution in original post

6 Replies
keshavkant
Enthusiast
Enthusiast
‎08-25-2015 02:08 AM
Jump to solution

Host can be disconnect randomely becouse of  Root certificate component ...You can try the solution that you must either connect to the Internet or turn off the Update Root Certificates component.

To turn off the Update Root Certificates component:

 

  1. Log into vCenter Server.
  2. Click Start > Settings > Control Panel > Add or Remove Programs.
  3. Click Add/Remove Windows Components.
  4. Deselect the Update Root Certificates checkbox, click Next, then Finish.

On Windows 2008 R2, turn off Automatic Root Certificates Update:

To turn off Automatic Root Certificates Update:

  1. Click Start > Run, type gpedit.msc, and click OK.
  2. If the User Account Control dialog box appears, confirm that the action it displays is what you want, and click Continue.
  3. Double-click Administrative Templates > System > Internet Communication Management, and click Internet Communication settings.
  4. Double-click Turn off Automatic Root Certificates Update, click Enabled, and click OK.
  5. Close the Local Group Policy Editor.
0 Kudos
Wallace44
Contributor
Contributor
‎08-25-2015 06:12 PM
Jump to solution

Thanks. I did find that article, but didn't think it related.

I've disabled the updates via gpedit on our Vcentre server and will keep an eye on it to see how it goes.

I will report back.

0 Kudos
Wallace44
Contributor
Contributor
‎08-25-2015 06:36 PM
Jump to solution

After applying the fix, another host disconnected and reconnected randomly.

This is the VPXD logs from this host disconnection;

2015-08-26T09:22:24.485+08:00 [09832 info 'vpxdvpxdVmomi'] [ClientAdapterBase::InvokeOnSoap] Invoke done (192.168.101.171, vmodl.query.PropertyCollector.waitForUpdates)


2015-08-26T09:22:24.488+08:00 [09832 info 'vpxdvpxdVmomi'] [ClientAdapterBase::InvokeOnSoap] Invoke done (192.168.101.171, vmodl.query.PropertyCollector.waitForUpdates)


2015-08-26T09:22:24.492+08:00 [09832 info 'vpxdvpxdVmomi'] [ClientAdapterBase::InvokeOnSoap] Invoke done (192.168.101.171, vmodl.query.PropertyCollector.waitForUpdates)


2015-08-26T09:22:24.495+08:00 [09832 info 'vpxdvpxdVmomi'] [ClientAdapterBase::InvokeOnSoap] Invoke done (192.168.101.171, vmodl.query.PropertyCollector.waitForUpdates)


2015-08-26T09:22:24.499+08:00 [09832 info 'vpxdvpxdVmomi'] [ClientAdapterBase::InvokeOnSoap] Invoke done (192.168.101.171, vmodl.query.PropertyCollector.waitForUpdates)


2015-08-26T09:22:24.503+08:00 [09832 info 'vpxdvpxdVmomi'] [ClientAdapterBase::InvokeOnSoap] Invoke done (192.168.101.171, vmodl.query.PropertyCollector.waitForUpdates)


2015-08-26T09:22:24.506+08:00 [09832 info 'vpxdvpxdVmomi'] [ClientAdapterBase::InvokeOnSoap] Invoke done (192.168.101.171, vmodl.query.PropertyCollector.waitForUpdates)


2015-08-26T09:22:24.509+08:00 [09832 info 'vpxdvpxdVmomi'] [ClientAdapterBase::InvokeOnSoap] Invoke done (192.168.101.171, vmodl.query.PropertyCollector.waitForUpdates)


2015-08-26T09:22:24.513+08:00 [09832 info 'vpxdvpxdVmomi'] [ClientAdapterBase::InvokeOnSoap] Invoke done (192.168.101.171, vmodl.query.PropertyCollector.waitForUpdates)


2015-08-26T09:22:24.526+08:00 [09832 info 'vpxdvpxdVmomi'] [ClientAdapterBase::InvokeOnSoap] Invoke done (192.168.101.171, vmodl.query.PropertyCollector.waitForUpdates)


2015-08-26T09:22:24.899+08:00 [09832 info 'vpxdvpxdVmomi'] [ClientAdapterBase::InvokeOnSoap] Invoke done (192.168.101.171, vmodl.query.PropertyCollector.waitForUpdates)


2015-08-26T09:22:24.915+08:00 [09852 info 'vpxdvpxdVmomi' opID=HB-host-42@145228-30bc65e1] [ClientAdapterBase::InvokeOnSoap] Invoke done (192.168.101.171, vmodl.query.PropertyCollector.Filter.destroy)


2015-08-26T09:22:24.917+08:00 [09852 info 'vpxdvpxdVmomi' opID=HB-host-42@145228-30bc65e1] [ClientAdapterBase::InvokeOnSoap] Invoke done (192.168.101.171, vmodl.query.PropertyCollector.cancelWaitForUpdates)


2015-08-26T09:22:24.917+08:00 [09852 warning 'VpxProfiler' opID=HB-host-42@145228-30bc65e1] [VpxdHostSync] GetChanges host:192.168.101.171 (192.168.101.171) [GetChangesTime] took 102881 ms


2015-08-26T09:22:24.917+08:00 [09852 info 'vpxdvpxdInvtHostCnx' opID=HB-host-42@145228-30bc65e1] [VpxdHostSync] Retrieved host update to 145228 for host-42


2015-08-26T09:22:24.917+08:00 [09852 info 'vpxdvpxdMoLicenseManager' opID=HB-host-42@145228-30bc65e1] [LicMgr] Re-acquiring licenses for Host 'host-42'.Old Host Type: 'embeddedEsx', Old License Product Name: 'VMware ESX Server', Old Version: '5.5.0',Old License Product Version: '5.0', Old File Version: '5.5.2.0',New Host Type: 'embeddedEsx', New License Product Name: 'VMware ESX Server', New Version: '5.5.0',New License Product Version: '5.0', New File Version: '5.5.2.0'


2015-08-26T09:22:24.949+08:00 [09852 info 'vpxdvpxdVmomi' opID=HB-host-42@145228-30bc65e1] [ClientAdapterBase::InvokeOnSoap] Invoke done (192.168.101.171, vim.EnvironmentBrowser.queryProvisioningPolicy)


2015-08-26T09:22:25.085+08:00 [09852 info 'vpxdvpxdMoHost' opID=HB-host-42@145228-30bc65e1] [HostMo::SetComputeCompatibilityDirty] Marked host-42 as dirty.


2015-08-26T09:22:25.085+08:00 [09852 info 'clustervpxdMoCluster' opID=HB-host-42@145228-30bc65e1] [ClusterMo::SetDasCompatDirty] Marked domain-c26 as dirty.


2015-08-26T09:22:25.100+08:00 [09852 info 'vpxdvpxdMoTransEVCManager' opID=HB-host-42@145228-30bc65e1] [MoTransEVCManager::GetMaxModeForHostCapabilities] 192.168.101.171 supports EVC mode intel-ivybridge.


2015-08-26T09:22:25.100+08:00 [09852 info 'vpxdvpxdMoTransEVCManager' opID=HB-host-42@145228-30bc65e1] [MoTransEVCManager::GetMaxModeForHostCapabilities] 192.168.101.171 supports EVC mode intel-merom.


2015-08-26T09:22:25.100+08:00 [09852 info 'vpxdvpxdMoTransEVCManager' opID=HB-host-42@145228-30bc65e1] [MoTransEVCManager::GetMaxModeForHostCapabilities] 192.168.101.171 supports EVC mode intel-nehalem.


2015-08-26T09:22:25.100+08:00 [09852 info 'vpxdvpxdMoTransEVCManager' opID=HB-host-42@145228-30bc65e1] [MoTransEVCManager::GetMaxModeForHostCapabilities] 192.168.101.171 supports EVC mode intel-penryn.


2015-08-26T09:22:25.100+08:00 [09852 info 'vpxdvpxdMoTransEVCManager' opID=HB-host-42@145228-30bc65e1] [MoTransEVCManager::GetMaxModeForHostCapabilities] 192.168.101.171 supports EVC mode intel-sandybridge.


2015-08-26T09:22:25.100+08:00 [09852 info 'vpxdvpxdMoTransEVCManager' opID=HB-host-42@145228-30bc65e1] [MoTransEVCManager::GetMaxModeForHostCapabilities] 192.168.101.171 supports EVC mode intel-westmere.


2015-08-26T09:22:25.100+08:00 [09852 info 'vpxdvpxdMoTransEVCManager' opID=HB-host-42@145228-30bc65e1] [MoTransEVCManager::GetMaxModeForHostCapabilities] 192.168.101.171 supports a maximum EVC mode of intel-ivybridge.


2015-08-26T09:22:25.488+08:00 [16416 info 'commonvpxLro' opID=HB-host-42@145228-30bc65e1-64] [VpxLRO] -- BEGIN task-internal-57845 -- host-42 -- VsanHostConfigIssuesLRO -- 


2015-08-26T09:22:25.488+08:00 [09832 info 'commonvpxLro' opID=HB-host-42@145228-30bc65e1-24] [VpxLRO] -- BEGIN task-internal-57846 -- host-43 -- VsanHostConfigIssuesLRO -- 


2015-08-26T09:22:25.488+08:00 [02844 info 'commonvpxLro' opID=HB-host-42@145228-30bc65e1-a8] [VpxLRO] -- BEGIN task-internal-57847 -- host-47 -- VsanHostConfigIssuesLRO -- 


2015-08-26T09:22:25.488+08:00 [09832 info 'commonvpxLro' opID=HB-host-42@145228-30bc65e1-24] [VpxLRO] -- FINISH task-internal-57846 -- host-43 -- VsanHostConfigIssuesLRO -- 


2015-08-26T09:22:25.488+08:00 [02844 info 'commonvpxLro' opID=HB-host-42@145228-30bc65e1-a8] [VpxLRO] -- FINISH task-internal-57847 -- host-47 -- VsanHostConfigIssuesLRO -- 


2015-08-26T09:22:25.559+08:00 [16416 info 'commonvpxLro' opID=HB-host-42@145228-30bc65e1-64] [VpxLRO] -- FINISH task-internal-57845 -- host-42 -- VsanHostConfigIssuesLRO -- 


2015-08-26T09:22:25.577+08:00 [09852 warning 'vpxdvpxdMoHost' opID=HB-host-42@145228-30bc65e1] [HostMo::GetLinkedVmInt] Getting linked vm for vmid 4 on host 192.168.101.171 - COULD NOT FIND IT


2015-08-26T09:22:25.577+08:00 [09852 warning 'vpxdvpxdMoHost' opID=HB-host-42@145228-30bc65e1] [HostMo::GetLinkedVmInt] Getting linked vm for vmid 6 on host 192.168.101.171 - COULD NOT FIND IT


2015-08-26T09:22:25.577+08:00 [09852 warning 'vpxdvpxdMoHost' opID=HB-host-42@145228-30bc65e1] [HostMo::GetLinkedVmInt] Getting linked vm for vmid 7 on host 192.168.101.171 - COULD NOT FIND IT


2015-08-26T09:22:25.577+08:00 [16416 info 'commonvpxLro' opID=HB-host-42@145228-30bc65e1-f3] [VpxLRO] -- BEGIN task-internal-57848 --  -- update-localization-catalog -- 


2015-08-26T09:22:25.577+08:00 [09852 warning 'vpxdvpxdMoHost' opID=HB-host-42@145228-30bc65e1] [HostMo::GetLinkedVmInt] Getting linked vm for vmid 27 on host 192.168.101.171 - COULD NOT FIND IT


2015-08-26T09:22:25.577+08:00 [09852 warning 'vpxdvpxdMoHost' opID=HB-host-42@145228-30bc65e1] [HostMo::GetLinkedVmInt] Getting linked vm for vmid 28 on host 192.168.101.171 - COULD NOT FIND IT


2015-08-26T09:22:25.577+08:00 [09852 warning 'vpxdvpxdMoHost' opID=HB-host-42@145228-30bc65e1] [HostMo::GetLinkedVmInt] Getting linked vm for vmid 29 on host 192.168.101.171 - COULD NOT FIND IT


2015-08-26T09:22:25.577+08:00 [09852 warning 'vpxdvpxdMoHost' opID=HB-host-42@145228-30bc65e1] [HostMo::GetLinkedVmInt] Getting linked vm for vmid 30 on host 192.168.101.171 - COULD NOT FIND IT


2015-08-26T09:22:25.577+08:00 [16416 info 'Localization' opID=HB-host-42@145228-30bc65e1-f3] [VpxdUpdateLocaleLro::ProcessHostUpdate]Processing update for host 


2015-08-26T09:22:25.577+08:00 [09852 warning 'vpxdvpxdMoHost' opID=HB-host-42@145228-30bc65e1] [HostMo::GetLinkedVmInt] Getting linked vm for vmid 31 on host 192.168.101.171 - COULD NOT FIND IT


2015-08-26T09:22:25.577+08:00 [09852 warning 'vpxdvpxdMoHost' opID=HB-host-42@145228-30bc65e1] [HostMo::GetLinkedVmInt] Getting linked vm for vmid 32 on host 192.168.101.171 - COULD NOT FIND IT


2015-08-26T09:22:25.578+08:00 [16416 info 'commonvpxLro' opID=HB-host-42@145228-30bc65e1-f3] [VpxLRO] -- FINISH task-internal-57848 --  -- update-localization-catalog -- 


2015-08-26T09:22:25.586+08:00 [09852 info 'vpxdoverheadMemory' opID=HB-host-42@145228-30bc65e1] [OverheadCache::Invalidate] VM vim.VirtualMachine:vm-70: config changed, invalidating cache...


2015-08-26T09:22:25.606+08:00 [09852 info 'vpxdoverheadMemory' opID=HB-host-42@145228-30bc65e1] [OverheadCache::Invalidate] VM vim.VirtualMachine:vm-77: config changed, invalidating cache...


2015-08-26T09:22:25.634+08:00 [09852 info 'vpxdoverheadMemory' opID=HB-host-42@145228-30bc65e1] [OverheadCache::Invalidate] VM vim.VirtualMachine:vm-80: config changed, invalidating cache...


2015-08-26T09:22:25.654+08:00 [09852 info 'vpxdoverheadMemory' opID=HB-host-42@145228-30bc65e1] [OverheadCache::Invalidate] VM vim.VirtualMachine:vm-102: config changed, invalidating cache...


2015-08-26T09:22:25.676+08:00 [09852 info 'vpxdoverheadMemory' opID=HB-host-42@145228-30bc65e1] [OverheadCache::Invalidate] VM vim.VirtualMachine:vm-66: config changed, invalidating cache...


2015-08-26T09:22:25.714+08:00 [09852 info 'vpxdoverheadMemory' opID=HB-host-42@145228-30bc65e1] [OverheadCache::Invalidate] VM vim.VirtualMachine:vm-145: config changed, invalidating cache...


2015-08-26T09:22:25.744+08:00 [09852 info 'vpxdoverheadMemory' opID=HB-host-42@145228-30bc65e1] [OverheadCache::Invalidate] VM vim.VirtualMachine:vm-147: config changed, invalidating cache...


2015-08-26T09:22:25.789+08:00 [09852 info 'vpxdoverheadMemory' opID=HB-host-42@145228-30bc65e1] [OverheadCache::Invalidate] VM vim.VirtualMachine:vm-164: config changed, invalidating cache...


2015-08-26T09:22:25.818+08:00 [09852 info 'vpxdoverheadMemory' opID=HB-host-42@145228-30bc65e1] [OverheadCache::Invalidate] VM vim.VirtualMachine:vm-73: config changed, invalidating cache...


2015-08-26T09:22:25.894+08:00 [09852 error 'drmLogger' opID=HB-host-42@145228-30bc65e1] Datastore 52ce36db-6fee25c2-3616-3cd92bf3e984 not found


2015-08-26T09:22:25.894+08:00 [09852 error 'drmLogger' opID=HB-host-42@145228-30bc65e1] Datastore with uuid 52ce36db-6fee25c2-3616-3cd92bf3e984 not found, ignoring for correlations.


2015-08-26T09:22:25.894+08:00 [09852 error 'drmLogger' opID=HB-host-42@145228-30bc65e1] Datastore with uuid 5433587b-9daed671-6f09-3cd92bf3e984 not found, ignoring for correlations.


2015-08-26T09:22:25.894+08:00 [09852 error 'drmLogger' opID=HB-host-42@145228-30bc65e1] Datastore 5433587b-9daed671-6f09-3cd92bf3e984 not found


2015-08-26T09:22:25.902+08:00 [09852 info 'vpxdvpxdInvtHostDb' opID=HB-host-42@145228-30bc65e1] [VpxdInvtHost::SaveFieldsToDb] IPMI info of host 192.168.101.171 is not set


2015-08-26T09:22:26.594+08:00 [16416 info 'commonvpxLro' opID=A7CDA854-0000006B-1e] [VpxLRO] -- BEGIN task-internal-57849 --  -- vmodl.query.PropertyCollector.retrieveContents -- 797a6059-95cf-5fd5-da29-2acdd9530c27(52f3ae38-3d5e-a3a2-879e-32f6770b2b13)


2015-08-26T09:22:26.595+08:00 [16416 info 'commonvpxLro' opID=A7CDA854-0000006B-1e] [VpxLRO] -- FINISH task-internal-57849 --  -- vmodl.query.PropertyCollector.retrieveContents -- 


2015-08-26T09:22:26.595+08:00 [16416 info 'commonvpxLro' opID=A7CDA854-0000006C-b4] [VpxLRO] -- BEGIN task-internal-57850 --  -- vmodl.query.PropertyCollector.retrieveContents -- 797a6059-95cf-5fd5-da29-2acdd9530c27(52f3ae38-3d5e-a3a2-879e-32f6770b2b13)


2015-08-26T09:22:26.595+08:00 [16416 info 'commonvpxLro' opID=A7CDA854-0000006C-b4] [VpxLRO] -- FINISH task-internal-57850 --  -- vmodl.query.PropertyCollector.retrieveContents -- 


2015-08-26T09:22:26.597+08:00 [17708 info 'commonvpxLro' opID=A7CDA854-0000006D-26] [VpxLRO] -- BEGIN task-internal-57851 --  -- vim.LicenseAssignmentManager.queryAssignedLicenses -- 797a6059-95cf-5fd5-da29-2acdd9530c27(52f3ae38-3d5e-a3a2-879e-32f6770b2b13)


2015-08-26T09:22:26.597+08:00 [17208 info 'commonvpxLro' opID=A7CDA854-0000006E-7b] [VpxLRO] -- BEGIN task-internal-57852 --  -- vim.LicenseAssignmentManager.queryAssignedLicenses -- 797a6059-95cf-5fd5-da29-2acdd9530c27(52f3ae38-3d5e-a3a2-879e-32f6770b2b13)


2015-08-26T09:22:26.597+08:00 [17708 info 'commonvpxLro' opID=A7CDA854-0000006D-26] [VpxLRO] -- FINISH task-internal-57851 --  -- vim.LicenseAssignmentManager.queryAssignedLicenses -- 


2015-08-26T09:22:26.598+08:00 [17208 info 'commonvpxLro' opID=A7CDA854-0000006E-7b] [VpxLRO] -- FINISH task-internal-57852 --  -- vim.LicenseAssignmentManager.queryAssignedLicenses -- 


2015-08-26T09:22:27.337+08:00 [09852 info 'vpxdvpxdVmomi' opID=HB-host-42@145228-30bc65e1] [ClientAdapterBase::InvokeOnSoap] Invoke done (192.168.101.171, vpxapi.VpxaService.setServer)


2015-08-26T09:22:27.338+08:00 [09852 info 'vpxdvpxdMoLicenseDataManager' opID=HB-host-42@145228-30bc65e1] [LicenseDataManager::Processor] Processed auto-assigned licenses for host 192.168.101.171


2015-08-26T09:22:27.338+08:00 [09852 info 'vpxdvpxdInvtHostCnx' opID=HB-host-42@145228-30bc65e1] [VpxdHostSync] Completed host synchronization for host-42


2015-08-26T09:22:27.343+08:00 [09852 warning 'VpxProfiler' opID=HB-host-42@145228-30bc65e1] [VpxdHostSync] DoHostSync:0000000009D0F6C0 [DoHostSyncTime] took 105308 ms


2015-08-26T09:22:27.343+08:00 [09852 info 'vpxdvpxdInvtHostCnx' opID=HB-host-42@145228-30bc65e1] [VpxdHostSpecSync] saving host update for [vim.HostSystem:host-42,192.168.101.171] to vpxaMasterSpecGenNo = 0, vpxdMasterSpecGenNo = 4737


2015-08-26T09:22:27.343+08:00 [09852 info 'drmLogger' opID=HB-host-42@145228-30bc65e1] [VpxDrmStatsManager::GetMetricsToCollect] No datastores requiring stats collection on host vim.HostSystem:host-42, early return


2015-08-26T09:22:27.345+08:00 [09852 info 'vpxdvpxdInvtHostCnx' opID=HB-host-42@145228-30bc65e1] [VpxdHostSpecSync]Updating vpxa for [vim.HostSystem:host-42,192.168.101.171] (spec. genno 4737)


2015-08-26T09:22:28.173+08:00 [16416 info 'Default' opID=SWI-45976d81] [ClusterCompatibilitySet::ComputeCompatibilitySet] Before lock Dirty hosts: 1 ; Dirty VMs: 0 ; Clean hosts: 2 ; Clean VMs: 24


2015-08-26T09:22:29.280+08:00 [09852 info 'vpxdvpxdVmomi' opID=HB-host-42@145228-30bc65e1] [ClientAdapterBase::InvokeOnSoap] Invoke done (192.168.101.171, vpxapi.VpxaService.setConfig)


2015-08-26T09:22:29.281+08:00 [09852 info 'vpxdvpxdInvtHostDb' opID=HB-host-42@145228-30bc65e1] [VpxdInvtHost::SaveFieldsToDb] IPMI info of host 192.168.101.171 is not set


2015-08-26T09:22:29.290+08:00 [09852 info 'DAS' opID=HB-host-42@145228-30bc65e1] [VpxdDas::FdmCleanUp] Host [vim.HostSystem:host-42,192.168.101.171] is in a HA cluster but doesn't have an unconfig error. Skipping the cleanup as it is not needed


2015-08-26T09:22:29.356+08:00 [09852 info 'vpxdvpxdMoHost' opID=HB-host-42@145228-30bc65e1] [HostMo] host connection state changed to [CONNECTED] for host-42


2015-08-26T09:22:29.357+08:00 [09852 info 'vmmoVm' opID=HB-host-42@145228-30bc65e1] [VmMo::SetComputeCompatibilityDirty] vm vm-77 is marked dirty


2015-08-26T09:22:29.357+08:00 [09852 info 'clustervpxdMoCluster' opID=HB-host-42@145228-30bc65e1] [ClusterMo::SetDasCompatDirty] Marked domain-c26 as dirty.


2015-08-26T09:22:29.357+08:00 [09852 info 'vmmoVm' opID=HB-host-42@145228-30bc65e1] [VmMo::SetComputeCompatibilityDirty] vm vm-66 is marked dirty


2015-08-26T09:22:29.358+08:00 [09852 info 'clustervpxdMoCluster' opID=HB-host-42@145228-30bc65e1] [ClusterMo::SetDasCompatDirty] Marked domain-c26 as dirty.


2015-08-26T09:22:29.358+08:00 [09852 info 'vmmoVm' opID=HB-host-42@145228-30bc65e1] [VmMo::SetComputeCompatibilityDirty] vm vm-145 is marked dirty


2015-08-26T09:22:29.358+08:00 [09852 info 'clustervpxdMoCluster' opID=HB-host-42@145228-30bc65e1] [ClusterMo::SetDasCompatDirty] Marked domain-c26 as dirty.


2015-08-26T09:22:29.358+08:00 [09852 info 'vmmoVm' opID=HB-host-42@145228-30bc65e1] [VmMo::SetComputeCompatibilityDirty] vm vm-102 is marked dirty


2015-08-26T09:22:29.358+08:00 [09852 info 'clustervpxdMoCluster' opID=HB-host-42@145228-30bc65e1] [ClusterMo::SetDasCompatDirty] Marked domain-c26 as dirty.


2015-08-26T09:22:29.367+08:00 [09852 info 'vmmoVm' opID=HB-host-42@145228-30bc65e1] [VmMo::SetComputeCompatibilityDirty] vm vm-80 is marked dirty


2015-08-26T09:22:29.367+08:00 [09852 info 'clustervpxdMoCluster' opID=HB-host-42@145228-30bc65e1] [ClusterMo::SetDasCompatDirty] Marked domain-c26 as dirty.


2015-08-26T09:22:29.368+08:00 [09852 info 'vmmoVm' opID=HB-host-42@145228-30bc65e1] [VmMo::SetComputeCompatibilityDirty] vm vm-147 is marked dirty


2015-08-26T09:22:29.368+08:00 [09852 info 'clustervpxdMoCluster' opID=HB-host-42@145228-30bc65e1] [ClusterMo::SetDasCompatDirty] Marked domain-c26 as dirty.


2015-08-26T09:22:29.368+08:00 [09852 info 'vmmoVm' opID=HB-host-42@145228-30bc65e1] [VmMo::SetComputeCompatibilityDirty] vm vm-73 is marked dirty


2015-08-26T09:22:29.368+08:00 [09852 info 'clustervpxdMoCluster' opID=HB-host-42@145228-30bc65e1] [ClusterMo::SetDasCompatDirty] Marked domain-c26 as dirty.


2015-08-26T09:22:29.369+08:00 [09852 info 'vmmoVm' opID=HB-host-42@145228-30bc65e1] [VmMo::SetComputeCompatibilityDirty] vm vm-164 is marked dirty


2015-08-26T09:22:29.369+08:00 [09852 info 'clustervpxdMoCluster' opID=HB-host-42@145228-30bc65e1] [ClusterMo::SetDasCompatDirty] Marked domain-c26 as dirty.


2015-08-26T09:22:29.369+08:00 [09852 info 'vmmoVm' opID=HB-host-42@145228-30bc65e1] [VmMo::SetComputeCompatibilityDirty] vm vm-70 is marked dirty


2015-08-26T09:22:29.369+08:00 [09852 info 'clustervpxdMoCluster' opID=HB-host-42@145228-30bc65e1] [ClusterMo::SetDasCompatDirty] Marked domain-c26 as dirty.


2015-08-26T09:22:29.369+08:00 [09852 info 'vpxdvpxdMoHost' opID=HB-host-42@145228-30bc65e1] [HostMo::SetComputeCompatibilityDirty] Marked host-42 as dirty.


2015-08-26T09:22:29.369+08:00 [09852 info 'clustervpxdMoCluster' opID=HB-host-42@145228-30bc65e1] [ClusterMo::SetDasCompatDirty] Marked domain-c26 as dirty.


2015-08-26T09:22:29.391+08:00 [04728 info 'vpxdvpxdVmomi' opID=HB-host-42@145228-30bc65e1-SWI-99337cb] [ClientAdapterBase::InvokeOnSoap] Invoke done (192.168.101.171, vim.dvs.HostDistributedVirtualSwitchManager.applyDvsList)


2015-08-26T09:22:29.393+08:00 [09852 warning 'VpxProfiler' opID=HB-host-42@145228-30bc65e1] InvtHostSyncLRO::StartWork [HostSyncTime] took 107358 ms


2015-08-26T09:22:29.393+08:00 [09852 info 'commonvpxLro' opID=HB-host-42@145228-30bc65e1] [VpxLRO] -- FINISH task-internal-57823 -- host-42 -- VpxdInvtHostSyncHostLRO.Synchronize -- 


2015-08-26T09:22:29.415+08:00 [21316 info 'vpxdvpxdVmomi' opID=HB-host-42@145228-30bc65e1-SWI-332ff19e] [ClientAdapterBase::InvokeOnSoap] Invoke done (192.168.101.171, vpxapi.VpxaService.ConfigureDatastoreIORMOnHost)


2015-08-26T09:22:29.423+08:00 [21316 info 'vpxdvpxdVmomi' opID=HB-host-42@145228-30bc65e1-SWI-332ff19e] [ClientAdapterBase::InvokeOnSoap] Invoke done (192.168.101.171, vpxapi.VpxaService.ConfigureDatastoreIORMOnHost)


2015-08-26T09:22:29.431+08:00 [21316 info 'vpxdvpxdVmomi' opID=HB-host-42@145228-30bc65e1-SWI-332ff19e] [ClientAdapterBase::InvokeOnSoap] Invoke done (192.168.101.171, vpxapi.VpxaService.ConfigureDatastoreIORMOnHost)


2015-08-26T09:22:29.439+08:00 [21316 info 'vpxdvpxdVmomi' opID=HB-host-42@145228-30bc65e1-SWI-332ff19e] [ClientAdapterBase::InvokeOnSoap] Invoke done (192.168.101.171, vpxapi.VpxaService.ConfigureDatastoreIORMOnHost)


2015-08-26T09:22:29.447+08:00 [21316 info 'vpxdvpxdVmomi' opID=HB-host-42@145228-30bc65e1-SWI-332ff19e] [ClientAdapterBase::InvokeOnSoap] Invoke done (192.168.101.171, vpxapi.VpxaService.ConfigureDatastoreIORMOnHost)


2015-08-26T09:22:29.455+08:00 [21316 info 'vpxdvpxdVmomi' opID=HB-host-42@145228-30bc65e1-SWI-332ff19e] [ClientAdapterBase::InvokeOnSoap] Invoke done (192.168.101.171, vpxapi.VpxaService.ConfigureDatastoreIORMOnHost)


2015-08-26T09:22:37.301+08:00 [09292 info 'vpxdvpxdInvtHostCnx' opID=SWI-10381aa1] [VpxdIntHost] Missed 12 heartbeats for host 192.168.101.171


2015-08-26T09:22:37.301+08:00 [09400 info 'commonvpxLro' opID=HB-host-42@145231-3b67679] [VpxLRO] -- BEGIN task-internal-57855 -- host-42 -- VpxdInvtHostSyncHostLRO.Synchronize -- 


2015-08-26T09:22:37.302+08:00 [09400 info 'vpxdvpxdInvtHostCnx' opID=HB-host-42@145231-3b67679] [VpxdHostSync] Synchronizing host: host-42 (192.168.101.171, 192.168.101.171)


2015-08-26T09:22:37.353+08:00 [09400 info 'vpxdvpxdVmomi' opID=HB-host-42@145231-3b67679] [ClientAdapterBase::InvokeOnSoap] Invoke done (192.168.101.171, vpxapi.VpxaService.retrieveChanges)


2015-08-26T09:22:37.358+08:00 [09400 info 'vpxdvpxdVmomi' opID=HB-host-42@145231-3b67679] [ClientAdapterBase::InvokeOnSoap] Invoke done (192.168.101.171, vmodl.query.PropertyCollector.createFilter)


2015-08-26T09:22:37.359+08:00 [10092 info 'vpxdvpxdVmomi'] [ClientAdapterBase::InvokeOnSoap] Invoke done (192.168.101.171, vmodl.query.PropertyCollector.waitForUpdates)


2015-08-26T09:22:37.727+08:00 [10092 info 'vpxdvpxdVmomi'] [ClientAdapterBase::InvokeOnSoap] Invoke done (192.168.101.171, vmodl.query.PropertyCollector.waitForUpdates)


2015-08-26T09:22:37.748+08:00 [10092 info 'vpxdvpxdVmomi'] [ClientAdapterBase::InvokeOnSoap] Invoke done (192.168.101.171, vmodl.query.PropertyCollector.waitForUpdates)


2015-08-26T09:22:37.751+08:00 [09400 info 'vpxdvpxdVmomi' opID=HB-host-42@145231-3b67679] [ClientAdapterBase::InvokeOnSoap] Invoke done (192.168.101.171, vmodl.query.PropertyCollector.Filter.destroy)


2015-08-26T09:22:37.753+08:00 [09400 info 'vpxdvpxdVmomi' opID=HB-host-42@145231-3b67679] [ClientAdapterBase::InvokeOnSoap] Invoke done (192.168.101.171, vmodl.query.PropertyCollector.cancelWaitForUpdates)


2015-08-26T09:22:37.754+08:00 [09400 info 'vpxdvpxdInvtHostCnx' opID=HB-host-42@145231-3b67679] [VpxdHostSync] Retrieved host update to 145231 for host-42


2015-08-26T09:22:37.754+08:00 [09400 info 'vpxdvpxdMoLicenseManager' opID=HB-host-42@145231-3b67679] [LicMgr] Re-acquiring licenses for Host 'host-42'.Old Host Type: 'embeddedEsx', Old License Product Name: 'VMware ESX Server', Old Version: '5.5.0',Old License Product Version: '5.0', Old File Version: '5.5.2.0',New Host Type: 'embeddedEsx', New License Product Name: 'VMware ESX Server', New Version: '5.5.0',New License Product Version: '5.0', New File Version: '5.5.2.0'


2015-08-26T09:22:37.817+08:00 [09400 info 'vpxdvpxdMoHost' opID=HB-host-42@145231-3b67679] [HostMo::SetComputeCompatibilityDirty] Marked host-42 as dirty.


2015-08-26T09:22:37.817+08:00 [09400 info 'clustervpxdMoCluster' opID=HB-host-42@145231-3b67679] [ClusterMo::SetDasCompatDirty] Marked domain-c26 as dirty.


2015-08-26T09:22:37.842+08:00 [09400 info 'vpxdvpxdInvtHostDb' opID=HB-host-42@145231-3b67679] [VpxdInvtHost::SaveFieldsToDb] IPMI info of host 192.168.101.171 is not set


2015-08-26T09:22:37.847+08:00 [09400 info 'vpxdvpxdInvtHostCnx' opID=HB-host-42@145231-3b67679] [VpxdHostSync] Completed host synchronization for host-42


2015-08-26T09:22:37.847+08:00 [09400 info 'vpxdvpxdInvtHostCnx' opID=HB-host-42@145231-3b67679] [VpxdHostSpecSync] saving host update for [vim.HostSystem:host-42,192.168.101.171] to vpxaMasterSpecGenNo = 4737, vpxdMasterSpecGenNo = 4741


2015-08-26T09:22:37.847+08:00 [09400 info 'drmLogger' opID=HB-host-42@145231-3b67679] [VpxDrmStatsManager::GetMetricsToCollect] No datastores requiring stats collection on host vim.HostSystem:host-42, early return


2015-08-26T09:22:37.848+08:00 [09400 info 'vpxdvpxdInvtHostCnx' opID=HB-host-42@145231-3b67679] [VpxdHostSpecSync]Updating vpxa for [vim.HostSystem:host-42,192.168.101.171] (spec. genno 4741)


2015-08-26T09:22:37.882+08:00 [09400 info 'vpxdvpxdVmomi' opID=HB-host-42@145231-3b67679] [ClientAdapterBase::InvokeOnSoap] Invoke done (192.168.101.171, vpxapi.VpxaService.setConfig)


2015-08-26T09:22:37.883+08:00 [09400 info 'vpxdvpxdInvtHostDb' opID=HB-host-42@145231-3b67679] [VpxdInvtHost::SaveFieldsToDb] IPMI info of host 192.168.101.171 is not set


2015-08-26T09:22:37.884+08:00 [09400 info 'commonvpxLro' opID=HB-host-42@145231-3b67679] [VpxLRO] -- FINISH task-internal-57855 -- host-42 -- VpxdInvtHostSyncHostLRO.Synchronize -- 


2015-08-26T09:22:37.934+08:00 [09852 warning 'Locale'] No message string to format object vim.option.OptionDef.


--> 


2015-08-26T09:22:37.936+08:00 [09852 warning 'Locale'] No message string to format object vim.option.OptionDef.


--> 


2015-08-26T09:22:39.541+08:00 [08824 info 'commonvpxLro' opID=HB-host-47@189559-6438f185] [VpxLRO] -- BEGIN task-internal-57856 -- host-47 -- VpxdInvtHostSyncHostLRO.Synchronize -- 


2015-08-26T09:22:39.541+08:00 [08824 info 'vpxdvpxdInvtHostCnx' opID=HB-host-47@189559-6438f185] [VpxdHostSync] Synchronizing host: host-47 (192.168.101.173, 192.168.101.173)


2015-08-26T09:22:39.545+08:00 [08824 info 'vpxdvpxdVmomi' opID=HB-host-47@189559-6438f185] [ClientAdapterBase::InvokeOnSoap] Invoke done (192.168.101.173, vpxapi.VpxaService.retrieveChanges)


2015-08-26T09:22:39.548+08:00 [08824 info 'vpxdvpxdVmomi' opID=HB-host-47@189559-6438f185] [ClientAdapterBase::InvokeOnSoap] Invoke done (192.168.101.173, vmodl.query.PropertyCollector.createFilter)


2015-08-26T09:22:39.571+08:00 [09832 info 'vpxdvpxdVmomi'] [ClientAdapterBase::InvokeOnSoap] Invoke done (192.168.101.173, vmodl.query.PropertyCollector.waitForUpdates)


2015-08-26T09:22:39.665+08:00 [09832 info 'vpxdvpxdVmomi'] [ClientAdapterBase::InvokeOnSoap] Invoke done (192.168.101.173, vmodl.query.PropertyCollector.waitForUpdates)


2015-08-26T09:22:39.670+08:00 [09832 info 'vpxdvpxdVmomi'] [ClientAdapterBase::InvokeOnSoap] Invoke done (192.168.101.173, vmodl.query.PropertyCollector.waitForUpdates)


2015-08-26T09:22:39.673+08:00 [08824 info 'vpxdvpxdVmomi' opID=HB-host-47@189559-6438f185] [ClientAdapterBase::InvokeOnSoap] Invoke done (192.168.101.173, vmodl.query.PropertyCollector.Filter.destroy)


2015-08-26T09:22:39.675+08:00 [08824 info 'vpxdvpxdVmomi' opID=HB-host-47@189559-6438f185] [ClientAdapterBase::InvokeOnSoap] Invoke done (192.168.101.173, vmodl.query.PropertyCollector.cancelWaitForUpdates)


2015-08-26T09:22:39.676+08:00 [08824 info 'vpxdvpxdInvtHostCnx' opID=HB-host-47@189559-6438f185] [VpxdHostSync] Retrieved host update to 189559 for host-47


2015-08-26T09:22:39.690+08:00 [08824 info 'vpxdvpxdInvtHostDb' opID=HB-host-47@189559-6438f185] [VpxdInvtHost::SaveFieldsToDb] IPMI info of host 192.168.101.173 is not set


2015-08-26T09:22:39.697+08:00 [08824 info 'vpxdvpxdInvtHostCnx' opID=HB-host-47@189559-6438f185] [VpxdHostSync] Completed host synchronization for host-47


2015-08-26T09:22:39.697+08:00 [08824 info 'commonvpxLro' opID=HB-host-47@189559-6438f185] [VpxLRO] -- FINISH task-internal-57856 -- host-47 -- VpxdInvtHostSyncHostLRO.Synchronize -- 


2015-08-26T09:22:47.304+08:00 [08824 info 'commonvpxLro' opID=HB-host-42@145233-35e1ad81] [VpxLRO] -- BEGIN task-internal-57857 -- host-42 -- VpxdInvtHostSyncHostLRO.Synchronize -- 


2015-08-26T09:22:47.304+08:00 [08824 info 'vpxdvpxdInvtHostCnx' opID=HB-host-42@145233-35e1ad81] [VpxdHostSync] Synchronizing host: host-42 (192.168.101.171, 192.168.101.171)


2015-08-26T09:22:47.309+08:00 [08824 info 'vpxdvpxdVmomi' opID=HB-host-42@145233-35e1ad81] [ClientAdapterBase::InvokeOnSoap] Invoke done (192.168.101.171, vpxapi.VpxaService.retrieveChanges)


2015-08-26T09:22:47.313+08:00 [08824 info 'vpxdvpxdVmomi' opID=HB-host-42@145233-35e1ad81] [ClientAdapterBase::InvokeOnSoap] Invoke done (192.168.101.171, vmodl.query.PropertyCollector.createFilter)


2015-08-26T09:22:47.326+08:00 [10564 info 'vpxdvpxdVmomi'] [ClientAdapterBase::InvokeOnSoap] Invoke done (192.168.101.171, vmodl.query.PropertyCollector.waitForUpdates)


2015-08-26T09:22:47.647+08:00 [10564 info 'vpxdvpxdVmomi'] [ClientAdapterBase::InvokeOnSoap] Invoke done (192.168.101.171, vmodl.query.PropertyCollector.waitForUpdates)


2015-08-26T09:22:47.668+08:00 [10564 info 'vpxdvpxdVmomi'] [ClientAdapterBase::InvokeOnSoap] Invoke done (192.168.101.171, vmodl.query.PropertyCollector.waitForUpdates)


2015-08-26T09:22:47.673+08:00 [08824 info 'vpxdvpxdVmomi' opID=HB-host-42@145233-35e1ad81] [ClientAdapterBase::InvokeOnSoap] Invoke done (192.168.101.171, vmodl.query.PropertyCollector.Filter.destroy)


2015-08-26T09:22:47.679+08:00 [08824 info 'vpxdvpxdVmomi' opID=HB-host-42@145233-35e1ad81] [ClientAdapterBase::InvokeOnSoap] Invoke done (192.168.101.171, vmodl.query.PropertyCollector.cancelWaitForUpdates)


2015-08-26T09:22:48.679+08:00 [08824 info 'vpxdvpxdInvtHostCnx' opID=HB-host-42@145233-35e1ad81] [VpxdHostSync] Retrieved host update to 145233 for host-42


2015-08-26T09:22:48.679+08:00 [08824 info 'vpxdvpxdMoLicenseManager' opID=HB-host-42@145233-35e1ad81] [LicMgr] Re-acquiring licenses for Host 'host-42'.Old Host Type: 'embeddedEsx', Old License Product Name: 'VMware ESX Server', Old Version: '5.5.0',Old License Product Version: '5.0', Old File Version: '5.5.2.0',New Host Type: 'embeddedEsx', New License Product Name: 'VMware ESX Server', New Version: '5.5.0',New License Product Version: '5.0', New File Version: '5.5.2.0'


2015-08-26T09:22:48.732+08:00 [08824 info 'vpxdvpxdMoHost' opID=HB-host-42@145233-35e1ad81] [HostMo::SetComputeCompatibilityDirty] Marked host-42 as dirty.


2015-08-26T09:22:48.732+08:00 [08824 info 'clustervpxdMoCluster' opID=HB-host-42@145233-35e1ad81] [ClusterMo::SetDasCompatDirty] Marked domain-c26 as dirty.


2015-08-26T09:22:49.120+08:00 [08824 info 'vpxdvpxdInvtHostDb' opID=HB-host-42@145233-35e1ad81] [VpxdInvtHost::SaveFieldsToDb] IPMI info of host 192.168.101.171 is not set


2015-08-26T09:22:49.129+08:00 [08824 info 'vpxdvpxdInvtHostCnx' opID=HB-host-42@145233-35e1ad81] [VpxdHostSync] Completed host synchronization for host-42


2015-08-26T09:22:49.129+08:00 [08824 info 'commonvpxLro' opID=HB-host-42@145233-35e1ad81] [VpxLRO] -- FINISH task-internal-57857 -- host-42 -- VpxdInvtHostSyncHostLRO.Synchronize -- 


2015-08-26T09:22:49.215+08:00 [08736 warning 'Locale'] No message string to format object vim.option.OptionDef.


--> 


2015-08-26T09:22:49.217+08:00 [08736 warning 'Locale'] No message string to format object vim.option.OptionDef.


--> 


2015-08-26T09:22:51.748+08:00 [09988 info 'vpxdvpxdVmomi' opID=SWI-1021c707] [ClientAdapterBase::InvokeOnSoap] Invoke done (192.168.101.171, vpxapi.VpxaService.fetchQuickStats)


2015-08-26T09:22:51.748+08:00 [09988 info 'vpxdvpxdVmomi' opID=SWI-1021c707] [ClientAdapterBase::InvokeOnSoap] Invoke done (192.168.101.172, vpxapi.VpxaService.fetchQuickStats)


2015-08-26T09:22:51.748+08:00 [09988 info 'vpxdvpxdVmomi' opID=SWI-1021c707] [ClientAdapterBase::InvokeOnSoap] Invoke done (192.168.101.173, vpxapi.VpxaService.fetchQuickStats)

Any other suggestions?

0 Kudos
Sivapb
Enthusiast
Enthusiast
‎08-26-2015 12:05 AM
Jump to solution

hi,

This issue occurs because the vCenter Server agent on the host (vpxa) fails to send heartbeats to the vCenter Server.

Thanks Siva.PB

0 Kudos
Wallace44
Contributor
Contributor
‎08-26-2015 12:42 AM
Jump to solution

Thanks for that info.

Any info on how to resolve it or is this a known issue?

0 Kudos
Wallace44
Contributor
Contributor
‎10-04-2015 07:21 PM
Jump to solution

This weekend just gone one of the hosts disconnected completely and wouldn't come back online.

To resolve we followed the info in this KB article:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=203779...

This has now been run on all the hosts and the random disconnects have disappeared. I will be looking to apply the updates mentioned in that article ASAP.

Just FYI for anyone else that stumbles across this.