I'm baffled by this one.... Hopefully someone can enlight me
The facts.
I have a VCSA 6.7 (6.7.0.21000) running with 2 clusters and 5 hosts running either ESXi 6.0 or 6.7.
Nothing fancy to it at all; selfsigned certs, AD over LDAP integrated. Very straight forward really.
All hosts and VCSA are on the same internal /24 subnet, and using 2 public DNS servers that my company controls - not my table though.
The history.
The VCSA was deployed back in December 18/January 19, and the 5 hosts were added to the vCenter with running VM's. The hosts were running 6.0 and some were later upgraded to 6.7.
The problem.
I want to add another host to the vCenter to balance the load of a cluster. But it's a nogo.
From the UI the task reaches 0% and times out after 15 minutes of idle time. "A general system error occured: Connection times out" Below I'll paste some findings from the logs.
Fun facts.
I installed the host with 6.7 Dell Customized. After troubleshotting the issue for a while, I decided to install 6.0 Dell Customized - but the same result. However.... for some reason I still have the old vSphere 6.0 Windows vCenter running. And the ESXi 6.0 connects JUST FINE to that one. Huh?
Disconnected the host from vCenter 6.0 and tried connection to VCSA 6.7 - still no luck. Baffled.... (The Windows vCenter is likewise on the same internal subnet)
All hosts are Dell PowerEdge M630 with 2 x 2-ports QLogic 57810 10G adapter. 2 ports are used for iSCSI storage and 2 for VM traffic, mgmt, vMotion etc.
Hostd.log
Accepted password for user root from 10.230.9.101 (VCSA)
2019-03-23T11:45:57.046Z info hostd[39A40B70] [Originator@6876 sub=Vimsvc.ha-eventmgr opID=jtlcpedv-5521-auto-49f-h5:70000258-bb-3d33 user=:VSPHERE.LOCAL\Administrator] Event 143 : User root@10.230.9.101 logged in as VMware-client/6.5.0
2019-03-23T11:45:57.062Z info hostd[392C1B70] [Originator@6876 sub=Hostsvc.VsanSystemProvider opID=jtlcpedv-5521-auto-49f-h5:70000258-bb-3d38 user=root:VSPHERE.LOCAL\Administrator] Complete, runtime info: (vim.vsan.host.VsanRuntimeInfo) {
--> accessGenNo = 0,
--> }
2019-03-23T11:45:57.097Z warning hostd[39AC2B70] [Originator@6876 sub=HTTP server] UnimplementedRequestHandler: HTTP method POST not supported for URI /fdm
2019-03-23T11:45:57.098Z info hostd[392C1B70] [Originator@6876 sub=Vimsvc.ha-eventmgr opID=jtlcpedv-5521-auto-49f-h5:70000258-bb-3d39 user=root:VSPHERE.LOCAL\Administrator] Event 144 : User root@10.230.9.101 logged out (login time: Saturday, 23 March, 2019 11:45:57, number of API invocations: 0, user agent: VMware-client/6.5.0)
[LikewiseGetDomainJoinInfo:355] QueryInformation(): ERROR_FILE_NOT_FOUND (2/0):
Accepted password for user root from 10.230.9.101
2019-03-23T11:45:58.605Z info hostd[392C1B70] [Originator@6876 sub=Vimsvc.ha-eventmgr opID=jtlcpedv-5524-auto-49h-h5:70000259-f-3d3c user=:VSPHERE.LOCAL\Administrator] Event 145 : User root@10.230.9.101 logged in as VMware-client/6.5.0
2019-03-23T11:45:58.622Z info hostd[39AC2B70] [Originator@6876 sub=Hostsvc.VsanSystemProvider opID=jtlcpedv-5524-auto-49h-h5:70000259-f-3d41 user=root:VSPHERE.LOCAL\Administrator] Complete, runtime info: (vim.vsan.host.VsanRuntimeInfo) {
--> accessGenNo = 0,
--> }
2019-03-23T11:45:58.656Z warning hostd[39A40B70] [Originator@6876 sub=HTTP server] UnimplementedRequestHandler: HTTP method POST not supported for URI /fdm
2019-03-23T11:45:58.657Z info hostd[39AC2B70] [Originator@6876 sub=Vimsvc.ha-eventmgr opID=jtlcpedv-5524-auto-49h-h5:70000259-f-3d42 user=root:VSPHERE.LOCAL\Administrator] Event 146 : User root@10.230.9.101 logged out (login time: Saturday, 23 March, 2019 11:45:58, number of API invocations: 0, user agent: VMware-client/6.5.0)
[LikewiseGetDomainJoinInfo:355] QueryInformation(): ERROR_FILE_NOT_FOUND (2/0):
Accepted password for user root from 10.230.9.101
2019-03-23T11:46:03.030Z info hostd[39AC2B70] [Originator@6876 sub=Vimsvc.ha-eventmgr opID=jtlcpedv-5530-auto-49n-h5:70000264-5e-3d45 user=:VSPHERE.LOCAL\Administrator] Event 147 : User root@10.230.9.101 logged in as VMware-client/6.5.0
2019-03-23T11:46:03.046Z info hostd[39AC2B70] [Originator@6876 sub=Hostsvc.VsanSystemProvider opID=jtlcpedv-5530-auto-49n-h5:70000264-5e-3d4a user=root:VSPHERE.LOCAL\Administrator] Complete, runtime info: (vim.vsan.host.VsanRuntimeInfo) {
--> accessGenNo = 0,
--> }
2019-03-23T11:46:03.080Z warning hostd[39A40B70] [Originator@6876 sub=HTTP server] UnimplementedRequestHandler: HTTP method POST not supported for URI /fdm
2019-03-23T11:46:03.081Z info hostd[39AC2B70] [Originator@6876 sub=Vimsvc.ha-eventmgr opID=jtlcpedv-5530-auto-49n-h5:70000264-5e-3d4b user=root:VSPHERE.LOCAL\Administrator] Event 148 : User root@10.230.9.101 logged out (login time: Saturday, 23 March, 2019 11:46:03, number of API invocations: 0, user agent: VMware-client/6.5.0)
[LikewiseGetDomainJoinInfo:355] QueryInformation(): ERROR_FILE_NOT_FOUND (2/0):
[LikewiseGetDomainJoinInfo:355] QueryInformation(): ERROR_FILE_NOT_FOUND (2/0):
Accepted password for user root from 10.230.9.101
2019-03-23T11:46:22.866Z info hostd[392C1B70] [Originator@6876 sub=Vimsvc.ha-eventmgr opID=jtlcpedv-5555-auto-4ad-h5:70000273-53-3d4e user=:VSPHERE.LOCAL\Administrator] Event 149 : User root@10.230.9.101 logged in as VMware-client/6.5.0
2019-03-23T11:46:22.882Z info hostd[38D80B70] [Originator@6876 sub=Hostsvc.VsanSystemProvider opID=jtlcpedv-5555-auto-4ad-h5:70000273-53-3d53 user=root:VSPHERE.LOCAL\Administrator] Complete, runtime info: (vim.vsan.host.VsanRuntimeInfo) {
--> accessGenNo = 0,
--> }
2019-03-23T11:46:22.916Z warning hostd[39AC2B70] [Originator@6876 sub=HTTP server] UnimplementedRequestHandler: HTTP method POST not supported for URI /fdm
2019-03-23T11:46:22.917Z info hostd[38D80B70] [Originator@6876 sub=Vimsvc.ha-eventmgr opID=jtlcpedv-5555-auto-4ad-h5:70000273-53-3d54 user=root:VSPHERE.LOCAL\Administrator] Event 150 : User root@10.230.9.101 logged out (login time: Saturday, 23 March, 2019 11:46:22, number of API invocations: 0, user agent: VMware-client/6.5.0)
[LikewiseGetDomainJoinInfo:355] QueryInformation(): ERROR_FILE_NOT_FOUND (2/0):
Accepted password for user root from 10.230.9.101
vpxd.log
2019-03-23T14:59:12.585+01:00 info vpxd[04603] [Originator@6876 sub=vpxLro opID=sps-Main-337003-648-8c] [VpxLRO] -- BEGIN lro-49503 -- AuthorizationManager -- vim.AuthorizationManager.hasPrivilegeOnEntities -- 529a744c-77ac-33c8-9b6b-b1f99c882732(523b42a3-d4bb-6b90-b234-bf94d7e1ff02)
2019-03-23T14:59:12.585+01:00 info vpxd[04603] [Originator@6876 sub=vpxLro opID=sps-Main-337003-648-8c] [VpxLRO] -- FINISH lro-49503
2019-03-23T14:59:12.634+01:00 info vpxd[04679] [Originator@6876 sub=vpxLro opID=sps-Main-337003-648-cd] [VpxLRO] -- BEGIN lro-49504 -- AuthorizationManager -- vim.AuthorizationManager.hasPrivilegeOnEntities -- 529a744c-77ac-33c8-9b6b-b1f99c882732(523b42a3-d4bb-6b90-b234-bf94d7e1ff02)
2019-03-23T14:59:12.634+01:00 info vpxd[04679] [Originator@6876 sub=vpxLro opID=sps-Main-337003-648-cd] [VpxLRO] -- FINISH lro-49504
2019-03-23T14:59:12.637+01:00 info vpxd[04579] [Originator@6876 sub=vpxLro opID=sps-Main-337003-648-W1311-44] [VpxLRO] -- BEGIN lro-49505 -- ViewManager -- vim.view.ViewManager.createContainerView -- 529a744c-77ac-33c8-9b6b-b1f99c882732(523b42a3-d4bb-6b90-b234-bf94d7e1ff02)
2019-03-23T14:59:12.638+01:00 info vpxd[04579] [Originator@6876 sub=vpxLro opID=sps-Main-337003-648-W1311-44] [VpxLRO] -- FINISH lro-49505
2019-03-23T14:59:12.647+01:00 info vpxd[04595] [Originator@6876 sub=vpxLro opID=sps-Main-337003-648-W1311-2a] [VpxLRO] -- BEGIN lro-49508 -- session[529a744c-77ac-33c8-9b6b-b1f99c882732]52c4ac54-178a-5b89-9464-aeb591df4733 -- vim.view.View.destroy -- 529a744c-77ac-33c8-9b6b-b1f99c882732(523b42a3-d4bb-6b90-b234-bf94d7e1ff02)
2019-03-23T14:59:12.648+01:00 info vpxd[04595] [Originator@6876 sub=vpxLro opID=sps-Main-337003-648-W1311-2a] [VpxLRO] -- FINISH lro-49508
2019-03-23T14:59:12.874+01:00 info vpxd[04615] [Originator@6876 sub=vpxLro opID=jtlcpedv-fn9-h5:70000060-3f] [VpxLRO] -- BEGIN lro-49509 -- ChangeLogCollector -- vim.cdc.ChangeLogCollector.waitForChanges -- 528002ab-3f19-683e-e337-da65afb967f8(52b2af15-ed53-1948-5d0c-ce653bd61ac4)
2019-03-23T14:59:15.875+01:00 info vpxd[04582] [Originator@6876 sub=vpxLro opID=jtlcpedv-fn9-h5:70000060-3f] [VpxLRO] -- FINISH lro-49509
2019-03-23T14:59:16.878+01:00 info vpxd[04628] [Originator@6876 sub=vpxLro opID=jtlcpedv-fna-h5:70000060-88] [VpxLRO] -- BEGIN lro-49511 -- ChangeLogCollector -- vim.cdc.ChangeLogCollector.waitForChanges -- 528002ab-3f19-683e-e337-da65afb967f8(52b2af15-ed53-1948-5d0c-ce653bd61ac4)
2019-03-23T14:59:17.019+01:00 info vpxd[04628] [Originator@6876 sub=vpxLro opID=lro-2-6d7d4b3-1a3] [VpxLRO] -- BEGIN lro-49512 -- -- AskResSettingRecLro --
2019-03-23T14:59:17.024+01:00 info vpxd[04628] [Originator@6876 sub=vpxLro opID=lro-2-6d7d4b3-1a3] [VpxLRO] -- FINISH lro-49512
2019-03-23T14:59:17.378+01:00 info vpxd[04598] [Originator@6876 sub=vpxLro opID=jtlcpedv-fna-h5:70000060-88] [VpxLRO] -- FINISH lro-49511
2019-03-23T14:59:18.024+01:00 info vpxd[04641] [Originator@6876 sub=vpxLro opID=lro-2-6d7d4b3-1a4] [VpxLRO] -- BEGIN lro-49513 -- -- AskResSettingRecLro --
2019-03-23T14:59:18.025+01:00 info vpxd[04641] [Originator@6876 sub=vpxLro opID=lro-2-6d7d4b3-1a4] [VpxLRO] -- FINISH lro-49513
2019-03-23T14:59:18.237+01:00 info vpxd[04626] [Originator@6876 sub=vpxLro opID=jtlcpedv-20279-auto-fnc-h5:70000554-8e] [VpxLRO] -- BEGIN lro-49514 -- datacenter-2 -- vim.Datacenter.queryConnectionInfo -- 528002ab-3f19-683e-e337-da65afb967f8(52b2af15-ed53-1948-5d0c-ce653bd61ac4)
2019-03-23T14:59:18.248+01:00 info vpxd[04626] [Originator@6876 sub=vpxLro opID=jtlcpedv-20279-auto-fnc-h5:70000554-8e] [VpxLRO] -- FINISH lro-49514
2019-03-23T14:59:18.248+01:00 info vpxd[04626] [Originator@6876 sub=Default opID=jtlcpedv-20279-auto-fnc-h5:70000554-8e] [VpxLRO] -- ERROR lro-49514 -- datacenter-2 -- vim.Datacenter.queryConnectionInfo: vim.fault.SSLVerifyFault:
--> Result:
--> (vim.fault.SSLVerifyFault) {
--> faultCause = (vmodl.MethodFault) null,
--> faultMessage = <unset>,
--> selfSigned = false,
--> thumbprint = "11:DA:6A:67:22:D2:01:C7:C4:46:8E:5C:77:44:6F:23:E5:7D:A7:4E"
--> msg = ""
--> }
--> Args:
-->
--> Arg hostname:
--> "esx01.SENSORED.net"
--> Arg port:
--> -1
--> Arg username:
--> "root"
--> Arg password:
--> (not shown)
-->
--> Arg sslThumbprint:
-->
2019-03-23T14:59:18.251+01:00 info vpxd[05502] [Originator@6876 sub=vpxLro opID=jtlcpedv-20279-auto-fnd-h5:70000554-fa] [VpxLRO] -- BEGIN lro-49515 -- datacenter-2 -- vim.Datacenter.queryConnectionInfo -- 528002ab-3f19-683e-e337-da65afb967f8(52b2af15-ed53-1948-5d0c-ce653bd61ac4)
2019-03-23T14:59:18.356+01:00 info vpxd[05502] [Originator@6876 sub=vpxLro opID=jtlcpedv-20279-auto-fnd-h5:70000554-fa] [VpxLRO] -- FINISH lro-49515
2019-03-23T14:59:18.402+01:00 info vpxd[04625] [Originator@6876 sub=vpxLro opID=jtlcpedv-fne-h5:70000060-6a] [VpxLRO] -- BEGIN lro-49516 -- ChangeLogCollector -- vim.cdc.ChangeLogCollector.waitForChanges -- 528002ab-3f19-683e-e337-da65afb967f8(52b2af15-ed53-1948-5d0c-ce653bd61ac4)
2019-03-23T14:59:18.402+01:00 info vpxd[04625] [Originator@6876 sub=vpxLro opID=jtlcpedv-fne-h5:70000060-6a] [VpxLRO] -- FINISH lro-49516
2019-03-23T14:59:19.416+01:00 info vpxd[04625] [Originator@6876 sub=vpxLro opID=jtlcpedv-fnf-h5:70000060-21] [VpxLRO] -- BEGIN lro-49517 -- ChangeLogCollector -- vim.cdc.ChangeLogCollector.waitForChanges -- 528002ab-3f19-683e-e337-da65afb967f8(52b2af15-ed53-1948-5d0c-ce653bd61ac4)
2019-03-23T14:59:19.584+01:00 info vpxd[04625] [Originator@6876 sub=vpxLro opID=HB-SpecSync-host-167@582642-25f65cec] [VpxLRO] -- BEGIN lro-49518 -- -- SpecSyncLRO.Synchronize --
2019-03-23T14:59:19.618+01:00 info vpxd[04625] [Originator@6876 sub=vpxLro opID=HB-SpecSync-host-167@582642-25f65cec] [VpxLRO] -- FINISH lro-49518
2019-03-23T14:59:20.394+01:00 info vpxd[04671] [Originator@6876 sub=vpxLro opID=jtlcpedv-20284-auto-fnh-h5:70000555-6c] [VpxLRO] -- BEGIN lro-49519 -- datacenter-2 -- vim.Datacenter.queryConnectionInfo -- 528002ab-3f19-683e-e337-da65afb967f8(52b2af15-ed53-1948-5d0c-ce653bd61ac4)
2019-03-23T14:59:20.493+01:00 info vpxd[04671] [Originator@6876 sub=vpxLro opID=jtlcpedv-20284-auto-fnh-h5:70000555-6c] [VpxLRO] -- FINISH lro-49519
2019-03-23T14:59:22.585+01:00 info vpxd[04603] [Originator@6876 sub=vpxLro opID=sps-Main-337003-648-28] [VpxLRO] -- BEGIN lro-49520 -- AuthorizationManager -- vim.AuthorizationManager.hasPrivilegeOnEntities -- 529a744c-77ac-33c8-9b6b-b1f99c882732(523b42a3-d4bb-6b90-b234-bf94d7e1ff02)
2019-03-23T14:59:22.585+01:00 info vpxd[04603] [Originator@6876 sub=vpxLro opID=sps-Main-337003-648-28] [VpxLRO] -- FINISH lro-49520
2019-03-23T14:59:22.588+01:00 info vpxd[05319] [Originator@6876 sub=vpxLro opID=sps-Main-337003-648-W1312-39] [VpxLRO] -- BEGIN lro-49521 -- ViewManager -- vim.view.ViewManager.createContainerView -- 529a744c-77ac-33c8-9b6b-b1f99c882732(523b42a3-d4bb-6b90-b234-bf94d7e1ff02)
2019-03-23T14:59:22.589+01:00 info vpxd[05319] [Originator@6876 sub=vpxLro opID=sps-Main-337003-648-W1312-39] [VpxLRO] -- FINISH lro-49521
2019-03-23T14:59:22.604+01:00 info vpxd[05223] [Originator@6876 sub=vpxLro opID=sps-Main-337003-648-W1312-22] [VpxLRO] -- BEGIN lro-49524 -- session[529a744c-77ac-33c8-9b6b-b1f99c882732]520260a1-e12e-9179-2fd4-02bb941966bf -- vim.view.View.destroy -- 529a744c-77ac-33c8-9b6b-b1f99c882732(523b42a3-d4bb-6b90-b234-bf94d7e1ff02)
2019-03-23T14:59:22.604+01:00 info vpxd[05223] [Originator@6876 sub=vpxLro opID=sps-Main-337003-648-W1312-22] [VpxLRO] -- FINISH lro-49524
2019-03-23T14:59:24.267+01:00 info vpxd[05464] [Originator@6876 sub=vpxLro opID=vb-51648:RefAsRoot:01-4f] [VpxLRO] -- BEGIN lro-49525 -- ResourceModel -- cis.data.provider.ResourceModel.query -- 528002ab-3f19-683e-e337-da65afb967f8(52b2af15-ed53-1948-5d0c-ce653bd61ac4)
2019-03-23T14:59:24.267+01:00 info vpxd[05464] [Originator@6876 sub=vpxLro opID=vb-51648:RefAsRoot:01-4f] [VpxLRO] -- FINISH lro-49525
2019-03-23T14:59:24.417+01:00 info vpxd[04625] [Originator@6876 sub=vpxLro opID=vb-51649:DcVmFolder:01-eb] [VpxLRO] -- BEGIN lro-49526 -- ResourceModel -- cis.data.provider.ResourceModel.query -- 528002ab-3f19-683e-e337-da65afb967f8(52b2af15-ed53-1948-5d0c-ce653bd61ac4)
2019-03-23T14:59:24.421+01:00 info vpxd[04625] [Originator@6876 sub=vpxLro opID=vb-51649:DcVmFolder:01-eb] [VpxLRO] -- FINISH lro-49526
2019-03-23T14:59:24.423+01:00 info vpxd[04590] [Originator@6876 sub=vpxLro opID=vb-51649:DcVmFolder:02-4d] [VpxLRO] -- BEGIN lro-49527 -- ResourceModel -- cis.data.provider.ResourceModel.query -- 528002ab-3f19-683e-e337-da65afb967f8(52b2af15-ed53-1948-5d0c-ce653bd61ac4)
2019-03-23T14:59:24.424+01:00 info vpxd[04590] [Originator@6876 sub=vpxLro opID=vb-51649:DcVmFolder:02-4d] [VpxLRO] -- FINISH lro-49527
2019-03-23T14:59:26.128+01:00 info vpxd[04646] [Originator@6876 sub=vpxLro opID=q-51650:h5ui-getProperties:urn:vmomi:Datacenter:datacenter-2:444d0aa9-7ea0-4a53-ab11-57d3286ff4da:1820018042:01-4c] [VpxLRO] -- BEGIN lro-49528 -- ResourceModel -- cis.data.provider.ResourceModel.query -- 528002ab-3f19-683e-e337-da65afb967f8(52b2af15-ed53-1948-5d0c-ce653bd61ac4)
2019-03-23T14:59:26.128+01:00 info vpxd[04646] [Originator@6876 sub=vpxLro opID=q-51650:h5ui-getProperties:urn:vmomi:Datacenter:datacenter-2:444d0aa9-7ea0-4a53-ab11-57d3286ff4da:1820018042:01-4c] [VpxLRO] -- FINISH lro-49528
2019-03-23T14:59:26.264+01:00 info vpxd[04608] [Originator@6876 sub=vpxLro opID=vb-51651:urn:vmomi:Datacenter:datacenter-2:444d0aa9-7ea0-4a53-ab11-57d3286ff4da.properties:01-d4] [VpxLRO] -- BEGIN lro-49529 -- ResourceModel -- cis.data.provider.ResourceModel.query -- 528002ab-3f19-683e-e337-da65afb967f8(52b2af15-ed53-1948-5d0c-ce653bd61ac4)
2019-03-23T14:59:26.264+01:00 info vpxd[04608] [Originator@6876 sub=vpxLro opID=vb-51651:urn:vmomi:Datacenter:datacenter-2:444d0aa9-7ea0-4a53-ab11-57d3286ff4da.properties:01-d4] [VpxLRO] -- FINISH lro-49529
2019-03-23T14:59:26.266+01:00 info vpxd[04601] [Originator@6876 sub=vpxLro opID=jtlcpedv-20290-auto-fnn-h5:70000560-83] [VpxLRO] -- BEGIN lro-49530 -- datacenter-2 -- vim.Datacenter.queryConnectionInfo -- 528002ab-3f19-683e-e337-da65afb967f8(52b2af15-ed53-1948-5d0c-ce653bd61ac4)
2019-03-23T14:59:26.370+01:00 info vpxd[04601] [Originator@6876 sub=vpxLro opID=jtlcpedv-20290-auto-fnn-h5:70000560-83] [VpxLRO] -- FINISH lro-49530
2019-03-23T14:59:26.380+01:00 info vpxd[04637] [Originator@6876 sub=vpxLro opID=vb-51652:auto-name-33245:01-d] [VpxLRO] -- BEGIN lro-49532 -- ResourceModel -- cis.data.provider.ResourceModel.query -- 528002ab-3f19-683e-e337-da65afb967f8(52b2af15-ed53-1948-5d0c-ce653bd61ac4)
2019-03-23T14:59:26.380+01:00 info vpxd[04637] [Originator@6876 sub=vpxLro opID=vb-51652:auto-name-33245:01-d] [VpxLRO] -- FINISH lro-49532
2019-03-23T14:59:26.396+01:00 info vpxd[04771] [Originator@6876 sub=vpxLro opID=jtlcpedv-20290-auto-fno-h5:70000560-54] [VpxLRO] -- BEGIN task-14609 -- group-h4 -- vim.Folder.addStandaloneHost -- 528002ab-3f19-683e-e337-da65afb967f8(52b2af15-ed53-1948-5d0c-ce653bd61ac4)
2019-03-23T14:59:26.397+01:00 info vpxd[05240] [Originator@6876 sub=vpxLro opID=vb-51655:auto-name-33247:01-ff] [VpxLRO] -- BEGIN lro-49534 -- ResourceModel -- cis.data.provider.ResourceModel.query -- 528002ab-3f19-683e-e337-da65afb967f8(52b2af15-ed53-1948-5d0c-ce653bd61ac4)
2019-03-23T14:59:26.397+01:00 info vpxd[05240] [Originator@6876 sub=vpxLro opID=vb-51655:auto-name-33247:01-ff] [VpxLRO] -- FINISH lro-49534
2019-03-23T14:59:26.418+01:00 info vpxd[04796] [Originator@6876 sub=vpxLro opID=jtlcpedv-fnf-h5:70000060-21] [VpxLRO] -- FINISH lro-49517
2019-03-23T14:59:26.442+01:00 info vpxd[04748] [Originator@6876 sub=vpxLro opID=vsan-PC-584c149988f78-6c] [VpxLRO] -- BEGIN lro-49539 -- host-3017 -- vim.ManagedEntity.retrieveCustomValues -- 529a744c-77ac-33c8-9b6b-b1f99c882732(523b42a3-d4bb-6b90-b234-bf94d7e1ff02)
2019-03-23T14:59:26.442+01:00 info vpxd[04748] [Originator@6876 sub=vpxLro opID=vsan-PC-584c149988f78-6c] [VpxLRO] -- FINISH lro-49539
2019-03-23T14:59:26.449+01:00 info vpxd[05506] [Originator@6876 sub=vpxLro opID=vsan-PC-584c149988f78-2d] [VpxLRO] -- BEGIN lro-49542 -- host-3017 -- vim.ManagedEntity.retrieveCustomValues -- 529a744c-77ac-33c8-9b6b-b1f99c882732(523b42a3-d4bb-6b90-b234-bf94d7e1ff02)
2019-03-23T14:59:26.449+01:00 info vpxd[05506] [Originator@6876 sub=vpxLro opID=vsan-PC-584c149988f78-2d] [VpxLRO] -- FINISH lro-49542
2019-03-23T14:59:26.460+01:00 info vpxd[04579] [Originator@6876 sub=vpxLro opID=vsan-PC-584c149988f78-14] [VpxLRO] -- BEGIN lro-49545 -- host-3017 -- vim.ManagedEntity.retrieveCustomValues -- 529a744c-77ac-33c8-9b6b-b1f99c882732(523b42a3-d4bb-6b90-b234-bf94d7e1ff02)
2019-03-23T14:59:26.463+01:00 info vpxd[04579] [Originator@6876 sub=vpxLro opID=vsan-PC-584c149988f78-14] [VpxLRO] -- FINISH lro-49545
2019-03-23T14:59:26.603+01:00 info vpxd[04606] [Originator@6876 sub=vpxLro opID=vb-51658:auto-name-33249:01-27] [VpxLRO] -- BEGIN lro-49548 -- ResourceModel -- cis.data.provider.ResourceModel.query -- 528002ab-3f19-683e-e337-da65afb967f8(52b2af15-ed53-1948-5d0c-ce653bd61ac4)
2019-03-23T14:59:26.603+01:00 info vpxd[04606] [Originator@6876 sub=vpxLro opID=vb-51658:auto-name-33249:01-27] [VpxLRO] -- FINISH lro-49548
2019-03-23T14:59:26.809+01:00 info vpxd[04665] [Originator@6876 sub=vpxLro opID=vb-51661:auto-name-33251:01-54] [VpxLRO] -- BEGIN lro-49550 -- ResourceModel -- cis.data.provider.ResourceModel.query -- 528002ab-3f19-683e-e337-da65afb967f8(52b2af15-ed53-1948-5d0c-ce653bd61ac4)
2019-03-23T14:59:26.809+01:00 info vpxd[04665] [Originator@6876 sub=vpxLro opID=vb-51661:auto-name-33251:01-54] [VpxLRO] -- FINISH lro-49550
2019-03-23T14:59:27.020+01:00 info vpxd[04678] [Originator@6876 sub=vpxLro opID=vb-51664:auto-name-33253:01-86] [VpxLRO] -- BEGIN lro-49552 -- ResourceModel -- cis.data.provider.ResourceModel.query -- 528002ab-3f19-683e-e337-da65afb967f8(52b2af15-ed53-1948-5d0c-ce653bd61ac4)
2019-03-23T14:59:27.020+01:00 info vpxd[04678] [Originator@6876 sub=vpxLro opID=vb-51664:auto-name-33253:01-86] [VpxLRO] -- FINISH lro-49552
2019-03-23T14:59:27.227+01:00 info vpxd[04545] [Originator@6876 sub=vpxLro opID=vb-51667:auto-name-33255:01-3] [VpxLRO] -- BEGIN lro-49554 -- ResourceModel -- cis.data.provider.ResourceModel.query -- 528002ab-3f19-683e-e337-da65afb967f8(52b2af15-ed53-1948-5d0c-ce653bd61ac4)
2019-03-23T14:59:27.227+01:00 info vpxd[04545] [Originator@6876 sub=vpxLro opID=vb-51667:auto-name-33255:01-3] [VpxLRO] -- FINISH lro-49554
2019-03-23T14:59:27.420+01:00 info vpxd[04676] [Originator@6876 sub=vpxLro opID=jtlcpedv-fnv-h5:70000060-25] [VpxLRO] -- BEGIN lro-49555 -- ChangeLogCollector -- vim.cdc.ChangeLogCollector.waitForChanges -- 528002ab-3f19-683e-e337-da65afb967f8(52b2af15-ed53-1948-5d0c-ce653bd61ac4)
2019-03-23T14:59:27.420+01:00 info vpxd[04676] [Originator@6876 sub=vpxLro opID=jtlcpedv-fnv-h5:70000060-25] [VpxLRO] -- FINISH lro-49555
2019-03-23T14:59:27.436+01:00 info vpxd[05294] [Originator@6876 sub=vpxLro opID=vb-51670:auto-name-33257:01-a8] [VpxLRO] -- BEGIN lro-49557 -- ResourceModel -- cis.data.provider.ResourceModel.query -- 528002ab-3f19-683e-e337-da65afb967f8(52b2af15-ed53-1948-5d0c-ce653bd61ac4)
2019-03-23T14:59:27.436+01:00 info vpxd[05294] [Originator@6876 sub=vpxLro opID=vb-51670:auto-name-33257:01-a8] [VpxLRO] -- FINISH lro-49557
2019-03-23T14:59:27.443+01:00 info vpxd[04743] [Originator@6876 sub=vpxLro opID=vb-51671:DcCompResFolder:01-d2] [VpxLRO] -- BEGIN lro-49558 -- ResourceModel -- cis.data.provider.ResourceModel.query -- 528002ab-3f19-683e-e337-da65afb967f8(52b2af15-ed53-1948-5d0c-ce653bd61ac4)
2019-03-23T14:59:27.444+01:00 info vpxd[04743] [Originator@6876 sub=vpxLro opID=vb-51671:DcCompResFolder:01-d2] [VpxLRO] -- FINISH lro-49558
2019-03-23T14:59:27.444+01:00 info vpxd[04743] [Originator@6876 sub=vpxLro opID=vb-51671:DcCluster:01-38] [VpxLRO] -- BEGIN lro-49559 -- ResourceModel -- cis.data.provider.ResourceModel.query -- 528002ab-3f19-683e-e337-da65afb967f8(52b2af15-ed53-1948-5d0c-ce653bd61ac4)
2019-03-23T14:59:27.444+01:00 info vpxd[04743] [Originator@6876 sub=vpxLro opID=vb-51671:DcCluster:01-38] [VpxLRO] -- FINISH lro-49559
2019-03-23T14:59:27.445+01:00 info vpxd[04589] [Originator@6876 sub=vpxLro opID=vb-51671:DcStandaloneHost:01-32] [VpxLRO] -- BEGIN lro-49560 -- ResourceModel -- cis.data.provider.ResourceModel.query -- 528002ab-3f19-683e-e337-da65afb967f8(52b2af15-ed53-1948-5d0c-ce653bd61ac4)
2019-03-23T14:59:27.445+01:00 info vpxd[04589] [Originator@6876 sub=vpxLro opID=vb-51671:DcStandaloneHost:01-32] [VpxLRO] -- FINISH lro-49560
2019-03-23T14:59:27.449+01:00 info vpxd[04681] [Originator@6876 sub=vpxLro opID=vb-51671:DcCluster:02-9c] [VpxLRO] -- BEGIN lro-49561 -- ResourceModel -- cis.data.provider.ResourceModel.query -- 528002ab-3f19-683e-e337-da65afb967f8(52b2af15-ed53-1948-5d0c-ce653bd61ac4)
2019-03-23T14:59:27.449+01:00 info vpxd[05502] [Originator@6876 sub=vpxLro opID=vb-51671:DcCompResFolder:02-38] [VpxLRO] -- BEGIN lro-49562 -- ResourceModel -- cis.data.provider.ResourceModel.query -- 528002ab-3f19-683e-e337-da65afb967f8(52b2af15-ed53-1948-5d0c-ce653bd61ac4)
2019-03-23T14:59:27.449+01:00 info vpxd[05502] [Originator@6876 sub=vpxLro opID=vb-51671:DcCompResFolder:02-38] [VpxLRO] -- FINISH lro-49562
2019-03-23T14:59:27.449+01:00 info vpxd[04681] [Originator@6876 sub=vpxLro opID=vb-51671:DcCluster:02-9c] [VpxLRO] -- FINISH lro-49561
2019-03-23T14:59:27.449+01:00 info vpxd[04681] [Originator@6876 sub=vpxLro opID=vb-51671:DcStandaloneHost:02-eb] [VpxLRO] -- BEGIN lro-49563 -- ResourceModel -- cis.data.provider.ResourceModel.query -- 528002ab-3f19-683e-e337-da65afb967f8(52b2af15-ed53-1948-5d0c-ce653bd61ac4)
2019-03-23T14:59:27.450+01:00 info vpxd[04681] [Originator@6876 sub=vpxLro opID=vb-51671:DcStandaloneHost:02-eb] [VpxLRO] -- FINISH lro-49563
2019-03-23T14:59:27.452+01:00 info vpxd[04659] [Originator@6876 sub=vpxLro opID=vb-51671:DcStandaloneHost:03:rel-01-9b] [VpxLRO] -- BEGIN lro-49564 -- ResourceModel -- cis.data.provider.ResourceModel.query -- 528002ab-3f19-683e-e337-da65afb967f8(52b2af15-ed53-1948-5d0c-ce653bd61ac4)
2019-03-23T14:59:27.452+01:00 info vpxd[04659] [Originator@6876 sub=vpxLro opID=vb-51671:DcStandaloneHost:03:rel-01-9b] [VpxLRO] -- FINISH lro-49564
2019-03-23T14:59:27.454+01:00 info vpxd[04622] [Originator@6876 sub=vpxLro opID=vb-51671:DcStandaloneHost:03:rel-02-1c] [VpxLRO] -- BEGIN lro-49565 -- ResourceModel -- cis.data.provider.ResourceModel.query -- 528002ab-3f19-683e-e337-da65afb967f8(52b2af15-ed53-1948-5d0c-ce653bd61ac4)
2019-03-23T14:59:27.454+01:00 info vpxd[04622] [Originator@6876 sub=vpxLro opID=vb-51671:DcStandaloneHost:03:rel-02-1c] [VpxLRO] -- FINISH lro-49565
2019-03-23T14:59:27.454+01:00 info vpxd[04622] [Originator@6876 sub=vpxLro opID=vb-51671:DcStandaloneHost:03:rel-02-c2] [VpxLRO] -- BEGIN lro-49566 -- ResourceModel -- cis.data.provider.ResourceModel.query -- 528002ab-3f19-683e-e337-da65afb967f8(52b2af15-ed53-1948-5d0c-ce653bd61ac4)
2019-03-23T14:59:27.455+01:00 info vpxd[04622] [Originator@6876 sub=vpxLro opID=vb-51671:DcStandaloneHost:03:rel-02-c2] [VpxLRO] -- FINISH lro-49566
2019-03-23T14:59:27.644+01:00 info vpxd[04647] [Originator@6876 sub=vpxLro opID=vb-51674:auto-name-33259:01-d7] [VpxLRO] -- BEGIN lro-49568 -- ResourceModel -- cis.data.provider.ResourceModel.query -- 528002ab-3f19-683e-e337-da65afb967f8(52b2af15-ed53-1948-5d0c-ce653bd61ac4)
2019-03-23T14:59:27.644+01:00 info vpxd[04647] [Originator@6876 sub=vpxLro opID=vb-51674:auto-name-33259:01-d7] [VpxLRO] -- FINISH lro-49568
2019-03-23T14:59:27.850+01:00 info vpxd[04684] [Originator@6876 sub=vpxLro opID=vb-51677:auto-name-33261:01-ff] [VpxLRO] -- BEGIN lro-49570 -- ResourceModel -- cis.data.provider.ResourceModel.query -- 528002ab-3f19-683e-e337-da65afb967f8(52b2af15-ed53-1948-5d0c-ce653bd61ac4)
From the time around the timeout:
2019-03-23T15:14:56.790+01:00 info vpxd[05294] [Originator@6876 sub=vpxLro opID=vb-65148:auto-name-42237:01-98] [VpxLRO] -- FINISH lro-59564
2019-03-23T15:14:56.996+01:00 info vpxd[04613] [Originator@6876 sub=vpxLro opID=vb-65151:auto-name-42239:01-e6] [VpxLRO] -- BEGIN lro-59566 -- ResourceModel -- cis.data.provider.ResourceModel.query -- 528002ab-3f19-683e-e337-da65afb967f8(52b2af15-ed53-1948-5d0c-ce653bd61ac4)
2019-03-23T15:14:56.997+01:00 info vpxd[04613] [Originator@6876 sub=vpxLro opID=vb-65151:auto-name-42239:01-e6] [VpxLRO] -- FINISH lro-59566
2019-03-23T15:14:57.203+01:00 info vpxd[04605] [Originator@6876 sub=vpxLro opID=vb-65154:auto-name-42241:01-40] [VpxLRO] -- BEGIN lro-59568 -- ResourceModel -- cis.data.provider.ResourceModel.query -- 528002ab-3f19-683e-e337-da65afb967f8(52b2af15-ed53-1948-5d0c-ce653bd61ac4)
2019-03-23T15:14:57.203+01:00 info vpxd[04605] [Originator@6876 sub=vpxLro opID=vb-65154:auto-name-42241:01-40] [VpxLRO] -- FINISH lro-59568
2019-03-23T15:14:57.409+01:00 info vpxd[05224] [Originator@6876 sub=vpxLro opID=vb-65157:auto-name-42243:01-10] [VpxLRO] -- BEGIN lro-59570 -- ResourceModel -- cis.data.provider.ResourceModel.query -- 528002ab-3f19-683e-e337-da65afb967f8(52b2af15-ed53-1948-5d0c-ce653bd61ac4)
2019-03-23T15:14:57.409+01:00 info vpxd[05224] [Originator@6876 sub=vpxLro opID=vb-65157:auto-name-42243:01-10] [VpxLRO] -- FINISH lro-59570
2019-03-23T15:14:57.513+01:00 error vpxd[04771] [Originator@6876 sub=InvtHost opID=jtlcpedv-20290-auto-fno-h5:70000560-54] Failed to reconnect to cleanup before host removal: N3Vim5Fault12InvalidLogin9ExceptionE(Fault cause: vim.fault.InvalidLogin
--> )
--> [context]zKq7AVECAAAAAAnxsgAWdnB4ZAAAiLEqbGlidm1hY29yZS5zbwAAQDcbAI67GAFeRlN2cHhkAAEiwl4B9253AWZ+dwFgenYBgmy9AQdvvQGtz3UBMNB1gvFNAQFsaWJ2aW0tdHlwZXMuc28AAWvbcQG233ABZ+1wAQ5IcQCVHyMAXiIjABEJKwPUcwBsaWJwdGhyZWFkLnNvLjAABJ2MDmxpYmMuc28uNgA=[/context]
2019-03-23T15:14:57.516+01:00 error vpxd[04771] [Originator@6876 sub=Iofilter opID=jtlcpedv-20290-auto-fno-h5:70000560-54] host [vim.HostSystem:host-3017,esx01.CENSORED.net] has null config
2019-03-23T15:14:57.518+01:00 info vpxd[04771] [Originator@6876 sub=moHostProfileMgr opID=jtlcpedv-20290-auto-fno-h5:70000560-54] [DeleteHostSpecificationInt]: Host spec for DeleteHostSpecificationInt removed: vim.HostSystem:host-3017
2019-03-23T15:14:57.523+01:00 info vpxd[05529] [Originator@6876 sub=vpxLro opID=jtlcpedv-j2b-h5:70000060-eb] [VpxLRO] -- FINISH lro-59170
2019-03-23T15:14:57.523+01:00 info vpxd[04771] [Originator@6876 sub=Vsan opID=jtlcpedv-20290-auto-fno-h5:70000560-54] Destroying VSAN dynamic MOs for host-3017
2019-03-23T15:14:57.528+01:00 info vpxd[04681] [Originator@6876 sub=licenseClient opID=jtlcpedv-20290-auto-fno-h5:70000560-54-LicenseClientUnregisterHostAsync-23253d46] Unregister host: 'host-3017'
2019-03-23T15:14:57.536+01:00 info vpxd[04771] [Originator@6876 sub=vpxLro opID=jtlcpedv-20290-auto-fno-h5:70000560-54] [VpxLRO] -- FINISH task-14609
2019-03-23T15:14:57.536+01:00 info vpxd[04771] [Originator@6876 sub=Default opID=jtlcpedv-20290-auto-fno-h5:70000560-54] [VpxLRO] -- ERROR task-14609 -- group-h4 -- vim.Folder.addStandaloneHost: vmodl.fault.SystemError:
--> Result:
--> (vmodl.fault.SystemError) {
--> faultCause = (vmodl.MethodFault) null,
--> faultMessage = <unset>,
--> reason = "Connection timed out"
--> msg = ""
--> }
--> Args:
-->
--> Arg spec:
--> (vim.host.ConnectSpec) {
--> hostName = "esx01.CENSORED.net",
--> port = <unset>,
--> sslThumbprint = "11:DA:6A:67:22:D2:01:C7:C4:46:8E:5C:77:44:6F:23:E5:7D:A7:4E",
--> userName = "root",
--> password = (not shown),
--> vmFolder = 'vim.Folder:444d0aa9-7ea0-4a53-ab11-57d3286ff4da:group-v3',
--> force = true,
--> vimAccountName = <unset>,
--> vimAccountPassword = <unset>,
--> managementIp = <unset>,
--> lockdownMode = "lockdownDisabled",
--> hostGateway = (vim.host.GatewaySpec) null
--> }
--> Arg compResSpec:
-->
--> Arg addConnected:
--> true
--> Arg license:
--> "00000-00000-00000-00000-00000"
2019-03-23T15:14:57.558+01:00 info vpxd[04681] [Originator@6876 sub=licenseServiceProxy opID=jtlcpedv-20290-auto-fno-h5:70000560-54-LicenseClientUnregisterHostAsync-23253d46] Authenticating to LS old API for stubAdapter: [N5Vmomi19SoapStubAdapterImplE:0x00007f7fe0262640]
2019-03-23T15:14:57.611+01:00 info vpxd[04681] [Originator@6876 sub=licenseServiceProxy opID=jtlcpedv-20290-auto-fno-h5:70000560-54-LicenseClientUnregisterHostAsync-23253d46] Authenticated to LS old API for stubAdapter: [N5Vmomi19SoapStubAdapterImplE:0x00007f7fe0262640]
2019-03-23T15:14:57.615+01:00 info vpxd[04631] [Originator@6876 sub=vpxLro opID=vb-65160:auto-name-42245:01-2c] [VpxLRO] -- BEGIN lro-59574 -- ResourceModel -- cis.data.provider.ResourceModel.query -- 528002ab-3f19-683e-e337-da65afb967f8(52b2af15-ed53-1948-5d0c-ce653bd61ac4)
2019-03-23T15:14:57.616+01:00 info vpxd[04631] [Originator@6876 sub=vpxLro opID=vb-65160:auto-name-42245:01-2c] [VpxLRO] -- FINISH lro-59574
2019-03-23T15:14:57.684+01:00 error vpxd[04681] [Originator@6876 sub=licenseClient opID=jtlcpedv-20290-auto-fno-h5:70000560-54-LicenseClientUnregisterHostAsync-23253d46] Swallowing exception 'Fault cause: vim.fault.LicenseEntityNotFound
--> ' thrown when unregister entity 'host-3017'
2019-03-23T15:14:58.530+01:00 info vpxd[04586] [Originator@6876 sub=vpxLro opID=jtlcpedv-j7h-h5:70000060-81] [VpxLRO] -- BEGIN lro-59575 -- ChangeLogCollector -- vim.cdc.ChangeLogCollector.waitForChanges -- 528002ab-3f19-683e-e337-da65afb967f8(52b2af15-ed53-1948-5d0c-ce653bd61ac4)
2019-03-23T15:14:58.530+01:00 info vpxd[04586] [Originator@6876 sub=vpxLro opID=jtlcpedv-j7h-h5:70000060-81] [VpxLRO] -- FINISH lro-59575
2019-03-23T15:14:58.569+01:00 info vpxd[04794] [Originator@6876 sub=vpxLro opID=vb-65161:DcCluster:01-4e] [VpxLRO] -- BEGIN lro-59576 -- ResourceModel -- cis.data.provider.ResourceModel.query -- 528002ab-3f19-683e-e337-da65afb967f8(52b2af15-ed53-1948-5d0c-ce653bd61ac4)
2019-03-23T15:14:58.569+01:00 info vpxd[04794] [Originator@6876 sub=vpxLro opID=vb-65161:DcCluster:01-4e] [VpxLRO] -- FINISH lro-59576
2019-03-23T15:14:58.569+01:00 info vpxd[04618] [Originator@6876 sub=vpxLro opID=vb-65161:DcCompResFolder:01-26] [VpxLRO] -- BEGIN lro-59577 -- ResourceModel -- cis.data.provider.ResourceModel.query -- 528002ab-3f19-683e-e337-da65afb967f8(52b2af15-ed53-1948-5d0c-ce653bd61ac4)
2019-03-23T15:14:58.570+01:00 info vpxd[04618] [Originator@6876 sub=vpxLro opID=vb-65161:DcCompResFolder:01-26] [VpxLRO] -- FINISH lro-59577
2019-03-23T15:14:58.570+01:00 info vpxd[04544] [Originator@6876 sub=vpxLro opID=vb-65161:DcStandaloneHost:01-d4] [VpxLRO] -- BEGIN lro-59578 -- ResourceModel -- cis.data.provider.ResourceModel.query -- 528002ab-3f19-683e-e337-da65afb967f8(52b2af15-ed53-1948-5d0c-ce653bd61ac4)
2019-03-23T15:14:58.570+01:00 info vpxd[04544] [Originator@6876 sub=vpxLro opID=vb-65161:DcStandaloneHost:01-d4] [VpxLRO] -- FINISH lro-59578
2019-03-23T15:14:58.576+01:00 info vpxd[04615] [Originator@6876 sub=vpxLro opID=vb-65161:DcCluster:02-40] [VpxLRO] -- BEGIN lro-59579 -- ResourceModel -- cis.data.provider.ResourceModel.query -- 528002ab-3f19-683e-e337-da65afb967f8(52b2af15-ed53-1948-5d0c-ce653bd61ac4)
2019-03-23T15:14:58.576+01:00 info vpxd[04615] [Originator@6876 sub=vpxLro opID=vb-65161:DcCluster:02-40] [VpxLRO] -- FINISH lro-59579
2019-03-23T15:14:58.580+01:00 info vpxd[04615] [Originator@6876 sub=vpxLro opID=vb-65161:DcCompResFolder:02-3e] [VpxLRO] -- BEGIN lro-59580 -- ResourceModel -- cis.data.provider.ResourceModel.query -- 528002ab-3f19-683e-e337-da65afb967f8(52b2af15-ed53-1948-5d0c-ce653bd61ac4)
2019-03-23T15:14:58.580+01:00 info vpxd[04615] [Originator@6876 sub=vpxLro opID=vb-65161:DcCompResFolder:02-3e] [VpxLRO] -- FINISH lro-59580
2019-03-23T15:14:58.580+01:00 info vpxd[04615] [Originator@6876 sub=vpxLro opID=vb-65161:DcStandaloneHost:02-7d] [VpxLRO] -- BEGIN lro-59581 -- ResourceModel -- cis.data.provider.ResourceModel.query -- 528002ab-3f19-683e-e337-da65afb967f8(52b2af15-ed53-1948-5d0c-ce653bd61ac4)