Update 6.5U1e to 6.5U2d failed / could not connect to one or more vCenter Server Systems:https://vCenterFQDN:443/sdk

[SebastianGrugel]

Hello.

Yesterday i performed update vCSA:

From: 6.5.0, 7515524, U1e

To:  VC-6.5.0U2d-Appliance-FP (version from VAMI Appliance GUI)

I used for this VAMI GUI interface according documenation https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.upgrade.doc/GUID-E5E78149-4AC8-4DD7...

When update was completed. I compared this in logs and GUI VCSA ask me to reboot appliance.

After reboot services starting very long... and:

1. on begining user and passwords (from AD domain and for administrator@vsphere.local) not worked in booth clients

2. login page loaded very slow

3. IAfter some time we started received bellow errors in Web client.

4. In HTML5 client we received error bellow:

5. CPU was overload:

6. Next started problems with VPXD service

Next we decided to revert to snapshot before update.

We started VCSA which starting 40 minutes.

Next i logged to HTML5 and WebInterface with success but performance was very low.

in Web client plugin for NSX manager loaded not properly but maybe this is not related

CPU was still on high level:

and problem back again:

VPXD service was stopped:

In next post i will try paste some logs from VPXD log.... but still this is not clear for use why this back after revert to good snapshoot.

[SebastianGrugel]

logs from VPXD.logs

root@vcsa [ /var/log/vmware/vpxd ]# tail -f ./vpxd.log | grep -i failed

2018-12-05T09:56:56.553+01:00 error vpxd[7FA01AFD5800] [Originator@6876 sub=AuthzStorageProvider] [AuthzStorageProvider::CreateAuthzMgr] Failed to connect to IS: <N5Vmomi5Fault17HostCommunication9ExceptionE(vmodl.fault.HostCommunication)

2018-12-05T09:56:56.557+01:00 error vpxd[7FA01AFD5800] [Originator@6876 sub=Default] Failed to instantiate AuthzStorageProvider: N5Vmomi5Fault17HostCommunication9ExceptionE(vmodl.fault.HostCommunication)

2018-12-05T09:56:56.561+01:00 error vpxd[7FA01AFD5800] [Originator@6876 sub=Authorize] Failed to initialize authorizeManager

2018-12-05T09:56:56.561+01:00 error vpxd[7FA01AFD5800] [Originator@6876 sub=Default] Failed to start VMware VirtualCenter. Shutting down

2018-12-05T11:52:08.767+01:00 info vpxd[7F6A5C20B800] [Originator@6876 sub=AuthzStorageProvider] [AuthzStorageProvider::CreateAuthzMgr] Retry for this error: attempt count 4

2018-12-05T11:52:12.045+01:00 error vpxd[7F6A5C20B800] [Originator@6876 sub=httpUtil] [HttpUtil::ExecuteRequest] Error in sending request - Connection refused

2018-12-05T11:52:12.045+01:00 error vpxd[7F6A5C20B800] [Originator@6876 sub=ServerAccess] Remote login failed: N3Vim5Fault9HttpFault9ExceptionE(vim.fault.HttpFault)

2018-12-05T11:52:12.050+01:00 error vpxd[7F6A5C20B800] [Originator@6876 sub=AuthzStorageProvider] [AuthzStorageProvider::CreateAuthzMgr] Failed to connect to IS: <N5Vmomi5Fault17HostCommunication9ExceptionE(vmodl.fault.HostCommunication)

[SebastianGrugel]

Logs are similar to VMware Knowledge Base but not tested yet.

Sebastian

vSebastian.net

[SebastianGrugel]

Manually started vpxd services and GUI start worked but CPU is overload and in logs we can see many unsiccesfull requests to DB:

==> ./vpxd.log <==

2018-12-05T14:55:00.097+01:00 warning vpxd[7F5EA68D1700] [Originator@6876 sub=Default opID=23e562b3] [VdbStatement] SQL execution took too long: SELECT sc.stat_id, d.device_name, sm.sample_time,                   sm.sample_interval, st.stat_val           FROM  vpx_stat_counter sc, vpx_sample_time1 sm,                 vpx_device d, vpx_hist_stat1 st           WHERE  sc.entity_id = ? AND sc.stat_id = ? AND d.device_name = '' AND                  sc.counter_id = st.counter_id AND coalesce(sc.device_id,1) = coalesce(d.device_id,1) AND                  st.time_id = sm.time_id AND sm.sample_time > ? AND sm.sample_time <= ?           ORDER BY sm.sample_time

2018-12-05T14:55:00.097+01:00 warning vpxd[7F5EA68D1700] [Originator@6876 sub=Default opID=23e562b3] [VdbStatement] Execution elapsed time: 60779 ms

2018-12-05T14:55:00.143+01:00 warning vpxd[7F5E4B1DF700] [Originator@6876 sub=Default opID=4903f188] [VdbStatement] SQL execution took too long: SELECT sc.stat_id, d.device_name, sm.sample_time,                   sm.sample_interval, st.stat_val           FROM  vpx_stat_counter sc, vpx_sample_time1 sm,                 vpx_device d, vpx_hist_stat1 st           WHERE  sc.entity_id = ? AND sc.stat_id = ? AND d.device_name = '' AND                  sc.counter_id = st.counter_id AND coalesce(sc.device_id,1) = coalesce(d.device_id,1) AND                  st.time_id = sm.time_id AND sm.sample_time > ? AND sm.sample_time <= ?           ORDER BY sm.sample_time

2018-12-05T14:55:00.143+01:00 warning vpxd[7F5E4B1DF700] [Originator@6876 sub=Default opID=4903f188] [VdbStatement] Execution elapsed time: 51934 ms

2018-12-05T14:55:00.741+01:00 warning vpxd[7F5E4982C700] [Originator@6876 sub=Default opID=63bfadab] [VdbStatement] SQL execution took too long: SELECT sc.stat_id, d.device_name, sm.sample_time,                   sm.sample_interval, st.stat_val           FROM  vpx_stat_counter sc, vpx_sample_time1 sm,                 vpx_device d, vpx_hist_stat1 st           WHERE  sc.entity_id = ? AND sc.stat_id = ? AND d.device_name = '' AND                  sc.counter_id = st.counter_id AND coalesce(sc.device_id,1) = coalesce(d.device_id,1) AND                  st.time_id = sm.time_id AND sm.sample_time > ? AND sm.sample_time <= ?           ORDER BY sm.sample_time

2018-12-05T14:55:00.741+01:00 warning vpxd[7F5E4982C700] [Originator@6876 sub=Default opID=63bfadab] [VdbStatement] Execution elapsed time: 47784 ms

2018-12-05T14:55:00.757+01:00 info vpxd[7F5EA68D1700] [Originator@6876 sub=vpxLro opID=23e562b3] [VpxLRO] -- FINISH lro-12415

2018-12-05T14:55:00.757+01:00 info vpxd[7F5EA68D1700] [Originator@6876 sub=vpxLro opID=3330381c] [VpxLRO] -- BEGIN lro-12950 -- PerfMgr -- vim.PerformanceManager.queryStats -- 524992c7-2651-bdbc-bd3f-ca226bebf152(52727c42-e7ac-e71c-7c4e-cc51010491c6)

2018-12-05T14:55:00.786+01:00 error vpxd[7F5E3DEC2700] [Originator@6876 sub=Default opID=7eb10347] [Vdb::GetConnection] Failed to obtain connection: 08001:101

2018-12-05T14:55:00.826+01:00 error vpxd[7F5E454A6700] [Originator@6876 sub=Default opID=766f6583] [Vdb::GetConnection] Failed to obtain database connection. Retrying...

2018-12-05T14:55:00.848+01:00 error vpxd[7F5E40187700] [Originator@6876 sub=Default opID=57963a4e] [Vdb::GetConnection] Failed to obtain connection: 08001:101

2018-12-05T14:55:00.868+01:00 error vpxd[7F5E45E39700] [Originator@6876 sub=Default opID=606a7a59] [Vdb::GetConnection] Failed to obtain connection: 08001:101

2018-12-05T14:55:00.935+01:00 error vpxd[7F5E3A8D7700] [Originator@6876 sub=MoHost] Response from host host-35936 is null, even though no error received

2018-12-05T14:55:00.935+01:00 error vpxd[7F5E3A8D7700] [Originator@6876 sub=MoHost] [HandleAsyncQueryPerfResultsDoFinally] The RPC for host host-35936 completed with status code 3

2018-12-05T14:55:00.935+01:00 info vpxd[7F5E4B1DF700] [Originator@6876 sub=vpxLro opID=4903f188] [VpxLRO] -- FINISH lro-13960

2018-12-05T14:55:00.954+01:00 error vpxd[7F5EA7A74700] [Originator@6876 sub=Default opID=2c286574] [Vdb::GetConnection] Failed to obtain connection: 08001:101

2018-12-05T14:55:00.958+01:00 info vpxd[7F5E4B1DF700] [Originator@6876 sub=vpxLro opID=f29f56b] [VpxLRO] -- BEGIN lro-14630 -- PerfMgr -- vim.PerformanceManager.queryStats -- 52fa5c25-3dab-49aa-2cdd-99a423bd992c(52e3da1a-47b1-810d-5390-c6b2be5c0239)

2018-12-05T14:55:00.974+01:00 warning vpxd[7F5E49BB3700] [Originator@6876 sub=Default opID=49a97245] [VdbStatement] SQL execution took too long: SELECT sc.stat_id, d.device_name, sm.sample_time,                   sm.sample_interval, st.stat_val           FROM  vpx_stat_counter sc, vpx_sample_time1 sm,                 vpx_device d, vpx_hist_stat1 st           WHERE  sc.entity_id = ? AND sc.stat_id = ? AND d.device_name = '' AND                  sc.counter_id = st.counter_id AND coalesce(sc.device_id,1) = coalesce(d.device_id,1) AND                  st.time_id = sm.time_id AND sm.sample_time > ? AND sm.sample_time <= ?           ORDER BY sm.sample_time

2018-12-05T14:55:00.974+01:00 warning vpxd[7F5E49BB3700] [Originator@6876 sub=Default opID=49a97245] [VdbStatement] Execution elapsed time: 47515 ms

2018-12-05T14:55:00.975+01:00 error vpxd[7F5E48A91700] [Originator@6876 sub=Default opID=4fef9ce4] [Vdb::GetConnection] Failed to obtain connection: 08001:101

[daphnissov]

It looks like someone has monkied with your SSO config at some point in the past. You may want to open an SR on this one...