VMware Cloud Community
jhouser01
Contributor
Contributor

VCSA 6.5 upgrade to 6.7 U1 - Fails at 57%

Having issues with my upgrade from 6.5 to 6.7 U1. Not getting too far with VMware support so figured the experts here might have some suggestions for me.

This is the error I receive when upgrading? Note: The external PSC upgrade went fine.

VMware VirtualCenter failed firstboot.

An error occurred while invoking external command : 'Command: ['/usr/sbin/vpxd', '-L'] Stderr: '

I have the log bundle, what should I provide here that might help?

~Jake

18 Replies
daphnissov
Immortal
Immortal

Is your root password on the vCenter appliance expired?

Reply
0 Kudos
jhouser01
Contributor
Contributor

root password is actually set to not expire, so no.

Reply
0 Kudos
a_p_
Leadership
Leadership

I haven't seen this error myself yet, but the Knowledge Base has an entry for it.

Please check whether this applies https://kb.vmware.com/s/article/2147547

André

Reply
0 Kudos
jhouser01
Contributor
Contributor

Yeah, I saw that KB, but couldn't find those errors in the /var/log/vmware/vpxd/vpxd-#.log at all.

Is that even the correct log file I should be looking at?

Reply
0 Kudos
jhouser01
Contributor
Contributor

This is what I see in the vpxd log file:

2018-11-06T19:08:43.461Z info vpxd[18127] [Originator@6876 sub=profileUtil] [DeserializeFromFile] reading data from file: /etc/vmware-vpx/vsan_mo.xml

2018-11-06T19:08:43.461Z warning vpxd[18127] [Originator@6876 sub=Vsan] Failed to find type vim.host.VsanHealthSystem in vSAN dynamic type list.Retry on global type list.

2018-11-06T19:08:43.461Z warning vpxd[18127] [Originator@6876 sub=Vsan] Failed to find type vim.cluster.VsanClusterHealthSystem in vSAN dynamic type list.Retry on global type list.

2018-11-06T19:08:43.461Z warning vpxd[18127] [Originator@6876 sub=Vsan] Failed to find type vim.host.VsanStretchedClusterSystem in vSAN dynamic type list.Retry on global type list.

2018-11-06T19:08:43.461Z warning vpxd[18127] [Originator@6876 sub=Vsan] Failed to find type vim.host.VsanDiskManagementSystem in vSAN dynamic type list.Retry on global type list.

2018-11-06T19:08:43.461Z info vpxd[18127] [Originator@6876 sub=Locale] LoadResources path (/etc/vmware-vpx/extensions/com.vmware.vim.ls/locale/) locale (en) removeable (false) resMap (00007f0a2407cff0) extKeys (0000000000000000)

2018-11-06T19:08:43.461Z info vpxd[18127] [Originator@6876 sub=Locale] LoadResources path (/etc/vmware-vpx/coreLocale/) locale (en) removeable (false) resMap (00007f0a2407cff0) extKeys (0000000000000000)

2018-11-06T19:08:43.461Z info vpxd[18127] [Originator@6876 sub=Locale] LoadResources path (/etc/vmware-vpx/extensions/com.vmware.vim.stats.report/locale/) locale (en) removeable (false) resMap (00007f0a2407cff0) extKeys (0000000000000000)

2018-11-06T19:08:43.461Z info vpxd[18127] [Originator@6876 sub=Locale] LoadResources path (/etc/vmware-vpx/extensions/com.vmware.vim.vsm/locale/) locale (en) removeable (false) resMap (00007f0a2407cff0) extKeys (0000000000000000)

2018-11-06T19:08:43.462Z info vpxd[18127] [Originator@6876 sub=Locale] LoadResources path (/etc/vmware-vpx/extensions/com.vmware.vim.sms/locale/) locale (en) removeable (false) resMap (00007f0a2407cff0) extKeys (0000000000000000)

2018-11-06T19:08:43.462Z info vpxd[18127] [Originator@6876 sub=Locale] LoadResources path (/etc/vmware-vpx/extensions/com.vmware.vim.sps/locale/) locale (en) removeable (false) resMap (00007f0a2407cff0) extKeys (0000000000000000)

2018-11-06T19:08:43.462Z info vpxd[18127] [Originator@6876 sub=Locale] LoadResources path (/etc/vmware-vpx/locale/) locale (en) removeable (false) resMap (00007f0a2407cff0) extKeys (0000000000000000)

2018-11-06T19:08:43.489Z info vpxd[18127] [Originator@6876 sub=Locale] LoadResources path (/etc/vmware-vpx/extensions/VirtualCenter/locale/) locale (en) removeable (false) resMap (00007f0a2407cff0) extKeys (0000000000000000)

2018-11-06T19:08:43.494Z info vpxd[18127] [Originator@6876 sub=Locale] LoadResources path (/etc/vmware-vpx/extensions/com.vmware.vim.vcha/locale/) locale (en) removeable (false) resMap (00007f0a2407cff0) extKeys (0000000000000000)

2018-11-06T19:08:43.494Z info vpxd[18127] [Originator@6876 sub=Locale] LoadResources path (/etc/vmware-vpx/extensions/hostdiag/locale/) locale (en) removeable (false) resMap (00007f0a2407cff0) extKeys (0000000000000000)

2018-11-06T19:08:43.507Z info vpxd[18127] [Originator@6876 sub=Locale] LoadResources path (/etc/vmware-vpx/extensions/hostprofiles/locale/) locale (en) removeable (true) resMap (00007f0a2407cff0) extKeys (00007f0a240ba060)

2018-11-06T19:08:43.525Z info vpxd[18127] [Originator@6876 sub=LSClient] Creating lookup service stub adapter at URL: https://vcenter.corp.local/lookupservice/sdk

2018-11-06T19:08:43.532Z error vpxd[18127] [Originator@6876 sub=Main] Unable to enumerate trusted roots from VECS localhost, error: 87

2018-11-06T19:08:43.537Z info vpxd[18136] [Originator@6876 sub=ThreadPool] Spawning additional worker - allocated: 118, idle: 2

2018-11-06T19:08:43.537Z info vpxd[18276] [Originator@6876 sub=ThreadPool] Thread enlisted

2018-11-06T19:08:43.537Z info vpxd[18276] [Originator@6876 sub=ThreadPool] Entering worker thread loop

2018-11-06T19:08:43.538Z info vpxd[18276] [Originator@6876 sub=ThreadPool] Spawning additional worker - allocated: 119, idle: 3

2018-11-06T19:08:43.538Z info vpxd[18277] [Originator@6876 sub=ThreadPool] Thread enlisted

2018-11-06T19:08:43.538Z info vpxd[18277] [Originator@6876 sub=ThreadPool] Entering worker thread loop

2018-11-06T19:08:43.546Z info vpxd[18146] [Originator@6876 sub=ThreadPool] Spawning additional worker - allocated: 120, idle: 4

2018-11-06T19:08:43.546Z info vpxd[18278] [Originator@6876 sub=ThreadPool] Thread enlisted

2018-11-06T19:08:43.546Z info vpxd[18278] [Originator@6876 sub=ThreadPool] Entering worker thread loop

2018-11-06T19:08:43.546Z info vpxd[18278] [Originator@6876 sub=ThreadPool] Spawning additional worker - allocated: 121, idle: 4

2018-11-06T19:08:43.547Z info vpxd[18279] [Originator@6876 sub=ThreadPool] Thread enlisted

2018-11-06T19:08:43.547Z info vpxd[18279] [Originator@6876 sub=ThreadPool] Entering worker thread loop

2018-11-06T19:08:43.547Z error vpxd[18127] [Originator@6876 sub=LSClient] Caught exception while creating LS client adapter: N7Vmacore3Ssl18SSLVerifyExceptionE(SSL Exception: Verification parameters:

--> PeerThumbprint: 95:92:B2:5D:60:79:3B:19:E7:0E:07:A0:4E:BF:27:D8:3F:5F:D1:31

--> ExpectedThumbprint:

--> ExpectedPeerName: vcenter.corp.local

--> The remote host certificate has these problems:

-->

--> * self signed certificate in certificate chain)

--> [context]zKq7AVECAAAAAPlSnAANdnB4ZAAAlLAqbGlidm1hY29yZS5zbwAAIDcbAG67GACm9yEAcSgiACP8IQDbACIAaVEjAKEeIwBqISMAHQgrAdRzAGxpYnB0aHJlYWQuc28uMAACnYwObGliYy5zby42AA==[/context]

2018-11-06T19:08:43.558Z info vpxd[18127] [Originator@6876 sub=LSClient] Creating lookup service stub adapter at URL: https://vcenter.corp.local/lookupservice/sdk

2018-11-06T19:08:43.565Z error vpxd[18127] [Originator@6876 sub=Main] Unable to enumerate trusted roots from VECS localhost, error: 87

2018-11-06T19:08:43.569Z info vpxd[18146] [Originator@6876 sub=ThreadPool] Spawning additional worker - allocated: 122, idle: 6

2018-11-06T19:08:43.569Z info vpxd[18282] [Originator@6876 sub=ThreadPool] Thread enlisted

2018-11-06T19:08:43.569Z info vpxd[18282] [Originator@6876 sub=ThreadPool] Entering worker thread loop

2018-11-06T19:08:43.570Z info vpxd[18146] [Originator@6876 sub=ThreadPool] Spawning additional worker - allocated: 123, idle: 7

2018-11-06T19:08:43.570Z info vpxd[18283] [Originator@6876 sub=ThreadPool] Thread enlisted

2018-11-06T19:08:43.570Z info vpxd[18283] [Originator@6876 sub=ThreadPool] Entering worker thread loop

2018-11-06T19:08:43.578Z info vpxd[18146] [Originator@6876 sub=ThreadPool] Spawning additional worker - allocated: 124, idle: 8

2018-11-06T19:08:43.578Z info vpxd[18284] [Originator@6876 sub=ThreadPool] Thread enlisted

2018-11-06T19:08:43.578Z info vpxd[18284] [Originator@6876 sub=ThreadPool] Entering worker thread loop

2018-11-06T19:08:43.578Z info vpxd[18284] [Originator@6876 sub=ThreadPool] Spawning additional worker - allocated: 125, idle: 8

2018-11-06T19:08:43.578Z info vpxd[18285] [Originator@6876 sub=ThreadPool] Thread enlisted

2018-11-06T19:08:43.578Z info vpxd[18285] [Originator@6876 sub=ThreadPool] Entering worker thread loop

2018-11-06T19:08:43.578Z error vpxd[18127] [Originator@6876 sub=LSClient] Caught exception while creating LS client adapter: N7Vmacore3Ssl18SSLVerifyExceptionE(SSL Exception: Verification parameters:

--> PeerThumbprint: 95:92:B2:5D:60:79:3B:19:E7:0E:07:A0:4E:BF:27:D8:3F:5F:D1:31

--> ExpectedThumbprint:

--> ExpectedPeerName: vcenter.corp.local

--> The remote host certificate has these problems:

-->

--> * self signed certificate in certificate chain)

--> [context]zKq7AVECAAAAAPlSnAANdnB4ZAAAlLAqbGlidm1hY29yZS5zbwAAIDcbAG67GACm9yEAcSgiACP8IQDbACIAaVEjAKEeIwBqISMAHQgrAdRzAGxpYnB0aHJlYWQuc28uMAACnYwObGliYy5zby42AA==[/context]

2018-11-06T19:08:43.579Z warning vpxd[18127] [Originator@6876 sub=LSClient] Caught exception while retrieve endpoint. N7Vmacore9ExceptionE(Cannot initialize service registration stub)

--> [context]zKq7AVECAAAAAPlSnAAMdnB4ZAAAlLAqbGlidm1hY29yZS5zbwAAIDcbAIWzGAFvyWR2cHhkAAGuzmQBntBkAQ7bZAETglMB18ZTAUYZUgJwBQJsaWJjLnNvLjYAAdUSUg==[/context]

2018-11-06T19:08:43.582Z warning vpxd[18127] [Originator@6876 sub=LSClient] Endpoint not found for Product: com.vmware.cis, Type: cs.identity, EndPointType:  com.vmware.cis.cs.identity.sso

2018-11-06T19:08:43.582Z warning vpxd[18127] [Originator@6876 sub=LSClient] Caught exception while retrieve endpoint. N7Vmacore9ExceptionE(Cannot initialize service registration stub)

--> [context]zKq7AVECAAAAAPlSnAAMdnB4ZAAAlLAqbGlidm1hY29yZS5zbwAAIDcbAIWzGAFvyWR2cHhkAAGuzmQBntBkAQ7bZAG2glMB18ZTAUYZUgJwBQJsaWJjLnNvLjYAAdUSUg==[/context]

2018-11-06T19:08:43.584Z warning vpxd[18127] [Originator@6876 sub=LSClient] Endpoint not found for Product: com.vmware.cis, Type: cs.identity, EndPointType:  com.vmware.cis.cs.identity.admin

2018-11-06T19:08:43.584Z warning vpxd[18127] [Originator@6876 sub=LSClient] Caught exception while retrieve endpoint. N7Vmacore9ExceptionE(Cannot initialize service registration stub)

--> [context]zKq7AVECAAAAAPlSnAAMdnB4ZAAAlLAqbGlidm1hY29yZS5zbwAAIDcbAIWzGAFvyWR2cHhkAAGuzmQBntBkAQ7bZAFsg1MB18ZTAUYZUgJwBQJsaWJjLnNvLjYAAdUSUg==[/context]

2018-11-06T19:08:43.587Z warning vpxd[18127] [Originator@6876 sub=LSClient] Endpoint not found for Product: com.vmware.cis, Type: cs.identity, EndPointType:  com.vmware.cis.cs.identity.groupcheck

2018-11-06T19:08:43.587Z info vpxd[18127] [Originator@6876 sub=HostGateway] Created ComponentManagerGatewaySource!

2018-11-06T19:08:43.587Z info vpxd[18127] [Originator@6876 sub=HostGateway] Created CmConnectionFSM

2018-11-06T19:08:43.587Z info vpxd[18127] [Originator@6876 sub=HostGateway] Created ComponentManagerClient.

2018-11-06T19:08:43.587Z info vpxd[18127] [Originator@6876 sub=HostGateway] CmConnectionFSM::RunFSM(ST_INIT)

2018-11-06T19:08:43.588Z info vpxd[18127] [Originator@6876 sub=HostGateway] stsUrlFromConfig: http://localhost:7080/sts/STSService/vsphere.local ssoAdminUrlFromConfig: http://localhost:7080/sso-adminserver/sdk/vsphere.local

2018-11-06T19:08:43.588Z warning vpxd[18127] [Originator@6876 sub=LSClient] Caught exception while retrieve endpoint. N7Vmacore9ExceptionE(Cannot initialize service registration stub)

--> [context]zKq7AVECAAAAAPlSnAATdnB4ZAAAlLAqbGlidm1hY29yZS5zbwAAIDcbAIWzGAFvyWR2cHhkAAGuzmQBntBkAQ7bZAH5pIsBBb+LATjBiwEmlIsBkZaLAcRniwHXXosBx4VTAdfGUwFGGVICcAUCbGliYy5zby42AAHVElI=[/context]

2018-11-06T19:08:43.592Z warning vpxd[18127] [Originator@6876 sub=LSClient] Endpoint not found for Product: com.vmware.cis, Type: cs.identity, EndPointType:  com.vmware.cis.cs.identity.sso

2018-11-06T19:08:43.592Z warning vpxd[18127] [Originator@6876 sub=LSClient] Caught exception while retrieve endpoint. N7Vmacore9ExceptionE(Cannot initialize service registration stub)

--> [context]zKq7AVECAAAAAPlSnAATdnB4ZAAAlLAqbGlidm1hY29yZS5zbwAAIDcbAIWzGAFvyWR2cHhkAAGuzmQBntBkAQ7bZAGVpYsBBb+LATjBiwEmlIsBkZaLAcRniwHXXosBx4VTAdfGUwFGGVICcAUCbGliYy5zby42AAHVElI=[/context]

2018-11-06T19:08:43.594Z warning vpxd[18127] [Originator@6876 sub=LSClient] Endpoint not found for Product: com.vmware.cis, Type: cs.identity, EndPointType:  com.vmware.cis.cs.identity.admin

2018-11-06T19:08:43.594Z info vpxd[18127] [Originator@6876 sub=HostGateway] stsUrlFromLs:  ssoAdminUrlFromLs:

2018-11-06T19:08:43.594Z info vpxd[18127] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] Try to connect to SSO VMOMI endpoint

2018-11-06T19:08:43.597Z info vpxd[18146] [Originator@6876 sub=ThreadPool] Spawning additional worker - allocated: 126, idle: 10

2018-11-06T19:08:43.597Z info vpxd[18150] [Originator@6876 sub=ThreadPool] Spawning additional worker - allocated: 127, idle: 11

2018-11-06T19:08:43.597Z info vpxd[18287] [Originator@6876 sub=ThreadPool] Thread enlisted

2018-11-06T19:08:43.597Z info vpxd[18287] [Originator@6876 sub=ThreadPool] Entering worker thread loop

2018-11-06T19:08:43.597Z warning vpxd[18150] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x00007f0a14002f10, h:23, <TCP '127.0.0.1 : 43984'>, <TCP '127.0.0.1 : 7080'>>, e: 111(Connection refused)

2018-11-06T19:08:43.597Z info vpxd[18288] [Originator@6876 sub=ThreadPool] Thread enlisted

2018-11-06T19:08:43.597Z info vpxd[18288] [Originator@6876 sub=ThreadPool] Entering worker thread loop

2018-11-06T19:08:43.598Z warning vpxd[18127] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] [RetryOnConnectionFailure] Failed to connect to SSO; uri: http://localhost:7080/sso-adminserver/sdk/vsphere.local, reason: SystemException, ex: N7Vmacore15SystemExceptionE(Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.)

--> [context]zKq7AVECAAAAAPlSnAAOdnB4ZAAAlLAqbGlidm1hY29yZS5zbwAAIDcbAG67GAD35yIAPIwlAISUJQAasSUA4bolAGlRIwChHiMAaiEjAB0IKwHUcwBsaWJwdGhyZWFkLnNvLjAAAp2MDmxpYmMuc28uNgA=[/context]

2018-11-06T19:08:43.598Z info vpxd[18127] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] Will attempt to connect to SSO VMOMI endpoint again in 10 seconds

2018-11-06T19:08:43.634Z info vpxd[18287] [Originator@6876 sub=ThreadPool] Spawning additional worker - allocated: 128, idle: 12

2018-11-06T19:08:43.634Z info vpxd[18289] [Originator@6876 sub=ThreadPool] Thread enlisted

2018-11-06T19:08:43.634Z info vpxd[18289] [Originator@6876 sub=ThreadPool] Entering worker thread loop

2018-11-06T19:08:44.634Z info vpxd[18153] [Originator@6876 sub=ThreadPool] Spawning additional worker - allocated: 129, idle: 13

2018-11-06T19:08:44.635Z info vpxd[18291] [Originator@6876 sub=ThreadPool] Thread enlisted

2018-11-06T19:08:44.635Z info vpxd[18291] [Originator@6876 sub=ThreadPool] Entering worker thread loop

2018-11-06T19:08:45.635Z info vpxd[18153] [Originator@6876 sub=ThreadPool] Spawning additional worker - allocated: 130, idle: 14

2018-11-06T19:08:45.635Z info vpxd[18296] [Originator@6876 sub=ThreadPool] Thread enlisted

2018-11-06T19:08:45.635Z info vpxd[18296] [Originator@6876 sub=ThreadPool] Entering worker thread loop

2018-11-06T19:08:46.635Z info vpxd[18153] [Originator@6876 sub=ThreadPool] Spawning additional worker - allocated: 131, idle: 15

2018-11-06T19:08:46.635Z info vpxd[18300] [Originator@6876 sub=ThreadPool] Thread enlisted

2018-11-06T19:08:46.635Z info vpxd[18300] [Originator@6876 sub=ThreadPool] Entering worker thread loop

2018-11-06T19:08:47.635Z info vpxd[18153] [Originator@6876 sub=ThreadPool] Spawning additional worker - allocated: 132, idle: 16

2018-11-06T19:08:47.636Z info vpxd[18302] [Originator@6876 sub=ThreadPool] Thread enlisted

2018-11-06T19:08:47.636Z info vpxd[18302] [Originator@6876 sub=ThreadPool] Entering worker thread loop

2018-11-06T19:08:48.636Z info vpxd[18159] [Originator@6876 sub=ThreadPool] Spawning additional worker - allocated: 133, idle: 17

2018-11-06T19:08:48.636Z info vpxd[18303] [Originator@6876 sub=ThreadPool] Thread enlisted

2018-11-06T19:08:48.636Z info vpxd[18303] [Originator@6876 sub=ThreadPool] Entering worker thread loop

2018-11-06T19:08:49.636Z info vpxd[18159] [Originator@6876 sub=ThreadPool] Spawning additional worker - allocated: 134, idle: 18

2018-11-06T19:08:49.637Z info vpxd[18310] [Originator@6876 sub=ThreadPool] Thread enlisted

2018-11-06T19:08:49.637Z info vpxd[18310] [Originator@6876 sub=ThreadPool] Entering worker thread loop

2018-11-06T19:08:53.598Z info vpxd[18127] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] Try to connect to SSO VMOMI endpoint

2018-11-06T19:08:53.601Z warning vpxd[18158] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x00007f09e0002eb0, h:23, <TCP '127.0.0.1 : 44036'>, <TCP '127.0.0.1 : 7080'>>, e: 111(Connection refused)

2018-11-06T19:08:53.602Z warning vpxd[18127] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] [RetryOnConnectionFailure] Failed to connect to SSO; uri: http://localhost:7080/sso-adminserver/sdk/vsphere.local, reason: SystemException, ex: N7Vmacore15SystemExceptionE(Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.)

--> [context]zKq7AVECAAAAAPlSnAAOdnB4ZAAAlLAqbGlidm1hY29yZS5zbwAAIDcbAG67GAD35yIAPIwlAISUJQAasSUA4bolAGlRIwChHiMAaiEjAB0IKwHUcwBsaWJwdGhyZWFkLnNvLjAAAp2MDmxpYmMuc28uNgA=[/context]

2018-11-06T19:08:53.603Z info vpxd[18127] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] Will attempt to connect to SSO VMOMI endpoint again in 10 seconds

2018-11-06T19:09:03.603Z info vpxd[18127] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] Try to connect to SSO VMOMI endpoint

2018-11-06T19:09:03.606Z warning vpxd[18171] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x00007f09d4003730, h:23, <TCP '127.0.0.1 : 44086'>, <TCP '127.0.0.1 : 7080'>>, e: 111(Connection refused)

2018-11-06T19:09:03.606Z warning vpxd[18127] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] [RetryOnConnectionFailure] Failed to connect to SSO; uri: http://localhost:7080/sso-adminserver/sdk/vsphere.local, reason: SystemException, ex: N7Vmacore15SystemExceptionE(Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.)

--> [context]zKq7AVECAAAAAPlSnAAOdnB4ZAAAlLAqbGlidm1hY29yZS5zbwAAIDcbAG67GAD35yIAPIwlAISUJQAasSUA4bolAGlRIwChHiMAaiEjAB0IKwHUcwBsaWJwdGhyZWFkLnNvLjAAAp2MDmxpYmMuc28uNgA=[/context]

2018-11-06T19:09:03.608Z info vpxd[18127] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] Will attempt to connect to SSO VMOMI endpoint again in 10 seconds

2018-11-06T19:09:13.545Z error vpxd[18191] [Originator@6876 sub=vmomi.soapStub[0]] Resetting stub adapter for server <cs p:00007f09580a8110, TCP:vcenter.corp.local:443> : service state request failed: N7Vmacore3Ssl18SSLVerifyExceptionE(SSL Exception: Verification parameters:

--> PeerThumbprint: 95:92:B2:5D:60:79:3B:19:E7:0E:07:A0:4E:BF:27:D8:3F:5F:D1:31

--> ExpectedThumbprint:

--> ExpectedPeerName: vcenter.corp.local

--> The remote host certificate has these problems:

-->

--> * self signed certificate in certificate chain)

--> [context]zKq7AVECAAAAAPlSnAANdnB4ZAAAlLAqbGlidm1hY29yZS5zbwAAIDcbAG67GACm9yEAcSgiACP8IQDbACIAaVEjAKEeIwBqISMAHQgrAdRzAGxpYnB0aHJlYWQuc28uMAACnYwObGliYy5zby42AA==[/context]

2018-11-06T19:09:13.578Z error vpxd[18192] [Originator@6876 sub=vmomi.soapStub[1]] Resetting stub adapter for server <cs p:00007f09580a8dc0, TCP:vcenter.corp.local:443> : service state request failed: N7Vmacore3Ssl18SSLVerifyExceptionE(SSL Exception: Verification parameters:

--> PeerThumbprint: 95:92:B2:5D:60:79:3B:19:E7:0E:07:A0:4E:BF:27:D8:3F:5F:D1:31

--> ExpectedThumbprint:

--> ExpectedPeerName: vcenter.corp.local

--> The remote host certificate has these problems:

-->

--> * self signed certificate in certificate chain)

--> [context]zKq7AVECAAAAAPlSnAANdnB4ZAAAlLAqbGlidm1hY29yZS5zbwAAIDcbAG67GACm9yEAcSgiACP8IQDbACIAaVEjAKEeIwBqISMAHQgrAdRzAGxpYnB0aHJlYWQuc28uMAACnYwObGliYy5zby42AA==[/context]

2018-11-06T19:09:13.580Z info vpxd[18194] [Originator@6876 sub=ThreadPool] Spawning additional worker - allocated: 135, idle: 18

2018-11-06T19:09:13.580Z info vpxd[18195] [Originator@6876 sub=ThreadPool] Spawning additional worker - allocated: 136, idle: 18

2018-11-06T19:09:13.580Z info vpxd[18381] [Originator@6876 sub=ThreadPool] Thread enlisted

2018-11-06T19:09:13.580Z info vpxd[18381] [Originator@6876 sub=ThreadPool] Entering worker thread loop

2018-11-06T19:09:13.580Z info vpxd[18382] [Originator@6876 sub=ThreadPool] Thread enlisted

2018-11-06T19:09:13.580Z info vpxd[18382] [Originator@6876 sub=ThreadPool] Entering worker thread loop

2018-11-06T19:09:13.608Z info vpxd[18127] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] Try to connect to SSO VMOMI endpoint

2018-11-06T19:09:13.610Z warning vpxd[18197] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x00007f095c09bc70, h:23, <TCP '127.0.0.1 : 44192'>, <TCP '127.0.0.1 : 7080'>>, e: 111(Connection refused)

2018-11-06T19:09:13.611Z warning vpxd[18127] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] [RetryOnConnectionFailure] Failed to connect to SSO; uri: http://localhost:7080/sso-adminserver/sdk/vsphere.local, reason: SystemException, ex: N7Vmacore15SystemExceptionE(Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.)

--> [context]zKq7AVECAAAAAPlSnAAOdnB4ZAAAlLAqbGlidm1hY29yZS5zbwAAIDcbAG67GAD35yIAPIwlAISUJQAasSUA4bolAGlRIwChHiMAaiEjAB0IKwHUcwBsaWJwdGhyZWFkLnNvLjAAAp2MDmxpYmMuc28uNgA=[/context]

2018-11-06T19:09:13.612Z info vpxd[18127] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] Will attempt to connect to SSO VMOMI endpoint again in 10 seconds

2018-11-06T19:09:23.612Z info vpxd[18127] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] Try to connect to SSO VMOMI endpoint

2018-11-06T19:09:23.615Z warning vpxd[18211] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x00007f0974003640, h:23, <TCP '127.0.0.1 : 44264'>, <TCP '127.0.0.1 : 7080'>>, e: 111(Connection refused)

2018-11-06T19:09:23.616Z warning vpxd[18127] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] [RetryOnConnectionFailure] Failed to connect to SSO; uri: http://localhost:7080/sso-adminserver/sdk/vsphere.local, reason: SystemException, ex: N7Vmacore15SystemExceptionE(Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.)

--> [context]zKq7AVECAAAAAPlSnAAOdnB4ZAAAlLAqbGlidm1hY29yZS5zbwAAIDcbAG67GAD35yIAPIwlAISUJQAasSUA4bolAGlRIwChHiMAaiEjAB0IKwHUcwBsaWJwdGhyZWFkLnNvLjAAAp2MDmxpYmMuc28uNgA=[/context]

2018-11-06T19:09:23.617Z error vpxd[18127] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] [RetryOnConnectionFailure] Max attempts (5) reached. Giving up ...

2018-11-06T19:09:23.617Z error vpxd[18127] [Originator@6876 sub=HostGateway] [CisConnection]: Error getting trusted STS certificates: Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.

2018-11-06T19:09:23.617Z warning vpxd[18127] [Originator@6876 sub=HostGateway] State(ST_INIT) failed with: Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.

2018-11-06T19:09:23.617Z warning vpxd[18127] [Originator@6876 sub=HostGateway] ComponentManager service is not available! Will attempt a lazy init of CmClient on first use!

2018-11-06T19:09:23.618Z info vpxd[18127] [Originator@6876 sub=HostGateway] HostGateway::GatewayRegistry created!

2018-11-06T19:09:23.618Z warning vpxd[18127] [Originator@6876 sub=VpxProfiler] Init [Vpxd::HostGateway::InitModule()] took 40030 ms

2018-11-06T19:09:23.618Z info vpxd[18127] [Originator@6876 sub=OMM] Overhead Memory Module (OMM) initialized

2018-11-06T19:09:23.618Z info vpxd[18127] [Originator@6876 sub=WorkQueue.vpxd.stats.persister.queue] Created: WorkQueue.vpxd.stats.persister.queue, type = serial, priority = 16 , itemWeight = 1

2018-11-06T19:09:23.676Z info vpxd[18127] [Originator@6876 sub=InvtDomain] DRS on cluster CantonProduction is enabled

2018-11-06T19:09:23.677Z info vpxd[18127] [Originator@6876 sub=InvtDomain] Initialized DRS automation level of cluster CantonProduction to manual

2018-11-06T19:09:23.750Z info vpxd[18127] [Originator@6876 sub=InvtDomain] DRS on cluster CantonLinux is disabled

2018-11-06T19:09:23.750Z info vpxd[18127] [Originator@6876 sub=InvtDomain] Initialized DRS automation level of cluster CantonLinux to fullyAutomated

2018-11-06T19:09:23.750Z info vpxd[18127] [Originator@6876 sub=WorkQueue.vpxd.ovfconsumers.queue] Created: WorkQueue.vpxd.ovfconsumers.queue, type = serial, priority = 16 , itemWeight = 1

2018-11-06T19:09:23.783Z info vpxd[18127] [Originator@6876 sub=Vsan] Destroying VSAN dynamic MOs for host-68

2018-11-06T19:09:23.783Z info vpxd[18127] [Originator@6876 sub=Vsan] Done creating VSAN dynamic MOs for -68.

2018-11-06T19:09:23.783Z info vpxd[18127] [Originator@6876 sub=MoHost] host [vim.HostSystem:host-68,host-68] connection state changed to CONNECTED

2018-11-06T19:09:23.784Z info vpxd[18127] [Originator@6876 sub=Vsan] Destroying VSAN dynamic MOs for host-61

2018-11-06T19:09:23.784Z info vpxd[18127] [Originator@6876 sub=Vsan] Done creating VSAN dynamic MOs for -61.

2018-11-06T19:09:23.784Z info vpxd[18127] [Originator@6876 sub=MoHost] host [vim.HostSystem:host-61,host-61] connection state changed to CONNECTED

2018-11-06T19:09:23.784Z error vpxd[18127] [Originator@6876 sub=InvtId] [VpxdInvtId::CheckIp] detected IP [iDRAC Network:169.254.0.2] conflict between host [host-61] (moId:host-61) nic:NetworkInfo.vnic[1].ipAddress and entity [host-68] (moId:host-68)

2018-11-06T19:09:23.784Z info vpxd[18127] [Originator@6876 sub=Vsan] Destroying VSAN dynamic MOs for host-171

2018-11-06T19:09:23.784Z info vpxd[18127] [Originator@6876 sub=Vsan] Done creating VSAN dynamic MOs for -171.

2018-11-06T19:09:23.784Z info vpxd[18127] [Originator@6876 sub=MoHost] host [vim.HostSystem:host-171,host-171] connection state changed to CONNECTED

2018-11-06T19:09:23.784Z error vpxd[18127] [Originator@6876 sub=InvtId] [VpxdInvtId::CheckIp] detected IP [iDRAC Network:169.254.0.2] conflict between host [host-171] (moId:host-171) nic:NetworkInfo.vnic[1].ipAddress and entity [host-68] (moId:host-68)

2018-11-06T19:09:23.785Z info vpxd[18127] [Originator@6876 sub=Vsan] Destroying VSAN dynamic MOs for host-66

2018-11-06T19:09:23.785Z info vpxd[18127] [Originator@6876 sub=Vsan] Done creating VSAN dynamic MOs for -66.

2018-11-06T19:09:23.785Z info vpxd[18127] [Originator@6876 sub=MoHost] host [vim.HostSystem:host-66,host-66] connection state changed to CONNECTED

2018-11-06T19:09:23.785Z error vpxd[18127] [Originator@6876 sub=InvtId] [VpxdInvtId::CheckIp] detected IP [iDRAC Network:169.254.0.2] conflict between host [host-66] (moId:host-66) nic:NetworkInfo.vnic[1].ipAddress and entity [host-68] (moId:host-68)

2018-11-06T19:09:23.785Z info vpxd[18127] [Originator@6876 sub=Vsan] Destroying VSAN dynamic MOs for host-987

2018-11-06T19:09:23.785Z info vpxd[18127] [Originator@6876 sub=Vsan] Done creating VSAN dynamic MOs for -987.

2018-11-06T19:09:23.785Z info vpxd[18127] [Originator@6876 sub=MoHost] host [vim.HostSystem:host-987,host-987] connection state changed to CONNECTED

2018-11-06T19:09:23.901Z info vpxd[18127] [Originator@6876 sub=MoOptionMgr] Invoking callbacks for key log.level, pre commit

2018-11-06T19:09:23.901Z info vpxd[18127] [Originator@6876 sub=MoOptionMgr] Invoking callbacks for key log.level, pre commit

2018-11-06T19:09:23.901Z info vpxd[18127] [Originator@6876 sub=MoOptionMgr] No change to log.level

2018-11-06T19:09:23.909Z info vpxd[18127] [Originator@6876 sub=Main] [VpxdMain] Setting OpenSSL verify locations CAFile= CAPath=/etc/ssl/certs

2018-11-06T19:09:23.909Z info vpxd[18127] [Originator@6876 sub=Default] Creating SSL Contexts

2018-11-06T19:09:24.057Z info vpxd[18127] [Originator@6876 sub=DbParallelLoader] Num rows: 42, num CPUs: 8, threads: 1, step: 42

2018-11-06T19:09:24.258Z info vpxd[18127] [Originator@6876 sub=corehostCapsule] Host [[vim.HostSystem:host-61,esx01corp.local]] joined the DVS [50 0f df fa 09 ea bc 22-3a 12 b1 3b 6f 72 be c7]

2018-11-06T19:09:24.258Z info vpxd[18127] [Originator@6876 sub=corehostCapsule] Host [[vim.HostSystem:host-171,esx04corp.local]] joined the DVS [50 0f df fa 09 ea bc 22-3a 12 b1 3b 6f 72 be c7]

2018-11-06T19:09:24.258Z info vpxd[18127] [Originator@6876 sub=corehostCapsule] Host [[vim.HostSystem:host-68,esx03corp.local]] joined the DVS [50 0f df fa 09 ea bc 22-3a 12 b1 3b 6f 72 be c7]

2018-11-06T19:09:24.258Z info vpxd[18127] [Originator@6876 sub=corehostCapsule] Host [[vim.HostSystem:host-987,cakplnxesx01corp.local]] joined the DVS [50 0f df fa 09 ea bc 22-3a 12 b1 3b 6f 72 be c7]

2018-11-06T19:09:24.258Z info vpxd[18127] [Originator@6876 sub=corehostCapsule] Host [[vim.HostSystem:host-66,esx02corp.local]] joined the DVS [50 0f df fa 09 ea bc 22-3a 12 b1 3b 6f 72 be c7]

2018-11-06T19:09:24.259Z info vpxd[18127] [Originator@6876 sub=corehostCapsule] Host [[vim.HostSystem:host-61,esx01corp.local]] joined the DVS [50 0f 91 92 47 18 c3 0d-f0 a2 7d 9d 84 4f fb 82]

2018-11-06T19:09:24.259Z info vpxd[18127] [Originator@6876 sub=corehostCapsule] Host [[vim.HostSystem:host-171,esx04corp.local]] joined the DVS [50 0f 91 92 47 18 c3 0d-f0 a2 7d 9d 84 4f fb 82]

2018-11-06T19:09:24.259Z info vpxd[18127] [Originator@6876 sub=corehostCapsule] Host [[vim.HostSystem:host-68,esx03corp.local]] joined the DVS [50 0f 91 92 47 18 c3 0d-f0 a2 7d 9d 84 4f fb 82]

2018-11-06T19:09:24.259Z info vpxd[18127] [Originator@6876 sub=corehostCapsule] Host [[vim.HostSystem:host-987,cakplnxesx01corp.local]] joined the DVS [50 0f 91 92 47 18 c3 0d-f0 a2 7d 9d 84 4f fb 82]

2018-11-06T19:09:24.259Z info vpxd[18127] [Originator@6876 sub=corehostCapsule] Host [[vim.HostSystem:host-66,esx02corp.local]] joined the DVS [50 0f 91 92 47 18 c3 0d-f0 a2 7d 9d 84 4f fb 82]

2018-11-06T19:09:24.338Z info vpxd[18127] [Originator@6876 sub=dbPortgroup] [VpxdInvtDVPortGroup::PreLoadDvpgConfig] loaded [32] dvpg config objects

2018-11-06T19:09:24.348Z info vpxd[18127] [Originator@6876 sub=[SSO][SsoFactory_CreateFacade]] Solution user set to: vpxd-8accb555-17b3-4b3f-bf5e-6ddce696b9b8@vsphere.local

2018-11-06T19:09:24.348Z info vpxd[18127] [Originator@6876 sub=[SSO][SsoFactory_CreateFacade]] VC's ServiceId in LookupService: 

2018-11-06T19:09:24.348Z info vpxd[18127] [Originator@6876 sub=[SSO][SsoFactory_CreateFacade]] STS URI set to: http://localhost:7080/sts/STSService/vsphere.local

2018-11-06T19:09:24.348Z info vpxd[18127] [Originator@6876 sub=[SSO][SsoFactory_CreateFacade]] Admin URI set to: http://localhost:7080/sso-adminserver/sdk/vsphere.local

2018-11-06T19:09:24.348Z info vpxd[18127] [Originator@6876 sub=[SSO][SsoFactory_CreateFacade]] Groupcheck URI set to: http://localhost:7080/sso-adminserver/sdk/vsphere.local

2018-11-06T19:09:24.348Z info vpxd[18127] [Originator@6876 sub=[SSO][SsoFactory_CreateFacade]] VC SSL certificate location: /etc/vmware-vpx/ssl/rui.crt

2018-11-06T19:09:24.349Z info vpxd[18127] [Originator@6876 sub=[SSO][CreateSsoFacade]] [CreateUserDirectory] STS URI set to: http://localhost:7080/sts/STSService/vsphere.local

2018-11-06T19:09:24.349Z info vpxd[18127] [Originator@6876 sub=[SSO][CreateSsoFacade]] [CreateUserDirectory] Admin URI set to: http://localhost:7080/sso-adminserver/sdk/vsphere.local

2018-11-06T19:09:24.349Z info vpxd[18127] [Originator@6876 sub=[SSO][CreateSsoFacade]] [CreateUserDirectory] Groupcheck URI set to: http://localhost:7080/sso-adminserver/sdk/vsphere.local

2018-11-06T19:09:24.349Z info vpxd[18127] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] Try to connect to SSO VMOMI endpoint

2018-11-06T19:09:24.352Z warning vpxd[18219] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x00007f098c027ff0, h:23, <TCP '127.0.0.1 : 44270'>, <TCP '127.0.0.1 : 7080'>>, e: 111(Connection refused)

2018-11-06T19:09:24.352Z warning vpxd[18127] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] [RetryOnConnectionFailure] Failed to connect to SSO; uri: http://localhost:7080/sso-adminserver/sdk/vsphere.local, reason: SystemException, ex: N7Vmacore15SystemExceptionE(Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.)

--> [context]zKq7AVECAAAAAPlSnAAOdnB4ZAAAlLAqbGlidm1hY29yZS5zbwAAIDcbAG67GAD35yIAPIwlAISUJQAasSUA4bolAGlRIwChHiMAaiEjAB0IKwHUcwBsaWJwdGhyZWFkLnNvLjAAAp2MDmxpYmMuc28uNgA=[/context]

2018-11-06T19:09:24.353Z info vpxd[18127] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] Will attempt to connect to SSO VMOMI endpoint again in 10 seconds

2018-11-06T19:09:34.354Z info vpxd[18127] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] Try to connect to SSO VMOMI endpoint

2018-11-06T19:09:34.357Z warning vpxd[18219] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x00007f0988002d80, h:23, <TCP '127.0.0.1 : 44334'>, <TCP '127.0.0.1 : 7080'>>, e: 111(Connection refused)

2018-11-06T19:09:34.357Z warning vpxd[18127] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] [RetryOnConnectionFailure] Failed to connect to SSO; uri: http://localhost:7080/sso-adminserver/sdk/vsphere.local, reason: SystemException, ex: N7Vmacore15SystemExceptionE(Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.)

--> [context]zKq7AVECAAAAAPlSnAAOdnB4ZAAAlLAqbGlidm1hY29yZS5zbwAAIDcbAG67GAD35yIAPIwlAISUJQAasSUA4bolAGlRIwChHiMAaiEjAB0IKwHUcwBsaWJwdGhyZWFkLnNvLjAAAp2MDmxpYmMuc28uNgA=[/context]

2018-11-06T19:09:34.358Z info vpxd[18127] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] Will attempt to connect to SSO VMOMI endpoint again in 10 seconds

2018-11-06T19:09:44.358Z info vpxd[18127] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] Try to connect to SSO VMOMI endpoint

2018-11-06T19:09:44.362Z warning vpxd[18232] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x00007f09c4003380, h:23, <TCP '127.0.0.1 : 44386'>, <TCP '127.0.0.1 : 7080'>>, e: 111(Connection refused)

2018-11-06T19:09:44.362Z warning vpxd[18127] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] [RetryOnConnectionFailure] Failed to connect to SSO; uri: http://localhost:7080/sso-adminserver/sdk/vsphere.local, reason: SystemException, ex: N7Vmacore15SystemExceptionE(Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.)

--> [context]zKq7AVECAAAAAPlSnAAOdnB4ZAAAlLAqbGlidm1hY29yZS5zbwAAIDcbAG67GAD35yIAPIwlAISUJQAasSUA4bolAGlRIwChHiMAaiEjAB0IKwHUcwBsaWJwdGhyZWFkLnNvLjAAAp2MDmxpYmMuc28uNgA=[/context]

2018-11-06T19:09:44.363Z info vpxd[18127] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] Will attempt to connect to SSO VMOMI endpoint again in 10 seconds

2018-11-06T19:09:54.363Z info vpxd[18127] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] Try to connect to SSO VMOMI endpoint

2018-11-06T19:09:54.366Z warning vpxd[18254] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x00007f0a200028d0, h:23, <TCP '127.0.0.1 : 44438'>, <TCP '127.0.0.1 : 7080'>>, e: 111(Connection refused)

2018-11-06T19:09:54.367Z warning vpxd[18127] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] [RetryOnConnectionFailure] Failed to connect to SSO; uri: http://localhost:7080/sso-adminserver/sdk/vsphere.local, reason: SystemException, ex: N7Vmacore15SystemExceptionE(Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.)

--> [context]zKq7AVECAAAAAPlSnAAOdnB4ZAAAlLAqbGlidm1hY29yZS5zbwAAIDcbAG67GAD35yIAPIwlAISUJQAasSUA4bolAGlRIwChHiMAaiEjAB0IKwHUcwBsaWJwdGhyZWFkLnNvLjAAAp2MDmxpYmMuc28uNgA=[/context]

2018-11-06T19:09:54.368Z info vpxd[18127] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] Will attempt to connect to SSO VMOMI endpoint again in 10 seconds

2018-11-06T19:10:04.368Z info vpxd[18127] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] Try to connect to SSO VMOMI endpoint

2018-11-06T19:10:04.371Z warning vpxd[18148] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x00007f0a28023690, h:23, <TCP '127.0.0.1 : 44492'>, <TCP '127.0.0.1 : 7080'>>, e: 111(Connection refused)

2018-11-06T19:10:04.371Z warning vpxd[18127] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] [RetryOnConnectionFailure] Failed to connect to SSO; uri: http://localhost:7080/sso-adminserver/sdk/vsphere.local, reason: SystemException, ex: N7Vmacore15SystemExceptionE(Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.)

--> [context]zKq7AVECAAAAAPlSnAAOdnB4ZAAAlLAqbGlidm1hY29yZS5zbwAAIDcbAG67GAD35yIAPIwlAISUJQAasSUA4bolAGlRIwChHiMAaiEjAB0IKwHUcwBsaWJwdGhyZWFkLnNvLjAAAp2MDmxpYmMuc28uNgA=[/context]

2018-11-06T19:10:04.372Z error vpxd[18127] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] [RetryOnConnectionFailure] Max attempts (5) reached. Giving up ...

2018-11-06T19:10:04.372Z error vpxd[18127] [Originator@6876 sub=[SSO][SsoFactory_CreateFacade]] Unable to create SSO facade: N7Vmacore15SystemExceptionE(Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.)

--> [context]zKq7AVECAAAAAPlSnAAOdnB4ZAAAlLAqbGlidm1hY29yZS5zbwAAIDcbAG67GAD35yIAPIwlAISUJQAasSUA4bolAGlRIwChHiMAaiEjAB0IKwHUcwBsaWJwdGhyZWFkLnNvLjAAAp2MDmxpYmMuc28uNgA=[/context].

2018-11-06T19:10:04.374Z warning vpxd[18127] [Originator@6876 sub=VpxProfiler] Init [Vpx::Common::Sso::SsoFactory_CreateFacade( sslContext, ssoFacadeConstPtr)] took 40025 ms

2018-11-06T19:10:04.374Z error vpxd[18127] [Originator@6876 sub=Main] Init failed. SystemError: N5Vmomi5Fault11SystemError9ExceptionE(Fault cause: vmodl.fault.SystemError

--> )

--> [context]zKq7AVECAAAAAPlSnAAKdnB4ZAAAlLAqbGlidm1hY29yZS5zbwAAIDcbAG67GAEeRlN2cHhkAAEZSFMBwZdTAdfGUwFGGVICcAUCbGliYy5zby42AAHVElI=[/context]

2018-11-06T19:10:04.376Z warning vpxd[18127] [Originator@6876 sub=VpxProfiler] ServerApp::Init [TotalTime] took 81731 ms

2018-11-06T19:10:04.376Z info vpxd[18127] [Originator@6876 sub=SupportMgr] Wrote uptime information

Reply
0 Kudos
Vijay2027
Expert
Expert

Attach complete log bundle please.

Reply
0 Kudos
jhouser01
Contributor
Contributor

I would be happy to provide the log bundle, but unfortunately that is against company policy. Is there a certain log file you would like to see? I can filter through it and redact servers names and such beforehand that way.

Hope you understand.

Reply
0 Kudos
Vijay2027
Expert
Expert

At which stage upgrade failed? Import, export or firstboot.

And let's start off with upgrade-runner.log

Location: /var/log/vmware/upgrade

Reply
0 Kudos
jhouser01
Contributor
Contributor

From the error... Firstboot.

Log attached. Let me know if you need another. 

Reply
0 Kudos
Vijay2027
Expert
Expert

Looks like export completed.

2018-11-06T18:57:56.994Z INFO UpgradeRunner Upgrade mode 'export' completed successfully

2018-11-06T18:57:56.995Z INFO root Exiting with exit-code 0

Need logs from /var/log/firstboot

Reply
0 Kudos
jhouser01
Contributor
Contributor

Here's a few of the log files from the Fastboot log folder. Are there any others you need?

Appreciate your help and diligence here. Thank you!

Reply
0 Kudos
Vijay2027
Expert
Expert

Sent DM.

Reply
0 Kudos
jhouser01
Contributor
Contributor

Here are screenshots of the errors.

Reply
0 Kudos
bglasgow
Contributor
Contributor

I'm getting this same error and it looks like it is due to my using a non-locally generated certificate.  I am using a cert generated by my internal MS CA.  I don't have the solution today yet - though I've opened a ticket and starting to search this morning...

-Brent

Reply
0 Kudos
mldeller
Enthusiast
Enthusiast

Running into this same issue, failing at 57%, also using MS CA certificates.  Did you ever come to a resolution on this?

Reply
0 Kudos
VMwareAndrew
VMware Employee
VMware Employee

I just helped another customer fix this issue.  The clue in the vpxd logs is here:

2018-11-06T19:08:43.532Z error vpxd[18127] [Originator@6876 sub=Main] Unable to enumerate trusted roots from VECS localhost, error: 87

For whatever reason, there was a failure to enumerate the SSL certificates stored in the TRUSTED_ROOTS store in VECS.  In this case, the customer was also using MS CA certificates, but that wasn't the specific issue.  There were some expired certificates in the store.  We removed the expired certificates and were able to upgrade successfully after that.  Unfortunately the error is not very descriptive but you most likely also have some expired certificates in TRUSTED_ROOTS.  There are probably also new versions of those certificates which is why everything still runs, but the old ones will break the upgrade as it will try to validate every cert in the store.

This procedure requires removing certificates from VECS and VMDIR which if you are not careful, can cause big problems if the wrong ones are removed.  I would recommend opening an SR and getting some help.

I believe the above also resolved the OP's issue as I think we found the related SR for that and it has the same resolution.

There is no KB for this yet, but I will write one as soon as I can.

LINCADMIN
Contributor
Contributor

Andrew,

I wonder if you helped me on this as I had 2 support cases opened with VMware, 1 for our PROD site and 1 for our DR site, as they were both failing when I attempted the upgrade. Eventually we were able to resolve the upgrade issues for our PROD site by doing what you provided below; unpublishing certs from the TRUSTED_ROOTS store. Since that worked for our PROD site, the case information was provided to our other case. We did the same thing for our DR site vCenter and that also resolved the upgrade issue.

I'm happy to say that both sites are now running 6.7 U1 and working well.

Here is an example of the commands we followed to clear up our issues; we only removed the old certs or certificates that were in the store from previous enhanced linked-mode configurations:

1. Take snapshot of vCenter nodes (including PSC if external)

2. SSH connection to vCenter.

3. Run the command : /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store trusted_roots --text| grep "Subject:" | less

    *This will show the certificates currently in the TRUSTED_ROOTS store*

4. Make a folder under the root.

    Command: mkdir /certificate

5. And the run the below commands:

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store TRUSTED_ROOTS --alias 7c25b666422cebc605c2229d4865454e9ac6b56a --output /certificate/7c25b666422cebc605c2229d4865454e9ac6b56a.crt

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store TRUSTED_ROOTS --alias 7be0557ebae2c62890f60bda6c07bc4ddd61be1e --output /certificate/7be0557ebae2c62890f60bda6c07bc4ddd61be1e.crt

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store TRUSTED_ROOTS --alias 2f904573d807d1f11e2527ebab25748b093b2b80 --output /certificate/2f904573d807d1f11e2527ebab25748b093b2b80.crt

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store TRUSTED_ROOTS --alias 49a9399a90a0ced1049e36bad13734d3cce4a79b --output /certificate/49a9399a90a0ced1049e36bad13734d3cce4a79b.crt

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store TRUSTED_ROOTS --alias cc9b170207044b72827cd9beb6096c8e6dd407cc --output /certificate/cc9b170207044b72827cd9beb6096c8e6dd407cc.crt

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store TRUSTED_ROOTS --alias 906d963cba872e7bd5d18460d7d7b6cacb9e3f4d --output /certificate/906d963cba872e7bd5d18460d7d7b6cacb9e3f4d.crt

/usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store TRUSTED_ROOTS --alias 7c25b666422cebc605c2229d4865454e9ac6b56a -y

/usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store TRUSTED_ROOTS --alias 7be0557ebae2c62890f60bda6c07bc4ddd61be1e -y

/usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store TRUSTED_ROOTS --alias 2f904573d807d1f11e2527ebab25748b093b2b80 -y

/usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store TRUSTED_ROOTS --alias 49a9399a90a0ced1049e36bad13734d3cce4a79b -y

/usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store TRUSTED_ROOTS --alias cc9b170207044b72827cd9beb6096c8e6dd407cc -y

/usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store TRUSTED_ROOTS --alias 906d963cba872e7bd5d18460d7d7b6cacb9e3f4d -y

/usr/lib/vmware-vmafd/bin/dir-cli trustedcert unpublish --cert "/certificate/7c25b666422cebc605c2229d4865454e9ac6b56a.crt" --login administrator@vsphere.local --password "<pass_here>"

/usr/lib/vmware-vmafd/bin/dir-cli trustedcert unpublish --cert "/certificate/7be0557ebae2c62890f60bda6c07bc4ddd61be1e.crt" --login administrator@vsphere.local --password "<pass_here>"

/usr/lib/vmware-vmafd/bin/dir-cli trustedcert unpublish --cert "/certificate/2f904573d807d1f11e2527ebab25748b093b2b80.crt" --login administrator@vsphere.local --password "<pass_here>"

/usr/lib/vmware-vmafd/bin/dir-cli trustedcert unpublish --cert "/certificate/49a9399a90a0ced1049e36bad13734d3cce4a79b.crt" --login administrator@vsphere.local --password "<pass_here>"

/usr/lib/vmware-vmafd/bin/dir-cli trustedcert unpublish --cert "/certificate/cc9b170207044b72827cd9beb6096c8e6dd407cc.crt" --login administrator@vsphere.local --password "<pass_here>"

/usr/lib/vmware-vmafd/bin/dir-cli trustedcert unpublish --cert "/certificate/906d963cba872e7bd5d18460d7d7b6cacb9e3f4d.crt" --login administrator@vsphere.local --password "<pass_here>"

Once that is done,

Run the command to verify the certs are no longer in the TRUSTED_ROOTS store:

/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store trusted_roots --text| grep "Subject:" | less

Reply
0 Kudos
VMwareAndrew
VMware Employee
VMware Employee

That was not me.

One note, you should unpublish from VMDIR before deleting from VECS, or if you happen to hit the timing just right, VMDIR will push them back to VECS after you remove them, by design.

That and you need to be very careful that you only remove unused or expired certificates from that store, which is why I recommend filing an SR.  You shouldn't remove every cert you see like in your example.  Also the aliases will be different for everyone.  If you remove the required Root CA used by SSO/etc. you can break your environment in a way that is extremely difficult to repair.

Reply
0 Kudos