I am trying to perform upgrade from
vCenter Appliance 6.7 Update 3o (6.7.0.50000)
To
vCenter Server 7.0U3a - 18778458
and I encounter problem on Stage 2 , I did log investigation and I catch the following in the new temporary vCenter Server (once I initialize the stage 2)
2021-12-09T15:45:17 PM UTC [6797]INFO:vmware.vherd.transport.authentication_manager:Initialized local authentication module
2021-12-09T15:45:17 PM UTC [6797]DEBUG:vmware.vherd.transport.authentication_manager:Authentication Modules = [<bound method SSOAuthentication.authenticateRequest of <vmware.appliance.extensions.authentication.authentica
2021-12-13T10:14:12 AM UTC [6797]INFO:vmware.vherd.transport.ssh_access_collector:[Unit Test] renewed 50 credits to post event
2021-12-13T10:14:12 AM UTC [6797]DEBUG:vmware.vherd.transport.authentication:Authentication Server Secret Renewed.
2021-12-13T10:14:12 AM UTC [6797]INFO:vmware.vherd.transport.post_sso_events:File /var/log/audit/sso-events/audit_events.log not detected, Exit.
2021-12-13T10:14:12 AM UTC [6797]INFO:vmware.vherd.transport.post_sso_events:File /var/log/audit/sso-events/operation_events.log not detected, Exit.
2021-12-13T10:14:12 AM UTC [6797]INFO:vmware.vherd.transport.ssh_access_collector:[Unit Test]Start collecting from sshinfo.log ...
2021-12-13T10:14:25 AM UTC [6797]INFO:vmware.appliance.vapi.auth:Authorization request for service_id: com.vmware.cis.session, operation_id: create
2021-12-13T10:14:25 AM UTC [6797]DEBUG:vmware.appliance.update.update_state:In State._get using state file /etc/applmgmt/appliance/software_update_state.conf
2021-12-13T10:14:25 AM UTC [6797]INFO:vmware.appliance.vapi.auth:Reset pam tally for root
2021-12-13T10:14:25 AM UTC [6797]DEBUG:vmware.appliance.vapi.auth:stdout: b'Login Failures Latest failure From\nroot 0 \n'
2021-12-13T10:15:07 AM UTC [6797]INFO:vmware.appliance.vapi.auth:Authorization request for service_id: com.vmware.vcenter.deployment.upgrade, operation_id: check
2021-12-13T10:15:07 AM UTC [6797]DEBUG:vmware.vherd.base.authorization_local:Local authorization initialized
2021-12-13T10:15:07 AM UTC [6797]ERROR:vmware.appliance.extensions.authorization.authorization_sso:Failed to get certificate or key from VECS
Traceback (most recent call last):
File "/usr/lib/applmgmt/lib/extensions/py/vmware/appliance/extensions/authorization/authorization_sso.py", line 424, in __init__
applmgmt_ks.load('machine')
File "/usr/lib/vmware/site-packages/identity/vmkeystore.py", line 99, in load
self._store_context = self._client.OpenCertStore(store_name, password)
RuntimeError: unidentifiable C++ exception
2021-12-13T10:15:07 AM UTC [6797]ERROR:root:Authorization module (authorization_sso) failed to initialize {unidentifiable C++ exception}
2021-12-13T10:15:07 AM UTC [6797]DEBUG:vmware.vherd.base.authorization_local:Local authorization initialized
2021-12-13T10:15:07 AM UTC [6797]DEBUG:vmware.vherd.base.authorization_local:Verify privileges user (root) privilege ['ModifyConfiguration']
2021-12-13T10:15:07 AM UTC [6797]DEBUG:root:Validated user privileges in localstore or SSO
2021-12-13T10:15:07 AM UTC [6797]DEBUG:vmware.appliance.update.update_state:In State._get using state file /etc/applmgmt/appliance/software_update_state.conf
2021-12-13T10:15:07 AM UTC [6797]INFO:vcenter.deploy_status:Deploy state: {'time': '2021-12-09T15:45:11.789Z', 'state': 'initialized'}
2021-12-13T10:15:07 AM UTC [6797]INFO:vcenter.deployment_impl:Appliance state retrieved: INITIALIZED.
2021-12-13T10:15:07 AM UTC [6797]INFO:vcenter.global_mutex:Using '/etc/vmware/deploy/lifecycle.global.mutex' as the global mutex file.
2021-12-13T10:15:07 AM UTC [6797]DEBUG:cis.filelock:Acquiring lock /etc/vmware/deploy/lifecycle.global.mutex.upgradeLock
2021-12-13T10:15:07 AM UTC [6797]INFO:vcenter.global_mutex:Global mutex successfully acquired.
2021-12-13T10:15:07 AM UTC [6797]INFO:vcenter.deploy_status:PhaseStatusWriter initialized
2021-12-13T10:15:07 AM UTC [6797]INFO:vcenter.common_utils:Returning deployment type as 'VCSA_EMBEDDED'
2021-12-13T10:15:07 AM UTC [6797]INFO:vcenter.spec_validators:Using default 'False' value for auto_answer mode.
2021-12-13T10:15:07 AM UTC [6797]INFO:vcenter.spec_validators:Checking given deployment type structure is valid for this deployment.
2021-12-13T10:15:07 AM UTC [6797]INFO:vcenter.spec_validators:Validating source_appliance: vcenter.iaa.local
2021-12-13T10:15:07 AM UTC [6797]INFO:vcenter.network_utils:Testing connection to vcenter.iaa.local:443.
2021-12-13T10:15:07 AM UTC [6797]INFO:vcenter.network_utils:Connection test to 'vcenter.iaa.local:443' was successful.
2021-12-13T10:15:07 AM UTC [6797]INFO:vcenter.spec_validators:Checking source_appliance SSL certificate: vcenter.iaa.local
2021-12-13T10:15:07 AM UTC [6797]INFO:vcenter.spec_validators:No verification is performed.
2021-12-13T10:15:09 AM UTC [6797]INFO:vmware.vherd.transport.ssh_access_collector:[Unit Test] renewed 50 credits to post event
2021-12-13T10:15:09 AM UTC [6797]INFO:vmware.vherd.transport.post_sso_events:File /var/log/audit/sso-events/audit_events.log not detected, Exit.
2021-12-13T10:15:09 AM UTC [6797]INFO:vmware.vherd.transport.ssh_access_collector:[Unit Test]Start collecting from sshinfo.log ...
2021-12-13T10:15:09 AM UTC [6797]INFO:vmware.vherd.transport.post_sso_events:File /var/log/audit/sso-events/operation_events.log not detected, Exit.
2021-12-13T10:15:09 AM UTC [6797]DEBUG:vmware.vherd.transport.authentication:Authentication Server Secret Renewed.
2021-12-13T10:15:24 AM UTC [6797]DEBUG:cis.filelock:Releasing lock /etc/vmware/deploy/lifecycle.global.mutex.upgradeLock
2021-12-13T10:15:24 AM UTC [6797]ERROR:vmware.vapi.provider.local:Error in invoking com.vmware.vcenter.deployment.upgrade in check - [Errno 104] Connection reset by peer
Traceback (most recent call last):
File "/usr/lib/applmgmt/vapi/lib/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/provider/local.py", line 277, in invoke
service_id, operation_id, input_value, ctx)
File "/usr/lib/applmgmt/vapi/lib/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/provider/local.py", line 248, in _invoke_int
method_result = iface.invoke(ctx, method_id, input_value)
File "/usr/lib/applmgmt/vapi/lib/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/bindings/skeleton.py", line 371, in invoke
meth_output = method(**meth_args)
File "/usr/lib/applmgmt/vcenter/py/vmware/vcenter/deployment_impl.py", line 334, in check
return validator.validate(spec, dt.get_type())
File "/usr/lib/applmgmt/vcenter/py/vmware/vcenter/spec_validators.py", line 1851, in validate
self._validate_source_appliance(spec, dep_type)
File "/usr/lib/applmgmt/vcenter/py/vmware/vcenter/spec_validators.py", line 1701, in _validate_source_appliance
err_message='bad.credentials.source.appliance')
File "/usr/lib/applmgmt/vcenter/py/vmware/vcenter/spec_validators.py", line 878, in validate_ssl
ssl_cert = self.net_utils.get_server_cert(host, port)
File "/usr/lib/applmgmt/vcenter/py/vmware/vcenter/network_utils.py", line 294, in get_server_cert
cert_str = self.get_cert_func((address, port))
File "/usr/lib/python3.7/ssl.py", line 1314, in get_server_certificate
with context.wrap_socket(sock) as sslsock:
File "/usr/lib/python3.7/ssl.py", line 423, in wrap_socket
session=session
File "/usr/lib/python3.7/ssl.py", line 870, in _create
self.do_handshake()
File "/usr/lib/python3.7/ssl.py", line 1139, in do_handshake
self._sslobj.do_handshake()
ConnectionResetError: [Errno 104] Connection reset by peer
2021-12-13T10:17:08 AM UTC [6797]DEBUG:vmware.vherd.transport.authentication_manager:HTTP METHOD b'POST'
2021-12-13T10:17:08 AM UTC [6797]DEBUG:vmware.vherd.transport.authentication_local:authenticateRequest: RPCPath = system.listMethods
so if we look on the log , the problem is
2021-12-13T10:15:24 AM UTC [6797]ERROR:vmware.vapi.provider.local:Error in invoking com.vmware.vcenter.deployment.upgrade in check - [Errno 104] Connection reset by peer
Traceback (most recent call last):
File "/usr/lib/applmgmt/vapi/lib/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/provider/local.py", line 277, in invoke
service_id, operation_id, input_value, ctx)
File "/usr/lib/applmgmt/vapi/lib/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/provider/local.py", line 248, in _invoke_int
method_result = iface.invoke(ctx, method_id, input_value)
File "/usr/lib/applmgmt/vapi/lib/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/bindings/skeleton.py", line 371, in invoke
meth_output = method(**meth_args)
File "/usr/lib/applmgmt/vcenter/py/vmware/vcenter/deployment_impl.py", line 334, in check
return validator.validate(spec, dt.get_type())
File "/usr/lib/applmgmt/vcenter/py/vmware/vcenter/spec_validators.py", line 1851, in validate
self._validate_source_appliance(spec, dep_type)
File "/usr/lib/applmgmt/vcenter/py/vmware/vcenter/spec_validators.py", line 1701, in _validate_source_appliance
err_message='bad.credentials.source.appliance')
File "/usr/lib/applmgmt/vcenter/py/vmware/vcenter/spec_validators.py", line 878, in validate_ssl
ssl_cert = self.net_utils.get_server_cert(host, port)
File "/usr/lib/applmgmt/vcenter/py/vmware/vcenter/network_utils.py", line 294, in get_server_cert
cert_str = self.get_cert_func((address, port))
File "/usr/lib/python3.7/ssl.py", line 1314, in get_server_certificate
with context.wrap_socket(sock) as sslsock:
File "/usr/lib/python3.7/ssl.py", line 423, in wrap_socket
session=session
File "/usr/lib/python3.7/ssl.py", line 870, in _create
self.do_handshake()
File "/usr/lib/python3.7/ssl.py", line 1139, in do_handshake
self._sslobj.do_handshake()
ConnectionResetError: [Errno 104] Connection reset by peer
and the error msg is
ConnectionResetError: [Errno 104] Connection reset by peer
what I already did but did not resolved the issue:
https://kb.vmware.com/s/article/80469
https://kb.vmware.com/s/article/78552
https://kb.vmware.com/s/article/68155
I have double check the SSL on the source vCenter Server.
the SSL certificate is not expired.
also it is self signed certificate that was installed by the vCenter Server (not Enterprise CA)
so the SSL certificate is good and not expired on the source vCenter Server
Also I checked:
new vCenter Server (temporary IP) can ssh to Old vCenter Server and vice versa , so it is not SSH problem !!!
both vCenter Servers are on the same subnet , so there is no firewall that filter traffic between them. so all the ports are open.
Let me know if someone already encounter such issue and the resolution path.
Hi,
try to upgrade the STS certs
I suggest you upgrade to version 7.0U2 as indicated by vmware instead of 7.0U3
vSphere 7.0 Update 3, Update 3a & Update 3b - Frequently Asked Questions (FAQ) (86398) (vmware.com)
Is this VC in linked mode ? If yes please check the vmdird state if it went to read only or some other issue.
If the vecs store have right inputs and permissions and certs are fine; then should be something with vmdird.
2021-12-13T10:15:07 AM UTC [6797]ERROR:vmware.appliance.extensions.authorization.authorization_sso:Failed to get certificate or key from VECS Traceback (most recent call last): File "/usr/lib/applmgmt/lib/extensions/py/vmware/appliance/extensions/authorization/authorization_sso.py", line 424, in __init__ applmgmt_ks.load('machine')