Upgrade vCenter Appliance 6.7 Update 3o (6.7.0.50000) To vCenter Server 7.0U3a - 18778458

[goldeneyez]

I am trying to perform upgrade from

vCenter Appliance 6.7 Update 3o (6.7.0.50000)

To

vCenter Server 7.0U3a - 18778458

and I encounter problem on Stage 2 , I did log investigation and I catch the following in the new temporary vCenter Server (once I initialize the stage 2)

2021-12-09T15:45:17 PM UTC [6797]INFO:vmware.vherd.transport.authentication_manager:Initialized local authentication module
2021-12-09T15:45:17 PM UTC [6797]DEBUG:vmware.vherd.transport.authentication_manager:Authentication Modules = [<bound method SSOAuthentication.authenticateRequest of <vmware.appliance.extensions.authentication.authentica
2021-12-13T10:14:12 AM UTC [6797]INFO:vmware.vherd.transport.ssh_access_collector:[Unit Test] renewed 50 credits to post event
2021-12-13T10:14:12 AM UTC [6797]DEBUG:vmware.vherd.transport.authentication:Authentication Server Secret Renewed.
2021-12-13T10:14:12 AM UTC [6797]INFO:vmware.vherd.transport.post_sso_events:File /var/log/audit/sso-events/audit_events.log not detected, Exit.
2021-12-13T10:14:12 AM UTC [6797]INFO:vmware.vherd.transport.post_sso_events:File /var/log/audit/sso-events/operation_events.log not detected, Exit.
2021-12-13T10:14:12 AM UTC [6797]INFO:vmware.vherd.transport.ssh_access_collector:[Unit Test]Start collecting from sshinfo.log ...
2021-12-13T10:14:25 AM UTC [6797]INFO:vmware.appliance.vapi.auth:Authorization request for service_id: com.vmware.cis.session, operation_id: create
2021-12-13T10:14:25 AM UTC [6797]DEBUG:vmware.appliance.update.update_state:In State._get using state file /etc/applmgmt/appliance/software_update_state.conf
2021-12-13T10:14:25 AM UTC [6797]INFO:vmware.appliance.vapi.auth:Reset pam tally for root
2021-12-13T10:14:25 AM UTC [6797]DEBUG:vmware.appliance.vapi.auth:stdout: b'Login           Failures Latest failure     From\nroot                0    \n'
2021-12-13T10:15:07 AM UTC [6797]INFO:vmware.appliance.vapi.auth:Authorization request for service_id: com.vmware.vcenter.deployment.upgrade, operation_id: check
2021-12-13T10:15:07 AM UTC [6797]DEBUG:vmware.vherd.base.authorization_local:Local authorization initialized
2021-12-13T10:15:07 AM UTC [6797]ERROR:vmware.appliance.extensions.authorization.authorization_sso:Failed to get certificate or key from VECS
Traceback (most recent call last):
  File "/usr/lib/applmgmt/lib/extensions/py/vmware/appliance/extensions/authorization/authorization_sso.py", line 424, in __init__
    applmgmt_ks.load('machine')
  File "/usr/lib/vmware/site-packages/identity/vmkeystore.py", line 99, in load
    self._store_context = self._client.OpenCertStore(store_name, password)
RuntimeError: unidentifiable C++ exception




2021-12-13T10:15:07 AM UTC [6797]ERROR:root:Authorization module (authorization_sso) failed to initialize {unidentifiable C++ exception}
2021-12-13T10:15:07 AM UTC [6797]DEBUG:vmware.vherd.base.authorization_local:Local authorization initialized
2021-12-13T10:15:07 AM UTC [6797]DEBUG:vmware.vherd.base.authorization_local:Verify privileges user (root) privilege ['ModifyConfiguration']
2021-12-13T10:15:07 AM UTC [6797]DEBUG:root:Validated user privileges in localstore or SSO
2021-12-13T10:15:07 AM UTC [6797]DEBUG:vmware.appliance.update.update_state:In State._get using state file /etc/applmgmt/appliance/software_update_state.conf
2021-12-13T10:15:07 AM UTC [6797]INFO:vcenter.deploy_status:Deploy state: {'time': '2021-12-09T15:45:11.789Z', 'state': 'initialized'}
2021-12-13T10:15:07 AM UTC [6797]INFO:vcenter.deployment_impl:Appliance state retrieved: INITIALIZED.
2021-12-13T10:15:07 AM UTC [6797]INFO:vcenter.global_mutex:Using '/etc/vmware/deploy/lifecycle.global.mutex' as the global mutex file.
2021-12-13T10:15:07 AM UTC [6797]DEBUG:cis.filelock:Acquiring lock /etc/vmware/deploy/lifecycle.global.mutex.upgradeLock
2021-12-13T10:15:07 AM UTC [6797]INFO:vcenter.global_mutex:Global mutex successfully acquired.
2021-12-13T10:15:07 AM UTC [6797]INFO:vcenter.deploy_status:PhaseStatusWriter initialized
2021-12-13T10:15:07 AM UTC [6797]INFO:vcenter.common_utils:Returning deployment type as 'VCSA_EMBEDDED'
2021-12-13T10:15:07 AM UTC [6797]INFO:vcenter.spec_validators:Using default 'False' value for auto_answer mode.
2021-12-13T10:15:07 AM UTC [6797]INFO:vcenter.spec_validators:Checking given deployment type structure is valid for this deployment.
2021-12-13T10:15:07 AM UTC [6797]INFO:vcenter.spec_validators:Validating source_appliance: vcenter.iaa.local
2021-12-13T10:15:07 AM UTC [6797]INFO:vcenter.network_utils:Testing connection to vcenter.iaa.local:443.
2021-12-13T10:15:07 AM UTC [6797]INFO:vcenter.network_utils:Connection test to 'vcenter.iaa.local:443' was successful.
2021-12-13T10:15:07 AM UTC [6797]INFO:vcenter.spec_validators:Checking source_appliance SSL certificate: vcenter.iaa.local
2021-12-13T10:15:07 AM UTC [6797]INFO:vcenter.spec_validators:No verification is performed.
2021-12-13T10:15:09 AM UTC [6797]INFO:vmware.vherd.transport.ssh_access_collector:[Unit Test] renewed 50 credits to post event
2021-12-13T10:15:09 AM UTC [6797]INFO:vmware.vherd.transport.post_sso_events:File /var/log/audit/sso-events/audit_events.log not detected, Exit.
2021-12-13T10:15:09 AM UTC [6797]INFO:vmware.vherd.transport.ssh_access_collector:[Unit Test]Start collecting from sshinfo.log ...
2021-12-13T10:15:09 AM UTC [6797]INFO:vmware.vherd.transport.post_sso_events:File /var/log/audit/sso-events/operation_events.log not detected, Exit.
2021-12-13T10:15:09 AM UTC [6797]DEBUG:vmware.vherd.transport.authentication:Authentication Server Secret Renewed.
2021-12-13T10:15:24 AM UTC [6797]DEBUG:cis.filelock:Releasing lock /etc/vmware/deploy/lifecycle.global.mutex.upgradeLock




2021-12-13T10:15:24 AM UTC [6797]ERROR:vmware.vapi.provider.local:Error in invoking com.vmware.vcenter.deployment.upgrade in check - [Errno 104] Connection reset by peer
Traceback (most recent call last):
  File "/usr/lib/applmgmt/vapi/lib/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/provider/local.py", line 277, in invoke
    service_id, operation_id, input_value, ctx)
  File "/usr/lib/applmgmt/vapi/lib/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/provider/local.py", line 248, in _invoke_int
    method_result = iface.invoke(ctx, method_id, input_value)
  File "/usr/lib/applmgmt/vapi/lib/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/bindings/skeleton.py", line 371, in invoke
    meth_output = method(**meth_args)
  File "/usr/lib/applmgmt/vcenter/py/vmware/vcenter/deployment_impl.py", line 334, in check
    return validator.validate(spec, dt.get_type())
  File "/usr/lib/applmgmt/vcenter/py/vmware/vcenter/spec_validators.py", line 1851, in validate
    self._validate_source_appliance(spec, dep_type)
  File "/usr/lib/applmgmt/vcenter/py/vmware/vcenter/spec_validators.py", line 1701, in _validate_source_appliance
    err_message='bad.credentials.source.appliance')
  File "/usr/lib/applmgmt/vcenter/py/vmware/vcenter/spec_validators.py", line 878, in validate_ssl
    ssl_cert = self.net_utils.get_server_cert(host, port)
  File "/usr/lib/applmgmt/vcenter/py/vmware/vcenter/network_utils.py", line 294, in get_server_cert
    cert_str = self.get_cert_func((address, port))
  File "/usr/lib/python3.7/ssl.py", line 1314, in get_server_certificate
    with context.wrap_socket(sock) as sslsock:
  File "/usr/lib/python3.7/ssl.py", line 423, in wrap_socket
    session=session
  File "/usr/lib/python3.7/ssl.py", line 870, in _create
    self.do_handshake()
  File "/usr/lib/python3.7/ssl.py", line 1139, in do_handshake
    self._sslobj.do_handshake()
ConnectionResetError: [Errno 104] Connection reset by peer
2021-12-13T10:17:08 AM UTC [6797]DEBUG:vmware.vherd.transport.authentication_manager:HTTP METHOD b'POST'
2021-12-13T10:17:08 AM UTC [6797]DEBUG:vmware.vherd.transport.authentication_local:authenticateRequest: RPCPath = system.listMethods

so if we look on the log , the problem is

2021-12-13T10:15:24 AM UTC [6797]ERROR:vmware.vapi.provider.local:Error in invoking com.vmware.vcenter.deployment.upgrade in check - [Errno 104] Connection reset by peer
Traceback (most recent call last):
  File "/usr/lib/applmgmt/vapi/lib/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/provider/local.py", line 277, in invoke
    service_id, operation_id, input_value, ctx)
  File "/usr/lib/applmgmt/vapi/lib/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/provider/local.py", line 248, in _invoke_int
    method_result = iface.invoke(ctx, method_id, input_value)
  File "/usr/lib/applmgmt/vapi/lib/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/bindings/skeleton.py", line 371, in invoke
    meth_output = method(**meth_args)
  File "/usr/lib/applmgmt/vcenter/py/vmware/vcenter/deployment_impl.py", line 334, in check
    return validator.validate(spec, dt.get_type())
  File "/usr/lib/applmgmt/vcenter/py/vmware/vcenter/spec_validators.py", line 1851, in validate
    self._validate_source_appliance(spec, dep_type)
  File "/usr/lib/applmgmt/vcenter/py/vmware/vcenter/spec_validators.py", line 1701, in _validate_source_appliance
    err_message='bad.credentials.source.appliance')
  File "/usr/lib/applmgmt/vcenter/py/vmware/vcenter/spec_validators.py", line 878, in validate_ssl
    ssl_cert = self.net_utils.get_server_cert(host, port)
  File "/usr/lib/applmgmt/vcenter/py/vmware/vcenter/network_utils.py", line 294, in get_server_cert
    cert_str = self.get_cert_func((address, port))
  File "/usr/lib/python3.7/ssl.py", line 1314, in get_server_certificate
    with context.wrap_socket(sock) as sslsock:
  File "/usr/lib/python3.7/ssl.py", line 423, in wrap_socket
    session=session
  File "/usr/lib/python3.7/ssl.py", line 870, in _create
    self.do_handshake()
  File "/usr/lib/python3.7/ssl.py", line 1139, in do_handshake
    self._sslobj.do_handshake()
ConnectionResetError: [Errno 104] Connection reset by peer

and the error msg is

ConnectionResetError: [Errno 104] Connection reset by peer

what I already did but did not resolved the issue:

https://kb.vmware.com/s/article/80469

https://kb.vmware.com/s/article/78552

https://www.dell.com/support/kbdoc/he-il/000183860/vxrail-code-upgrade-failing-with-error-the-machin...

https://vcloudvision.com/2019/05/13/how-to-fix-an-expired-vcsa-machine-ssl-certificate-with-a-bugged...

https://kb.vmware.com/s/article/68155

I have double check the SSL on the source vCenter Server.

the SSL certificate is not expired.

also it is self signed certificate that was installed by the vCenter Server (not Enterprise CA)

so the SSL certificate is good and not expired on the source vCenter Server

Also I checked:

new vCenter Server (temporary IP) can ssh to Old vCenter Server and vice versa , so it is not SSH problem !!!

both vCenter Servers are on the same subnet , so there is no firewall that filter traffic between them. so all the ports are open.

Let me know if someone already encounter such issue and the resolution path.

[kenobi79]

Hi,

try to upgrade the STS certs

https://luchodelorenzi.com/2020/05/28/proactively-checking-and-replacing-sts-certificate-on-vsphere-...

I suggest you upgrade to version 7.0U2 as indicated by vmware instead of 7.0U3

vSphere 7.0 Update 3, Update 3a & Update 3b - Frequently Asked Questions (FAQ) (86398) (vmware.com)

[Ajay1988]

Is this VC in linked mode ? If yes please check the vmdird state if it went to read only or some other issue.

If the vecs store have right inputs and permissions and certs are fine; then should be something with vmdird. 

2021-12-13T10:15:07 AM UTC [6797]ERROR:vmware.appliance.extensions.authorization.authorization_sso:Failed to get certificate or key from VECS
Traceback (most recent call last):
  File "/usr/lib/applmgmt/lib/extensions/py/vmware/appliance/extensions/authorization/authorization_sso.py", line 424, in __init__
    applmgmt_ks.load('machine')