Upgrade VCSA 6.5 to 6.7 failed at 60%

[ashvetsov]

I tried three time to upgrade VCSA 6.5U1g to 6.7 and always failed at 60%.

firstbootInfrastructure.log

2018-04-24T15:12:19.187Z INFO firstbootInfrastructure Running firstboot script: ['/opt/vmware/bin/python', '/usr/lib/vmware-vpx/firstboot/vpxd_firstboot.py', '--action', 'firstboot', '--compkey', 'vpxd', '--errlog', '/var/log/firstboot/vpxd_firstboot.py_9477_stderr.log']

2018-04-24T15:19:26.82Z INFO firstbootInfrastructure [Failed] /usr/lib/vmware-vpx/firstboot/vpxd_firstboot.py is complete

2018-04-24T15:19:26.86Z WARNING firstbootInfrastructure Bug component info file does not exist

2018-04-24T15:19:26.86Z INFO firstbootInfrastructure Firstboot duration: 1461 sec

2018-04-24T15:19:26.86Z INFO firstbootInfrastructure First boot is a failure

2018-04-24T15:19:26.93Z INFO firstbootInfrastructure Changing vMon default start profile to ALL

2018-04-24T15:19:26.93Z ERROR firstbootInfrastructure Installation of vCenter Server failed with error:

['Traceback (most recent call last):\n', '  File "/bin/run-firstboot-scripts", line 800, in <module>\n    add_vaos_dependencies()\n', '  File "/bin/run-firstboot-scripts", line 232, in add_vaos_dependencies\n    with open(conf_file + \'.tmp\', \'w\') as fp:\n', "FileNotFoundError: [Errno 2] No such file or directory: '/etc/systemd/system/vmware-vmon.service.d/dep_override.conf.tmp'\n"]

2018-04-24T15:19:26.95Z WARNING firstbootInfrastructure stopping status aggregation...

cloudvm.log

2018-04-24T14:39:13.189Z: Upating deployment node type embedded

2018-04-24T14:40:09.437Z: Upgrade CHECKS requested. Starting...

2018-04-24T14:42:20.839Z: Upgrade CHECKS succeeded.

2018-04-24T14:44:31.993Z: Upgrade EXPORT requested. Starting...

2018-04-24T14:55:03.925Z: Upgrade EXPORT succeeded.

2018-04-24T15:19:26.148Z: Failed to start services. Firstboot Error.

2018-04-24T15:19:26.150Z: Starting ssh...

fbInstall.json

    "totalSteps": 38,

    "failedSteps": "vpxd_firstboot",

    "stepsCompleted": 24,

    "stepsStarted": 24

vpxd_firstboot.py_9477_stderr.log

2018-04-24T15:19:26.009Z  Invoked command: ['/usr/sbin/vpxd', '-L']

2018-04-24T15:19:26.009Z  RC = 2

Stdout =

Stderr =

2018-04-24T15:19:26.010Z  VirtualCenter firstboot failed

Any ideas?

[ashwin_prakash]

Hello,

Please check for error that are reported on the firstboot logs for their respective services.

/var/log/firstboot/vmidentity-firstboot.py_#####_stdout.log

/var/log//firstboot/vmidentity-firstboot.py_####_stderr.log

/var/log//firstboot/vpxd-firstboot.py_#####_stdout.log

/var/log//firstboot/vpxd-firstboot.py_#####_stderr.log

Check the errors that are reported on the logs.

[ashvetsov]

I see some errors with vpxd but can't understand root cause.

vmidentity-firstboot.py_9477_stderr.log

2018-04-24T14:57:06.724Z  Failure setting accounting for vmware-sts-idmd. Err Failed to set unit properties on vmware-sts-idmd.service: Unit vmware-sts-idmd.service is not loaded.

2018-04-24T14:57:19.528Z  Failure setting accounting for vmware-stsd. Err Failed to set unit properties on vmware-stsd.service: Unit vmware-stsd.service is not loaded.

vmidentity-firstboot.py_9477_stdout

Successfully checked VMware STS.

2018-04-24T14:57:42.600Z  <<<<stdout

2018-04-24T14:57:42.600Z  ===Return code: 0

2018-04-24T14:57:42.600Z  Getting value for install-parameter: upgrade.import.directory

2018-04-24T14:57:42.601Z  VMware Identity Service bootstrap: importDirectory=/storage/seat/cis-export-folder/sso

2018-04-24T14:57:42.601Z  VMware Identity Service bootstrap: isUgprading=True

2018-04-24T14:57:42.601Z  Getting value for install-parameter: upgrade.import.directory

2018-04-24T14:57:42.601Z  VMware Identity Service bootstrap: importDirectory=/storage/seat/cis-export-folder/sso

2018-04-24T14:57:42.601Z  VMware Identity Service bootstrap: isUgprading=True

2018-04-24T14:57:42.601Z  Getting value for install-parameter: upgrade.import.directory

2018-04-24T14:57:42.601Z  VMware Identity Service bootstrap: importDirectory=/storage/seat/cis-export-folder/sso

2018-04-24T14:57:42.602Z  Registering sts and idm services

2018-04-24T14:57:42.602Z  Running command: ['chkconfig', 'vmware-sts-idmd', 'on']

2018-04-24T14:57:42.684Z  Done running command

2018-04-24T14:57:42.684Z  >>>>stdout:

2018-04-24T14:57:42.684Z 

2018-04-24T14:57:42.684Z  <<<<stdout

2018-04-24T14:57:42.684Z  ===Return code: 0

2018-04-24T14:57:42.684Z  Running command: ['chkconfig', 'vmware-stsd', 'on']

2018-04-24T14:57:42.804Z  Done running command

2018-04-24T14:57:42.804Z  >>>>stdout:

2018-04-24T14:57:42.804Z 

2018-04-24T14:57:42.804Z  <<<<stdout

2018-04-24T14:57:42.804Z  ===Return code: 0

2018-04-24T14:57:42.804Z  Registration of sts and idm services complete

2018-04-24T14:57:42.804Z  Running command: ['/bin/ln', '-s', '-f', '/usr/lib/vmware-sso/firewall/sso-firewall.json', '/etc/vmware/appliance/firewall/vmware-sso']

2018-04-24T14:57:42.807Z  Done running command

2018-04-24T14:57:42.807Z  >>>>stdout:

2018-04-24T14:57:42.807Z 

2018-04-24T14:57:42.807Z  <<<<stdout

2018-04-24T14:57:42.807Z  ===Return code: 0

2018-04-24T14:57:42.807Z  Running command: ['/usr/lib/applmgmt/networking/bin/firewall-reload']

2018-04-24T14:57:43.194Z  Done running command

2018-04-24T14:57:43.194Z  >>>>stdout:

2018-04-24T14:57:43.194Z 

2018-04-24T14:57:43.194Z  <<<<stdout

2018-04-24T14:57:43.194Z  ===Return code: 0

2018-04-24T14:57:43.195Z  Successfully processed VMware Identity Service bootstrap - FIRSTBOOT.

2018-04-24T14:57:43.195Z  VMware Identity Service bootstrap completed successfully.

vpxd_firstboot.py_9477_stderr.log

2018-04-24T15:19:26.009Z  Invoked command: ['/usr/sbin/vpxd', '-L']

2018-04-24T15:19:26.009Z  RC = 2

Stdout =

Stderr =

2018-04-24T15:19:26.010Z  VirtualCenter firstboot failed

vpxd_firstboot.py_9477_stdout.log

2018-04-24T15:17:58.665Z  Registered 'vpxd' as local service in SCA

2018-04-24T15:17:58.666Z  Running command: ['pidof', 'rhttpproxy']

2018-04-24T15:17:58.673Z  Done running command

2018-04-24T15:17:58.673Z  Running command: ['/bin/kill', '-HUP', '11614']

2018-04-24T15:17:58.676Z  Done running command

2018-04-24T15:17:58.676Z  Replacing solution-user.name

2018-04-24T15:17:58.676Z  Replacing vmdir.domain-name

2018-04-24T15:17:58.677Z  Getting value for install-parameter: vmdir.domain-name

2018-04-24T15:17:58.677Z  Replacing solution-user.name

2018-04-24T15:17:58.677Z  Replacing vpxd.serviceid

2018-04-24T15:17:58.677Z  Replacing sca.hostid

2018-04-24T15:17:58.677Z  Getting value for install-parameter: sca.hostid

2018-04-24T15:17:58.677Z  Replacing vc.control.script

2018-04-24T15:17:58.677Z  Replacing rhttpproxy.ext.port2

2018-04-24T15:17:58.678Z  Getting value for install-parameter: rhttpproxy.ext.port2

2018-04-24T15:17:58.678Z  Replacing vc.health.location

2018-04-24T15:17:58.678Z  Replacing rhttpproxy.ext.port2

2018-04-24T15:17:58.678Z  Getting value for install-parameter: rhttpproxy.ext.port2

2018-04-24T15:17:58.678Z  Replacing rhttpproxy.ext.port2

2018-04-24T15:17:58.678Z  Getting value for install-parameter: rhttpproxy.ext.port2

2018-04-24T15:17:58.678Z  Replacing rhttpproxy.ext.port2

2018-04-24T15:17:58.678Z  Getting value for install-parameter: rhttpproxy.ext.port2

2018-04-24T15:17:58.679Z  Replacing rhttpproxy.ext.port2

2018-04-24T15:17:58.679Z  Getting value for install-parameter: rhttpproxy.ext.port2

2018-04-24T15:17:58.679Z  Replacing vc.home.javapath

2018-04-24T15:17:58.679Z  Replacing vc.home.javapath

2018-04-24T15:17:58.679Z  Replacing vpxd.int.sdk-port

2018-04-24T15:17:58.679Z  Getting value for install-parameter: vpxd.int.sdk-port

2018-04-24T15:17:58.679Z  Replacing vpxd.int.sdk-tunnel-port

2018-04-24T15:17:58.679Z  Getting value for install-parameter: vpxd.int.sdk-tunnel-port

2018-04-24T15:17:58.680Z  Replacing vpxd.int.sdk-tunnel-port

2018-04-24T15:17:58.680Z  Getting value for install-parameter: vpxd.int.sdk-tunnel-port

2018-04-24T15:17:58.680Z  Replacing vpxd.int.sdk-tunnel-port

2018-04-24T15:17:58.680Z  Getting value for install-parameter: vpxd.int.sdk-tunnel-port

2018-04-24T15:17:58.680Z  Replacing vpxd.int.sdk-tunnel-port

2018-04-24T15:17:58.680Z  Getting value for install-parameter: vpxd.int.sdk-tunnel-port

2018-04-24T15:17:58.680Z  Replacing vpxd.int.sdk-tunnel-port

2018-04-24T15:17:58.680Z  Getting value for install-parameter: vpxd.int.sdk-tunnel-port

2018-04-24T15:17:58.680Z  Replacing vpxd.int.sdk-port

2018-04-24T15:17:58.680Z  Getting value for install-parameter: vpxd.int.sdk-port

2018-04-24T15:17:58.680Z  Getting value for install-parameter: sca.hostid

2018-04-24T15:17:58.754Z  Getting value for install-parameter: vmdir.username

2018-04-24T15:17:58.754Z  Getting value for install-parameter: vmdir.password

2018-04-24T15:17:58.754Z  Getting value for install-parameter: sca.hostid

2018-04-24T15:17:58.761Z  Getting value for install-parameter: vmdir.domain-dn

2018-04-24T15:17:58.761Z  Initiating new cloudvm_sso_cm_register operation

2018-04-24T15:18:01.027Z  Successfully performed tryupgrade operation for service b086700d-8401-48b4-a11a-c267fb5b1322

2018-04-24T15:18:01.027Z  Running command: ['/usr/lib/vmware-vmafd/bin/vmafd-cli', 'get-machine-id', '--server-name', 'localhost']

2018-04-24T15:18:01.038Z  Done running command

2018-04-24T15:18:01.038Z  Getting value for install-parameter: vmdir.domain-name

2018-04-24T15:18:01.171Z  Setting vc.instance.cfg.path install-parameter to: /etc/vmware-vpx/instance.cfg

2018-04-24T15:18:01.306Z  Getting value for install-parameter: vpxd.mac-allocation-scheme.prefix

2018-04-24T15:18:01.306Z  scheme_prefix=

2018-04-24T15:18:01.306Z  Getting value for install-parameter: vpxd.mac-allocation-scheme.prefix-length

2018-04-24T15:18:01.306Z  scheme_prefixlen=0

2018-04-24T15:18:01.306Z  Getting value for install-parameter: vpxd.mac-allocation-scheme.ranges

2018-04-24T15:18:01.306Z  range_scheme=

2018-04-24T15:18:01.306Z  Getting value for install-parameter: vsm.int.http

2018-04-24T15:18:01.307Z  Performing text substitutions in /etc/vmware-vpx/extensions/com.vmware.vim.vsm/extension.xml

2018-04-24T15:18:01.356Z  Skipped re-encrypting db for 6.0 vCenter.

2018-04-24T15:18:01.356Z  Migrating vpxd.cfg from version 650...

2018-04-24T15:18:01.357Z  Migrating logging config...

2018-04-24T15:18:01.357Z  Ignoring /config/alert/log/enabled property not found on destination - Key not found: /config/alert/log/enabled.

2018-04-24T15:18:01.357Z  Added /config/alert/log/enabled property with source value true.

2018-04-24T15:18:01.358Z  Ignoring /config/level[@id="CpuFeatures"]/logLevel property not found on source - Key not found: /config/level[@id="CpuFeatures"]/logLevel.

2018-04-24T15:18:01.358Z  Ignoring /config/level[@id="CpuFeatures"]/logLevel property not found on destination - Key not found: /config/level[@id="CpuFeatures"]/logLevel.

2018-04-24T15:18:01.358Z  Ignoring empty /config/level[@id="CpuFeatures"]/logLevel property value.

2018-04-24T15:18:01.358Z  Ignoring /config/level[@id="CpuFeatures"]/logName property not found on source - Key not found: /config/level[@id="CpuFeatures"]/logName.

2018-04-24T15:18:01.358Z  Ignoring /config/level[@id="CpuFeatures"]/logName property not found on destination - Key not found: /config/level[@id="CpuFeatures"]/logName.

2018-04-24T15:18:01.358Z  Ignoring empty /config/level[@id="CpuFeatures"]/logName property value.

2018-04-24T15:18:01.358Z  Ignoring /config/level[@id="VmCheck"]/logLevel property not found on source - Key not found: /config/level[@id="VmCheck"]/logLevel.

2018-04-24T15:18:01.359Z  Ignoring /config/level[@id="VmCheck"]/logLevel property not found on destination - Key not found: /config/level[@id="VmCheck"]/logLevel.

2018-04-24T15:18:01.359Z  Ignoring empty /config/level[@id="VmCheck"]/logLevel property value.

2018-04-24T15:18:01.359Z  Ignoring /config/level[@id="VmCheck"]/logName property not found on source - Key not found: /config/level[@id="VmCheck"]/logName.

2018-04-24T15:18:01.359Z  Ignoring /config/level[@id="VmCheck"]/logName property not found on destination - Key not found: /config/level[@id="VmCheck"]/logName.

2018-04-24T15:18:01.359Z  Ignoring empty /config/level[@id="VmCheck"]/logName property value.

2018-04-24T15:18:01.359Z  Ignoring unchanged /config/log/compressOnRoll property value.

2018-04-24T15:18:01.359Z  Ignoring unchanged /config/log/level property value.

2018-04-24T15:18:01.359Z  Ignoring unchanged /config/log/maxFileNum property value.

2018-04-24T15:18:01.359Z  Ignoring unchanged /config/log/maxFileSize property value.

2018-04-24T15:18:01.359Z  Migrating network config...

2018-04-24T15:18:01.360Z  Ignoring unchanged /config/vpxd/network/rollback property value.

2018-04-24T15:18:01.360Z  Ignoring /config/vpxd/network/rollbackTimeout property not found on source - Key not found: /config/vpxd/network/rollbackTimeout.

2018-04-24T15:18:01.360Z  Ignoring /config/vpxd/network/rollbackTimeout property not found on destination - Key not found: /config/vpxd/network/rollbackTimeout.

2018-04-24T15:18:01.360Z  Ignoring empty /config/vpxd/network/rollbackTimeout property value.

2018-04-24T15:18:01.360Z  Ignoring /config/vpxd/network/macReclaim property not found on source - Key not found: /config/vpxd/network/macReclaim.

2018-04-24T15:18:01.361Z  Ignoring /config/vpxd/network/macReclaim property not found on destination - Key not found: /config/vpxd/network/macReclaim.

2018-04-24T15:18:01.361Z  Ignoring empty /config/vpxd/network/macReclaim property value.

2018-04-24T15:18:01.361Z  Ignoring /config/vpxd/network/macReclaim/size property not found on source - Key not found: /config/vpxd/network/macReclaim/size.

2018-04-24T15:18:01.361Z  Ignoring /config/vpxd/network/macReclaim/size property not found on destination - Key not found: /config/vpxd/network/macReclaim/size.

2018-04-24T15:18:01.361Z  Ignoring empty /config/vpxd/network/macReclaim/size property value.

2018-04-24T15:18:01.362Z  Ignoring /config/vpxd/network/validateMacAlloc property not found on source - Key not found: /config/vpxd/network/validateMacAlloc.

2018-04-24T15:18:01.362Z  Ignoring /config/vpxd/network/validateMacAlloc property not found on destination - Key not found: /config/vpxd/network/validateMacAlloc.

2018-04-24T15:18:01.362Z  Ignoring empty /config/vpxd/network/validateMacAlloc property value.

2018-04-24T15:18:01.362Z  Migrating DVS config...

2018-04-24T15:18:01.362Z  Ignoring /config/vpxd/dvs/noVendorSpecificConfigOutOfSync property not found on source - Key not found: /config/vpxd/dvs/noVendorSpecificConfigOutOfSync.

2018-04-24T15:18:01.362Z  Ignoring /config/vpxd/dvs/noVendorSpecificConfigOutOfSync property not found on destination - Key not found: /config/vpxd/dvs/noVendorSpecificConfigOutOfSync.

2018-04-24T15:18:01.362Z  Ignoring empty /config/vpxd/dvs/noVendorSpecificConfigOutOfSync property value.

2018-04-24T15:18:01.363Z  Ignoring /config/vpxd/dvs/makeDvsParent property not found on source - Key not found: /config/vpxd/dvs/makeDvsParent.

2018-04-24T15:18:01.363Z  Ignoring /config/vpxd/dvs/makeDvsParent property not found on destination - Key not found: /config/vpxd/dvs/makeDvsParent.

2018-04-24T15:18:01.363Z  Ignoring empty /config/vpxd/dvs/makeDvsParent property value.

2018-04-24T15:18:01.363Z  Ignoring /config/vpxd/dvs/noPreLoadPorts property not found on source - Key not found: /config/vpxd/dvs/noPreLoadPorts.

2018-04-24T15:18:01.364Z  Ignoring /config/vpxd/dvs/noPreLoadPorts property not found on destination - Key not found: /config/vpxd/dvs/noPreLoadPorts.

2018-04-24T15:18:01.364Z  Ignoring empty /config/vpxd/dvs/noPreLoadPorts property value.

2018-04-24T15:18:01.364Z  Ignoring /config/vpxd/dvs/unitTest/portPersistence property not found on source - Key not found: /config/vpxd/dvs/unitTest/portPersistence.

2018-04-24T15:18:01.364Z  Ignoring /config/vpxd/dvs/unitTest/portPersistence property not found on destination - Key not found: /config/vpxd/dvs/unitTest/portPersistence.

2018-04-24T15:18:01.364Z  Ignoring empty /config/vpxd/dvs/unitTest/portPersistence property value.

2018-04-24T15:18:01.364Z  Ignoring /config/vpxd/dvs/allowMultipleDvsPerExtension property not found on source - Key not found: /config/vpxd/dvs/allowMultipleDvsPerExtension.

2018-04-24T15:18:01.365Z  Ignoring /config/vpxd/dvs/allowMultipleDvsPerExtension property not found on destination - Key not found: /config/vpxd/dvs/allowMultipleDvsPerExtension.

2018-04-24T15:18:01.365Z  Ignoring empty /config/vpxd/dvs/allowMultipleDvsPerExtension property value.

2018-04-24T15:18:01.365Z  Ignoring /config/vpxd/das/findMasterIntervalSec property not found on source - Key not found: /config/vpxd/das/findMasterIntervalSec.

2018-04-24T15:18:01.365Z  Ignoring /config/vpxd/das/findMasterIntervalSec property not found on destination - Key not found: /config/vpxd/das/findMasterIntervalSec.

2018-04-24T15:18:01.365Z  Ignoring empty /config/vpxd/das/findMasterIntervalSec property value.

2018-04-24T15:18:01.365Z  Migrating database space monitoring thresholds ...

2018-04-24T15:18:01.365Z  Ignoring /config/vpxd/vdb/space/alertMB property not found on source - Key not found: /config/vpxd/vdb/space/alertMB.

2018-04-24T15:18:01.366Z  Ignoring /config/vpxd/vdb/space/alertMB property not found on destination - Key not found: /config/vpxd/vdb/space/alertMB.

2018-04-24T15:18:01.366Z  Ignoring empty /config/vpxd/vdb/space/alertMB property value.

2018-04-24T15:18:01.366Z  Ignoring /config/vpxd/vdb/space/warningMB property not found on source - Key not found: /config/vpxd/vdb/space/warningMB.

2018-04-24T15:18:01.366Z  Ignoring /config/vpxd/vdb/space/warningMB property not found on destination - Key not found: /config/vpxd/vdb/space/warningMB.

2018-04-24T15:18:01.366Z  Ignoring empty /config/vpxd/vdb/space/warningMB property value.

2018-04-24T15:18:01.366Z  No 6.x properties to migrate.

2018-04-24T15:18:01.367Z  Normalizing access records...

2018-04-24T15:18:01.367Z  Getting value for install-parameter: db.type

2018-04-24T15:18:01.367Z  Running command: ['/usr/sbin/vpxd', '-L']

2018-04-24T15:19:26.009Z  Done running command

[ashwin_prakash]

Check vmware-sts-idmd logs for more errors.

You could also share the log bundle, so we can also try to find some information.

[msripada]

Hello,

Please share these two files

/var/log//firstboot/vpxd-firstboot.py_#####_stdout.log

/var/log//firstboot/vpxd-firstboot.py_#####_stderr.log

along with the vpxd logs  in var/log/vmware/vpxd

We can identify why vpxd failed to start

Thanks,

MS

[ashvetsov]

vmware-sts-idmd.err is empty

vmware-sts-idmd-perf.log

[2018-04-24T14:57:08.576Z                      IDM Startup                          INFO ] [IdmServer] Starting IDM Server...

[2018-04-24T14:57:08.579Z                      IDM Startup                          INFO ] [VmEventAppender] EventLog: source=[VMware Identity Server], tenant=[], eventid=[SERVER_STARTED], level=[INFO], category=[VMEVENT_CATEGORY_IDM], text=[SimpleMessage[message=IDM Server has started]], detailText=[null], corelationId=[IDM Startup], timestamp=[1524581828578]

[2018-04-24T14:57:08.579Z                      IDM Startup                          INFO ] [IdmServer] IDM Server has started

[2018-04-24T14:57:20.816Z                                                           ERROR] [STSHealthChecker] Connection refused (Connection refused)

vpxd log attached

[msripada]

vpxd fails to connect with sso resulting in the issue.. can you check the vmware-identity-sts.log and ssoAdminserver.log

2018-04-24T15:18:45.960Z info vpxd[22498] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] Try to connect to SSO VMOMI endpoint

2018-04-24T15:18:45.966Z warning vpxd[22498] [Originator@6876 sub=Default] Closing Response processing in unexpected state: 3

2018-04-24T15:18:45.966Z warning vpxd[22498] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] [RetryOnConnectionFailure] Failed to connect to SSO; uri: http://localhost:7080/sso-adminserver/sdk/vsphere.local, reason: HttpException, ex: N7Vmacore4Http13HttpExceptionE(HTTP error response: Not Found)

--> [context]zKq7AVECAAAAALGqfAATdnB4ZAAAMJwqbGlidm1hY29yZS5zbwAAUC4bAJ6yGAHmCRFsaWJ2bW9taS5zbwAB7loRAWNcEQEojxICSRENbGlic3NvLXR5cGVzLnNvAAMaaIR2cHhkAAM/bIQDN26EA8xuhAODMHEDgSdxA7d4UgODp1IDdvpQBHAFAmxpYmMuc28uNgADBfRQ[/context]

Thanks,

MS

[ashvetsov]

there are no problems in vmware-identity-sts.log, but many in ssoAdminserver.log

[ERROR][2018-04-24T14:58:04.101Z][] IdentityManager - Failed to add user [machine-f306db4f-4fd7-4546-9a77-68b7ab4e90ed@mycompany.local] to group [ComponentManager.Administrators] in tenant [mycompany.local]

[ERROR][2018-04-24T14:58:04.102Z][] ServerUtils - Exception 'com.vmware.identity.idm.MemberAlreadyExistException: group ComponentManager.Administrators currently has user CN=machine-f306db4f-4fd7-4546-9a77-68b7ab4e90ed,CN=ServicePrincipals,DC=mycompany,DC=local as its member'

...

[ERROR][2018-04-24T14:58:04.122Z][] IdentityManager - Failed to add user [Administrator@mycompany.LOCAL] to group [ComponentManager.Administrators] in tenant [mycompany.local]

[ERROR][2018-04-24T14:58:04.122Z][] ServerUtils - Exception 'com.vmware.identity.idm.MemberAlreadyExistException: group ComponentManager.Administrators currently has user cn=Administrator,cn=Users,dc=mycompany,dc=local as its member'

...

[ERROR][2018-04-24T15:18:03.596Z][] IdentityManager - Failed to get tenant [vsphere.local]

[ERROR][2018-04-24T15:18:03.596Z][] ServerUtils - Exception 'com.vmware.identity.idm.NoSuchTenantException: No such tenant [vsphere.local]'

...

[ERROR][2018-04-24T15:18:03.596Z][] EndpointValidator - HTTP code 404 returned:  Identity provider 'vsphere.local' specified at given URI '/sso-adminserver/sdk/vsphere.local' does not exist!

[ERROR][2018-04-24T15:18:13.612Z][] IdentityManager - Failed to get tenant [vsphere.local]

[ERROR][2018-04-24T15:18:13.612Z][] ServerUtils - Exception 'com.vmware.identity.idm.NoSuchTenantException: No such tenant [vsphere.local]'

...

[ERROR][2018-04-24T15:18:55.975Z][] EndpointValidator - HTTP code 404 returned:  Identity provider 'vsphere.local' specified at given URI '/sso-adminserver/sdk/vsphere.local' does not exist!

[ERROR][2018-04-24T15:19:05.983Z][] IdentityManager - Failed to get tenant [vsphere.local]

[ERROR][2018-04-24T15:19:05.983Z][] ServerUtils - Exception 'com.vmware.identity.idm.NoSuchTenantException: No such tenant [vsphere.local]'

but I use mycompanyname.local tenant

Seems to be that upgrade script do not support custom tenant

[daphnissov]

Yet another reason to not change the default SSO domain At this point, you should probably open an SR with VMware (if you haven't done so already).

[ashvetsov]

Any ideas else?

[daphnissov]

Open a support case. This is occurring for several others with no resolution I've seen thus far.

[ashvetsov]

I find root cause. That is a custom machine SSL certificate from Comodo.

I restored VMCA at 6.5, than successfully migrated to 6.7.

Now I've tried to install custom certificate, but get same error - vpxd not starting and rollback it.

[ashwin_prakash]

Is the custom certificates a Wildcard certificates?

What are the errors that you are observing in the certificate.manager.log

[ashvetsov]

Not wildcard.

service-control.log

Service-control failed. Error: Failed to start services in profile ALL. RC=2, stderr=Failed to start vpxd services. Error: Service crashed while starting

certificate-manager.log

2018-05-10T15:56:39.726Z INFO certificate-manager Running command :- service-control --start  --all

2018-05-10T15:56:39.726Z INFO certificate-manager please see service-control.log for service status

Service-control failed. Error: Failed to start services in profile ALL. RC=2, stderr=Failed to start vpxd services. Error: Service crashed while starting

2018-05-10T16:02:14.843Z ERROR certificate-manager None

2018-05-10T16:02:14.844Z ERROR certificate-manager Error while starting services, please see service-control log for more details

2018-05-10T16:02:14.844Z ERROR certificate-manager Error while replacing Machine SSL Cert, please see /var/log/vmware/vmcad/certificate-manager.log for more information.

2018-05-10T16:02:14.844Z ERROR certificate-manager {

    "problemId": null,

    "detail": [

        {

            "args": [

                "None"

            ],

            "id": "install.ciscommon.command.errinvoke",

            "localized": "An error occurred while invoking external command : 'None'",

            "translatable": "An error occurred while invoking external command : '%(0)s'"

        },

        "Error while starting services, please see service-control log for more details"

    ],

    "componentKey": null,

    "resolution": null

}

2018-05-10T16:02:14.844Z INFO certificate-manager Performing rollback of Machine SSL Cert...

vpxd.log has errors "unable to get issuer certificate" and "Host name does not match the subject name(s) in certificate", but certificate works fine at 6.5

[ashwin_prakash]

Check if the DNS records are correct.

Check the host name associated on VCSA 6.7

Create a new CSR make sure you are using the correct host name and fqdn

[shepart]

vmware knows the problem. They located the problem and in some days there will be an fix available.

regards

[ashvetsov]

6.7a has the same problem with certificates

[ashvetsov]

6.7.0с still not resolve the certificate issue. The same errors.

Despite of this:

The vSphere Certificate Manager utility fails to replace a machine SSL certificate if it contains extra details in the Subject Alternative Name (SAN) field

The vSphere Certificate Manager utility might fail to replace a machine SSL certificate if you specify additional details in the SAN field, such as sites, IP addresses and common names. With this fix, the vSphere Certificate Manager only checks the system name in the SAN field of machine SSL certificates.

[ashvetsov · Accepted solution]

This article helps me to understand root cause with comodo certificates.

http://engineering.pivotal.io/post/vcenter_6.7_tls/

So, I reverted self-signed certificate at 6.5, successfully upgrade to 6.7, then download comodo root certificate and install all comodo certs via web console.