1 2 Previous Next 18 Replies Latest reply on Aug 20, 2018 10:51 PM by ashvetsov

    Upgrade VCSA 6.5 to 6.7 failed at 60%

    ashvetsov Novice

      I tried three time to upgrade VCSA 6.5U1g to 6.7 and always failed at 60%.

       

      firstbootInfrastructure.log

      2018-04-24T15:12:19.187Z INFO firstbootInfrastructure Running firstboot script: ['/opt/vmware/bin/python', '/usr/lib/vmware-vpx/firstboot/vpxd_firstboot.py', '--action', 'firstboot', '--compkey', 'vpxd', '--errlog', '/var/log/firstboot/vpxd_firstboot.py_9477_stderr.log']

      2018-04-24T15:19:26.82Z INFO firstbootInfrastructure [Failed] /usr/lib/vmware-vpx/firstboot/vpxd_firstboot.py is complete

      2018-04-24T15:19:26.86Z WARNING firstbootInfrastructure Bug component info file does not exist

      2018-04-24T15:19:26.86Z INFO firstbootInfrastructure Firstboot duration: 1461 sec

      2018-04-24T15:19:26.86Z INFO firstbootInfrastructure First boot is a failure

      2018-04-24T15:19:26.93Z INFO firstbootInfrastructure Changing vMon default start profile to ALL

      2018-04-24T15:19:26.93Z ERROR firstbootInfrastructure Installation of vCenter Server failed with error:

      ['Traceback (most recent call last):\n', '  File "/bin/run-firstboot-scripts", line 800, in <module>\n    add_vaos_dependencies()\n', '  File "/bin/run-firstboot-scripts", line 232, in add_vaos_dependencies\n    with open(conf_file + \'.tmp\', \'w\') as fp:\n', "FileNotFoundError: [Errno 2] No such file or directory: '/etc/systemd/system/vmware-vmon.service.d/dep_override.conf.tmp'\n"]

      2018-04-24T15:19:26.95Z WARNING firstbootInfrastructure stopping status aggregation...

       

      cloudvm.log

      2018-04-24T14:39:13.189Z: Upating deployment node type embedded

      2018-04-24T14:40:09.437Z: Upgrade CHECKS requested. Starting...

      2018-04-24T14:42:20.839Z: Upgrade CHECKS succeeded.

      2018-04-24T14:44:31.993Z: Upgrade EXPORT requested. Starting...

      2018-04-24T14:55:03.925Z: Upgrade EXPORT succeeded.

      2018-04-24T15:19:26.148Z: Failed to start services. Firstboot Error.

      2018-04-24T15:19:26.150Z: Starting ssh...

       

      fbInstall.json

          "totalSteps": 38,

          "failedSteps": "vpxd_firstboot",

          "stepsCompleted": 24,

          "stepsStarted": 24

       

      vpxd_firstboot.py_9477_stderr.log

      2018-04-24T15:19:26.009Z  Invoked command: ['/usr/sbin/vpxd', '-L']

      2018-04-24T15:19:26.009Z  RC = 2

      Stdout =

      Stderr =

      2018-04-24T15:19:26.010Z  VirtualCenter firstboot failed

       

      Any ideas?

        • 1. Re: Upgrade VCSA 6.5 to 6.7 failed at 60%
          ashwin_prakash Expert
          GS Skyline Support Skyline Community 2/2VMware Employees

          Hello,

           

          Please check for error that are reported on the firstboot logs for their respective services.

           

          /var/log/firstboot/vmidentity-firstboot.py_#####_stdout.log

          /var/log//firstboot/vmidentity-firstboot.py_####_stderr.log

          /var/log//firstboot/vpxd-firstboot.py_#####_stdout.log

          /var/log//firstboot/vpxd-firstboot.py_#####_stderr.log

           

          Check the errors that are reported on the logs.

          • 2. Re: Upgrade VCSA 6.5 to 6.7 failed at 60%
            ashvetsov Novice

            I see some errors with vpxd but can't understand root cause.

             

            vmidentity-firstboot.py_9477_stderr.log

            2018-04-24T14:57:06.724Z  Failure setting accounting for vmware-sts-idmd. Err Failed to set unit properties on vmware-sts-idmd.service: Unit vmware-sts-idmd.service is not loaded.

            2018-04-24T14:57:19.528Z  Failure setting accounting for vmware-stsd. Err Failed to set unit properties on vmware-stsd.service: Unit vmware-stsd.service is not loaded.

             

            vmidentity-firstboot.py_9477_stdout

            Successfully checked VMware STS.

            2018-04-24T14:57:42.600Z  <<<<stdout

            2018-04-24T14:57:42.600Z  ===Return code: 0

            2018-04-24T14:57:42.600Z  Getting value for install-parameter: upgrade.import.directory

            2018-04-24T14:57:42.601Z  VMware Identity Service bootstrap: importDirectory=/storage/seat/cis-export-folder/sso

            2018-04-24T14:57:42.601Z  VMware Identity Service bootstrap: isUgprading=True

            2018-04-24T14:57:42.601Z  Getting value for install-parameter: upgrade.import.directory

            2018-04-24T14:57:42.601Z  VMware Identity Service bootstrap: importDirectory=/storage/seat/cis-export-folder/sso

            2018-04-24T14:57:42.601Z  VMware Identity Service bootstrap: isUgprading=True

            2018-04-24T14:57:42.601Z  Getting value for install-parameter: upgrade.import.directory

            2018-04-24T14:57:42.601Z  VMware Identity Service bootstrap: importDirectory=/storage/seat/cis-export-folder/sso

            2018-04-24T14:57:42.602Z  Registering sts and idm services

            2018-04-24T14:57:42.602Z  Running command: ['chkconfig', 'vmware-sts-idmd', 'on']

            2018-04-24T14:57:42.684Z  Done running command

            2018-04-24T14:57:42.684Z  >>>>stdout:

            2018-04-24T14:57:42.684Z 

            2018-04-24T14:57:42.684Z  <<<<stdout

            2018-04-24T14:57:42.684Z  ===Return code: 0

            2018-04-24T14:57:42.684Z  Running command: ['chkconfig', 'vmware-stsd', 'on']

            2018-04-24T14:57:42.804Z  Done running command

            2018-04-24T14:57:42.804Z  >>>>stdout:

            2018-04-24T14:57:42.804Z 

            2018-04-24T14:57:42.804Z  <<<<stdout

            2018-04-24T14:57:42.804Z  ===Return code: 0

            2018-04-24T14:57:42.804Z  Registration of sts and idm services complete

            2018-04-24T14:57:42.804Z  Running command: ['/bin/ln', '-s', '-f', '/usr/lib/vmware-sso/firewall/sso-firewall.json', '/etc/vmware/appliance/firewall/vmware-sso']

            2018-04-24T14:57:42.807Z  Done running command

            2018-04-24T14:57:42.807Z  >>>>stdout:

            2018-04-24T14:57:42.807Z 

            2018-04-24T14:57:42.807Z  <<<<stdout

            2018-04-24T14:57:42.807Z  ===Return code: 0

            2018-04-24T14:57:42.807Z  Running command: ['/usr/lib/applmgmt/networking/bin/firewall-reload']

            2018-04-24T14:57:43.194Z  Done running command

            2018-04-24T14:57:43.194Z  >>>>stdout:

            2018-04-24T14:57:43.194Z 

            2018-04-24T14:57:43.194Z  <<<<stdout

            2018-04-24T14:57:43.194Z  ===Return code: 0

            2018-04-24T14:57:43.195Z  Successfully processed VMware Identity Service bootstrap - FIRSTBOOT.

            2018-04-24T14:57:43.195Z  VMware Identity Service bootstrap completed successfully.

             

            vpxd_firstboot.py_9477_stderr.log

            2018-04-24T15:19:26.009Z  Invoked command: ['/usr/sbin/vpxd', '-L']

            2018-04-24T15:19:26.009Z  RC = 2

            Stdout =

            Stderr =

            2018-04-24T15:19:26.010Z  VirtualCenter firstboot failed

             

            vpxd_firstboot.py_9477_stdout.log

            2018-04-24T15:17:58.665Z  Registered 'vpxd' as local service in SCA

            2018-04-24T15:17:58.666Z  Running command: ['pidof', 'rhttpproxy']

            2018-04-24T15:17:58.673Z  Done running command

            2018-04-24T15:17:58.673Z  Running command: ['/bin/kill', '-HUP', '11614']

            2018-04-24T15:17:58.676Z  Done running command

            2018-04-24T15:17:58.676Z  Replacing solution-user.name

            2018-04-24T15:17:58.676Z  Replacing vmdir.domain-name

            2018-04-24T15:17:58.677Z  Getting value for install-parameter: vmdir.domain-name

            2018-04-24T15:17:58.677Z  Replacing solution-user.name

            2018-04-24T15:17:58.677Z  Replacing vpxd.serviceid

            2018-04-24T15:17:58.677Z  Replacing sca.hostid

            2018-04-24T15:17:58.677Z  Getting value for install-parameter: sca.hostid

            2018-04-24T15:17:58.677Z  Replacing vc.control.script

            2018-04-24T15:17:58.677Z  Replacing rhttpproxy.ext.port2

            2018-04-24T15:17:58.678Z  Getting value for install-parameter: rhttpproxy.ext.port2

            2018-04-24T15:17:58.678Z  Replacing vc.health.location

            2018-04-24T15:17:58.678Z  Replacing rhttpproxy.ext.port2

            2018-04-24T15:17:58.678Z  Getting value for install-parameter: rhttpproxy.ext.port2

            2018-04-24T15:17:58.678Z  Replacing rhttpproxy.ext.port2

            2018-04-24T15:17:58.678Z  Getting value for install-parameter: rhttpproxy.ext.port2

            2018-04-24T15:17:58.678Z  Replacing rhttpproxy.ext.port2

            2018-04-24T15:17:58.678Z  Getting value for install-parameter: rhttpproxy.ext.port2

            2018-04-24T15:17:58.679Z  Replacing rhttpproxy.ext.port2

            2018-04-24T15:17:58.679Z  Getting value for install-parameter: rhttpproxy.ext.port2

            2018-04-24T15:17:58.679Z  Replacing vc.home.javapath

            2018-04-24T15:17:58.679Z  Replacing vc.home.javapath

            2018-04-24T15:17:58.679Z  Replacing vpxd.int.sdk-port

            2018-04-24T15:17:58.679Z  Getting value for install-parameter: vpxd.int.sdk-port

            2018-04-24T15:17:58.679Z  Replacing vpxd.int.sdk-tunnel-port

            2018-04-24T15:17:58.679Z  Getting value for install-parameter: vpxd.int.sdk-tunnel-port

            2018-04-24T15:17:58.680Z  Replacing vpxd.int.sdk-tunnel-port

            2018-04-24T15:17:58.680Z  Getting value for install-parameter: vpxd.int.sdk-tunnel-port

            2018-04-24T15:17:58.680Z  Replacing vpxd.int.sdk-tunnel-port

            2018-04-24T15:17:58.680Z  Getting value for install-parameter: vpxd.int.sdk-tunnel-port

            2018-04-24T15:17:58.680Z  Replacing vpxd.int.sdk-tunnel-port

            2018-04-24T15:17:58.680Z  Getting value for install-parameter: vpxd.int.sdk-tunnel-port

            2018-04-24T15:17:58.680Z  Replacing vpxd.int.sdk-tunnel-port

            2018-04-24T15:17:58.680Z  Getting value for install-parameter: vpxd.int.sdk-tunnel-port

            2018-04-24T15:17:58.680Z  Replacing vpxd.int.sdk-port

            2018-04-24T15:17:58.680Z  Getting value for install-parameter: vpxd.int.sdk-port

            2018-04-24T15:17:58.680Z  Getting value for install-parameter: sca.hostid

            2018-04-24T15:17:58.754Z  Getting value for install-parameter: vmdir.username

            2018-04-24T15:17:58.754Z  Getting value for install-parameter: vmdir.password

            2018-04-24T15:17:58.754Z  Getting value for install-parameter: sca.hostid

            2018-04-24T15:17:58.761Z  Getting value for install-parameter: vmdir.domain-dn

            2018-04-24T15:17:58.761Z  Initiating new cloudvm_sso_cm_register operation

            2018-04-24T15:18:01.027Z  Successfully performed tryupgrade operation for service b086700d-8401-48b4-a11a-c267fb5b1322

            2018-04-24T15:18:01.027Z  Running command: ['/usr/lib/vmware-vmafd/bin/vmafd-cli', 'get-machine-id', '--server-name', 'localhost']

            2018-04-24T15:18:01.038Z  Done running command

            2018-04-24T15:18:01.038Z  Getting value for install-parameter: vmdir.domain-name

            2018-04-24T15:18:01.171Z  Setting vc.instance.cfg.path install-parameter to: /etc/vmware-vpx/instance.cfg

            2018-04-24T15:18:01.306Z  Getting value for install-parameter: vpxd.mac-allocation-scheme.prefix

            2018-04-24T15:18:01.306Z  scheme_prefix=

            2018-04-24T15:18:01.306Z  Getting value for install-parameter: vpxd.mac-allocation-scheme.prefix-length

            2018-04-24T15:18:01.306Z  scheme_prefixlen=0

            2018-04-24T15:18:01.306Z  Getting value for install-parameter: vpxd.mac-allocation-scheme.ranges

            2018-04-24T15:18:01.306Z  range_scheme=

            2018-04-24T15:18:01.306Z  Getting value for install-parameter: vsm.int.http

            2018-04-24T15:18:01.307Z  Performing text substitutions in /etc/vmware-vpx/extensions/com.vmware.vim.vsm/extension.xml

            2018-04-24T15:18:01.356Z  Skipped re-encrypting db for 6.0 vCenter.

            2018-04-24T15:18:01.356Z  Migrating vpxd.cfg from version 650...

            2018-04-24T15:18:01.357Z  Migrating logging config...

            2018-04-24T15:18:01.357Z  Ignoring /config/alert/log/enabled property not found on destination - Key not found: /config/alert/log/enabled.

            2018-04-24T15:18:01.357Z  Added /config/alert/log/enabled property with source value true.

            2018-04-24T15:18:01.358Z  Ignoring /config/level[@id="CpuFeatures"]/logLevel property not found on source - Key not found: /config/level[@id="CpuFeatures"]/logLevel.

            2018-04-24T15:18:01.358Z  Ignoring /config/level[@id="CpuFeatures"]/logLevel property not found on destination - Key not found: /config/level[@id="CpuFeatures"]/logLevel.

            2018-04-24T15:18:01.358Z  Ignoring empty /config/level[@id="CpuFeatures"]/logLevel property value.

            2018-04-24T15:18:01.358Z  Ignoring /config/level[@id="CpuFeatures"]/logName property not found on source - Key not found: /config/level[@id="CpuFeatures"]/logName.

            2018-04-24T15:18:01.358Z  Ignoring /config/level[@id="CpuFeatures"]/logName property not found on destination - Key not found: /config/level[@id="CpuFeatures"]/logName.

            2018-04-24T15:18:01.358Z  Ignoring empty /config/level[@id="CpuFeatures"]/logName property value.

            2018-04-24T15:18:01.358Z  Ignoring /config/level[@id="VmCheck"]/logLevel property not found on source - Key not found: /config/level[@id="VmCheck"]/logLevel.

            2018-04-24T15:18:01.359Z  Ignoring /config/level[@id="VmCheck"]/logLevel property not found on destination - Key not found: /config/level[@id="VmCheck"]/logLevel.

            2018-04-24T15:18:01.359Z  Ignoring empty /config/level[@id="VmCheck"]/logLevel property value.

            2018-04-24T15:18:01.359Z  Ignoring /config/level[@id="VmCheck"]/logName property not found on source - Key not found: /config/level[@id="VmCheck"]/logName.

            2018-04-24T15:18:01.359Z  Ignoring /config/level[@id="VmCheck"]/logName property not found on destination - Key not found: /config/level[@id="VmCheck"]/logName.

            2018-04-24T15:18:01.359Z  Ignoring empty /config/level[@id="VmCheck"]/logName property value.

            2018-04-24T15:18:01.359Z  Ignoring unchanged /config/log/compressOnRoll property value.

            2018-04-24T15:18:01.359Z  Ignoring unchanged /config/log/level property value.

            2018-04-24T15:18:01.359Z  Ignoring unchanged /config/log/maxFileNum property value.

            2018-04-24T15:18:01.359Z  Ignoring unchanged /config/log/maxFileSize property value.

            2018-04-24T15:18:01.359Z  Migrating network config...

            2018-04-24T15:18:01.360Z  Ignoring unchanged /config/vpxd/network/rollback property value.

            2018-04-24T15:18:01.360Z  Ignoring /config/vpxd/network/rollbackTimeout property not found on source - Key not found: /config/vpxd/network/rollbackTimeout.

            2018-04-24T15:18:01.360Z  Ignoring /config/vpxd/network/rollbackTimeout property not found on destination - Key not found: /config/vpxd/network/rollbackTimeout.

            2018-04-24T15:18:01.360Z  Ignoring empty /config/vpxd/network/rollbackTimeout property value.

            2018-04-24T15:18:01.360Z  Ignoring /config/vpxd/network/macReclaim property not found on source - Key not found: /config/vpxd/network/macReclaim.

            2018-04-24T15:18:01.361Z  Ignoring /config/vpxd/network/macReclaim property not found on destination - Key not found: /config/vpxd/network/macReclaim.

            2018-04-24T15:18:01.361Z  Ignoring empty /config/vpxd/network/macReclaim property value.

            2018-04-24T15:18:01.361Z  Ignoring /config/vpxd/network/macReclaim/size property not found on source - Key not found: /config/vpxd/network/macReclaim/size.

            2018-04-24T15:18:01.361Z  Ignoring /config/vpxd/network/macReclaim/size property not found on destination - Key not found: /config/vpxd/network/macReclaim/size.

            2018-04-24T15:18:01.361Z  Ignoring empty /config/vpxd/network/macReclaim/size property value.

            2018-04-24T15:18:01.362Z  Ignoring /config/vpxd/network/validateMacAlloc property not found on source - Key not found: /config/vpxd/network/validateMacAlloc.

            2018-04-24T15:18:01.362Z  Ignoring /config/vpxd/network/validateMacAlloc property not found on destination - Key not found: /config/vpxd/network/validateMacAlloc.

            2018-04-24T15:18:01.362Z  Ignoring empty /config/vpxd/network/validateMacAlloc property value.

            2018-04-24T15:18:01.362Z  Migrating DVS config...

            2018-04-24T15:18:01.362Z  Ignoring /config/vpxd/dvs/noVendorSpecificConfigOutOfSync property not found on source - Key not found: /config/vpxd/dvs/noVendorSpecificConfigOutOfSync.

            2018-04-24T15:18:01.362Z  Ignoring /config/vpxd/dvs/noVendorSpecificConfigOutOfSync property not found on destination - Key not found: /config/vpxd/dvs/noVendorSpecificConfigOutOfSync.

            2018-04-24T15:18:01.362Z  Ignoring empty /config/vpxd/dvs/noVendorSpecificConfigOutOfSync property value.

            2018-04-24T15:18:01.363Z  Ignoring /config/vpxd/dvs/makeDvsParent property not found on source - Key not found: /config/vpxd/dvs/makeDvsParent.

            2018-04-24T15:18:01.363Z  Ignoring /config/vpxd/dvs/makeDvsParent property not found on destination - Key not found: /config/vpxd/dvs/makeDvsParent.

            2018-04-24T15:18:01.363Z  Ignoring empty /config/vpxd/dvs/makeDvsParent property value.

            2018-04-24T15:18:01.363Z  Ignoring /config/vpxd/dvs/noPreLoadPorts property not found on source - Key not found: /config/vpxd/dvs/noPreLoadPorts.

            2018-04-24T15:18:01.364Z  Ignoring /config/vpxd/dvs/noPreLoadPorts property not found on destination - Key not found: /config/vpxd/dvs/noPreLoadPorts.

            2018-04-24T15:18:01.364Z  Ignoring empty /config/vpxd/dvs/noPreLoadPorts property value.

            2018-04-24T15:18:01.364Z  Ignoring /config/vpxd/dvs/unitTest/portPersistence property not found on source - Key not found: /config/vpxd/dvs/unitTest/portPersistence.

            2018-04-24T15:18:01.364Z  Ignoring /config/vpxd/dvs/unitTest/portPersistence property not found on destination - Key not found: /config/vpxd/dvs/unitTest/portPersistence.

            2018-04-24T15:18:01.364Z  Ignoring empty /config/vpxd/dvs/unitTest/portPersistence property value.

            2018-04-24T15:18:01.364Z  Ignoring /config/vpxd/dvs/allowMultipleDvsPerExtension property not found on source - Key not found: /config/vpxd/dvs/allowMultipleDvsPerExtension.

            2018-04-24T15:18:01.365Z  Ignoring /config/vpxd/dvs/allowMultipleDvsPerExtension property not found on destination - Key not found: /config/vpxd/dvs/allowMultipleDvsPerExtension.

            2018-04-24T15:18:01.365Z  Ignoring empty /config/vpxd/dvs/allowMultipleDvsPerExtension property value.

            2018-04-24T15:18:01.365Z  Ignoring /config/vpxd/das/findMasterIntervalSec property not found on source - Key not found: /config/vpxd/das/findMasterIntervalSec.

            2018-04-24T15:18:01.365Z  Ignoring /config/vpxd/das/findMasterIntervalSec property not found on destination - Key not found: /config/vpxd/das/findMasterIntervalSec.

            2018-04-24T15:18:01.365Z  Ignoring empty /config/vpxd/das/findMasterIntervalSec property value.

            2018-04-24T15:18:01.365Z  Migrating database space monitoring thresholds ...

            2018-04-24T15:18:01.365Z  Ignoring /config/vpxd/vdb/space/alertMB property not found on source - Key not found: /config/vpxd/vdb/space/alertMB.

            2018-04-24T15:18:01.366Z  Ignoring /config/vpxd/vdb/space/alertMB property not found on destination - Key not found: /config/vpxd/vdb/space/alertMB.

            2018-04-24T15:18:01.366Z  Ignoring empty /config/vpxd/vdb/space/alertMB property value.

            2018-04-24T15:18:01.366Z  Ignoring /config/vpxd/vdb/space/warningMB property not found on source - Key not found: /config/vpxd/vdb/space/warningMB.

            2018-04-24T15:18:01.366Z  Ignoring /config/vpxd/vdb/space/warningMB property not found on destination - Key not found: /config/vpxd/vdb/space/warningMB.

            2018-04-24T15:18:01.366Z  Ignoring empty /config/vpxd/vdb/space/warningMB property value.

            2018-04-24T15:18:01.366Z  No 6.x properties to migrate.

            2018-04-24T15:18:01.367Z  Normalizing access records...

            2018-04-24T15:18:01.367Z  Getting value for install-parameter: db.type

            2018-04-24T15:18:01.367Z  Running command: ['/usr/sbin/vpxd', '-L']

            2018-04-24T15:19:26.009Z  Done running command

            • 3. Re: Upgrade VCSA 6.5 to 6.7 failed at 60%
              ashwin_prakash Expert
              VMware EmployeesSkyline Community 2/2GS Skyline Support

              Check vmware-sts-idmd logs for more errors.

              You could also share the log bundle, so we can also try to find some information.

              • 4. Re: Upgrade VCSA 6.5 to 6.7 failed at 60%
                msripada Expert
                vExpert

                Hello,

                 

                Please share these two files

                /var/log//firstboot/vpxd-firstboot.py_#####_stdout.log

                /var/log//firstboot/vpxd-firstboot.py_#####_stderr.log

                along with the vpxd logs  in var/log/vmware/vpxd

                 

                We can identify why vpxd failed to start

                 

                Thanks,

                MS

                • 5. Re: Upgrade VCSA 6.5 to 6.7 failed at 60%
                  ashvetsov Novice

                  vmware-sts-idmd.err is empty

                   

                  vmware-sts-idmd-perf.log

                  [2018-04-24T14:57:08.576Z                      IDM Startup                          INFO ] [IdmServer] Starting IDM Server...

                  [2018-04-24T14:57:08.579Z                      IDM Startup                          INFO ] [VmEventAppender] EventLog: source=[VMware Identity Server], tenant=[], eventid=[SERVER_STARTED], level=[INFO], category=[VMEVENT_CATEGORY_IDM], text=[SimpleMessage[message=IDM Server has started]], detailText=[null], corelationId=[IDM Startup], timestamp=[1524581828578]

                  [2018-04-24T14:57:08.579Z                      IDM Startup                          INFO ] [IdmServer] IDM Server has started

                  [2018-04-24T14:57:20.816Z                                                           ERROR] [STSHealthChecker] Connection refused (Connection refused)

                   

                  vpxd log attached

                  • 6. Re: Upgrade VCSA 6.5 to 6.7 failed at 60%
                    msripada Expert
                    vExpert

                    vpxd fails to connect with sso resulting in the issue.. can you check the vmware-identity-sts.log and ssoAdminserver.log

                     

                    2018-04-24T15:18:45.960Z info vpxd[22498] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] Try to connect to SSO VMOMI endpoint

                    2018-04-24T15:18:45.966Z warning vpxd[22498] [Originator@6876 sub=Default] Closing Response processing in unexpected state: 3

                    2018-04-24T15:18:45.966Z warning vpxd[22498] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] [RetryOnConnectionFailure] Failed to connect to SSO; uri: http://localhost:7080/sso-adminserver/sdk/vsphere.local, reason: HttpException, ex: N7Vmacore4Http13HttpExceptionE(HTTP error response: Not Found)

                    --> [context]zKq7AVECAAAAALGqfAATdnB4ZAAAMJwqbGlidm1hY29yZS5zbwAAUC4bAJ6yGAHmCRFsaWJ2bW9taS5zbwAB7loRAWNcEQEojxICSRENbGlic3NvLXR5cGVzLnNvAAMaaIR2cHhkAAM/bIQDN26EA8xuhAODMHEDgSdxA7d4UgODp1IDdvpQBHAFAmxpYmMuc28uNgADBfRQ[/context]

                     

                    Thanks,

                    MS

                    • 7. Re: Upgrade VCSA 6.5 to 6.7 failed at 60%
                      ashvetsov Novice

                      there are no problems in vmware-identity-sts.log, but many in ssoAdminserver.log

                       

                      [ERROR][2018-04-24T14:58:04.101Z][] IdentityManager - Failed to add user [machine-f306db4f-4fd7-4546-9a77-68b7ab4e90ed@mycompany.local] to group [ComponentManager.Administrators] in tenant [mycompany.local]

                      [ERROR][2018-04-24T14:58:04.102Z][] ServerUtils - Exception 'com.vmware.identity.idm.MemberAlreadyExistException: group ComponentManager.Administrators currently has user CN=machine-f306db4f-4fd7-4546-9a77-68b7ab4e90ed,CN=ServicePrincipals,DC=mycompany,DC=local as its member'

                      ...

                      [ERROR][2018-04-24T14:58:04.122Z][] IdentityManager - Failed to add user [Administrator@mycompany.LOCAL] to group [ComponentManager.Administrators] in tenant [mycompany.local]

                      [ERROR][2018-04-24T14:58:04.122Z][] ServerUtils - Exception 'com.vmware.identity.idm.MemberAlreadyExistException: group ComponentManager.Administrators currently has user cn=Administrator,cn=Users,dc=mycompany,dc=local as its member'

                      ...

                      [ERROR][2018-04-24T15:18:03.596Z][] IdentityManager - Failed to get tenant [vsphere.local]

                      [ERROR][2018-04-24T15:18:03.596Z][] ServerUtils - Exception 'com.vmware.identity.idm.NoSuchTenantException: No such tenant [vsphere.local]'

                      ...

                      [ERROR][2018-04-24T15:18:03.596Z][] EndpointValidator - HTTP code 404 returned:  Identity provider 'vsphere.local' specified at given URI '/sso-adminserver/sdk/vsphere.local' does not exist!

                      [ERROR][2018-04-24T15:18:13.612Z][] IdentityManager - Failed to get tenant [vsphere.local]

                      [ERROR][2018-04-24T15:18:13.612Z][] ServerUtils - Exception 'com.vmware.identity.idm.NoSuchTenantException: No such tenant [vsphere.local]'

                      ...

                      [ERROR][2018-04-24T15:18:55.975Z][] EndpointValidator - HTTP code 404 returned:  Identity provider 'vsphere.local' specified at given URI '/sso-adminserver/sdk/vsphere.local' does not exist!

                      [ERROR][2018-04-24T15:19:05.983Z][] IdentityManager - Failed to get tenant [vsphere.local]

                      [ERROR][2018-04-24T15:19:05.983Z][] ServerUtils - Exception 'com.vmware.identity.idm.NoSuchTenantException: No such tenant [vsphere.local]'

                       

                      but I use mycompanyname.local tenant

                      Seems to be that upgrade script do not support custom tenant

                       

                       

                       

                      • 8. Re: Upgrade VCSA 6.5 to 6.7 failed at 60%
                        daphnissov Guru
                        vExpertCommunity Warriors

                        Yet another reason to not change the default SSO domain At this point, you should probably open an SR with VMware (if you haven't done so already).

                        • 9. Re: Upgrade VCSA 6.5 to 6.7 failed at 60%
                          ashvetsov Novice

                          Any ideas else?

                          • 10. Re: Upgrade VCSA 6.5 to 6.7 failed at 60%
                            daphnissov Guru
                            Community WarriorsvExpert

                            Open a support case. This is occurring for several others with no resolution I've seen thus far.

                            • 11. Re: Upgrade VCSA 6.5 to 6.7 failed at 60%
                              ashvetsov Novice

                              I find root cause. That is a custom machine SSL certificate from Comodo.

                              I restored VMCA at 6.5, than successfully migrated to 6.7.

                              Now I've tried to install custom certificate, but get same error - vpxd not starting and rollback it.

                               

                               

                              • 12. Re: Upgrade VCSA 6.5 to 6.7 failed at 60%
                                ashwin_prakash Expert
                                VMware EmployeesGS Skyline Support Skyline Community 2/2

                                Is the custom certificates a Wildcard certificates?

                                What are the errors that you are observing in the certificate.manager.log

                                • 13. Re: Upgrade VCSA 6.5 to 6.7 failed at 60%
                                  ashvetsov Novice

                                  Not wildcard.

                                   

                                  service-control.log

                                  Service-control failed. Error: Failed to start services in profile ALL. RC=2, stderr=Failed to start vpxd services. Error: Service crashed while starting

                                   

                                  certificate-manager.log

                                  2018-05-10T15:56:39.726Z INFO certificate-manager Running command :- service-control --start  --all

                                  2018-05-10T15:56:39.726Z INFO certificate-manager please see service-control.log for service status

                                  Service-control failed. Error: Failed to start services in profile ALL. RC=2, stderr=Failed to start vpxd services. Error: Service crashed while starting

                                  2018-05-10T16:02:14.843Z ERROR certificate-manager None

                                  2018-05-10T16:02:14.844Z ERROR certificate-manager Error while starting services, please see service-control log for more details

                                  2018-05-10T16:02:14.844Z ERROR certificate-manager Error while replacing Machine SSL Cert, please see /var/log/vmware/vmcad/certificate-manager.log for more information.

                                  2018-05-10T16:02:14.844Z ERROR certificate-manager {

                                      "problemId": null,

                                      "detail": [

                                          {

                                              "args": [

                                                  "None"

                                              ],

                                              "id": "install.ciscommon.command.errinvoke",

                                              "localized": "An error occurred while invoking external command : 'None'",

                                              "translatable": "An error occurred while invoking external command : '%(0)s'"

                                          },

                                          "Error while starting services, please see service-control log for more details"

                                      ],

                                      "componentKey": null,

                                      "resolution": null

                                  }

                                  2018-05-10T16:02:14.844Z INFO certificate-manager Performing rollback of Machine SSL Cert...

                                   

                                   

                                  vpxd.log has errors "unable to get issuer certificate" and "Host name does not match the subject name(s) in certificate", but certificate works fine at 6.5

                                  • 14. Re: Upgrade VCSA 6.5 to 6.7 failed at 60%
                                    ashwin_prakash Expert
                                    VMware EmployeesGS Skyline Support Skyline Community 2/2

                                    Check if the DNS records are correct.

                                    Check the host name associated on VCSA 6.7

                                    Create a new CSR make sure you are using the correct host name and fqdn

                                    1 2 Previous Next