VMware Cloud Community
dekoshal
Hot Shot
Hot Shot
Jump to solution

vcenter upgrade failed

From vcenter 6.0 4541947 vCenter Server 6.0 Update 2a

To vcenter 6.0 7037394 vCenter Server 6.0 Update 3c

uninstall.log

2017-02-18 02:23:30,903: CM search failed. ProductId=com.vmware.cis, TypeId=cs.identity

2017-02-18 02:23:30,903: Error connecting to CM: {

    "resolution": {

        "id": "install.default.failure.resolution", 

        "localized": "This is an unrecoverable error, please retry install. If you run into this error again, please collect a support bundle and open a support request.", 

        "translatable": "This is an unrecoverable error, please retry install. If you run into this error again, please collect a support bundle and open a support request."

    }, 

    "detail": [

        {

            "args": [

                "CM search failed. ProductId=com.vmware.cis, TypeId=cs.identity:503 Service Unavailable (Failed to connect to endpoint: [class Vmacore::Http::NamedPipeServiceSpec:00000000018056F0] _serverNamespace = / _isRedirect = false _pipeName =\\\\.\\pipe\\vmware-vpxd-webserver-pipe)"

            ], 

            "id": "install.ciscommon.internal.error", 

            "localized": "Encountered an internal error.\n\nCM search failed. ProductId=com.vmware.cis, TypeId=cs.identity:503 Service Unavailable (Failed to connect to endpoint: [class Vmacore::Http::NamedPipeServiceSpec:00000000018056F0] _serverNamespace = / _isRedirect = false _pipeName =\\\\.\\pipe\\vmware-vpxd-webserver-pipe)", 

            "translatable": "Encountered an internal error.\n\n%(0)s"

        }

    ], 

    "componentKey": null, 

    "problemId": null

}

2017-02-18 02:23:30,903: Failed to unregister node from CM. Error {

    "resolution": {

        "id": "install.default.failure.resolution", 

        "localized": "This is an unrecoverable error, please retry install. If you run into this error again, please collect a support bundle and open a support request.", 

        "translatable": "This is an unrecoverable error, please retry install. If you run into this error again, please collect a support bundle and open a support request."

    }, 

    "detail": [

        {

============================================================================================

soluser_firstboot.py_9276

2018-05-19T05:17:43.701Z   Invoked command: ['C:\\Program Files\\VMware\\vCenter Server\\vmafdd\\vecs-cli.exe', 'store', 'list']

2018-05-19T05:17:43.701Z   RC = 1021

Stdout = 

Stderr = vecs-cli failed. Error 1021: Cannot connect to vmafd service. 

2018-05-19T05:17:43.701Z   Error deleting solution user machine. Error {

    "resolution": null, 

    "detail": [

        {

            "args": [

                "Command: ['C:\\\\Program Files\\\\VMware\\\\vCenter Server\\\\vmafdd\\\\vecs-cli.exe', 'store', 'list']\nStderr: vecs-cli failed. Error 1021: Cannot connect to vmafd service. \r\n"

            ], 

            "id": "install.ciscommon.command.errinvoke", 

            "localized": "An error occurred while invoking external command : 'Command: ['C:\\\\Program Files\\\\VMware\\\\vCenter Server\\\\vmafdd\\\\vecs-cli.exe', 'store', 'list']\nStderr: vecs-cli failed. Error 1021: Cannot connect to vmafd service. \r\n'", 

            "translatable": "An error occurred while invoking external command : '%(0)s'"

        }, 

        {

            "id": "install.ciscommon.vecs.list.stores", 

            "localized": "Error in generating list of VECS store instances.", 

            "translatable": "Error in generating list of VECS store instances."

        }

    ], 

    "componentKey": null, 

    "problemId": null

=============================================================================================    

vmafd-firstboot.py_9276_stderr.log

2018-05-19T05:18:31.873Z   WARNING Value for install-parameter vmdir.password is empty

2018-05-19T05:18:31.873Z   WARNING Value for install-parameter machine.cert.dir is empty

2018-05-19T05:18:31.873Z   WARNING Value for install-parameter upgrade.import.directory is empty

2018-05-19T05:18:31.873Z   WARNING Value for install-parameter upgrade.source.platform is empty

2018-05-19T05:18:31.873Z   WARNING Value for install-parameter upgrade.import.directory is empty

2018-05-19T05:18:31.873Z   WARNING Value for install-parameter vmdir.password is empty

2018-05-19T05:18:31.889Z   WARNING Value for install-parameter vmdir.password is empty

2018-05-19T05:18:31.889Z   Exception: Traceback (most recent call last):

  File "C:\Program Files\VMware\vCenter Server\firstboot\vmafd-firstboot.py", line 121, in main

    controller.uninstall()

  File "C:\Program Files\VMware\vCenter Server\firstboot\vmafd-firstboot.py", line 71, in uninstall

    service.uninstall()

  File "C:\Program Files\VMware\vCenter Server\firstboot\identityinstall\vmcaInstall.py", line 191, in uninstall

    self.utils.stop_service(self.__VMCA_SERVICE_NAME, self.__VMCA_COMPONENT_NAME)

  File "C:\Program Files\VMware\vCenter Server\firstboot\identityinstall\WindowsUtils.py", line 92, in stop_service

    raise ServiceStopException(serviceName)

ServiceStopException: {

    "resolution": null, 

    "detail": [

        {

            "args": [

                "VMWareCertificateService"

            ], 

            "id": "install.ciscommon.service.failstop", 

            "localized": "An error occurred while stopping service 'VMWareCertificateService'", 

            "translatable": "An error occurred while stopping service '%(0)s'"

        }

    ], 

    "componentKey": null, 

    "problemId": null

}

                

vminst.log           

2018-05-18 23:17:31.236-06:00| vcsInstUtil-7037391| I: ParseStatusFile: STATUS: 87%: "Stopping VMware Single Sign-On User Creation..."

2018-05-18 23:17:31.236-06:00| vcsInstUtil-7037391| I: ParseStatusFile: Calling callback with 83 -> 87 "Stopping VMware Single Sign-On User Creation..."

2018-05-18 23:17:31.236-06:00| vcsInstUtil-7037391| I: Entering function: UpdatePkgMgrProgressFromStatusFile

2018-05-18 23:17:31.236-06:00| vcsInstUtil-7037391| I: ProcessMsiMsg: ACTIONDATA reporting "Stopping VMware Single Sign-On User Creation..."

2018-05-18 23:17:31.252-06:00| vcsInstUtil-7037391| I: Leaving function: UpdatePkgMgrProgressFromStatusFile

2018-05-18 23:17:31.252-06:00| vcsInstUtil-7037391| I: Leaving function: ParseStatusFile

2018-05-18 23:18:31.343-06:00| vcsInstUtil-7037391| I: Entering function: ParseStatusFile

2018-05-18 23:18:31.343-06:00| vcsInstUtil-7037391| I: ParseStatusFile: STATUS: 91%: "Stopping VMware Authentication Framework..."

2018-05-18 23:18:31.343-06:00| vcsInstUtil-7037391| I: ParseStatusFile: Calling callback with 87 -> 91 "Stopping VMware Authentication Framework..."

2018-05-18 23:18:31.343-06:00| vcsInstUtil-7037391| I: Entering function: UpdatePkgMgrProgressFromStatusFile

2018-05-18 23:18:31.343-06:00| vcsInstUtil-7037391| I: ProcessMsiMsg: ACTIONDATA reporting "Stopping VMware Authentication Framework..."

2018-05-18 23:18:31.359-06:00| vcsInstUtil-7037391| I: Leaving function: UpdatePkgMgrProgressFromStatusFile

2018-05-18 23:18:31.359-06:00| vcsInstUtil-7037391| I: Leaving function: ParseStatusFile

2018-05-18 23:18:31.998-06:00| vcsInstUtil-7037391| I: Entering function: ParseStatusFile

2018-05-18 23:18:32.014-06:00| vcsInstUtil-7037391| I: ParseStatusFile: curr error msg: "An error occurred while stopping service 'VMWareCertificateService'"

2018-05-18 23:18:32.014-06:00| vcsInstUtil-7037391| E: ParseStatusFile: Displaying error message for " ": "An error occurred while stopping service 'VMWareCertificateService'"

2018-05-18 23:18:49.408-06:00| vcsInstUtil-7037391| I: Entering function: ShowMessageQueue

2018-05-18 23:18:49.408-06:00| vcsInstUtil-7037391| I: Leaving function: ShowMessageQueue

2018-05-18 23:18:49.408-06:00| vcsInstUtil-7037391| I: Entering function: ShowMessageQueue

2018-05-18 23:18:49.408-06:00| vcsInstUtil-7037391| I: Leaving function: ShowMessageQueue

2018-05-18 23:18:49.408-06:00| vcsInstUtil-7037391| I: ParseStatusFile: got error or success

2018-05-18 23:18:49.423-06:00| vcsInstUtil-7037391| I: ParseStatusFile: keeping completed status file: C:\ProgramData\VMware\vCenterServer\logs\uninstall\fbInstall.json

2018-05-18 23:18:49.423-06:00| vcsInstUtil-7037391| I: Leaving function: ParseStatusFile

2018-05-18 23:18:49.423-06:00| vcsInstUtil-7037391| I: MonitorStatusFile: Parse callback gave final result: 0

2018-05-18 23:18:49.423-06:00| vcsInstUtil-7037391| I: MonitorStatusFile: called parse callback 10 times

2018-05-18 23:18:49.423-06:00| vcsInstUtil-7037391| I: MonitorStatusFile: Process still hasn't terminated, waiting

2018-05-18 23:42:03.177-06:00| vcsInstUtil-7037391| I: MonitorStatusFile: Wait on process has completed: 0

2018-05-18 23:42:03.177-06:00| vcsInstUtil-7037391| I: MonitorStatusFile: Process's job tree still hasn't terminated, waiting

2018-05-18 23:42:03.177-06:00| vcsInstUtil-7037391| I: MonitorStatusFile: Wait on process's job tree has completed: 0

2018-05-18 23:42:03.177-06:00| vcsInstUtil-7037391| I: Leaving function: MonitorStatusFile

2018-05-18 23:42:03.177-06:00| vcsInstUtil-7037391| E: LaunchProcAndMonitorStatus: Failure detected while monitoring status file

2018-05-18 23:42:03.177-06:00| vcsInstUtil-7037391| E: LaunchProcAndMonitorStatus: Job still alive, terminating

2018-05-18 23:42:03.177-06:00| vcsInstUtil-7037391| I: Leaving function: LaunchProcAndMonitorStatus

2018-05-18 23:42:03.177-06:00| vcsInstUtil-7037391| I: Leaving function: LaunchProcAndMonitorCommonStatusFile

2018-05-18 23:42:03.177-06:00| vcsInstUtil-7037391| E: RunFirstLastUpdateboot: Failed to run boot script: "C:\Windows\system32\cmd.exe /S /C ""C:\Program Files\VMware\vCenter Server\bin\run-uninstall-scripts.bat" --operation ROLLBACK""

2018-05-18 23:42:03.177-06:00| vcsInstUtil-7037391| I: Leaving function: VM_RunLastBoot

2018-05-18 23:42:03.520-06:00| vcsInstUtil-7037391| E: wWinMain: MSI result of install of "Z:\vCenter-Server\Packages\vcsservicemanager.msi" may have failed: 1603 (0x00000643)

2018-05-18 23:42:03.520-06:00| vcsInstUtil-7037391| E: LaunchPkgMgr: Operation on vcsservicemanager.msi appears to have failed: 1603 (0x00000643)

2018-05-18 23:42:03.520-06:00| vcsInstUtil-7037391| I: PitCA_MessageBox: Displaying message: "Installation of component VCSServiceManager failed with error code '1603'. Check the logs for more details."

2018-05-18 23:42:20.306-06:00| vcsInstUtil-7037391| I: LaunchPkgMgr: Telling child to revert transaction

2018-05-18 23:42:20.306-06:00| vcsInstUtil-7037391| I: InstallStatus_SetStage: install stage: install-packages / revert-transaction

2018-05-18 23:46:04.104-06:00| vcsInstUtil-7037391| I: EndTransaction: Chain transaction 1 rollback result 3010

2018-05-18 23:46:04.104-06:00| vcsInstUtil-7037391| I: wWinMain: We've been told our work is done, terminating

2018-05-18 23:46:04.104-06:00| vcsInstUtil-7037391| I: wWinMain: Releasing install mutex

2018-05-18 23:46:04.104-06:00| vcsInstUtil-7037391| I: Leaving function: wWinMain

2018-05-18 23:46:04.104-06:00| vcsInstUtil-7037391| I: LaunchPkgMgr: DLL got exe done: 3010

2018-05-18 23:46:04.104-06:00| vcsInstUtil-7037391| I: LaunchPkgMgr: disconnected pipe

2018-05-18 23:46:04.104-06:00| vcsInstUtil-7037391| I: LaunchPkgMgr: Starting services on update rollback.

2018-05-18 23:46:04.104-06:00| vcsInstUtil-7037391| I: Entering function: VM_StartVcsServices

2018-05-18 23:46:04.104-06:00| vcsInstUtil-7037391| I: PitCA_RefreshVMEnvironmentVars: Updating PATH environment variable.

2018-05-18 23:46:04.104-06:00| vcsInstUtil-7037391| I: PitCA_RefreshVMEnvironmentVars: Var: Path, Val: C:\Program Files (x86)\Java\jre7\bin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program Files\MIT\Kerberos\bin;D:\Program Files\VMware\vCenter Server\openSSL;C:\Program Files\VMware\vCenter Server\openSSL

2018-05-18 23:46:04.104-06:00| vcsInstUtil-7037391| I: StartStopVCSServices: Running command '"C:\Windows\system32\cmd.exe" /S /C ""C:\Program Files\VMware\vCenter Server\bin\service-control.bat"" --start --all'...

2018-05-18 23:46:04.104-06:00| vcsInstUtil-7037391| I: StartStopVCSServices: Waiting for VC services to start...

2018-05-18 23:46:09.002-06:00| vcsInstUtil-7037391| E: StartStopVCSServices: Unable to start VC services

2018-05-18 23:46:09.002-06:00| vcsInstUtil-7037391| I: Leaving function: VM_StartVcsServices

2018-05-18 23:46:09.049-06:00| vcsInstUtil-7037391| E: LaunchPkgMgr: Overall operation has failed

Reply
0 Kudos
1 Solution

Accepted Solutions
dekoshal
Hot Shot
Hot Shot
Jump to solution

Issue fixed by VMware support by removing stale registry  entry from vCenter server.

View solution in original post

Reply
0 Kudos
7 Replies
Devi94
Hot Shot
Hot Shot
Jump to solution

Sounds like a certificate issue. Are you running on VCSA or windows vcenter ?

Is that embedded PSC or external ?

Reply
0 Kudos
msripada
Virtuoso
Virtuoso
Jump to solution

By seeing this error message, it looks like the timeout value to stop service exceeded than expected and lead to this problem.. Can you stop and start the service manually before upgrade and disable any antivirus and disable UAC (before upgrade).

2018-05-18 23:18:32.014-06:00| vcsInstUtil-7037391| I: ParseStatusFile: curr error msg: "An error occurred while stopping service 'VMWareCertificateService'"

2018-05-18 23:18:32.014-06:00| vcsInstUtil-7037391| E: ParseStatusFile: Displaying error message for " ": "An error occurred while stopping service 'VMWareCertificateService'"

Thanks,

MS

Reply
0 Kudos
dekoshal
Hot Shot
Hot Shot
Jump to solution

We are able to start and stop service before and after the upgrade errors out. We have a open SR with VMware on this but so far no progress has been made.

Reply
0 Kudos
dekoshal
Hot Shot
Hot Shot
Jump to solution

We are using external PSC for windows vcenter server

Reply
0 Kudos
dekoshal
Hot Shot
Hot Shot
Jump to solution

firstbootStatusjson.log

{

    "totalSteps": 24, 

    "stepsCompleted": 23, 

    "stepsStarted": 23, 

    "finalStatus": "failure", 

    "failedSteps": "vmafd-firstboot", 

    "stepsCompletedList": "vmsyslogcollector_firstboot,vsanhealth_firstboot,ngc_firstboot_wait,eam_firstboot,mbcs_firstboot,netdump-firstboot,perfcharts_firstboot,autodeploy-firstboot,sms_spbm_firstboot,vapi_firstboot,vdc-firstboot,workflow_firstboot,vsm_firstboot,vws_firstboot,vpxd_firstboot,ngc_firstboot,invsvc_firstboot,dbconfig_firstboot,scafirstboot,cmfirstboot,rhttpproxy_firstboot,soluser_firstboot,vmafd-firstboot"

}

fbinstalljson.log

{

    "status": "error",

    "info": [],

    "question": null,

    "progress_message": {

        "args": [

            "VMware Authentication Framework"

        ],

        "id": "install.ciscommon.component.stopping",

        "localized": "Stopping VMware Authentication Framework...",

        "translatable": "Stopping %(0)s..."

    },

    "warning": [],

    "error": {

        "resolution": null,

        "detail": [

            {

                "args": [

                    "VMWareCertificateService"

                ],

                "id": "install.ciscommon.service.failstop",

                "localized": "An error occurred while stopping service 'VMWareCertificateService'",

                "translatable": "An error occurred while stopping service '%(0)s'"

            }

        ],

        "componentKey": "vmafd",

        "problemId": null

    },

    "progress": 91

}

Reply
0 Kudos
dekoshal
Hot Shot
Hot Shot
Jump to solution

Issue fixed by VMware support by removing stale registry  entry from vCenter server.

Reply
0 Kudos
TechThai
Contributor
Contributor
Jump to solution

where do i can get the solution or the steps to perform in removing the stale reg entry

Reply
0 Kudos