"Installation of component VCSServiceManager failed with error code '1603'" during a vCenter upgrade 5.5 to 6.0u3

[lborgognon]

Hi experts,

I'm trying an in place upgrade of my vCenter server from 5.5u3 to 6.0u3 and I get this following final error in vminst.log :

2017-08-02 17:27:43.614+02:00| vcsInstUtil-5318198| E: wWinMain: MSI result of install of "E:\vCenter-Server\Packages\vcsservicemanager.msi" may have failed: 1603 (0x00000643) 
2017-08-02 17:27:43.615+02:00| vcsInstUtil-5318198| E: LaunchPkgMgr: Operation on vcsservicemanager.msi appears to have failed: 1603 (0x00000643) 
2017-08-02 17:27:43.617+02:00| vcsInstUtil-5318198| I: PitCA_MessageBox: Displaying message: "Installation of component VCSServiceManager failed with error code '1603'. Check the logs for more details."

if I check more in details vminst.log, I also have this error which is triggered before above error :

2017-08-02 17:20:03.157+02:00| vcsInstUtil-5318198| I: ParseStatusFile: curr error msg: "Encountered an internal error. 
Traceback (most recent call last): 
  File "C:\Program Files\VMware\vCenter Server\firstboot\vmafd-firstboot.py", line 110, in main 
    controller.firstboot() 
  File "C:\Program Files\VMware\vCenter Server\firstboot\vmafd-firstboot.py", line 51, in firstboot 
    self.post_init() 
  File "C:\Program Files\VMware\vCenter Server\firstboot\vmafd-firstboot.py", line 59, in post_init 
    service.post_init() 
  File "C:\Program Files\VMware\vCenter Server\firstboot\identityinstall\vmafdInstall.py", line 561, in post_init 
    self.get_vmca_root_cert(self._vmdir_dc_name) 
  File "C:\Program Files\VMware\vCenter Server\firstboot\identityinstall\vmafdInstall.py", line 370, in get_vmca_root_cert 
    problemId = "install.vmafd.get_trusted_root_failed") 
SystemError: 'finally' pops bad exception 
" 
2017-08-02 17:20:03.157+02:00| vcsInstUtil-5318198| E: ParseStatusFile: Displaying error message for " ": "Encountered an internal error. 
Traceback (most recent call last): 
  File "C:\Program Files\VMware\vCenter Server\firstboot\vmafd-firstboot.py", line 110, in main 
    controller.firstboot() 
  File "C:\Program Files\VMware\vCenter Server\firstboot\vmafd-firstboot.py", line 51, in firstboot 
    self.post_init() 
  File "C:\Program Files\VMware\vCenter Server\firstboot\vmafd-firstboot.py", line 59, in post_init 
    service.post_init() 
  File "C:\Program Files\VMware\vCenter Server\firstboot\identityinstall\vmafdInstall.py", line 561, in post_init 
    self.get_vmca_root_cert(self._vmdir_dc_name) 
  File "C:\Program Files\VMware\vCenter Server\firstboot\identityinstall\vmafdInstall.py", line 370, in get_vmca_root_cert 
    problemId = "install.vmafd.get_trusted_root_failed") 
SystemError: 'finally' pops bad exception

I searched trough multiples posts about "Installation of component VCSServiceManager failed with error code '1603'" . and unfortunately most of them are details an 1603 error that not exactly match my issue. For the moment, the only information that is most matching my issue is a reddit post https://www.reddit.com/r/vmware/comments/5ij7h2/upgrading_windows_based_vcenter_55_to_60_fails/

On this post, error seems occur when IPv6 is disable. So I naturally checked my IPv6 stack and I activated it except tunnels (6to4, isatap, teredo) but I this issue is still occur.

I opened a case and VMware support also pointing something around IPv6 but for the moment they didn't find-out the issue.

Maybe, I should enable IPv6 tunnels but I don't see why an vCenter upgrade need them ? or perhaps someone have another idea about how to find-out  the issue ?

Thanks

[lborgognon]

I checked a little bit more deeper into installation logs, and I noticed something in vmafd-firstboot.py_5284_stdout.log. I have an error when Importing VMCA root cert to VECS :

2017-08-02T15:20:02.970Z   vmca init upgrade 
2017-08-02T15:20:02.970Z   In Post Init Phase 
2017-08-02T15:20:02.970Z   Importing VMCA root cert to VECS. 
Error Code : 382312694 
Message :UNKNOWN 
2017-08-02T15:20:03.116Z   VMAFD Firstboot failed 

But I also saw in the same log that VMCA service is stopped, updated and then restarted 20sec before this error:

2017-08-02T15:19:42.149Z   Stopping VMCA 
2017-08-02T15:19:42.149Z   Stopping service [VMWareCertificateService] 
2017-08-02T15:19:42.150Z   VMWareCertificateService is already Stopped 
2017-08-02T15:19:42.150Z   Performing vmca upgrade 
2017-08-02T15:19:42.150Z   Getting value for install-parameter: upgrade.import.directory 
2017-08-02T15:19:42.168Z   vmca upgrade success 
2017-08-02T15:19:42.168Z   Starting VMCA
2017-08-02T15:19:42.168Z   Starting service [VMWareCertificateService] 

And when I checked the VMCA log file (vmca.log.0), there is an error when starting VMCA service. It seems like there is an error when accessing to certs.db. 
 

2017-08-02T15:19:42.524Z    Informational    Restricting access to directory C:\ProgramData/VMware/vCenterServer/data/vmca/ 
2017-08-02T15:19:42.526Z    Informational    Initializing database: [C:\ProgramData\VMware\vCenterServer\data\vmca\\certs.db] 
2017-08-02T15:19:42.563Z    Warning    error code: 0x000010d8
2017-08-02T15:19:42.564Z    Informational    Directory sync thread starting 
2017-08-02T15:19:42.564Z    Warning    error code: 0x00000002 
2017-08-02T15:19:42.564Z    Warning    error code: 0x00000002 
2017-08-02T15:19:42.564Z    Warning    error code: 0x00000002 
2017-08-02T15:19:42.564Z    Error    Failed to update root certs due to error [2] 
2017-08-02T15:19:42.610Z    Informational    rpc_server_use_protseq_ep() succeeded, protoSeq (ncalrpc), endPoint(vmcasvc). 
2017-08-02T15:19:42.629Z    Informational    rpc_server_use_protseq_ep() succeeded, protoSeq (ncacn_ip_tcp), endPoint(2014). 
2017-08-02T15:20:20.315Z    Informational    Directory sync thread exiting 

It seems that the error when importing VMCA root cert is caused by the error when VMCA service is started.

With those new informations, I think that the IPv6 configuration is not the cause of the problem and I continue to search something else...

[klabiak]

Did you find the solution? I have very similar problem.