Highlighted
Enthusiast
Enthusiast

Decommission External 6.7 Update 1 PSC Error

Hello All

I'm moving from vCenter 6.5 Update 1g with external PSC  >>  6.7 Update 1 and converging the PSC into vCenter.

Steps I've followed:

1.  Upgrade PSC 6.5u1g  >>  PSC 6.7u1   [success]

2.  Upgrade vCenter (Appliance) 6.5u1g  >>  vCenter (Appliance) 6.7u1   [success]

3.  Converge PSC to vCenter: vcsa-util.exe converge --log-dir d:\logs --verbose --no-ssl-certificate-verification --backup-taken converge.json   [success]

4.  Decommission External PSC:  vcsa-util.exe decommission --log-dir d:\logs --verbose --no-ssl-certificate-verification decommission_psc.json   [FAIL]

I've tested both converge.json and decommission_psc.json in our DEV environment and the decom went through with no errors.  The pre-prod decommission got a long way through the process including shutting down the PSC.  If you log into the vCenter Administration / System Configuration, you can see only the vCenter with embedded PSC.

As this is fairly new, I can't find much PSC decom failures on Google / VM community.

IS THIS ANYTHING TO WORRY ABOUT - DO I NEED TO RE-RUN  THE DECOM???

This is the output from the logs showing the failure:

2019-01-10 14:14:53,021 - vCSACliConvergeLogger - DEBUG - Powering off PSC machine...

2019-01-10 14:14:55,038 - vCSACliConvergeLogger - INFO - PSC machine powered off successfully.

2019-01-10 14:14:55,039 - vCSACliConvergeLogger - DEBUG - Proceed with certificate thumbprint check...

2019-01-10 14:14:55,937 - vCSACliConvergeLogger - INFO - Decommissioning PSC node. This may take some time. Please wait..

2019-01-10 14:14:55,938 - vCSACliConvergeLogger - DEBUG - Running command on vm vcenter: /bin/bash --login -c '/usr/bin/cmsso-util unregister --node-pnid psc.domain --username administrator@vsphere.local --passwd *** --debug'

2019-01-10 14:22:47,614 - vCSACliConvergeLogger - ERROR - DecommissionTask: Caught exception Traceback (most recent call last):

  File "d:\build\ob\bora-9914372\bora\install\vcsa-installer\vcsaConvergeCli\cli_tasks\decommission\decommission_task.py", line 123, in execute

  File "d:\build\ob\bora-9914372\bora\install\vcsa-installer\vcsaConvergeCli\utils\esxvim.py", line 407, in run_command_wait

  File "d:\build\ob\bora-9914372\bora\install\vcsa-installer\vcsaConvergeCli\utils\esxvim.py", line 432, in CheckProcesses

  File "d:\build\ob\bora-9914372\bora\install\vcsa-installer\vcsaConvergeCli\deps\pyVmomi\pyVmomi\VmomiSupport.py", line 557, in <lambda>

  File "d:\build\ob\bora-9914372\bora\install\vcsa-installer\vcsaConvergeCli\deps\pyVmomi\pyVmomi\VmomiSupport.py", line 363, in _InvokeMethod

  File "d:\build\ob\bora-9914372\bora\install\vcsa-installer\vcsaConvergeCli\deps\pyVmomi\pyVmomi\SoapAdapter.py", line 1346, in InvokeMethod

pyVmomi.VmomiSupport.vmodl.fault.SystemError: (vmodl.fault.SystemError) {

   dynamicType = <unset>,

   dynamicProperty = (vmodl.DynamicProperty) [],

   msg = 'A general system error occurred: vix error codes = (3016, 0).\n',

   faultCause = <unset>,

   faultMessage = (vmodl.LocalizableMessage) [],

   reason = 'vix error codes = (3016, 0).\n'

}

2019-01-10 14:22:47,622 - vCSACliConvergeLogger - INFO - ================ [FAILED] Task: DecommissionTask: Running DecommissionTask execution failed at 14:22:47 ================

2019-01-10 14:22:47,623 - vCSACliConvergeLogger - DEBUG - Task 'DecommissionTask: Running DecommissionTask' execution failed because [(vmodl.fault.SystemError) {

   dynamicType = <unset>,

   dynamicProperty = (vmodl.DynamicProperty) [],

   msg = 'A general system error occurred: vix error codes = (3016, 0).\n',

   faultCause = <unset>,

   faultMessage = (vmodl.LocalizableMessage) [],

   reason = 'vix error codes = (3016, 0).\n'

}], possible resolution is []

2019-01-10 14:22:47,626 - vCSACliConvergeLogger - INFO - ================================================================================

2019-01-10 14:22:57,006 - vCSACliConvergeLogger - ERROR - Error occurred. See logs for details.

2019-01-10 14:22:57,007 - vCSACliConvergeLogger - DEBUG - Exception message: com.vmware.vcsa.installer.decommission.task: (vmodl.fault.SystemError) {

   dynamicType = <unset>,

   dynamicProperty = (vmodl.DynamicProperty) [],

   msg = 'A general system error occurred: vix error codes = (3016, 0).\n',

   faultCause = <unset>,

   faultMessage = (vmodl.LocalizableMessage) [],

   reason = 'vix error codes = (3016, 0).\n'

}

Trace:   File "d:\build\ob\bora-9914372\bora\install\vcsa-installer\vcsaConvergeCli\cli_tasks\decommission\decommission_task.py", line 123, in execute

  File "d:\build\ob\bora-9914372\bora\install\vcsa-installer\vcsaConvergeCli\utils\esxvim.py", line 407, in run_command_wait

  File "d:\build\ob\bora-9914372\bora\install\vcsa-installer\vcsaConvergeCli\utils\esxvim.py", line 432, in CheckProcesses

  File "d:\build\ob\bora-9914372\bora\install\vcsa-installer\vcsaConvergeCli\deps\pyVmomi\pyVmomi\VmomiSupport.py", line 557, in <lambda>

  File "d:\build\ob\bora-9914372\bora\install\vcsa-installer\vcsaConvergeCli\deps\pyVmomi\pyVmomi\VmomiSupport.py", line 363, in _InvokeMethod

  File "d:\build\ob\bora-9914372\bora\install\vcsa-installer\vcsaConvergeCli\deps\pyVmomi\pyVmomi\SoapAdapter.py", line 1346, in InvokeMethod

2019-01-10 14:22:57,015 - vCSACliConvergeLogger - INFO - =================================== 14:22:57 ===================================

2019-01-10 14:22:57,016 - vCSACliConvergeLogger - INFO - Result and log file information...

2019-01-10 14:22:57,017 - vCSACliConvergeLogger - INFO - WorkFlow log directory: d:\logs\workflow_1547129680951

6 Replies
Highlighted
Contributor
Contributor

I have an SR with VMware about this issue.

The Tech told me that it is a known issue with 6.7u1 , and that the fix will be released by end of Q1 2019 in Vcenter 6.7U2

0 Kudos
Highlighted
Contributor
Contributor

Did you get anywhere with this?

I've just had the same problem.

It's all very well being told that there is a fix coming Q1 2019 in the next build, but what will this affect now?

Thanks

0 Kudos
Highlighted
Contributor
Contributor

No side effect.

The converge process was successful, therefore I have now 1 VCSA with PSC embedded, and the 'old' PSC. We did shutdown the 'old' PSC for now, and wait to decommission it after the patch is released

Highlighted
Contributor
Contributor

Thanks.  Have converged all my PCSs now and have powered off and removed them.  All seems to be working fine.

Cheers.

0 Kudos
Highlighted
Contributor
Contributor

if anyone is still having an issue decommission a psc after the converge process, the following works from the command line to unregister the old PSC's

cmsso-util unregister --node-pnid fqndofoldpcservertoremove --username username@vsphere.local --passwd 'passwordgoeshere'

I ran into an issue of being unable to retrieve thumbprint on both vCenters when trying to decommission using the GUI.

Process will unregister, remove replication partner and restart vSphere services.

You can verify replication partners with the following command, after execution

/usr/lib/vmware-vmdir/bin/vdcrepadmin -f showpartners -h vcenterandpscfqdn -u username@vsphere.local

Highlighted
Champion
Champion

In my experience a lot of the problems decommissioning PSCs were solved by U2 and U3 so if you're having problems the first thing I would do is to upgrade.

Lars

0 Kudos