Upgrading vCenter Server from 6.5u1 to 6.7

[ericdude101]

Hello,

Im trying to upgrade my VCSA appliance to 6.7 but its continuously failing and i'm at a loss for info. In the final portion of the upgrade while running step 2 "Set up target vCenter Server and start services" it fails at the step "Starting VMware Analytics Service" and displays "A problem has occurred. The source vCenter Server might have been Powered Off during this process. Click on Messages for more information."

Upon reviewing logs I found what appears to be an authentication error in "analytics_firstboot.py":

==============================================================================

INFO:root:Register service with LS.

2018-05-04T17:49:18.154Z  Failed to register Analytics Service with Component Manager: SoapException:

faultcode: ns0:FailedAuthentication

faultstring: Invalid credentials

faultxml: <?xml version='1.0' encoding='UTF-8'?><S:Envelope xmlns:S="http://schemas.xmlsoap.org/soap/envelope/"><S:Body><S:Fault xmlns:ns4="http://www.w3.org/2003/05/soap-envelope"><faultcode xmlns:ns0="http://docs.oasis-open.org/ws-sx/ws-trust/200512">ns0:FailedAuthentication</faultcode><faultstring>Invalid credentials</faultstring></S:Fault></S:Body></S:Envelope>

2018-05-04T17:49:18.157Z  Traceback (most recent call last):

  File "/usr/lib/vmware-analytics/firstboot/analytics_firstboot.py", line 161, in register_with_cm

    cloudvm_sso_cm_register(keystore, cisreg_spec, key_alias, dyn_vars, isPatch=is_patch)

  File "/usr/lib/vmware-cm/bin/cloudvmcisreg.py", line 700, in cloudvm_sso_cm_register

    serviceId = do_lsauthz_operation(cisreg_opts_dict)

  File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 1044, in do_lsauthz_operation

    ls_obj.register_service(svc_id, svc_create_spec)

  File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 340, in add_securityctx_to_requests

    with self._sso_client.securityctx_modifier(self._stub):

  File "/usr/lib/python3.5/contextlib.py", line 59, in __enter__

    return next(self.gen)

  File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 240, in securityctx_modifier

    self._update_saml_token()

  File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 223, in _update_saml_token

    self._uname, self._passwd, token_duration=120)

  File "/usr/lib/vmware/site-packages/pyVim/sso.py", line 317, in get_bearer_saml_assertion

    ssl_context)

  File "/usr/lib/vmware/site-packages/pyVim/sso.py", line 256, in perform_request

    raise SoapException(fault, *parsed_fault)

pyVim.sso.SoapException: SoapException:

faultcode: ns0:FailedAuthentication

faultstring: Invalid credentials

faultxml: <?xml version='1.0' encoding='UTF-8'?><S:Envelope xmlns:S="http://schemas.xmlsoap.org/soap/envelope/"><S:Body><S:Fault xmlns:ns4="http://www.w3.org/2003/05/soap-envelope"><faultcode xmlns:ns0="http://docs.oasis-open.org/ws-sx/ws-trust/200512">ns0:FailedAuthentication</faultcode><faultstring>Invalid credentials</faultstring></S:Fault></S:Body></S:Envelope>

2018-05-04T17:49:18.157Z  Exception: Traceback (most recent call last):

  File "/usr/lib/vmware-analytics/firstboot/analytics_firstboot.py", line 161, in register_with_cm

    cloudvm_sso_cm_register(keystore, cisreg_spec, key_alias, dyn_vars, isPatch=is_patch)

  File "/usr/lib/vmware-cm/bin/cloudvmcisreg.py", line 700, in cloudvm_sso_cm_register

    serviceId = do_lsauthz_operation(cisreg_opts_dict)

  File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 1044, in do_lsauthz_operation

    ls_obj.register_service(svc_id, svc_create_spec)

  File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 340, in add_securityctx_to_requests

    with self._sso_client.securityctx_modifier(self._stub):

  File "/usr/lib/python3.5/contextlib.py", line 59, in __enter__

    return next(self.gen)

  File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 240, in securityctx_modifier

    self._update_saml_token()

  File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 223, in _update_saml_token

    self._uname, self._passwd, token_duration=120)

  File "/usr/lib/vmware/site-packages/pyVim/sso.py", line 317, in get_bearer_saml_assertion

    ssl_context)

  File "/usr/lib/vmware/site-packages/pyVim/sso.py", line 256, in perform_request

    raise SoapException(fault, *parsed_fault)

pyVim.sso.SoapException: SoapException:

faultcode: ns0:FailedAuthentication

faultstring: Invalid credentials

faultxml: <?xml version='1.0' encoding='UTF-8'?><S:Envelope xmlns:S="http://schemas.xmlsoap.org/soap/envelope/"><S:Body><S:Fault xmlns:ns4="http://www.w3.org/2003/05/soap-envelope"><faultcode xmlns:ns0="http://docs.oasis-open.org/ws-sx/ws-trust/200512">ns0:FailedAuthentication</faultcode><faultstring>Invalid credentials</faultstring></S:Fault></S:Body></S:Envelope>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "/usr/lib/vmware-analytics/firstboot/analytics_firstboot.py", line 260, in main

    fb.register_with_cm(analytics_int_http, is_patch)

  File "/usr/lib/vmware-analytics/firstboot/analytics_firstboot.py", line 172, in register_with_cm

    problem_id='install.analytics.cmregistration.failed')

cis.baseCISException.BaseInstallException: {

    "resolution": {

        "id": "install.analytics.cmregistration.failed.res",

        "localized": "Please search for these symptoms in the VMware Knowledge Base for any known issues and possible resolutions. If none can be found, collect a support bundle and open a support request.",

        "translatable": "Please search for these symptoms in the VMware Knowledge Base for any known issues and possible resolutions. If none can be found, collect a support bundle and open a support request."

    },

    "componentKey": "analytics",

    "detail": [

        {

            "id": "install.analytics.cmregistration.failed",

            "localized": "Analytics Service registration with Component Manager failed.",

            "translatable": "Analytics Service registration with Component Manager failed."

        }

    ],

    "problemId": "install.analytics.cmregistration.failed"

}

2018-05-04T17:49:18.158Z  VMware Analytics Service firstboot failed

==============================================================================

Any ideas what I might be up against here?

[daphnissov]

Have you changed your SSO domain from the default of vsphere.local?

[ericdude101]

Yes, but it was done that way with the initial deployment and been that way since. How did you gather that from that little snippet?

[daphnissov]

I didn't, but what I have observed from similar reports that fail in this manner is that they all seem to have in common the fact that the SSO domain has been altered from default.

[ericdude101]

That makes sense. If one of the setup scripts has administrator@vsphere.local hard coded into itself and is ignoring the supplied username from the setup UI, that would definitely cause an authentication error.

Is there a workaround available, or is everyone with this error currently dead in the water?

[daphnissov]

I haven't seen of any workarounds as of yet, so it may be worthwhile to log an SR with VMware and see what they come back and say.

[ericdude101]

Unfortunately enough, we don't have a current support agreement. I was considering upping our license and renewing our support contract, but wanted to test things out first. Hopefully a resolution surfaces soon.

[pez098]

I'm having the same problem, however we haven't changed from the default sso domain

upgrading from 6.0 to 6.7

[hwaltari]

Most of the problems I've seen are related to the DNS and/or using dhcp. I would make absolutely certain that dns points to correct host, including reverse record.

Another thing to check is that the temporary migration time ip-address you need to give to the new appliance is unique and not in use in any way. I would suggest to use static address as temporary address instead of dhcp.

Another point of error is that if you connect to esxi instead of vcenter, then DRS might relocate the new appliance and thus it will be unavailable. Disable DRS during migration.

[ericdude101]

In my case I am using all static IP's and have validated both FQDN name resolution as well as reverse lookup, unfortunately i'm still getting the authentication error.

[hwaltari]

Are You sure that the credentials used in source / destination vcenters are correct?

[ericdude101]

Of course. The destination vCenter credentials are set during the upgrade installation so it's not really possible to be wrong because it's going to proceed with whatever I told it to set to. The source vCenter had to be read in order for me to select destination host, datastore, and network settings which it did without issue.

[pez098]

I was just advised by vmware support to try 6.7 EP1, though they are also checking my logs and have found some errors which they're going to get back to me about today.

Looking at the download options, there is a 6.7.0a which has been released yesterday (22/5/2018) so I'm hoping that's it and am about to give it a go.

[pez098]

that version failed with a different error.

Analytics Service registration with Component Manager failed.

now to wait on them to get back to me

[pez098]

in my case, the problem was the certificates on the windows vcenter server. vmware support, via webex, reset them and now it's gotten past the point where it previously failed and looks like it is now working. I've emailed them asking how the certificates were reset, as my notes are a bit brief and I don't know how to do it.

this was my secondary/DR site. upgrading my primary site next so hopefully it will go more smoothly, or the certificate reset will work

notes from the vmware engineer:

D:\Program Files\VMware\vCenter Server\vmcad>certificate-manager.bat

                 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

                |                                                                     |

                |      *** Welcome to the vSphere 6.0 Certificate Manager  ***        |

                |                                                                     |

                |                   -- Select Operation --                            |

                |                                                                     |

                |      1. Replace Machine SSL certificate with Custom Certificate     |

                |                                                                     |

                |      2. Replace VMCA Root certificate with Custom Signing           |

                |         Certificate and replace all Certificates                    |

                |                                                                     |

                |      3. Replace Machine SSL certificate with VMCA Certificate       |

                |                                                                     |

                |      4. Regenerate a new VMCA Root Certificate and                  |

                |         replace all certificates                                    |

                |                                                                     |

                |      5. Replace Solution user certificates with                     |

                |         Custom Certificate                                          |

                |                                                                     |

                |      6. Replace Solution user certificates with VMCA certificates   |

                |                                                                     |

                |      7. Revert last performed operation by re-publishing old        |

                |         certificates                                                |

                |                                                                     |

                |      8. Reset all Certificates                                      |

                |_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _|

Note : Use Ctrl-Z and hit Enter to exit.

Option[1 to 8]: 8

Do you wish to generate all certificates using configuration file : Option[Y/N] ? : N

Please provide valid SSO and VC priviledged user credential to perform certificate operations.

Enter username [Administrator@vsphere.local]:administrator@vsphere.local

Enter password:

You are going to reset by regenerating Root Certificate and replace all certificates using VMCA

Please configure root.cfg with proper values before proceeding to next step.

Press Enter key to skip optional parameters or use Default value.

Enter proper value for 'Country' [Default value : US] : US

Enter proper value for 'Name' [Default value : CA] : CA

Enter proper value for 'Organization' [Default value : VMware] : VMware

Enter proper value for 'OrgUnit' [Default value : VMware] : VMware

Enter proper value for 'State' [Default value : California] : California

Enter proper value for 'Locality' [Default value : Palo Alto] : Palo Alto

Please configure MACHINE_SSL_CERT.cfg with proper values before proceeding to next step.

Press Enter key to skip optional parameters or use Default value.

Enter proper value for 'Country' [Default value : US] : US

Enter proper value for 'Name' [Default value : CA] : CA

Enter proper value for 'Organization' [Default value : VMware] : VMware

Enter proper value for 'OrgUnit' [Default value : VMware] : VMware-MachineSSL

Enter proper value for 'State' [Default value : California] : California

Enter proper value for 'Locality' [Default value : Palo Alto] : Palo Alto

Enter proper value for 'IPAddress' [optional] : 10.1.3.28

Enter proper value for 'Email' [Default value : email@acme.com] :

Enter proper value for 'Hostname' [Enter valid Fully Qualified Domain Name(FQDN), For Example : example.domain.com] : drvcentre.scc.shoalhaven.nsw.gov.au

Continue operation : Option[Y/N] ? : Y

You are going to reset by regenerating Root Certificate and replace all certificates using VMCA

Continue operation : Option[Y/N] ? : Y

Get site nameCompleted [Reset Machine SSL Cert...]

flinders

Lookup all services

Get service flinders:c03abcce-b544-46c9-ae13-8b5ffd9e71a2

Don't update service flinders:c03abcce-b544-46c9-ae13-8b5ffd9e71a2

Get service flinders:d83bf477-3e01-43c8-a87c-ce98d4847a9a

Don't update service flinders:d83bf477-3e01-43c8-a87c-ce98d4847a9a

Get service flinders:8407421e-aed4-4d39-a664-ea67fc07e9cd

Don't update service flinders:8407421e-aed4-4d39-a664-ea67fc07e9cd

Get service flinders:cd416255-9bb2-4f09-96d6-934453c138a7

Don't update service flinders:cd416255-9bb2-4f09-96d6-934453c138a7

Get service flinders:65d93da4-6a13-46d8-be5b-8daec3592068

Don't update service flinders:65d93da4-6a13-46d8-be5b-8daec3592068

Get service flinders:43856c9a-ac64-4f02-8214-8fa03dcf1ad0

Don't update service flinders:43856c9a-ac64-4f02-8214-8fa03dcf1ad0

Get service flinders:8cad0b61-926f-43ed-b941-c7f69623d116

Don't update service flinders:8cad0b61-926f-43ed-b941-c7f69623d116

Get service 78231D60-591A-4B08-BBF7-F5CDFFDE0C9D_com.vmware.vcIntegrity

Don't update service 78231D60-591A-4B08-BBF7-F5CDFFDE0C9D_com.vmware.vcIntegrity

Get service dfba4a1a-9ef7-461f-b638-7e6c642c2faa

Update service dfba4a1a-9ef7-461f-b638-7e6c642c2faa; spec: c:\users\andrew~1\appdata\local\temp\svcspec_2yv_da

Get service 9f58601a-e873-4b68-901e-3d09bbf615a1

Update service 9f58601a-e873-4b68-901e-3d09bbf615a1; spec: c:\users\andrew~1\appdata\local\temp\svcspec_rcujye

Get service e0eebcc7-537f-4997-96e6-85518e3e4bbf

Update service e0eebcc7-537f-4997-96e6-85518e3e4bbf; spec: c:\users\andrew~1\appdata\local\temp\svcspec_ktsbhn

Get service 63497e67-c830-4c2d-ab7d-183da6149f53

Update service 63497e67-c830-4c2d-ab7d-183da6149f53; spec: c:\users\andrew~1\appdata\local\temp\svcspec_oz9q2t

Get service b47d9a38-2dbf-412b-b8f4-350a57216dd9

Update service b47d9a38-2dbf-412b-b8f4-350a57216dd9; spec: c:\users\andrew~1\appdata\local\temp\svcspec_d7y8r2

Get service aa284864-8d41-4a13-8523-56656839d7b8

Update service aa284864-8d41-4a13-8523-56656839d7b8; spec: c:\users\andrew~1\appdata\local\temp\svcspec_7vmx8l

Get service 420adcb1-f828-466d-9d84-95666a8782b2

Update service 420adcb1-f828-466d-9d84-95666a8782b2; spec: c:\users\andrew~1\appdata\local\temp\svcspec_76s65z

Get service a46a214a-e1ed-4dc5-91c8-24f9cea5edec

Update service a46a214a-e1ed-4dc5-91c8-24f9cea5edec; spec: c:\users\andrew~1\appdata\local\temp\svcspec_t_3kji

Get service 183b45cb-0de1-451d-8a84-13f7eaa119aa

Update service 183b45cb-0de1-451d-8a84-13f7eaa119aa; spec: c:\users\andrew~1\appdata\local\temp\svcspec_bi9yh2

Get service 3c31990d-e7c2-44fe-aa40-fe0ef07ee45e

Update service 3c31990d-e7c2-44fe-aa40-fe0ef07ee45e; spec: c:\users\andrew~1\appdata\local\temp\svcspec_1vjfug

Get service 4a193fc7-e18c-4567-8243-e6d7b90b9504

Update service 4a193fc7-e18c-4567-8243-e6d7b90b9504; spec: c:\users\andrew~1\appdata\local\temp\svcspec_2z40ae

Get service 508485c8-1d8a-4106-900d-601d81ca125f

Update service 508485c8-1d8a-4106-900d-601d81ca125f; spec: c:\users\andrew~1\appdata\local\temp\svcspec_1r7zfp

Get service aa284864-8d41-4a13-8523-56656839d7b8_authz

Update service aa284864-8d41-4a13-8523-56656839d7b8_authz; spec: c:\users\andrew~1\appdata\local\temp\svcspec_bodhoi

Get service df3cede6-bdc9-4def-81d5-f8fb439a6e35

Update service df3cede6-bdc9-4def-81d5-f8fb439a6e35; spec: c:\users\andrew~1\appdata\local\temp\svcspec_8dqcbp

Get service 6d7e6057-7ba0-46a8-9c9e-bdda510077b5

Update service 6d7e6057-7ba0-46a8-9c9e-bdda510077b5; spec: c:\users\andrew~1\appdata\local\temp\svcspec_ive8dc

Get service 78231D60-591A-4B08-BBF7-F5CDFFDE0C9D

Update service 78231D60-591A-4B08-BBF7-F5CDFFDE0C9D; spec: c:\users\andrew~1\appdata\local\temp\svcspec_sqlonz

Get service aa284864-8d41-4a13-8523-56656839d7b8_kv

Update service aa284864-8d41-4a13-8523-56656839d7b8_kv; spec: c:\users\andrew~1\appdata\local\temp\svcspec_frm_yo

Get service 25fe70a4-a44a-48e1-b15b-c325e5c76c1f

Update service 25fe70a4-a44a-48e1-b15b-c325e5c76c1f; spec: c:\users\andrew~1\appdata\local\temp\svcspec_zqmfmp

Get service fc4d2de6-7c14-4c54-9d62-3442f9199fb6

Update service fc4d2de6-7c14-4c54-9d62-3442f9199fb6; spec: c:\users\andrew~1\appdata\local\temp\svcspec_ndryez

Get service 5d85d28d-5f1a-47de-b624-dab3667ca75f

Update service 5d85d28d-5f1a-47de-b624-dab3667ca75f; spec: c:\users\andrew~1\appdata\local\temp\svcspec_l_p0xh

Get service 78231D60-591A-4B08-BBF7-F5CDFFDE0C9D_com.vmware.vsan.health

Don't update service 78231D60-591A-4B08-BBF7-F5CDFFDE0C9D_com.vmware.vsan.health

Get service 618efccd-6ac9-40a5-9920-1954e158fc37

Update service 618efccd-6ac9-40a5-9920-1954e158fc37; spec: c:\users\andrew~1\appdata\local\temp\svcspec_jqzqz7

Updated 21 service(s)

Status : 55% Completed [Reset vpxd Cert...]

Reset status : 70% Completed [stopping services...]

Reset status : 85% Completed [starting services...]

Reset status : 100% Completed [Reset completed successfully]