Highlighted
Contributor
Contributor

VCSA 6.7 - vpxd doesn't start after replacing machine SSL certs

Creating a new VCSA 6.5.0 vm using win32 GUI.

After installation completed, I want to replace machine SSL certificates using HTML5 webgui.

I imported Terena CA and then replaced machine SSL cert (key & crt). After rebooting, all works fine.

Deleting this VM, and creating a new VCSA 6.7 VM using win32 GUI and exactly the same paramaters as before (fqdn, ip, ...). DNS entries are ok (FQDN to IP & IP to FQDN).

After installation completed, I imported the same certificate as before. After rebooting, when I try to access the web GUI, I've got the following error :

503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x00007f3890084700] _serverNamespace = / action = Allow _pipeName =/var/run/vmware/vpxd-webserver-pipe)

Trying to replace de certificate from CLI using certificate-manager :

Updated 34 service(s)

Status : 70% Completed [stopping services...]

Status : 85% Completed [starting services...]

Error while starting services, please see service-control log for more details

Status : 0% Completed [Operation failed, performing automatic rollback]              

Error while replacing Machine SSL Cert, please see /var/log/vmware/vmcad/certificate-manager.log for more information.

Performing rollback of Machine SSL Cert...

Get site nameus : 0% Completed [Rollback Machine SSL Cert...]    

This is the /var/log/vmware/vmcad/certificate-manager.log log :

2019-12-06T13:19:16.509Z INFO certificate-manager None

2019-12-06T13:19:26.519Z INFO certificate-manager Running command :- service-control --start  --all

2019-12-06T13:19:26.519Z INFO certificate-manager please see service-control.log for service status

Service-control failed. Error: Failed to start services in profile ALL. RC=2, stderr=Failed to start vpxd services. Error: Service crashed while starting

2019-12-06T13:25:38.27Z ERROR certificate-manager None

This is the vpxd.log :

--> [context]zKq7AVECAAAAAGC34QANdnB4ZAAA4AArbGlidm1hY29yZS5zbwAAWCUbAP6dGACeQCIAaXEiABtFIgDTSSIAOaIjAHFvIwA6ciMAnVYrAdRzAGxpYnB0aHJlYWQuc28uMAAC3Y4ObGliYy5zby42AA==[/context]

2019-12-06T13:23:09.269Z error vpxd[59800] [Originator@6876 sub=AuthzStorageProvider] [AuthzStorageProvider::CreateAuthzMgr] Failed to connect to IS: <N5Vmomi5Fault17HostCommunication9ExceptionE(Fault cause: vmodl.fault.HostCommunication

--> )

--> [context]zKq7AVECAAAAAGC34QASdnB4ZAAA4AArbGlidm1hY29yZS5zbwAAWCUbAP6dGAHu8VN2cHhkAAHu1VoBzsNjATdPoAGuOKACwO0BbGliYXV0aHpjbGllbnQuc28AAmkGAgLijQICxIUCAb3XngE6CVQBimhUARnGUgOQBQJsaWJjLnNvLjYAAaW+Ug==[/context]>

2019-12-06T13:23:09.270Z info vpxd[59800] [Originator@6876 sub=AuthzStorageProvider] [AuthzStorageProvider::CreateAuthzMgr] Retry for this error: attempt count 29

2019-12-06T13:23:12.314Z warning vpxd[59800] [Originator@6876 sub=VpxdAuthClient] [ConnectAndLogin] Failed to loginBySamlToken: N7Vmacore3Ssl18SSLVerifyExceptionE(SSL Exception: Verification parameters:

--> PeerThumbprint: 6B:B6:1F:29:7C:01:E8:65:09:A1:49:C2:46:71:BC:54:11:FB:7F:A8

--> ExpectedThumbprint:

--> ExpectedPeerName: localhost

--> The remote host certificate has these problems:

-->

--> * Host name does not match the subject name(s) in certificate.)

I don't know why ExpectedPeerName is searching for localhost, I always used fqdn and real ip during process and DNS is correctly resolving IP address & FQDN.

Either using webgui or cli for replacing the machine certificate, vpxd doesn't launch after.

Are there new prerequisites for installing a custom SSL certificate since 6.7.0 ?

0 Kudos
12 Replies
Highlighted
Hot Shot
Hot Shot

Here you have an answers:

Host name does not match the subject name(s) in certificate.

Your certificate must contain your vCenter FQDN and if you are using more names (friendly name etc. ) you should also use Subject Alternative Names in cerificates.

A little help:

VMware Knowledge Base

0 Kudos
Highlighted
Contributor
Contributor

The certificate contains vCenter FQDN. There is no additionnal names used for this vcenter. I never specified localhost for any parameters during the installation process.

This certificate works with VCSA 6.5 for the same parameters so I don't understang why it doesn't works  with VCSA 6.7.

0 Kudos
Highlighted
Contributor
Contributor

Did you ever find a solution to this?  We have the exact same issue.  Not why it is looking for localhost in the name

0 Kudos
Highlighted
Contributor
Contributor

Struggling with the same issue. Did you find a solution? Thank you very much in advance.

0 Kudos
Highlighted
Enthusiast
Enthusiast

We have exactly the same issue with our VCSA in our datacenter after installing custom SSL certificates.

Had to do a complete rollback to get vpxd up again.

Has anyone a solution for this issue?

0 Kudos
Highlighted
VMware Employee
VMware Employee

What did VMware support say when you opened a request with them?


Forum Usage Guidelines: https://communities.vmware.com/docs/DOC-12286
VMware Training & Certification blog: http://vmwaretraining.blogspot.com
0 Kudos
Highlighted
VMware Employee
VMware Employee

What did VMware support say when you opened a request with them?


Forum Usage Guidelines: https://communities.vmware.com/docs/DOC-12286
VMware Training & Certification blog: http://vmwaretraining.blogspot.com
0 Kudos
Highlighted
Enthusiast
Enthusiast

Did not open a support case yet, because it was detected last Friday evening.

We are using this version:

vCenter Appliance 6.7 Update 3 (6.7.0.40000)2019-08-201436773714368073

I will perform an upgrade with the vCenter clone in our lab environment to check if this helps before creating a ticket.

0 Kudos
Highlighted
Commander
Commander

This is because is maybe not matching with the PNID configured in the vCenter Server during the installation time, could you please run the next command and show us the output: /usr/lib/vmware-vmafd/bin/vmafd-cli get-pnid --server-name localhost

0 Kudos
Highlighted
Contributor
Contributor

Just tried your command. Result is the FQDN of the vcsa (used during the installation and still used)

I opened a ticket with those insights (nothing worked by the way😞

  1. Check the connectivity of the port 902 UDP and TCP between VC and the hosts.
    Did that - ports are opened
    1. VCSA to ESXI -> curl -v telnet <ESXi host IP/FQDN>:port
    2. ESXI to VCSA -> nc -uz <VC FQDN/IP> port
  2. Make sure the name resolution of VC and host works
  3. Make sure this is changed in web client: vCenter server object -> Configure -> General -> Run timesettings -> vCenter Server managed address -> New ip address information.
  4. Make sure this is changed in web client: vCenter server object -> Configure -> Advanced setting -> config.registry.key_managedIP -> New ip address information.
  5. https://kb.vmware.com/s/article/2121116?lang=en_US

Should have a remote session today. Unfortunately, vmware didn't contacted me as agreed. Any ideas folks? Smiley Happy

0 Kudos
Highlighted
Commander
Commander

Could you try to run the next command to search for duplicated certificates under the same serial numbers:

/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text| grep -A 6 -i alias | less

Also i was reading a little bit and it seems that the resolution for some of the folks was updating vCenter Server to 6.7 U3g which is build 16046470

0 Kudos
Highlighted
Contributor
Contributor

Tried it and no duplicated certificates.

Nonetheless, I had a remote session with vmware. According to the support a subnet change of a vcsa is not recommended and probably won't work. Checked the info:

  • change of ip address in a different subnet didn't work for me
  • change of ip address within the same subnet worked without any problems

If you have two-sites, as in our case, the recommend to set up a second vcsa and run both vcsa instances in a linked mode. However, anser and problem isn't directly the main question of the thread but has the same error log

0 Kudos