VMware Cloud Community
Dorin54
Contributor
Contributor

VCSA 6.7 - vpxd doesn't start after replacing machine SSL certs

Creating a new VCSA 6.5.0 vm using win32 GUI.

After installation completed, I want to replace machine SSL certificates using HTML5 webgui.

I imported Terena CA and then replaced machine SSL cert (key & crt). After rebooting, all works fine.

Deleting this VM, and creating a new VCSA 6.7 VM using win32 GUI and exactly the same paramaters as before (fqdn, ip, ...). DNS entries are ok (FQDN to IP & IP to FQDN).

After installation completed, I imported the same certificate as before. After rebooting, when I try to access the web GUI, I've got the following error :

503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x00007f3890084700] _serverNamespace = / action = Allow _pipeName =/var/run/vmware/vpxd-webserver-pipe)

Trying to replace de certificate from CLI using certificate-manager :

Updated 34 service(s)

Status : 70% Completed [stopping services...]

Status : 85% Completed [starting services...]

Error while starting services, please see service-control log for more details

Status : 0% Completed [Operation failed, performing automatic rollback]              

Error while replacing Machine SSL Cert, please see /var/log/vmware/vmcad/certificate-manager.log for more information.

Performing rollback of Machine SSL Cert...

Get site nameus : 0% Completed [Rollback Machine SSL Cert...]    

This is the /var/log/vmware/vmcad/certificate-manager.log log :

2019-12-06T13:19:16.509Z INFO certificate-manager None

2019-12-06T13:19:26.519Z INFO certificate-manager Running command :- service-control --start  --all

2019-12-06T13:19:26.519Z INFO certificate-manager please see service-control.log for service status

Service-control failed. Error: Failed to start services in profile ALL. RC=2, stderr=Failed to start vpxd services. Error: Service crashed while starting

2019-12-06T13:25:38.27Z ERROR certificate-manager None

This is the vpxd.log :

--> [context]zKq7AVECAAAAAGC34QANdnB4ZAAA4AArbGlidm1hY29yZS5zbwAAWCUbAP6dGACeQCIAaXEiABtFIgDTSSIAOaIjAHFvIwA6ciMAnVYrAdRzAGxpYnB0aHJlYWQuc28uMAAC3Y4ObGliYy5zby42AA==[/context]

2019-12-06T13:23:09.269Z error vpxd[59800] [Originator@6876 sub=AuthzStorageProvider] [AuthzStorageProvider::CreateAuthzMgr] Failed to connect to IS: <N5Vmomi5Fault17HostCommunication9ExceptionE(Fault cause: vmodl.fault.HostCommunication

--> )

--> [context]zKq7AVECAAAAAGC34QASdnB4ZAAA4AArbGlidm1hY29yZS5zbwAAWCUbAP6dGAHu8VN2cHhkAAHu1VoBzsNjATdPoAGuOKACwO0BbGliYXV0aHpjbGllbnQuc28AAmkGAgLijQICxIUCAb3XngE6CVQBimhUARnGUgOQBQJsaWJjLnNvLjYAAaW+Ug==[/context]>

2019-12-06T13:23:09.270Z info vpxd[59800] [Originator@6876 sub=AuthzStorageProvider] [AuthzStorageProvider::CreateAuthzMgr] Retry for this error: attempt count 29

2019-12-06T13:23:12.314Z warning vpxd[59800] [Originator@6876 sub=VpxdAuthClient] [ConnectAndLogin] Failed to loginBySamlToken: N7Vmacore3Ssl18SSLVerifyExceptionE(SSL Exception: Verification parameters:

--> PeerThumbprint: 6B:B6:1F:29:7C:01:E8:65:09:A1:49:C2:46:71:BC:54:11:FB:7F:A8

--> ExpectedThumbprint:

--> ExpectedPeerName: localhost

--> The remote host certificate has these problems:

-->

--> * Host name does not match the subject name(s) in certificate.)

I don't know why ExpectedPeerName is searching for localhost, I always used fqdn and real ip during process and DNS is correctly resolving IP address & FQDN.

Either using webgui or cli for replacing the machine certificate, vpxd doesn't launch after.

Are there new prerequisites for installing a custom SSL certificate since 6.7.0 ?

Reply
0 Kudos
14 Replies
KocPawel
Hot Shot
Hot Shot

Here you have an answers:

Host name does not match the subject name(s) in certificate.

Your certificate must contain your vCenter FQDN and if you are using more names (friendly name etc. ) you should also use Subject Alternative Names in cerificates.

A little help:

VMware Knowledge Base

Reply
0 Kudos
Dorin54
Contributor
Contributor

The certificate contains vCenter FQDN. There is no additionnal names used for this vcenter. I never specified localhost for any parameters during the installation process.

This certificate works with VCSA 6.5 for the same parameters so I don't understang why it doesn't works  with VCSA 6.7.

Reply
0 Kudos
slairipp
Contributor
Contributor

Did you ever find a solution to this?  We have the exact same issue.  Not why it is looking for localhost in the name

Reply
0 Kudos
mrgecco
Contributor
Contributor

Struggling with the same issue. Did you find a solution? Thank you very much in advance.

Reply
0 Kudos
mkaetm
Enthusiast
Enthusiast

We have exactly the same issue with our VCSA in our datacenter after installing custom SSL certificates.

Had to do a complete rollback to get vpxd up again.

Has anyone a solution for this issue?

Reply
0 Kudos
scott28tt
VMware Employee
VMware Employee

What did VMware support say when you opened a request with them?


-------------------------------------------------------------------------------------------------------------------------------------------------------------

Although I am a VMware employee I contribute to VMware Communities voluntarily (ie. not in any official capacity)
VMware Training & Certification blog
Reply
0 Kudos
scott28tt
VMware Employee
VMware Employee

What did VMware support say when you opened a request with them?


-------------------------------------------------------------------------------------------------------------------------------------------------------------

Although I am a VMware employee I contribute to VMware Communities voluntarily (ie. not in any official capacity)
VMware Training & Certification blog
Reply
0 Kudos
mkaetm
Enthusiast
Enthusiast

Did not open a support case yet, because it was detected last Friday evening.

We are using this version:

vCenter Appliance 6.7 Update 3 (6.7.0.40000)2019-08-201436773714368073

I will perform an upgrade with the vCenter clone in our lab environment to check if this helps before creating a ticket.

Reply
0 Kudos
Lalegre
Virtuoso
Virtuoso

This is because is maybe not matching with the PNID configured in the vCenter Server during the installation time, could you please run the next command and show us the output: /usr/lib/vmware-vmafd/bin/vmafd-cli get-pnid --server-name localhost

Reply
0 Kudos
mrgecco
Contributor
Contributor

Just tried your command. Result is the FQDN of the vcsa (used during the installation and still used)

I opened a ticket with those insights (nothing worked by the way😞

  1. Check the connectivity of the port 902 UDP and TCP between VC and the hosts.
    Did that - ports are opened
    1. VCSA to ESXI -> curl -v telnet <ESXi host IP/FQDN>:port
    2. ESXI to VCSA -> nc -uz <VC FQDN/IP> port
  2. Make sure the name resolution of VC and host works
  3. Make sure this is changed in web client: vCenter server object -> Configure -> General -> Run timesettings -> vCenter Server managed address -> New ip address information.
  4. Make sure this is changed in web client: vCenter server object -> Configure -> Advanced setting -> config.registry.key_managedIP -> New ip address information.
  5. https://kb.vmware.com/s/article/2121116?lang=en_US

Should have a remote session today. Unfortunately, vmware didn't contacted me as agreed. Any ideas folks? Smiley Happy

Reply
0 Kudos
Lalegre
Virtuoso
Virtuoso

Could you try to run the next command to search for duplicated certificates under the same serial numbers:

/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text| grep -A 6 -i alias | less

Also i was reading a little bit and it seems that the resolution for some of the folks was updating vCenter Server to 6.7 U3g which is build 16046470

Reply
0 Kudos
mrgecco
Contributor
Contributor

Tried it and no duplicated certificates.

Nonetheless, I had a remote session with vmware. According to the support a subnet change of a vcsa is not recommended and probably won't work. Checked the info:

  • change of ip address in a different subnet didn't work for me
  • change of ip address within the same subnet worked without any problems

If you have two-sites, as in our case, the recommend to set up a second vcsa and run both vcsa instances in a linked mode. However, anser and problem isn't directly the main question of the thread but has the same error log

Reply
0 Kudos
mosesd
Contributor
Contributor

While this seems to be an old thread, I just wanted to say what my fix for this issue appears to be.  Late last year, we upgrade the firmware on our FC SAN array, 3Par, and I had to remove and re-register the VASA provider.  The 3Par is operating in a mode where it handles the certs and not vCenter.  Apparently, something about the way it registered the cert with vCenter put an alias in the trusted root store that is invalid.  It used the URL for the VASA provider as the alias, so the alias is 'https://ip_of_vasa:9997/vasa' as the alias and it apparently doesn't like the special characters.  GSS walked through removing and unregistering the cert on the VCSA cli and it restarted without issue.  I've got a ticket open with HPE currently on how to change the 3Par so it will use the vCenter generated cert instead of its own self-signed cert.

Ran this command to see the alias:

for i in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list); do echo STORE $i; /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $i --text | egrep "Alias|Not After"; done

and you can see the problem alias below:

STORE TRUSTED_ROOTS
Alias : eb34...
Not After : Jan 15 15:41:25 2030 GMT
Alias : e14...
Not After : Feb 9 15:02:55 2027 GMT
Alias : fca....
Not After : Sep 19 17:37:41 2027 GMT
Alias : https://IP_ADDR:9997/vasa
Not After : Sep 19 17:37:41 2027 GMT

Reply
0 Kudos
beegandhi
Contributor
Contributor

Just wanted to say thank you!! this solution WORKED. After having three different GSS one of knew about it and removed the alias and start the VPXD service which started right away. 

Reply
0 Kudos