VMware Cloud Community
LearnerUser
Contributor
Contributor
Jump to solution

VAPI-ENDPOINT service fails to start after certificates update

I inherited a server running ESXi managed by VCentre with literally no documentation. Within weeks the certificates expired. Earlier this week I updated the STS certificate using https://kb.vmware.com/s/article/76719. After this I used certificate-manager as per https://kb.vmware.com/s/article/2097936 using option 8.

Now vapi-endpoint won't start. Looking at endpoint.log I can see that it fails on SoapBindingImp - log extract below. For some reason it is trying to talk to VCSA-01.something.somethingelse when the name of the VCSA instance is VCSA-01a.something.somethingelse. In the updated certificates I used the value O=VCSA-01a.something.somethingelse.

I assume this is something to do with the new certificates though I cannot find anything online which matches vapi-endpoint failing on SoapBindingImp online. Any thoughts on what the issue might be?

_______________

EDIT - I have also included a list of stopped services in case that gives a clue. Also, vmware-vpxd-svcs fails with the same error. This time it is looking for the name VCSA-01a.something.somethingelse.

_______________

vapi-endpoint Log extract:

2021-12-09T16:36:16.039Z | INFO | state-manager1 | DefaultStateManager | Invoking http-server
2021-12-09T16:36:16.040Z | INFO | state-manager1 | BaseServerBuilder | Creating endpoint with name 'default' on address(es): 127.0.0.1, ::1 with port: 12346
2021-12-09T16:36:16.057Z | INFO | state-manager1 | DefaultJettyServer | Logging initialized @4087ms to com.vmware.vapi.endpoint.http.DefaultJettyServer$JettyLogWrapper
2021-12-09T16:36:16.134Z | WARN | state-manager1 | BaseServerBuilder | Failed to bind /0:0:0:0:0:0:0:1:12346 while testing the endpoint validity
java.net.SocketException: Protocol family unavailable
...
2021-12-09T16:36:16.136Z | WARN | state-manager1 | BaseServerBuilder | Hostname ::1 was found to be invalid and removed from the configuration
2021-12-09T16:36:16.305Z | INFO | state-manager1 | BaseServerBuilder | Starting endpoint with name 'default' on address(es): 127.0.0.1, ::1 with port: 12346
2021-12-09T16:36:16.360Z | INFO | state-manager1 | DefaultJettyServer | Starting jetty server.
2021-12-09T16:36:16.431Z | INFO | state-manager1 | BaseServerBuilder | Started endpoint with name 'default' on address(es): 127.0.0.1, ::1 with port: 12346.
2021-12-09T16:36:16.431Z | INFO | state-manager1 | DefaultStateManager | Invoking cis-sso-settings-builder
2021-12-09T16:36:16.747Z | INFO | state-manager1 | CertificateUtil | Creating anonymous SSO Admin Client for URI https://VCSA-01.something.somethingelse/sso-adminserver/sdk/vsphere.local
2021-12-09T16:36:17.017Z | INFO | state-manager1 | DefaultStateManager | Invoking sts-builder
2021-12-09T16:36:17.776Z | ERROR | state-manager1 | SoapBindingImpl | Error communicating to the remote server https://VCSA-01.something.somethingelse/sts/STSService/vsphere.local
com.sun.xml.internal.ws.client.ClientTransportException: HTTP transport error: javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: No subject alternative DNS name matching VCSA-01.something.somethingelse found.
at com.sun.xml.internal.ws.transport.http.client.HttpClientTransport.getOutput(HttpClientTransport.java:117)
  

vmware-vpxd-svcs Log extract:

2021-12-10T11:55:49.834Z [main ERROR com.vmware.vim.sso.client.impl.SoapBindingImpl opId=] Error communicating to the remote server https://VCSA-01a.something.somethingelse/sts/STSService/vsphere.local
com.sun.xml.internal.ws.client.ClientTransportException: HTTP transport error: javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: No subject alternative DNS name matching VCSA-01a.something.somethingelse found.

 

Stopped Services:

vmcam vmware-content-library vmware-imagebuilder vmware-mbcs vmware-netdumper vmware-rbd-watchdog vmware-sca vmware-sps vmware-updatemgr vmware-vapi-endpoint vmware-vcha vmware-vpxd vmware-vpxd-svcs vmware-vsan-health vsan-dps

0 Kudos
1 Solution

Accepted Solutions
Ajay1988
Expert
Expert
Jump to solution

X509v3 Subject Alternative Name:
email:email@acme.com, DNS:ytdesxi650-1.something.somethingelse

ytdesxi650-1.something.somethingelse is the name of the ESXi instance that hosts VCSA-01a.something.somethingelse.
 
>> This is wrong. This should be VCSA-01a.something.somethingelse(PNID) in the certificate SAN name. Replace all certs again using the correct DNS/FDQN.
If you think your queries have been answered
Mark this response as "Correct" or "Helpful".

Regards,
AJ

View solution in original post

5 Replies
Ajay1988
Expert
Expert
Jump to solution

/usr/lib/vmware-vmafd/bin/vmafd-cli get-pnid --server-name localhost ( to check the PNID :- this should match the FQDN of VCSA)
The above PNID is to be used in certificate replacement too. 

If you think your queries have been answered
Mark this response as "Correct" or "Helpful".

Regards,
AJ
LearnerUser
Contributor
Contributor
Jump to solution


@Ajay1988 wrote:

/usr/lib/vmware-vmafd/bin/vmafd-cli get-pnid --server-name localhost ( to check the PNID :- this should match the FQDN of VCSA)
The above PNID is to be used in certificate replacement too. 

Running that I get:

VCSA-01a.something.somethingelse

This is the correct server name, not the VCSA-01.something.somethingelse as seen in the logs.

0 Kudos
Ajay1988
Expert
Expert
Jump to solution

Can you validate the hostname from VAMI (https://<vp-ip/fqdn:5480) also ?

And run /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store MACHINE_SSL_CERT --text | grep -A1 Alternative 

/opt/vmware/share/vami/vami_config_net

Also do nslook to IP and fqdn to check records.

See if it matches VCSA-01a.something.somethingelse.

 

 

If you think your queries have been answered
Mark this response as "Correct" or "Helpful".

Regards,
AJ
0 Kudos
LearnerUser
Contributor
Contributor
Jump to solution


@Ajay1988 wrote:

Can you validate the hostname from VAMI (https://<vp-ip/fqdn:5480) also ?

Going to https://VCSA-01a.something.somethingelse:5480/ takes me to VMware Appliance Manager login page from a browser.


@Ajay1988 wrote:

And run /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store MACHINE_SSL_CERT --text | grep -A1 Alternative 

/opt/vmware/share/vami/vami_config_net


I get the result

X509v3 Subject Alternative Name:
email:email@acme.com, DNS:ytdesxi650-1.something.somethingelse

ytdesxi650-1.something.somethingelse is the name of the ESXi instance that hosts VCSA-01a.something.somethingelse.
 

@Ajay1988 wrote:

Also do nslook to IP and fqdn to check records.

See if it matches VCSA-01a.something.somethingelse.


 

I get the following output. (10.0.X.Y and 10.0.X.Z are correct IP addresses and are working properly)

Server: 127.0.0.1
Address: 127.0.0.1#53

Name: something.somethingelse
Address: 10.0.X.Y
Name: something.somethingelse
Address: 10.0.X.Z

And

1.2.0.10.in-addr.arpa name = VCSA-01a.something.somethingelse.

 

0 Kudos
Ajay1988
Expert
Expert
Jump to solution

X509v3 Subject Alternative Name:
email:email@acme.com, DNS:ytdesxi650-1.something.somethingelse

ytdesxi650-1.something.somethingelse is the name of the ESXi instance that hosts VCSA-01a.something.somethingelse.
 
>> This is wrong. This should be VCSA-01a.something.somethingelse(PNID) in the certificate SAN name. Replace all certs again using the correct DNS/FDQN.
If you think your queries have been answered
Mark this response as "Correct" or "Helpful".

Regards,
AJ