I inherited a server running ESXi managed by VCentre with literally no documentation. Within weeks the certificates expired. Earlier this week I updated the STS certificate using https://kb.vmware.com/s/article/76719. After this I used certificate-manager as per https://kb.vmware.com/s/article/2097936 using option 8.
Now vapi-endpoint won't start. Looking at endpoint.log I can see that it fails on SoapBindingImp - log extract below. For some reason it is trying to talk to VCSA-01.something.somethingelse when the name of the VCSA instance is VCSA-01a.something.somethingelse. In the updated certificates I used the value O=VCSA-01a.something.somethingelse.
I assume this is something to do with the new certificates though I cannot find anything online which matches vapi-endpoint failing on SoapBindingImp online. Any thoughts on what the issue might be?
_______________
EDIT - I have also included a list of stopped services in case that gives a clue. Also, vmware-vpxd-svcs fails with the same error. This time it is looking for the name VCSA-01a.something.somethingelse.
_______________
vapi-endpoint Log extract:
2021-12-09T16:36:16.039Z | INFO | state-manager1 | DefaultStateManager | Invoking http-server
2021-12-09T16:36:16.040Z | INFO | state-manager1 | BaseServerBuilder | Creating endpoint with name 'default' on address(es): 127.0.0.1, ::1 with port: 12346
2021-12-09T16:36:16.057Z | INFO | state-manager1 | DefaultJettyServer | Logging initialized @4087ms to com.vmware.vapi.endpoint.http.DefaultJettyServer$JettyLogWrapper
2021-12-09T16:36:16.134Z | WARN | state-manager1 | BaseServerBuilder | Failed to bind /0:0:0:0:0:0:0:1:12346 while testing the endpoint validity
java.net.SocketException: Protocol family unavailable
...
2021-12-09T16:36:16.136Z | WARN | state-manager1 | BaseServerBuilder | Hostname ::1 was found to be invalid and removed from the configuration
2021-12-09T16:36:16.305Z | INFO | state-manager1 | BaseServerBuilder | Starting endpoint with name 'default' on address(es): 127.0.0.1, ::1 with port: 12346
2021-12-09T16:36:16.360Z | INFO | state-manager1 | DefaultJettyServer | Starting jetty server.
2021-12-09T16:36:16.431Z | INFO | state-manager1 | BaseServerBuilder | Started endpoint with name 'default' on address(es): 127.0.0.1, ::1 with port: 12346.
2021-12-09T16:36:16.431Z | INFO | state-manager1 | DefaultStateManager | Invoking cis-sso-settings-builder
2021-12-09T16:36:16.747Z | INFO | state-manager1 | CertificateUtil | Creating anonymous SSO Admin Client for URI https://VCSA-01.something.somethingelse/sso-adminserver/sdk/vsphere.local
2021-12-09T16:36:17.017Z | INFO | state-manager1 | DefaultStateManager | Invoking sts-builder
2021-12-09T16:36:17.776Z | ERROR | state-manager1 | SoapBindingImpl | Error communicating to the remote server https://VCSA-01.something.somethingelse/sts/STSService/vsphere.local
com.sun.xml.internal.ws.client.ClientTransportException: HTTP transport error: javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: No subject alternative DNS name matching VCSA-01.something.somethingelse found.
at com.sun.xml.internal.ws.transport.http.client.HttpClientTransport.getOutput(HttpClientTransport.java:117)
vmware-vpxd-svcs Log extract:
2021-12-10T11:55:49.834Z [main ERROR com.vmware.vim.sso.client.impl.SoapBindingImpl opId=] Error communicating to the remote server https://VCSA-01a.something.somethingelse/sts/STSService/vsphere.local
com.sun.xml.internal.ws.client.ClientTransportException: HTTP transport error: javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: No subject alternative DNS name matching VCSA-01a.something.somethingelse found.
Stopped Services:
vmcam vmware-content-library vmware-imagebuilder vmware-mbcs vmware-netdumper vmware-rbd-watchdog vmware-sca vmware-sps vmware-updatemgr vmware-vapi-endpoint vmware-vcha vmware-vpxd vmware-vpxd-svcs vmware-vsan-health vsan-dps
X509v3 Subject Alternative Name:
email:email@acme.com, DNS:ytdesxi650-1.something.somethingelse
/usr/lib/vmware-vmafd/bin/vmafd-cli get-pnid --server-name localhost ( to check the PNID :- this should match the FQDN of VCSA)
The above PNID is to be used in certificate replacement too.
@Ajay1988 wrote:/usr/lib/vmware-vmafd/bin/vmafd-cli get-pnid --server-name localhost ( to check the PNID :- this should match the FQDN of VCSA)
The above PNID is to be used in certificate replacement too.
Running that I get:
VCSA-01a.something.somethingelse
This is the correct server name, not the VCSA-01.something.somethingelse as seen in the logs.
Can you validate the hostname from VAMI (https://<vp-ip/fqdn:5480) also ?
And run /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store MACHINE_SSL_CERT --text | grep -A1 Alternative
/opt/vmware/share/vami/vami_config_net
Also do nslook to IP and fqdn to check records.
See if it matches VCSA-01a.something.somethingelse.
@Ajay1988 wrote:Can you validate the hostname from VAMI (https://<vp-ip/fqdn:5480) also ?
Going to https://VCSA-01a.something.somethingelse:5480/ takes me to VMware Appliance Manager login page from a browser.
@Ajay1988 wrote:And run /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store MACHINE_SSL_CERT --text | grep -A1 Alternative
/opt/vmware/share/vami/vami_config_net
I get the result
X509v3 Subject Alternative Name:
email:email@acme.com, DNS:ytdesxi650-1.something.somethingelse
@Ajay1988 wrote:Also do nslook to IP and fqdn to check records.
See if it matches VCSA-01a.something.somethingelse.
Server: 127.0.0.1
Address: 127.0.0.1#53
Name: something.somethingelse
Address: 10.0.X.Y
Name: something.somethingelse
Address: 10.0.X.Z
And
1.2.0.10.in-addr.arpa name = VCSA-01a.something.somethingelse.
X509v3 Subject Alternative Name:
email:email@acme.com, DNS:ytdesxi650-1.something.somethingelse