VCSA 6.0U3 SSL woes

[taranga]

Hello everyone,

Yesterday I started having trouble signing in to the VCSA 6.0U3 Flash ("flex") client, seemingly out of nowhere. Yes, I would like to upgrade to 6.5, but we have no support contract for two years...

The Windows "fat" client lets me log in, and if I SSH in and restart all services, my FIRST login succeeds. After that if I attempt to login again or from another machine I get the blue screen and spinning clock indefinitely.

The most promising error messages I can are from websso.log:

javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

ssoAdmin:

com.vmware.identity.admin.server.ims.ServerConfigurationException: Failed to get issuers certificates

and STS:

2020-08-11T17:16:09.961-04:00 | ERROR

| opId-8d5efa48-949b-47ed-8c13-5dd383b74896 | vdcs-background-executor-4 | StsTrustChainImpl

| Error retrieving trusted root certificates.

java.lang.NullPointerException

	at com.vmware.provider.VecsKeyStoreEngine.engineAliases(VecsKeyStoreEngine.java:71)
	at java.security.KeyStore.aliases(Unknown Source)
	at com.vmware.vcde.common.services.sso.impl.StsTrustChainImpl.refresh(StsTrustChainImpl.java:56)
	at com.vmware.vcde.common.services.sso.impl.StsTrustChainImpl.access$0(StsTrustChainImpl.java:51)
	at com.vmware.vcde.common.services.sso.impl.StsTrustChainImpl$1.run(StsTrustChainImpl.java:46)

I haven't made any modifications to the certs, and things were working prior to yesterday afternoon. All the certs I can find are valid through 2024 or 2025. I've poked through the management interface, through the PSC, manually verified the certs on the VCSA with openssl. My suspicion is that some cert expired but I can't find any that are expired.

I did reboot the VCSA, and when it came back up it wiped out eam.properties so I did rebuild that and have verified that vmware-eam is running, and that the vapi endpoint health check returns okay.

This is so strange because, once I rebooted and/or restart all services, the first login succeeds in the web interface, but I only get one. The fat client works. The PSC lets me log in.

Has anyone seen this before?

Thank you,

Don

[sjesse]

Based on the error this kb probably applies, it was written after 6.0 support was dropped. Just make sure you backup the appliance before doing anything.

VMware Knowledge Base

[taranga]

Thank you for answering - I've seen that KB already; the STS signing certs expire in 2025.

[sjesse]

Its something do with the cert or the time, I'd do a google search and look for

com.vmware.vcde.common.services.sso.impl.StsTrustChainImpl.refresh

there are a bunch of kbs that come up that might help.

[taranga]

The VCSA and PSC are on the same host; its time is managed via NTP and is correct by my laptop.

If I restart all services, I can sign in via the UI once, either as administrator@vsphere or using my AD account. In websso.log I can see successive sign-in attempts verifying my token, but the web interface never returns. I too suspect a cert somewhere but I'm really coming up at a loss.

[scott28tt]

Moderator: Thread moved to the vCenter Server area.

[Lalegre]

What identity provider are you using to do the authentication? Are you using LDAPs by any chance?

[taranga]

It's authenticating against Active Directory, no LDAP, but through the web client I was unable to log in as administrator@vsphere.local either.

I learned a lot of VCSA in the past few days, and suspect I was hitting two problems:

First, I learned that the embedded PSC maintains separate NTP settings from the appliance VM. In my case, the PSC was still pointing at an NTP server the university had decommissioned. I also learned that a clock skew of greater than one second can cause token validation problems while establishing a session. Then I learned that correcting the NTP setting and restarting all services with service-control is not enough - one must reboot the whole shebang.

Secondly, I had to rebuild eam.conf with what I thought were the correct settings, and in the process I used the FQDN for the SDK connector, as directed, though prior I had always used the ipv4 in the URL (I inherited these systems, and with no money for support or for replacement systems I've been simply trying to maintain state for quite a while). I was no longer able to log in to the web client using the ipv4, which makes sense when the connectors now point to the service name used for the certs. I use https://service-name and I can log in.

Those were the only two config changes I made in the VM, but between the two of them I'm again able to login, so I'm relieved.

Thanks to all for their suggestions and help.

D