vRO Plugin for vSphere Client

[burivay]

I have been trying out the beta plugin for the HTML5 vSphere Client which allows executing workflows directly from vCenter similar to the old plugin for the Flash-based Web Client in the past.

It works great in a lab environment, but I'm running into an issue when trying to use it on a vCenter server where the machine SSL certificate has been replaced (from our Active Directory CA). Even though the vRO appliance has no trouble connecting to the vCenter for authentication with this cert, when trying to run a workflow via the plugin, I'm getting the following:

[ERROR] tp-nio-127.0.0.1-5090-exec-4  com.vmware.vcoin.certificates.VcTrustManager  Server certificate chain is not trusted and thumbprint doesn't match
[ERROR] tp-nio-127.0.0.1-5090-exec-4  com.vmware.vcoin.exceptions.GlobalExceptionHandler  Exception handled while processing request for /ui/vcoin-ui/rest/vcobjects:  com.sun.xml.internal.ws.client.ClientTransportException: HTTP transport error: javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: Server certificate chain is not trusted and thumbprint doesn't match

This is from the vsphere_client_virgo log. In vCenter itself, all it says is "Problem in communication with one or more Orchestrator servers" which isn't very helpful.

I'm not aware of any official place to send feedback for this kind of thing, so I'm trying here. Let me know if there's someplace better!

Thanks,

Andrew

[nikol]

Hi Andrew,

the vRO plugin makes verification agains the VC thumbprint. At what point was the machine SSL cert replaced? Does re-login to the vSphere client help?

Thanks,

Irina

[burivay]

Hi Irina,

We replaced the certificate when upgrading to vSphere 6.5 about two years ago. In fact, the certificate is due for renewal in September, so I went ahead and replaced it with a new certificate in the hope that it might help, but I'm getting the same behavior after renewal. I also ran the ls_update_cert script on the vCenter after replacing the certificate to make sure all the endpoint certs are properly updated. I reconnected the vRealize Orchestrator appliance via vSphere Authentication after this and it is working as expected, but the plugin still fails.

In vCenter, I can see the vRealize Orchestrator appliance. It correctly reports the version info and the number of running/failed workflows. I can also see the other linked vCenters there. From this page, it looks like everything is working, but when I select an object in vCenter and go to vRealize Orchestrator -> Run Workflow, that's when I get the "Problem in communication with one or more Orchestrator servers" error message.

I've tried logging out and back in using different accounts - that hasn't changed the behavior any.

Thanks,

Andrew

[nikol]

Hi Andrew,

the UI error is actually wrong and misleading. The problem in this case has nothing to do with the connection to the vRO server. There is something wrong with the communication between the vRO UI plugin and the vCenter Server. The plugin fails to match the certificate from the client requests with the VC certificate.

Although you have run ls_update_certs.py, can you please verify again that the new machine SSL certificate is present in the Lookup Service registrations. Login with SSH, and check that the current certificate is the same as the one present in the Lookup Service registration. Just run these commands and compare their output:

   /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store MACHINE_SSL_CERT

   python /usr/lib/vmidentity/tools/scripts/lstool.py list --url http://localhost:7080/lookupservice/sdk --type vcenterserver --product com.vmware.cis // Search for "SSL trust:" in the output

   cat /etc/vmware-rhttpproxy/ssl/rui.crt

The certificate must be the same in all these outputs.

Also, can you confirm that you have restarted all services after running the ls_update_certs tool?

Thanks,

Irina

[burivay]

Hey Irina,

All of the certificate outputs appear to match. I did restart the server after running the replacement script, but the behavior continues.

The SSL Trust outputs are consistent for each vCenter across all the entries I see listed.

The only thing that might be different is that the MACHINE_SSL_CERT is a fully chained cert. It has the signing CA's certificate (our AD server) appended to it to complete the chain. The endpoint SSL Trust entries do not include this additional certificate, but only the main cert issued for the vCenter. Would this cause the issues I'm seeing?

Thanks,

Andrew

[nikol]

Hi Andrew,

yes, this could be causing the problems. We can confirm this by comparing the thumbprints of the web client certificate and the SSL trust entries from the Lookup Service.

1) Get the web client's certificate thumbprint - you can see it in the browser after you have logged into the web client - notice and copy the SHA 1 thumbrint

2) SSH to your VCSA

3) Get the SSL trust entry of the VC registration in LS. Copy the SSL trust string from this output:

    /usr/lib/vmidentity/tools/scripts/lstool.py list --url https://localhost/lookupservice/sdk --no-check-cert --ep-type com.vmware.vim 2>/dev/null

4) Create a file which contains this SSL trust entry. Append -----BEGIN CERTIFICATE----- to the beginning of the text string, and append -----END CERTIFICATE----- to the end of the text string.

5) Extract the thumbprint for the SSL trust:

    openssl x509 -in <path to your SSL trust file> -noout -sha1 -fingerprint

6) Compare the thumprints from steps 1 and 6. If they don't match, then this is why the vRO plugin doesn't work. We'll think of a workaround if this is the case.

Thanks,

Irina

[burivay]

Hi Irina,

I compared the thumbprints between the browser and the output from the VCSA shell and they do match. The only difference is that the browser displays the full certificate hierarchy in the Details tab, whereas the SSL Trust output only has the issued webserver certificate.

Thanks,

Andrew

[nikol]

Hi Andrew,

if thumbprints match, I'm afraid I can't think of a workaround at this point. We'll think of an option to be able to disable cert validation in future versions and have a more detailed logging.

Thanks,

Irina

[simon1234]

Hi, please check this: https://kb.vmware.com/s/article/52540 - i was able to get around this issue. But in my case it seems, that intermediate ca certificates resulting in this error. With certificate chain in vecs access to web client works correctly. Without intermediate certifiate, the plugin works, but users cannot access web client without errors about the trust chain.

Every time again, vmware and certificates is a big mess.

Regards, Simon