7 Replies Latest reply on Aug 26, 2019 7:48 AM by nikol

    vRO Plugin for vSphere Client

    burivay Lurker

      I have been trying out the beta plugin for the HTML5 vSphere Client which allows executing workflows directly from vCenter similar to the old plugin for the Flash-based Web Client in the past.

      It works great in a lab environment, but I'm running into an issue when trying to use it on a vCenter server where the machine SSL certificate has been replaced (from our Active Directory CA). Even though the vRO appliance has no trouble connecting to the vCenter for authentication with this cert, when trying to run a workflow via the plugin, I'm getting the following:

       

       

      [ERROR] tp-nio-127.0.0.1-5090-exec-4  com.vmware.vcoin.certificates.VcTrustManager  Server certificate chain is not trusted and thumbprint doesn't match
      [ERROR] tp-nio-127.0.0.1-5090-exec-4  com.vmware.vcoin.exceptions.GlobalExceptionHandler  Exception handled while processing request for /ui/vcoin-ui/rest/vcobjects:  com.sun.xml.internal.ws.client.ClientTransportException: HTTP transport error: javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: Server certificate chain is not trusted and thumbprint doesn't match
      

       

      This is from the vsphere_client_virgo log. In vCenter itself, all it says is "Problem in communication with one or more Orchestrator servers" which isn't very helpful.

       

      I'm not aware of any official place to send feedback for this kind of thing, so I'm trying here. Let me know if there's someplace better!

       

      Thanks,

      Andrew

        • 1. Re: vRO Plugin for vSphere Client
          nikol Enthusiast
          VMware Employees

          Hi Andrew,

          the vRO plugin makes verification agains the VC thumbprint. At what point was the machine SSL cert replaced? Does re-login to the vSphere client help?

           

          Thanks,

          Irina

          • 2. Re: vRO Plugin for vSphere Client
            burivay Lurker

            Hi Irina,

             

            We replaced the certificate when upgrading to vSphere 6.5 about two years ago. In fact, the certificate is due for renewal in September, so I went ahead and replaced it with a new certificate in the hope that it might help, but I'm getting the same behavior after renewal. I also ran the ls_update_cert script on the vCenter after replacing the certificate to make sure all the endpoint certs are properly updated. I reconnected the vRealize Orchestrator appliance via vSphere Authentication after this and it is working as expected, but the plugin still fails.

             

            In vCenter, I can see the vRealize Orchestrator appliance. It correctly reports the version info and the number of running/failed workflows. I can also see the other linked vCenters there. From this page, it looks like everything is working, but when I select an object in vCenter and go to vRealize Orchestrator -> Run Workflow, that's when I get the "Problem in communication with one or more Orchestrator servers" error message.

             

            I've tried logging out and back in using different accounts - that hasn't changed the behavior any.

             

            Thanks,

            Andrew

            • 3. Re: vRO Plugin for vSphere Client
              nikol Enthusiast
              VMware Employees

              Hi Andrew,

              the UI error is actually wrong and misleading. The problem in this case has nothing to do with the connection to the vRO server. There is something wrong with the communication between the vRO UI plugin and the vCenter Server. The plugin fails to match the certificate from the client requests with the VC certificate.

              Although you have run ls_update_certs.py, can you please verify again that the new machine SSL certificate is present in the Lookup Service registrations. Login with SSH, and check that the current certificate is the same as the one present in the Lookup Service registration. Just run these commands and compare their output:

                 /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store MACHINE_SSL_CERT

                 python /usr/lib/vmidentity/tools/scripts/lstool.py list --url http://localhost:7080/lookupservice/sdk --type vcenterserver --product com.vmware.cis // Search for "SSL trust:" in the output

                 cat /etc/vmware-rhttpproxy/ssl/rui.crt

               

              The certificate must be the same in all these outputs.

               

              Also, can you confirm that you have restarted all services after running the ls_update_certs tool?

               

              Thanks,

              Irina

              • 4. Re: vRO Plugin for vSphere Client
                burivay Lurker

                Hey Irina,

                 

                All of the certificate outputs appear to match. I did restart the server after running the replacement script, but the behavior continues.

                 

                The SSL Trust outputs are consistent for each vCenter across all the entries I see listed.

                The only thing that might be different is that the MACHINE_SSL_CERT is a fully chained cert. It has the signing CA's certificate (our AD server) appended to it to complete the chain. The endpoint SSL Trust entries do not include this additional certificate, but only the main cert issued for the vCenter. Would this cause the issues I'm seeing?

                 

                Thanks,

                Andrew

                • 5. Re: vRO Plugin for vSphere Client
                  nikol Enthusiast
                  VMware Employees

                  Hi Andrew,

                  yes, this could be causing the problems. We can confirm this by comparing the thumbprints of the web client certificate and the SSL trust entries from the Lookup Service.

                  1) Get the web client's certificate thumbprint - you can see it in the browser after you have logged into the web client - notice and copy the SHA 1 thumbrint

                  2) SSH to your VCSA

                  3) Get the SSL trust entry of the VC registration in LS. Copy the SSL trust string from this output:

                      /usr/lib/vmidentity/tools/scripts/lstool.py list --url https://localhost/lookupservice/sdk --no-check-cert --ep-type com.vmware.vim 2>/dev/null

                  4) Create a file which contains this SSL trust entry. Append -----BEGIN CERTIFICATE----- to the beginning of the text string, and append -----END CERTIFICATE----- to the end of the text string.

                  5) Extract the thumbprint for the SSL trust:

                      openssl x509 -in <path to your SSL trust file> -noout -sha1 -fingerprint

                  6) Compare the thumprints from steps 1 and 6. If they don't match, then this is why the vRO plugin doesn't work. We'll think of a workaround if this is the case.

                   

                  Thanks,

                  Irina

                   

                   

                   

                   

                   

                  • 6. Re: vRO Plugin for vSphere Client
                    burivay Lurker

                    Hi Irina,

                     

                    I compared the thumbprints between the browser and the output from the VCSA shell and they do match. The only difference is that the browser displays the full certificate hierarchy in the Details tab, whereas the SSL Trust output only has the issued webserver certificate.

                     

                    Thanks,

                    Andrew

                    • 7. Re: vRO Plugin for vSphere Client
                      nikol Enthusiast
                      VMware Employees

                      Hi Andrew,

                      if thumbprints match, I'm afraid I can't think of a workaround at this point. We'll think of an option to be able to disable cert validation in future versions and have a more detailed logging.

                       

                      Thanks,

                      Irina