Hi VMWare Community,
In order to develop and debug our custom plugin (as a team), we need to deploy and run it on our local machines. Here are some details:
What do we do:
Local environment setup:
- We download the vSphere Client SDK 7.0.0.10100 or 6.7.20000 to local machine (depending on vcenter version)
- We unpack the content locally and set required environment variables: VMWARE_CFG_DIR, VSPHERE_SDK_HOME
- We have correct JAVA_HOME and MAVEN_HOME defined
- We run the server-registration.bat script from vcenter SDK to generate three required files (store.jks, webclient.properties and ds.properties) and we store them under correct locations in ProgramFIles\VMWare\vCenterServer\cfg\
- server-registration.bat -vcip [our-vcenter-IP] -u [user] -pw [thePassword]
- At the very end, we run the startup.bat script from \html-client-sdk\vsphere-ui\server\bin in order to locally run the environment with debug option
- startup.bat -debug 6969
- After a couple of minutes, if everything works ok, setup script is completed and we can access the local environment by entering the https://localhost:9443/ui , login and see the content of the vcenter locally.
What does not work:
In spite of the fact that all team members follow same steps, some group of them are not able to start the plugin correctly. Here you can see the symptoms:
1) Installation process hangs on the following phase and doesn't process any further
2) When trying to login we can see an error message indicating a authentication problem:
3) In virgo logs we can observe:
[2020-08-04T07:30:24.368+02:00] [ERROR] cm-catalog-manager-pool-12 com.vmware.vim.sso.client.impl.SoapBindingImpl SOAP fault com.sun.xml.internal.ws.fault.Ser
verSOAPFaultException: Client received SOAP Fault from server: Signature is invalid. Please see the server log to find more detail regarding exact cause of the failure.
at com.sun.xml.internal.ws.fault.SOAP11Fault.getProtocolException(SOAP11Fault.java:178)
at com.sun.xml.internal.ws.fault.SOAPFaultBuilder.createException(SOAPFaultBuilder.java:116)
at com.sun.xml.internal.ws.client.dispatch.DispatchImpl.doInvoke(DispatchImpl.java:259)
Main question:
According to us, there is something wrong with certificates on local machines (only few people are affected in the team). Not sure what exactly. Should we think about any other certificates than these in Java keystore (according to JAVA_HOME) or store.jks (generated from vcenter) ? Do you have any ideas what can be the cause or where should we look for it?
What we've already tried (with no effect):
- Going through the same procedure (using the same vcenters) by different team members
- Generating the store.jks by people who do not have problems any use it by other user
- Copying the jre keystore (from JAVA_HOME) from people who do not have problems
- Members facing this problem tried to use different vcenters as well
- Examining the local firewall logs did not revealed any blocked traffic
- We also try to perform the entire setup procedure from scratch, many times removing the following files/dirs before:
- C:\ProgramData\VMware\vCenterServer\cfg\vsphere-client\cmCatalog
- C:\ProgramData\VMware\vCenterServer\cfg\vsphere-client\cm-service-packages
- C:\ProgramData\VMware\vCenterServer\cfg\vsphere-client\vc-packages
- C:\ProgramData\VMware\vCenterServer\cfg\vsphere-client\compatability-matrix.xml.
Please let me know if I should add any other information. All help is appreciated.
Regards,
Radoslaw
Hi Vladi,
thanks a lot for your answer. Unfortunately we still haven't solved the issue. I will refer you your suggestions inline:
"1) This is OK and is not stuck. The screenshot shows that the local plugin sample is deployed and this is the last phase of running the vSphere Client application server locally. You will see more output when the Client is opened in the browser (you don't get to this point)."
We are not sure about that but maybe I'm wrong - see "observation 2" at the bottom.
- "The certificate keystore that is at play here is the store.jks that got copied from the vCenter. Please check the location it is present on is correct."
We tried two ways - generating the store.jks (as well as ds.properties and webclient propeties) by running the registration script on local machine and by doing it on vcenter (and then, copying the output into correct locations). Regardless of the method, the store.jks is always in C:\ProgramData\VMware\vCenterServer\cfg (vcenter 6.7)
- "You mention using both 7.0 and 6.7 SDKs. Please note that SDK dev environment setup is not an API and cannot be backward compatible in the general case. This means that folks working with a 7.0 vCenter should use the 7.0 SDK and ones with 6.7 vCenter should use the 6.7 SDK. Please check if this is the case in the failing local environments."
We always tried the following configurations: sdk 6.7 when using vcenter 6.7.0.10000/30000 OR sdk 7.0 for vcenters 7.0.0.10000.
- "Here is the dev setup documentation: Online Documentation - Developing Local Plug-ins with the vSphere Client SDK - VMware {code} . It has a manual and automatic flow that can be tried. In any case please check the specifics required for the OS - there are slight setup differences between Windows and MacOS (though most are abstracted out by the server-registration script)."
Actually, we do everything according the instruction. We use only Windows 10 on our local machines where we want to deploy the plugin. I guess platform differences are not the case here.
- "Test if the startup script works without --debug <port>"
Unfortunately no effect as well - the same situation appear for startup without debug
Nevertheless, we have some more observations regarding the installation process:
Observation 1: When you take a look at the screen with the startup output (windows console) you will see that the procedure logs maximal the "plugin-deploy8" phase. By me, where the problem does not appear, I can see also next phases up to "plugin-deploy11" which is not reached by problematic installations.
Observation 2: Moreover, in case of the problematic installations, the "C:\ProgramData\VMware\vCenterServer\cfg\vsphere-client" directory DOES NOT contain "cmCatalog" and "vc-packages" dirs which, I guess, should be generated during the startup. See the screen below - this is a screen from a proper installation. In problematic installations cmCatalog and vc-packages are not present:
It looks like the startup is not fully completed, am I right?
Observation 3: in virgo logs:
[2020-08-24T21:14:43.914+02:00] [INFO ] cm-catalog-manager-pool-5 c.v.v.s.c.impl.SecurityTokenServiceImpl$RequestResponseProcessor Request signature is not valid. Check if the confirmation certificate matches the given private key.
[2020-08-24T21:14:43.914+02:00] [ERROR] cm-catalog-manager-pool-5 com.vmware.vise.vim.security.sso.impl.NgcSolutionUser Solution user login into domain vsphere.local failed. com.vmware.vim.sso.client.exception.AuthenticationFailedException: Request signature is not valid. Check if the confirmation certificate matches the given private key.[2020-08-24T21:14:43.914+02:00] [INFO ] cm-catalog-manager-pool-5 c.v.v.s.c.impl.SecurityTokenServiceImpl$RequestResponseProcessor Request signature is not valid. Check if the confirmation certificate matches the given private key.
[2020-08-24T21:14:43.914+02:00] [ERROR] cm-catalog-manager-pool-5 com.vmware.vise.vim.security.sso.impl.NgcSolutionUser Solution user login into domain vsphere.local failed. com.vmware.vim.sso.client.exception.AuthenticationFailedException: Request signature is not valid. Check if the confirmation certificate matches the given private key.
Maybe this additional info can be a tip regarding the potential root cause? Please let us know if more info is required...
Cheers,
Radoslaw
Hi Vladi,
Sorry for so late response. First and foremost thank you for your offer of help - really kind of you 
Nevertheless, we managed to find a solution, actually by mistake. The main cause was a newer version of JDK 8 used as JAVA_HOME on machines we were deploying the local plugin. After some investigation we noticed, that there were some changes introduced in Java 8 updates (between u192 and u241) regarding certificates, authorities etc... Using the older jdk temporary solved the issue and revealed the main cause.
Nevertheless, thanks for support
Have a nice day
Radoslaw