topic Re: Failed to deploy a local plugin on vSphere 8.0 in vSphere Client SDK Discussions

Failed to deploy a local plugin on vSphere 8.0

MK1506 — Thu, 06 Oct 2022 08:48:02 GMT

Below logs observed in "vsphere-ui-runtime.log.stdout" while deploying our local plugin on vSphere 8.0.

[2022-10-03T12:55:35.631Z] [ERROR] plugin-deploy19 [Subsystem Event] Error when starting subsystem: com.vcplugin.plugin org.osgi.service.subsystem.SubsystemException: Framework could not resolve the bundles: [org.osgi.service.subsystem.region.context.11_1.0.0 [359], org.eclipse.virgo.web.dm_1.0.0 [361], com.vmware.wsvim25_8.0.0 [360]]

OSGI reported missing dependencies for "org.eclipse.virgo.web.dm_1.0.0" only, but above logs showing errors for three bundles .

g! ss

360 RESOLVED com.vmware.wsvim25_8.0.0

361 INSTALLED org.eclipse.virgo.web.dm_1.0.0

g! diag 361

org.eclipse.virgo.web.dm [361]

Unresolved requirement: Import-Package: org.springframework.aop

Importing all packages reported by "packages org.springframework.aop" in respective pom.xml and Manifest file is not resolving deployment issue.

Re: Failed to deploy a local plugin on vSphere 8.0

ChrisSmith2 — Tue, 11 Oct 2022 09:27:25 GMT

vSphere 8.0 requires changes to local plugins to support library isolation:

https://vdc-repo.vmware.com/vmwb-repository/dcr-public/77dd6491-bb96-47f3-8c2e-a5a4655f078b/b916bc3c-0168-4487-a9e5-c89b50373c18/Local-Plugins-Library-Isolation.pdf

and FIPS compliance:

https://vdc-download.vmware.com/vmwb-repository/dcr-public/0a357b43-fe36-4478-a82b-2ae33ff971f2/cbce5d49-b1be-49f2-8514-d37b6cd910fe/Preparing-Local-Plug-ins-for-FIPS-Compliance.html

Re: Failed to deploy a local plugin on vSphere 8.0

msolinski — Thu, 15 Dec 2022 17:36:11 GMT

I also have a problem with my old local plugin and vSphere 8.0. I followed the instructions in the "Local plugin library isolation - vSphere 8.0" document. I moved third party libraries (Spring and others) to a separate "libraries" directory. My plugin deploys in vSphere but doesn't work.

In the OSGi console, my plugin bundle has the status 'INSTALLED'. The 'diag' command displays the message: "Unresolved requirement: Import-Package: com.vmware.vim.sso.http". But also in the same console I see this bundle (after typing 'ss' command):

71 ACTIVE com.vmware.vim.sso.http_0.0.1.dev

Do I need to copy any libraries from VMware to the "libraries" directory in addition to third-party libraries? I found the httpSsoAuth.jar file containing bundle com.vmware.vim.sso.http on vSphere server. But copying this file to my plugin's "libraries" directory seems incorrect to me.

Has anyone had a problem like this?

Re: Failed to deploy a local plugin on vSphere 8.0

Denis_Chorbadzh — Wed, 04 Jan 2023 12:57:32 GMT

Hi,

The package "com.vmware.vim.sso.http" is an internal package and should not be used. The code that uses classes from this package needs to be rewritten accordingly.

Best,

Denis

Re: Failed to deploy a local plugin on vSphere 8.0

Denis_Chorbadzh — Wed, 04 Jan 2023 13:08:03 GMT

Hi,

You will need to place the following bundles inside your "libraries" folder:

https://mvnrepository.com/artifact/org.apache.servicemix.bundles/org.apache.servicemix.bundles.spring-aop/5.3.24_1

https://mvnrepository.com/artifact/org.apache.servicemix.bundles/org.apache.servicemix.bundles.spring-core/5.3.24_1

Best,

Denis

Re: Failed to deploy a local plugin on vSphere 8.0

ShivamVerma380 — Tue, 07 Feb 2023 08:56:24 GMT

Hi,

We have tried to add the bundles mentioned above in the "libraries" folder which is parallel to the "plugins" folder on VCSA 8.0.

Still, the issue persists on the OSGI console and we have 2 unresolved dependencies i.e. org.springframework.orm and org.eclipse.virgo.web.dm.

org.springframework.orm is always in the "INSTALLED" state even if the plugin is not installed on VCSA 8.0.
Looking forward to your help on this,
Regards Shivam

Re: Failed to deploy a local plugin on vSphere 8.0

Denis_Chorbadzh — Thu, 09 Feb 2023 08:42:29 GMT

Hi @ShivamVerma380

If you use Spring MVC controllers in your plugin, here is a list of the bundles required to resolve Spring MVC dependencies:

org.apache.servicemix.bundles.aopalliance-1.0_6
org.apache.servicemix.bundles.spring-aop-5.2.22.RELEASE_1
org.apache.servicemix.bundles.spring-beans-5.2.22.RELEASE_1
org.apache.servicemix.bundles.spring-context-5.2.22.RELEASE_1
rg.apache.servicemix.bundles.spring-context-support-5.2.22.RELEASE_1
org.apache.servicemix.bundles.spring-core-5.2.22.RELEASE_1
org.apache.servicemix.bundles.spring-expression-5.2.22.RELEASE_1
org.apache.servicemix.bundles.spring-web-5.2.22.RELEASE_1
org.apache.servicemix.bundles.spring-webmvc-5.2.22.RELEASE_1

Best Regards,

Denis

Re: Failed to deploy a local plugin on vSphere 8.0

ShivamVerma380 — Wed, 22 Feb 2023 05:58:44 GMT

Hi @Denis_Chorbadzh

I have added the libraries folder where the plugin is installed at /etc/vmware/vsphere-ui/vc-packages/vsphere-client-serenity/com.vcplugin.plugin-1.2.1.27 path.

With the above packages added to the libraries folder, I was able to resolve all the OSGI dependencies except for the org.springframework.orm package. This issue persists on the OSGI console even if the plugin is uninstalled.

After restarting the VCSA 8.0 we get plugin deployment timeout, which eventually fails as shown below.

The plugin code is written in Springboot having the Spring version 4.3.9.RELEASE and SpringBoot version 1.5.9.RELEASE. Now, this plugin works perfectly on VCSA 7.0 but not on VCSA 8.0. I have a finding that the issue that might cause the plugin to not be compatible with VCSA 8.0 is the old spring version as mentioned in the article: https://kb.vmware.com/s/article/87880.

So I upgraded the plugin to Spring version 5.2.22.RELEASE and SpringBoot version to 2.0.9.RELEASE.

After this major upgrade, many packages were deprecated as SpringBoot was upgraded from 1. x to 2. x. After this major upgrade, I did some package name changes as suggested in the official Spring Documentation. After that also the plugin gets timed out.

Another thing is on UI we see the plugin type as local but from my observation plugin should be treated as remote and this might be an issue from UI.

Here is the plugin-package.xml:

Equinox.log:

vsphere_client_virgo.log:

vsphere-ui-runtime.log.stdout:

In Brief @Denis_Chorbadzh please let me know the answers to the following questions:

1. Is vSphere Client VCSA 8.0 compatible with Spring Version 4.3.9.RELEASE or do we need to upgrade Spring to 5.2.22_RELEASE ?

2. With help from the above attachments do you have any idea why is the plugin getting deployment timed-out after the upgrade?

3. Plugin type is seen as local on UI. But from my understanding this plugin is a remote plugin as my plugin is not running from VCSA. So is this affecting the deployment on VCSA 8.0?

4. Also after the spring boot upgrade to 2.0.9.RELEASE two dependencies again go into the INSTALLED state i.e. vcplugin-controller and vcplugin-ui. Do we need to add any jar to the libraries folder to resolve them?

5. If the plugin is getting installed on the VCSA 8.0, is there any way to access it just like APIs from postman? How to verify that it is successfully deployed/installed on the VCSA 8.0 and that the only issue lies on the UI side?

6. Any open-source repository for developing remote plugins on the VMware client plugin VCSA 8.0?

Best Regards,

Shivam

Re: Failed to deploy a local plugin on vSphere 8.0

Denis_Chorbadzh — Wed, 22 Feb 2023 11:20:46 GMT

Hi @ShivamVerma380 ,

First, you need to integrate the "libraries" folder in your plugin.zip and not modify directly the plugin cache in "/etc/vmware/vsphere-ui/vc-packages/vsphere-client-serenity/". This directory is used for downloading and unarchiving the plugin deliverables.

Regarding Spring 4.3.9, it is vulnerable to Spring4Shell, and we highly recommend you upgrade to 5.3.XX (LTS).

For vcplugin-ui, you need:
- https://mvnrepository.com/artifact/org.codehaus.woodstox/woodstox-core-asl/4.4.1
and
- https://mvnrepository.com/artifact/org.codehaus.woodstox/stax2-api
Don’t mind the "org.springramework.orm" not being resolved: it is a red herring. It has bundle id of 129, typically plugin bundle ids are in 300 range.

For vcplugin-controller, you need to resolve "org.springframwork.beans" jar first (it is the provider of "org.springframework.beans.factory.annotation").

From the attachments it is not clear why your “org.springframwork.beans” is not resolved (Note: there will be more than one “org.springframwork.beans”, yours will have id in the 300+ range).

It is also not clear why the plugin deployment timed out. It was probably due to the unresolved plugin bundles.

Regarding the type of the plugin, your plugin is a "local" plugin. This means it will be downloaded from the remote location and will be placed inside the VCSA's tomcat server. To be a "remote" plugin, it needs to run on your own server and just provide its views to the vSphere Client. Remote plugins also use a different manifest format, called plugin.json. You can find more information about the remote plugin architecture here. Have in mind that the local plugin architecture is being deprecated and the next major vSphere Release will support only remote plugins.

Finally, you can verify your plugin is successfully installed by finding it in Administration -> Client Plug-Ins and checking its status. Additionally, you can inspect the vsphere_client_virgo.log file and search for your plugin ID to verify there are not errors and the deployment is successful.

Best Regards,

Denis

Re: Failed to deploy a local plugin on vSphere 8.0

ShivamVerma380 — Mon, 27 Feb 2023 17:00:09 GMT

Hi @Denis_Chorbadzh ,

As you suggested, we've added stax2, woodstox jars, and integrated the libraries directory into plugin.zip.

Now we can see that the libraries folder getting extracted into /etc/vmware/vsphere-ui/vc-packages/vsphere-client-serenity/. Even with this change, we are still facing a timeout issue with our plugin. The org.springramework.orm unresolved dependency is still there but we're ignoring it as suggested and all the other dependencies are resolved/active.
At this stage, we are able to see plugin entry into /mob and "Failed" entry at Administration -> Client Plug-Ins. The vsphere_client_virgo.log shows an ERROR for deployment mentioning timeout but does not tell anything else.
Is there any other log to check or any debugging method to root cause the timeout issue? Please advice.


[2023-02-24T13:35:26.515Z] [INFO ] sdk-plugin-deployer-93 com.vmware.vise.plugin.status.LocalPluginStatusServiceImpl DEPLOYING: Deploying plugin package com.vcplugin.plugin:1.2.1.27 ...
[2023-02-24T13:35:26.516Z] [WARN ] sdk-plugin-deployer-93 com.vmware.vise.extensionfw.impl.PackageManifestParser Plugin version mismatch for com.vcplugin.plugin between the plugin registration info (1.2.1.27)
[2023-02-24T13:35:26.531Z] [INFO ] sdk-plugin-deployer-93 com.vmware.vise.extensionfw.impl.sandboxing.EsaBuilder Creating OSGi sandbox for package com.vcplugin.plugin with scope fullyIsolated
[2023-02-24T13:35:27.476Z] [INFO ] plugin-deploy13 com.vmware.vise.extensionfw.impl.PackagesDeployer Deploying plugin package 'com.vcplugin.plugin:1.2.1.27'.
[2023-02-24T13:35:27.476Z] [INFO ] plugin-deploy13 com.vmware.vise.extensionfw.impl.DeploymentEventLoggerImpl Event=PACKAGE_DEPLOY_BEGIN; Params=com.vcplugin.plugin, 1.2.1.27
[2023-02-24T13:45:26.841Z] [INFO ] plugin-discovery-bus-thread com.vmware.vise.vim.extension.PluginStatusTaskManager User: VSPHERE.LOCAL\vsphere-webclient-1c7d92d8-3b3b-4ec1-85ad-da68e934700d initiated plugin DOWNLOAD task for plugin com.vcplugin.plugin:1.2.1.27.
[2023-02-24T13:45:26.863Z] [INFO ] plugin-discovery-bus-thread com.vmware.vise.vim.extension.PluginStatusTaskManager User: VSPHERE.LOCAL\vsphere-webclient-1c7d92d8-3b3b-4ec1-85ad-da68e934700d initiated plugin DEPLOYMENT task for plugin com.vcplugin.plugin:1.2.1.27.
[2023-02-24T13:45:27.476Z] [ERROR] sdk-plugin-deployer-93 com.vmware.vise.extensionfw.impl.PackagesDeployer Package deployer timed out waiting for com.vcplugin.plugin:1.2.1.27;
[2023-02-24T13:45:27.489Z] [ERROR] sdk-plugin-deployer-93 com.vmware.vise.plugin.status.LocalPluginStatusServiceImpl DEPLOYMENT_FAILED: Error deploying plugin package com.vcplugin.plugin:1.2.1.27. Reason: Plugin deployment has timed out.
[2023-02-24T13:45:27.489Z] [INFO ] plugin-validation1 com.vmware.vise.extensionfw.impl.OsgiUsageValidationService Started validating com.vcplugin.plugin:1.2.1.27
[2023-02-24T13:45:27.489Z] [INFO ] plugin-validation1 com.vmware.vise.extensionfw.impl.OsgiUsageValidationService Begin VmwareNameMisuseCheck on com.vcplugin.plugin:1.2.1.27
[2023-02-24T13:45:27.490Z] [ERROR] plugin-deploy13 com.vmware.vise.extensionfw.impl.PackagesDeployer Error deploying one of more bundles for the plugin package /etc/vmware/vsphere-ui/vc-packages/vsphere-client-serenity/com.vcplugin.plugin-1.2.1.27 java.lang.InterruptedException: null
[2023-02-24T13:45:27.493Z] [INFO ] plugin-validation1 com.vmware.vise.extensionfw.impl.OsgiUsageValidationService End VmwareNameMisuseCheck on com.vcplugin.plugin:1.2.1.27
[2023-02-24T13:45:27.493Z] [INFO ] plugin-validation1 com.vmware.vise.extensionfw.impl.OsgiUsageValidationService Begin DynamicImportUsageCheck on com.vcplugin.plugin:1.2.1.27
[2023-02-24T13:45:27.493Z] [INFO ] plugin-validation1 com.vmware.vise.extensionfw.impl.OsgiUsageValidationService End DynamicImportUsageCheck on com.vcplugin.plugin:1.2.1.27
[2023-02-24T13:45:27.493Z] [INFO ] plugin-validation1 com.vmware.vise.extensionfw.impl.OsgiUsageValidationService Begin RequireBundleUsageCheck on com.vcplugin.plugin:1.2.1.27
[2023-02-24T13:45:27.493Z] [INFO ] plugin-validation1 com.vmware.vise.extensionfw.impl.OsgiUsageValidationService End RequireBundleUsageCheck on com.vcplugin.plugin:1.2.1.27
[2023-02-24T13:45:27.493Z] [INFO ] plugin-validation1 com.vmware.vise.extensionfw.impl.OsgiUsageValidationService Begin UnversionedPackageImportCheck on com.vcplugin.plugin:1.2.1.27
[2023-02-24T13:45:27.493Z] [ERROR] sdk-plugin-deployer-93 com.vmware.vise.vim.extension.VcExtensionManager Deployment of plugin package: 'com.vcplugin.plugin:1.2.1.27' registered in vCenter Server: 'VCSA80.test.com (a25c1537-d8c0-4ff4-bd69-09e97a946de0)' has failed. java.util.concurrent.CompletionException: com.vmware.vise.plugin.deploy.PluginDeploymentException: Deployment of plugin: 'com.vcplugin.plugin:1.2.1.27' in vCenter server: 'VCSA80.test.com (a25c1537-d8c0-4ff4-bd69-09e97a946de0)' failed!
Caused by: com.vmware.vise.plugin.deploy.PluginDeploymentException: Deployment of plugin: 'com.vcplugin.plugin:1.2.1.27' in vCenter server: 'VCSA80.test.com (a25c1537-d8c0-4ff4-bd69-09e97a946de0)' failed!
[2023-02-24T13:45:27.493Z] [INFO ] sdk-plugin-deployer-93 com.vmware.vise.plugin.registry.VcExtensionStateRegistry Updating entry: Plugin: 'com.vcplugin.plugin:1.2.1.27', State: 'FAILED_DEPLOY'
[2023-02-24T13:45:27.502Z] [INFO ] plugin-validation1 com.vmware.vise.extensionfw.impl.OsgiUsageValidationService End UnversionedPackageImportCheck on com.vcplugin.plugin:1.2.1.27
[2023-02-24T13:45:27.502Z] [INFO ] plugin-validation1 com.vmware.vise.extensionfw.impl.OsgiUsageValidationService Begin PluginPackageGlobalScopeCheck on com.vcplugin.plugin:1.2.1.27
[2023-02-24T13:45:27.502Z] [INFO ] plugin-validation1 com.vmware.vise.extensionfw.impl.OsgiUsageValidationService End PluginPackageGlobalScopeCheck on com.vcplugin.plugin:1.2.1.27
[2023-02-24T13:45:27.502Z] [INFO ] plugin-validation1 com.vmware.vise.extensionfw.impl.OsgiUsageValidationService Begin PluginDeploymentExceptionCheck on com.vcplugin.plugin:1.2.1.27
[2023-02-24T13:45:27.502Z] [INFO ] plugin-validation1 com.vmware.vise.extensionfw.impl.OsgiUsageValidationService End PluginDeploymentExceptionCheck on com.vcplugin.plugin:1.2.1.27
[2023-02-24T13:45:27.502Z] [INFO ] plugin-validation1 com.vmware.vise.extensionfw.impl.OsgiUsageValidationService Finished validating com.vcplugin.plugin:1.2.1.27
[2023-02-24T13:45:27.519Z] [WARN ] vice-extension-mngr-pool-144 com.vmware.vise.vim.extension.PluginStatusTaskManager Skip posting tasks for plugin package com.vcplugin.plugin:1.2.1.27: No task to update. This has no impact on plugin package deployment but is expected only during plugin development. In Production the plugin package should be registered as a vCenter extension.
[2023-02-24T13:45:27.519Z] [ERROR] plugin-deploy13 com.vmware.vise.plugin.status.LocalPluginStatusServiceImpl DEPLOYMENT_FAILED: Error deploying plugin package com.vcplugin.plugin:1.2.1.27. Reason: Deployment error. java.lang.InterruptedException: null
[2023-02-24T13:45:27.675Z] [ERROR] plugin-deploy13 com.vmware.vise.extensionfw.impl.DeploymentEventLoggerImpl Event=PACKAGE_DEPLOY_FAIL; Params=com.vcplugin.plugin, 1.2.1.27

The second thing is as mentioned in the document https://vdc-repo.vmware.com/vmwb-repository/dcr-public/77dd6491-bb96-47f3-8c2e-a5a4655f078b/b916bc3c-0168-4487-a9e5-c89b50373c18/Local-Plugins-Library-Isolation.pdf the libraries folder should be added in /usr/lib/vmware-vsphere-ui/plugin-packages/<Sample Plugin>. But we don't find any <Sample Plugin> package there.

Note: VCSA80.test.com is just for example. In our environment, it is FQDN and is resolvable.

Best Regards,

Shivam

Re: Failed to deploy a local plugin on vSphere 8.0

ShivamVerma380 — Wed, 08 Mar 2023 10:22:29 GMT

Hi @Denis_Chorbadzh ,

Any input on this? Also is it necessary to have the META-INF folder in the plugin zip file in VCSA 8.0 as we don't have one and we get the following WARNING:

vsphere_client_virgo.log:[2023-03-08T06:17:46.707Z] [WARN ] sdk-plugin-deployer-91 com.vmware.vise.plugin.signature.SignatureStateServiceImpl Detected an invalid signature for plugin 'com.vcplugin.plugin:1.2.1.27'. com.vmware.vise.extensionfw.signing.PluginSignatureException: No META-INF/MANIFEST.MF entry found in the plugin zip file.

We don't have the META-INF folder after the plugin is deployed on VCSA 7.0 but still we are able to successfully deploy the plugin.

Best Regards,

Shivam

Re: Failed to deploy a local plugin on vSphere 8.0

ShivamVerma380 — Tue, 14 Mar 2023 05:17:50 GMT

Hi @Denis_Chorbadzh ,

Please reply to the above questions.

Best Regards,

Shivam Verma

Re: Failed to deploy a local plugin on vSphere 8.0

Denis_Chorbadzh — Thu, 16 Mar 2023 12:49:37 GMT

Hi @ShivamVerma380 ,

The META_INF folder is required for VMware certified plugins. It does not impact your plugin's deployment.

Re: Failed to deploy a local plugin on vSphere 8.0

Denis_Chorbadzh — Thu, 16 Mar 2023 13:47:50 GMT

Hi @ShivamVerma380 ,

Looking at the logs:

This has no impact on plugin package deployment but is expected only during plugin development. In Production the plugin package should be registered as a vCenter extension

I suspect you have placed your plugin in "/usr/lib/vmware-vsphere-ui/plugin-packages/". It should be registered as a vCenter Server extension and not present in the plugin-packages folder.

The "libraries" folder should be place in your plugin package and be a part of the plugin zip file. It will be downloaded by the vSphere Client and unpacked in "/etc/vmware/vsphere-ui/vc-packages/vsphere-client-serenity/".

Best Regards,

Denis

Re: Failed to deploy a local plugin on vSphere 8.0

rajeshd1508 — Mon, 20 Mar 2023 07:33:36 GMT

Hi @Denis_Chorbadzh , thanks for clarifying our queries.

To add to @ShivamVerma380's comment, we are NOT putting anything into /usr/lib/vmware-vsphere-ui/plugin-packages/ directory. The contents of this directory are same before and after our plugin deployment.

root@vcsa8 [ ~ ]# ll /usr/lib/vmware-vsphere-ui/plugin-packages/ total 32K drwxr-xr-x 3 root root 4.0K Mar 14 16:30 cis-data-service-plugin drwxr-xr-x 3 root root 4.0K Mar 14 16:30 license-client drwxr-xr-x 3 root root 4.0K Mar 14 16:30 opsmgmt-client drwxr-xr-x 3 root root 4.0K Mar 14 16:30 root-app drwxr-xr-x 3 root root 4.0K Mar 14 16:30 sr-upload drwxr-xr-x 3 root root 4.0K Mar 14 16:30 sso-admin-ui drwxr-xr-x 3 root root 4.0K Mar 14 16:30 telemetry drwxr-xr-x 3 root root 4.0K Mar 14 16:30 vsphere-client root@vcsa8 [ ~ ]# root@vcsa8 [ ~ ]# ll /usr/lib/vmware-vsphere-ui/plugin-packages/vsphere-client/ total 8.0K -rwxr-xr-x 1 root root 2.9K Jan 11 12:39 plugin-package.xml drwxr-xr-x 2 root root 4.0K Mar 14 16:30 plugins root@vcsa8 [ ~ ]#

As you've mentioned, post installation, the plugin.zip is getting unpacked at /etc/vmware/vsphere-ui/vc-packages/vsphere-client-serenity by VCSA which has /libraries and /plugins directories with respective contents.

root@vcsa8 [ ~ ]# ll /etc/vmware/vsphere-ui/vc-packages/vsphere-client-serenity total 28K drwxr-x--- 3 vsphere-ui users 4.0K Jan 6 00:44 com.vmware.cloud.provider.services.plugin-1.0.0.0 drwxr-x--- 3 vsphere-ui users 4.0K Jan 6 00:56 com.vmware.vsan.client-8.0.200.10000 drwxr-xr-x 3 vsphere-ui users 4.0K Mar 14 16:47 com.vmware.lcm.client-8.0.0.10200 drwxr-xr-x 3 vsphere-ui users 4.0K Mar 14 16:47 com.vmware.vlcm.client-8.0.0.21216066 drwxr-xr-x 2 vsphere-ui users 4.0K Mar 15 05:07 com.example.plugin-1.2.1.27 drwxr-x--- 4 vsphere-ui users 4.0K Mar 15 05:18 com.vmware.vrops.install-8.0.200.10000 drwxr-xr-x 5 vsphere-ui users 4.0K Mar 19 14:22 com.vcplugin.plugin-1.2.1.27 root@vcsa8 [ ~ ]# root@vcsa8 [ ~ ]# ll /etc/vmware/vsphere-ui/vc-packages/vsphere-client-serenity/com.vcplugin.plugin-1.2.1.27/ total 16M -rw-r--r-- 1 vsphere-ui users 16M Mar 19 14:22 com.vcplugin.plugin-1.2.1.27.zip -rw-r--r-- 1 vsphere-ui users 498 Mar 19 14:22 plugin-package.xml drwxr-xr-x 2 vsphere-ui users 4.0K Mar 19 14:22 libraries drwxr-xr-x 2 vsphere-ui users 4.0K Mar 19 14:22 plugins drwxr-xr-x 2 vsphere-ui users 4.0K Mar 19 14:22 META-INF root@vcsa8 [ ~ ]#

Though we can see in https://vcsa80.example.com/mob/?moid=ExtensionManager that plugin is getting registered, the deployment tasks is failing with time out for deploying a bundle.

Snippet from the vsphere_client_virgo.log

[2023-03-19T14:22:30.047Z] [INFO ] plugin-discovery-bus-thread com.vmware.vise.plugin.registry.VcExtensionRegistryImpl Associated plugin com.vcplugin.plugin:1.2.1.27 with VC vcsa8.example.com with guid 90e06f45-343c-4a44-9581-f0ae24277e6a. [2023-03-19T14:22:30.047Z] [INFO ] plugin-discovery-bus-thread com.vmware.vise.plugin.registry.VcExtensionStateRegistry Adding registry entry for plugin: 'com.vcplugin.plugin:1.2.1.27'. [2023-03-19T14:22:30.047Z] [INFO ] plugin-discovery-bus-thread com.vmware.vise.plugin.registry.VcExtensionStateRegistry Registering plugin: 'com.vcplugin.plugin:1.2.1.27' with vCenter: 'vcsa8.example.com (90e06f45-343c-4a44-9581-f0ae24277e6a) [2023-03-19T14:22:30.047Z] [INFO ] plugin-discovery-bus-thread com.vmware.vise.plugin.registry.VcExtensionStateRegistry Checking entry existence: Plugin: 'com.vcplugin.plugin:1.2.1.27', State: '[FAILED_DOWNLOAD]'. [2023-03-19T14:22:30.048Z] [INFO ] plugin-discovery-bus-thread com.vmware.vise.plugin.registry.VcExtensionStateRegistry Updating entry: Plugin: 'com.vcplugin.plugin:1.2.1.27', State: 'DOWNLOADING' [2023-03-19T14:22:30.048Z] [INFO ] sdk-plugin-deployer-173 com.vmware.vise.plugin.async.task.PluginDownloadTask Starting download of plugin package: 'com.vcplugin.plugin:1.2.1.27'. [2023-03-19T14:22:30.229Z] [INFO ] sdk-plugin-deployer-173 com.vmware.vise.vim.extension.PluginStatusTaskManager User: VSPHERE.LOCAL\vsphere-webclient-4ced4907-5f14-4884-89f8-7a206a1598df initiated plugin DOWNLOAD task for plugin com.vcplugin.plugin:1.2.1.27. [2023-03-19T14:22:30.229Z] [INFO ] sdk-plugin-deployer-173 com.vmware.vise.plugin.status.LocalPluginStatusServiceImpl DOWNLOADING: Downloading plugin package com.vcplugin.plugin:1.2.1.27 from https://10.235.139.10:18080/storage/vcplugin-bundle.zip ... [2023-03-19T14:22:30.233Z] [INFO ] sdk-plugin-deployer-173 com.vmware.vise.plugin.download.PluginDownloadServiceImpl Downloading plug-in package 'com.vcplugin.plugin:1.2.1.27' with temporary name '/etc/vmware/vsphere-ui/vc-packages/vsphere-client-serenity/com.vcplugin.plugin-1.2.1.27/plugin-package.tmp'. [2023-03-19T14:22:30.234Z] [INFO ] sdk-plugin-deployer-173 com.vmware.vise.plugin.download.PluginDownloadServiceImpl Downloading plugin package from https://10.235.139.10:18080/storage/vcplugin-bundle.zip [2023-03-19T14:22:31.572Z] [INFO ] sdk-plugin-deployer-173 com.vmware.vise.plugin.download.PluginDownloadServiceImpl Done downloading plugin package 'com.vcplugin.plugin:1.2.1.27' from 'https://10.235.139.10:18080/storage/vcplugin-bundle.zip' [2023-03-19T14:22:31.572Z] [INFO ] sdk-plugin-deployer-173 com.vmware.vise.plugin.download.PluginDownloadServiceImpl Renaming downloaded plug-in '/etc/vmware/vsphere-ui/vc-packages/vsphere-client-serenity/com.vcplugin.plugin-1.2.1.27/plugin-package.tmp' to '/etc/vmware/vsphere-ui/vc-packages/vsphere-client-serenity/com.vcplugin.plugin-1.2.1.27/com.vcplugin.plugin-1.2.1.27.zip' [2023-03-19T14:22:31.572Z] [INFO ] sdk-plugin-deployer-173 com.vmware.vise.plugin.download.PluginDownloadServiceImpl Successfully renamed /etc/vmware/vsphere-ui/vc-packages/vsphere-client-serenity/com.vcplugin.plugin-1.2.1.27/plugin-package.tmp to /etc/vmware/vsphere-ui/vc-packages/vsphere-client-serenity/com.vcplugin.plugin-1.2.1.27/com.vcplugin.plugin-1.2.1.27.zip. [2023-03-19T14:22:31.572Z] [INFO ] sdk-plugin-deployer-173 com.vmware.vise.plugin.download.PluginDownloadServiceImpl Unzipping plug-in '/etc/vmware/vsphere-ui/vc-packages/vsphere-client-serenity/com.vcplugin.plugin-1.2.1.27/com.vcplugin.plugin-1.2.1.27.zip' into '/etc/vmware/vsphere-ui/vc-packages/vsphere-client-serenity/com.vcplugin.plugin-1.2.1.27' folder. [2023-03-19T14:22:31.813Z] [INFO ] sdk-plugin-deployer-173 com.vmware.vise.plugin.download.PluginDownloadServiceImpl Successfully unzipped plug-in '/etc/vmware/vsphere-ui/vc-packages/vsphere-client-serenity/com.vcplugin.plugin-1.2.1.27/com.vcplugin.plugin-1.2.1.27.zip' into '/etc/vmware/vsphere-ui/vc-packages/vsphere-client-serenity/com.vcplugin.plugin-1.2.1.27' folder. [2023-03-19T14:22:31.846Z] [ERROR] sdk-plugin-deployer-173 com.vmware.vise.plugin.signature.SignatureStateServiceImpl Unable to verify the signature for plugin 'com.vcplugin.plugin:1.2.1.27'. com.vmware.vise.extensionfw.signing.PluginSignatureGeneralException: The signature verification process failed. [2023-03-19T14:22:31.846Z] [INFO ] sdk-plugin-deployer-173 com.vmware.vise.plugin.download.PluginDownloadServiceImpl Successfully downloaded plug-in package 'com.vcplugin.plugin:1.2.1.27'. [2023-03-19T14:22:31.881Z] [INFO ] sdk-plugin-deployer-173 com.vmware.vise.plugin.status.LocalPluginStatusServiceImpl DOWNLOADED: Downloaded plugin package com.vcplugin.plugin:1.2.1.27 from https://10.235.139.10:18080/storage/vcplugin-bundle.zip successfully. [2023-03-19T14:22:31.882Z] [INFO ] sdk-plugin-deployer-173 com.vmware.vise.vim.extension.VcExtensionManager Handling successfully downloaded plugin: 'com.vcplugin.plugin:1.2.1.27' registered in vCenter: 'vcsa8.example.com (90e06f45-343c-4a44-9581-f0ae24277e6a)'. [2023-03-19T14:22:31.882Z] [INFO ] sdk-plugin-deployer-173 com.vmware.vise.plugin.registry.VcExtensionStateRegistry Updating entry: Plugin: 'com.vcplugin.plugin:1.2.1.27', State: 'DOWNLOADED' [2023-03-19T14:22:31.882Z] [INFO ] sdk-plugin-deployer-173 com.vmware.vise.plugin.registry.VcExtensionStateRegistry Updating entry: Plugin: 'com.vcplugin.plugin:1.2.1.27', State: 'DEPLOYING' [2023-03-19T14:22:31.882Z] [INFO ] sdk-plugin-deployer-177 com.vmware.vise.plugin.async.task.PluginDeployTask Starting deployment of plugin package: 'com.vcplugin.plugin:1.2.1.27' in vCenter server: 'vcsa8.example.com (90e06f45-343c-4a44-9581-f0ae24277e6a)'. [2023-03-19T14:22:31.923Z] [INFO ] sdk-plugin-deployer-177 com.vmware.vise.vim.extension.PluginStatusTaskManager User: VSPHERE.LOCAL\vsphere-webclient-4ced4907-5f14-4884-89f8-7a206a1598df initiated plugin DEPLOYMENT task for plugin com.vcplugin.plugin:1.2.1.27. [2023-03-19T14:22:31.923Z] [INFO ] sdk-plugin-deployer-177 com.vmware.vise.plugin.status.LocalPluginStatusServiceImpl DEPLOYING: Deploying plugin package com.vcplugin.plugin:1.2.1.27 ... [2023-03-19T14:22:31.928Z] [INFO ] sdk-plugin-deployer-177 com.vmware.vise.extensionfw.impl.sandboxing.EsaBuilder Creating OSGi sandbox for package com.vcplugin.plugin with scope fullyIsolated [2023-03-19T14:22:32.997Z] [INFO ] plugin-deploy16 com.vmware.vise.extensionfw.impl.PackagesDeployer Deploying plugin package 'com.vcplugin.plugin:1.2.1.27'. [2023-03-19T14:22:32.997Z] [INFO ] plugin-deploy16 com.vmware.vise.extensionfw.impl.DeploymentEventLoggerImpl Event=PACKAGE_DEPLOY_BEGIN; Params=com.vcplugin.plugin, 1.2.1.27 [2023-03-19T14:32:32.997Z] [ERROR] sdk-plugin-deployer-177 com.vmware.vise.extensionfw.impl.PackagesDeployer Package deployer timed out waiting for com.vcplugin.plugin:1.2.1.27; [2023-03-19T14:32:33.040Z] [ERROR] sdk-plugin-deployer-177 com.vmware.vise.plugin.status.LocalPluginStatusServiceImpl DEPLOYMENT_FAILED: Error deploying plugin package com.vcplugin.plugin:1.2.1.27. Reason: Plugin deployment has timed out. [2023-03-19T14:32:33.042Z] [ERROR] plugin-deploy16 com.vmware.vise.extensionfw.impl.PackagesDeployer Error deploying one of more bundles for the plugin package /etc/vmware/vsphere-ui/vc-packages/vsphere-client-serenity/com.vcplugin.plugin-1.2.1.27 java.lang.InterruptedException: null [2023-03-19T14:32:33.042Z] [INFO ] plugin-validation1 com.vmware.vise.extensionfw.impl.OsgiUsageValidationService Started validating com.vcplugin.plugin:1.2.1.27 [2023-03-19T14:32:33.042Z] [INFO ] plugin-validation1 com.vmware.vise.extensionfw.impl.OsgiUsageValidationService Begin VmwareNameMisuseCheck on com.vcplugin.plugin:1.2.1.27 [2023-03-19T14:32:33.042Z] [INFO ] plugin-validation1 com.vmware.vise.extensionfw.impl.OsgiUsageValidationService End VmwareNameMisuseCheck on com.vcplugin.plugin:1.2.1.27 [2023-03-19T14:32:33.042Z] [INFO ] plugin-validation1 com.vmware.vise.extensionfw.impl.OsgiUsageValidationService Begin DynamicImportUsageCheck on com.vcplugin.plugin:1.2.1.27 [2023-03-19T14:32:33.042Z] [INFO ] plugin-validation1 com.vmware.vise.extensionfw.impl.OsgiUsageValidationService End DynamicImportUsageCheck on com.vcplugin.plugin:1.2.1.27 [2023-03-19T14:32:33.042Z] [WARN ] plugin-deploy16 com.vmware.vise.vim.extension.PluginStatusTaskManager Skip posting tasks for plugin package com.vcplugin.plugin:1.2.1.27: No task to update. This has no impact on plugin package deployment but is expected only during plugin development. In Production the plugin package should be registered as a vCenter extension. [2023-03-19T14:32:33.043Z] [INFO ] plugin-validation1 com.vmware.vise.extensionfw.impl.OsgiUsageValidationService Begin RequireBundleUsageCheck on com.vcplugin.plugin:1.2.1.27 [2023-03-19T14:32:33.043Z] [INFO ] plugin-validation1 com.vmware.vise.extensionfw.impl.OsgiUsageValidationService End RequireBundleUsageCheck on com.vcplugin.plugin:1.2.1.27 [2023-03-19T14:32:33.043Z] [INFO ] plugin-validation1 com.vmware.vise.extensionfw.impl.OsgiUsageValidationService Begin UnversionedPackageImportCheck on com.vcplugin.plugin:1.2.1.27 [2023-03-19T14:32:33.043Z] [ERROR] plugin-deploy16 com.vmware.vise.plugin.status.LocalPluginStatusServiceImpl DEPLOYMENT_FAILED: Error deploying plugin package com.vcplugin.plugin:1.2.1.27. Reason: Deployment error. java.lang.InterruptedException: null [2023-03-19T14:32:33.043Z] [ERROR] sdk-plugin-deployer-177 com.vmware.vise.vim.extension.VcExtensionManager Deployment of plugin package: 'com.vcplugin.plugin:1.2.1.27' registered in vCenter Server: 'vcsa8.example.com (90e06f45-343c-4a44-9581-f0ae24277e6a)' has failed. java.util.concurrent.CompletionException: com.vmware.vise.plugin.deploy.PluginDeploymentException: Deployment of plugin: 'com.vcplugin.plugin:1.2.1.27' in vCenter server: 'vcsa8.example.com (90e06f45-343c-4a44-9581-f0ae24277e6a)' failed! Caused by: com.vmware.vise.plugin.deploy.PluginDeploymentException: Deployment of plugin: 'com.vcplugin.plugin:1.2.1.27' in vCenter server: 'vcsa8.example.com (90e06f45-343c-4a44-9581-f0ae24277e6a)' failed! [2023-03-19T14:32:33.043Z] [INFO ] plugin-validation1 com.vmware.vise.extensionfw.impl.OsgiUsageValidationService End UnversionedPackageImportCheck on com.vcplugin.plugin:1.2.1.27 [2023-03-19T14:32:33.043Z] [INFO ] sdk-plugin-deployer-177 com.vmware.vise.plugin.registry.VcExtensionStateRegistry Updating entry: Plugin: 'com.vcplugin.plugin:1.2.1.27', State: 'FAILED_DEPLOY' [2023-03-19T14:32:33.043Z] [INFO ] plugin-validation1 com.vmware.vise.extensionfw.impl.OsgiUsageValidationService Begin PluginPackageGlobalScopeCheck on com.vcplugin.plugin:1.2.1.27 [2023-03-19T14:32:33.043Z] [INFO ] plugin-validation1 com.vmware.vise.extensionfw.impl.OsgiUsageValidationService End PluginPackageGlobalScopeCheck on com.vcplugin.plugin:1.2.1.27 [2023-03-19T14:32:33.043Z] [INFO ] plugin-validation1 com.vmware.vise.extensionfw.impl.OsgiUsageValidationService Begin PluginDeploymentExceptionCheck on com.vcplugin.plugin:1.2.1.27 [2023-03-19T14:32:33.043Z] [INFO ] plugin-validation1 com.vmware.vise.extensionfw.impl.OsgiUsageValidationService End PluginDeploymentExceptionCheck on com.vcplugin.plugin:1.2.1.27 [2023-03-19T14:32:33.043Z] [INFO ] plugin-validation1 com.vmware.vise.extensionfw.impl.OsgiUsageValidationService Finished validating com.vcplugin.plugin:1.2.1.27 [2023-03-19T14:32:33.187Z] [ERROR] plugin-deploy16 com.vmware.vise.extensionfw.impl.DeploymentEventLoggerImpl Event=PACKAGE_DEPLOY_FAIL; Params=com.vcplugin.plugin, 1.2.1.27

The eventlog.log for same duration

[2023-03-19T14:22:33.654Z] [INFO ] plugin-deploy16 [BundleEvent] Bundle=org.eclipse.virgo.web.dm:1.0.0 Type=RESOLVED [2023-03-19T14:22:33.655Z] [INFO ] plugin-deploy16 [BundleEvent] Bundle=com.vmware.wsvim25:8.0.0 Type=RESOLVED [2023-03-19T14:22:33.656Z] [INFO ] plugin-deploy16 [BundleEvent] Bundle=OSGIBundle:1.0.0 Type=RESOLVED [2023-03-19T14:22:33.659Z] [INFO ] plugin-deploy16 [BundleEvent] Bundle=org.eclipse.virgo.web.dm:1.0.0 Type=STARTING [2023-03-19T14:22:33.701Z] [INFO ] plugin-deploy16 [BundleEvent] Bundle=org.eclipse.virgo.web.dm:1.0.0 Type=STARTED [2023-03-19T14:22:33.702Z] [INFO ] plugin-deploy16 [BundleEvent] Bundle=OSGIBundle:1.0.0 Type=STARTING [2023-03-19T14:22:33.702Z] [INFO ] plugin-deploy16 [BundleEvent] Bundle=OSGIBundle:1.0.0 Type=STARTED [2023-03-19T14:22:33.703Z] [INFO ] plugin-deploy16 [BundleEvent] Bundle=com.vmware.wsvim25:8.0.0 Type=STARTING [2023-03-19T14:22:33.711Z] [INFO ] plugin-deploy16 [BundleEvent] Bundle=com.vmware.wsvim25:8.0.0 Type=STARTED [2023-03-19T14:32:33.079Z] [INFO ] plugin-deploy16 [BundleEvent] Bundle=com.vmware.wsvim25:8.0.0 Type=STOPPING [2023-03-19T14:32:33.081Z] [INFO ] plugin-deploy16 [BundleEvent] Bundle=com.vmware.wsvim25:8.0.0 Type=STOPPED [2023-03-19T14:32:33.081Z] [INFO ] plugin-deploy16 [BundleEvent] Bundle=OSGIBundle:1.0.0 Type=STOPPING [2023-03-19T14:32:33.082Z] [INFO ] plugin-deploy16 [BundleEvent] Bundle=OSGIBundle:1.0.0 Type=STOPPED [2023-03-19T14:32:33.082Z] [INFO ] plugin-deploy16 [BundleEvent] Bundle=org.eclipse.virgo.web.dm:1.0.0 Type=STOPPING [2023-03-19T14:32:33.083Z] [INFO ] plugin-deploy16 [BundleEvent] Bundle=org.eclipse.virgo.web.dm:1.0.0 Type=STOPPED [2023-03-19T14:32:33.129Z] [INFO ] plugin-deploy16 [BundleEvent] Bundle=OSGIBundle:1.0.0 Type=UNRESOLVED [2023-03-19T14:32:33.133Z] [INFO ] plugin-deploy16 [BundleEvent] Bundle=OSGIBundle:1.0.0 Type=UNINSTALLED [2023-03-19T14:32:33.138Z] [INFO ] plugin-deploy16 [BundleEvent] Bundle=org.eclipse.virgo.web.dm:1.0.0 Type=UNRESOLVED [2023-03-19T14:32:33.156Z] [INFO ] plugin-deploy16 [BundleEvent] Bundle=org.eclipse.virgo.web.dm:1.0.0 Type=UNINSTALLED [2023-03-19T14:32:33.160Z] [INFO ] plugin-deploy16 [BundleEvent] Bundle=com.vmware.wsvim25:8.0.0 Type=UNRESOLVED [2023-03-19T14:32:33.164Z] [INFO ] plugin-deploy16 [BundleEvent] Bundle=com.vmware.wsvim25:8.0.0 Type=UNINSTALLED [2023-03-19T14:32:33.169Z] [INFO ] plugin-deploy16 [BundleEvent] Bundle=org.osgi.service.subsystem.region.context.7:1.0.0 Type=STOPPING [2023-03-19T14:32:33.169Z] [INFO ] plugin-deploy16 [BundleEvent] Bundle=org.osgi.service.subsystem.region.context.7:1.0.0 Type=STOPPED [2023-03-19T14:32:33.170Z] [INFO ] plugin-deploy16 [BundleEvent] Bundle=org.osgi.service.subsystem.region.context.7:1.0.0 Type=UNRESOLVED [2023-03-19T14:32:33.172Z] [INFO ] plugin-deploy16 [BundleEvent] Bundle=org.osgi.service.subsystem.region.context.7:1.0.0 Type=UNINSTALLED

From these logs, it can be seen that the Bundle=com.vmware.wsvim25:8.0.0 is getting STOPPED and plugin deployment is getting timed out around same time, but no other log is showing the cause or reason for this time out. Can you please point us to what can cause this timeout?

Few points to be NOTED:

1. vcsa8.example.com is just for representation purpose. The actual VCSA hostname is resolvable FQDN.
2. Nothing suspicious seen from tcpdump on VCSA.
3. We've created single OSGIBundle:1.0.0 bundle having all the dependencies.

Re: Failed to deploy a local plugin on vSphere 8.0

Denis_Chorbadzh — Tue, 21 Mar 2023 14:11:38 GMT

Hi,

From the provided log snippets, we can't infer what might be the issue.

We would need the VC support bundle and the plugin package in order to investigate.

If you can't provide the whole plugin package due to legal concerns, you can provide a stripped down version or modify one of the local plugin samples in order to reproduce the issue.

Best Regards,

Denis