Errors with vCD Plug-In under vCO 5.5

[kmzimm]

Hello,

I'm attempting to run the "Add a vDC" workflow under the vCD 5.1-558 plugin in vCO 5.5, and I'm receiving the following error:

Could not call sun.net.www.http.KeepAliveCache.writeObject() : Could not call sun.net.www.http.ClientVector.writeObject() : null

---- Debugging information ----

message             : Could not call sun.net.www.http.ClientVector.writeObject()

cause-exception     : java.io.NotSerializableException

cause-message       : null

-------------------------------

message             : Could not call sun.net.www.http.KeepAliveCache.writeObject()

cause-exception     : com.thoughtworks.xstream.converters.ConversionException

cause-message       : Could not call sun.net.www.http.ClientVector.writeObject() : null

-------------------------------

From the server.log:

2013-11-01 00:22:55.826+0000 [WorkflowExecutorPool-Thread-4] WARN  {mike@kmzhomenet.local:Add a vDC:46808080808080808080808080808080A880808001266312685778e677578588f:ff8080814210fce40142110a8b510027} [WorkflowItemTaskRunner] Script execution error on workflow : Add a vDC / 'Create Trigger'(item1) : Could not call sun.net.www.http.KeepAliveCache.writeObject() : Could not call sun.net.www.http.ClientVector.writeObject() : null

---- Debugging information ----

message             : Could not call sun.net.www.http.ClientVector.writeObject()

cause-exception     : java.io.NotSerializableException

cause-message       : null

-------------------------------

message             : Could not call sun.net.www.http.KeepAliveCache.writeObject()

cause-exception     : com.thoughtworks.xstream.converters.ConversionException

cause-message       : Could not call sun.net.www.http.ClientVector.writeObject() : null

It appears to be stopped at the Create Trigger action within the Wait for a task sub workflow. The creation of the vDC actually succeeds in vCD and no errors are thrown there. It worked previously without any change to the workflow as late as yesterday, but has since stopped.

I've done everything I can think of, including reinstalling vCO completely, but no luck. I'd be happy for any ideas.

[alazarov]

The problem is a known issue and I'm currently working on it.

[kmzimm]

Thank you! It seems getting to that step can either trigger that error, or a spin lock that leads to a java heap error. That also seems to lead to a dump that creates .hprof files large enough to fill up the vCO appliance's tiny drive.

I'd be happy to be a guinea pig for a fix if you need one.

[kmzimm]

By way of an update, I tried this again in the 559 build and the same issue is still present.

[alazarov]

Hi,

I tried the following workflows

- Power on a vApp

- Power off a vApp

- Add a vDC

and they ran & finished successfully.

My environment: 

vCO vCD plug-in 5.1 Build #559

vCO 5.5.0 Build #1281930

vCD 5.1.0 Build #810718

I've just closed the bug about this problem it appears to be an upgrade issue, make fresh install of vCO 5.5 and vCO vCD plug-in 5.1 build#559 an everything will work just fine.

Thanks,

Alex

[kmzimm]

Alex,

I just fresh installed the vCO and the plugin, and still ran into this while running Add a vDC:

Could not call sun.net.www.http.KeepAliveCache.writeObject() : Could not call sun.net.www.http.ClientVector.writeObject() : null

---- Debugging information ----

message             : Could not call sun.net.www.http.ClientVector.writeObject()

cause-exception     : java.io.NotSerializableException

cause-message       : null

-------------------------------

message             : Could not call sun.net.www.http.KeepAliveCache.writeObject()

cause-exception     : com.thoughtworks.xstream.converters.ConversionException

cause-message       : Could not call sun.net.www.http.ClientVector.writeObject() : null

-------------------------------

The differential is I'm running it against vCD 5.5, but I'm not sure it will make a difference necessarily. I'll try a 5.1.2 install just in case.

[alazarov]

If you are running against vCD 5.5 I would recommend to use the 5.5 vCO vCD plug-in. It will be available in a couple of days.

[kmzimm]

Alex,

Unfortunately I'm having this same issue with the vCD 5.5 plugin on a fresh install of vCO. Any ideas would be appreciated. When executing the "Add a vDC" workflow, the create trigger action returns this:

Could not call sun.net.www.http.KeepAliveCache.writeObject() : Could not call sun.net.www.http.ClientVector.writeObject() : null

---- Debugging information ----

message             : Could not call sun.net.www.http.ClientVector.writeObject()

cause-exception     : java.io.NotSerializableException

cause-message       : null

-------------------------------

message             : Could not call sun.net.www.http.KeepAliveCache.writeObject()

cause-exception     : com.thoughtworks.xstream.converters.ConversionException

cause-message       : Could not call sun.net.www.http.ClientVector.writeObject() : null

-------------------------------

The vDC creation completes normally.

Components:

vCO Appliance 5.5.0 build 1281930

vCloud Director Plugin 5.5.0.1421257

vCD 5.5.0.1323688

Not sure if it's relevant, but the connection type is Shared Session with an admin user logged into the System org.

logs:

2013-11-20 21:19:38.919+0000 [http-bio-0.0.0.0-8281-exec-5] INFO  {} [Execution] Executing workflow 'Add a vDC'

2013-11-20 21:19:38.934+0000 [org.springframework.jms.listener.DefaultMessageListenerContainer#0-2] INFO  {} [Execution] Invoking workflow handler.ff8080814277558101427761f9ad0013

2013-11-20 21:19:42.497+0000 [pool-13-thread-2] INFO  {} [VCloudPluginNotificationMediatorTask] Operation: Creating Virtual Datacenter ytrytt2(331db463-9bd6-461c-ae0b-ec8075d5c5ea) on host https://vcd.lookee.org:443 (admin@System) is in state running [task: https://vcd.lookee.org/api/task/d3da5627-f19c-44c7-b836-f4ead0eab03e, progress: null %]

2013-11-20 21:19:43.301+0000 [WorkflowExecutorPool-Thread-3] WARN  {mike@kmzhomenet.local:Add a vDC:46808080808080808080808080808080A880808001266312685778e677578588f:ff8080814277558101427761f9ad0013} [WorkflowItemTaskRunner] Script execution error on workflow : Add a vDC / 'Create Trigger'(item1) : Could not call sun.net.www.http.KeepAliveCache.writeObject() : Could not call sun.net.www.http.ClientVector.writeObject() : null

---- Debugging information ----

message             : Could not call sun.net.www.http.ClientVector.writeObject()

cause-exception     : java.io.NotSerializableException

cause-message       : null

-------------------------------

message             : Could not call sun.net.www.http.KeepAliveCache.writeObject()

cause-exception     : com.thoughtworks.xstream.converters.ConversionException

cause-message       : Could not call sun.net.www.http.ClientVector.writeObject() : null

-------------------------------

2013-11-20 21:19:44.580+0000 [pool-13-thread-2] INFO  {} [VCloudPluginNotificationMediatorTask] Operation: Creating Virtual Datacenter ytrytt2(331db463-9bd6-461c-ae0b-ec8075d5c5ea) on host https://vcd.lookee.org:443 (admin@System) is in state running [task: https://vcd.lookee.org/api/task/d3da5627-f19c-44c7-b836-f4ead0eab03e, progress: null %]

2013-11-20 21:19:49.614+0000 [pool-13-thread-2] INFO  {} [VCloudPluginNotificationMediatorTask] Operation: Creating Virtual Datacenter ytrytt2(331db463-9bd6-461c-ae0b-ec8075d5c5ea) on host https://vcd.lookee.org:443 (admin@System) is in state running [task: https://vcd.lookee.org/api/task/d3da5627-f19c-44c7-b836-f4ead0eab03e, progress: null %]

2013-11-20 21:19:57.646+0000 [pool-13-thread-2] INFO  {} [VCloudPluginNotificationMediatorTask] Operation: Created Virtual Datacenter ytrytt2(331db463-9bd6-461c-ae0b-ec8075d5c5ea) on host https://vcd.lookee.org:443 (admin@System) is in state success [task: https://vcd.lookee.org/api/task/d3da5627-f19c-44c7-b836-f4ead0eab03e, progress: 100 %]

Thanks.

[alazarov]

Hi,

We are working with the same environment as yours and still we haven't hit such an issue.

I'm suggesting you to contact our GSS and open a SR, after that will make a WebEx to investigate further.

I'm also very curious what's going wrong on your site!

Regards

[kmzimm]

Alex,

     I've opened up case 13411042012 for this. Just FYI, this is happening in both my lab environment and a PoC environment set up for our use case with vCO, so it doesn't appear to be isolated at this point.

[MatthiasGallant]

Hi,

after an upgrade from 5.1 to 5.5 we have now:

vCO 5.5.0 build 1281930

vCloud Director (plugin) 5.5.0.1421257

vCD 5.5.0.1323688

We are experiencing the same exact error.

Can you post the solution if/when it comes along ?

Or should I also open a case ?

Regards

[kmzimm]

Unfortunately I haven't heard back either here or through the ticket. I'd suggest you open one as well for visibility.

[Stephan84]

Same Problem here. New vCenter Orchestrator installation / appliance, vCloud 5.1 Upgraded to 5.5, Workflow: "Instantiate a vApp template", Shared Session.

Regards

*edit*

Also upgrading hw version to 10 and upgrading VMware Tools of the VMs did not solve the problem.

[trodemaster]

Any updates on this? We have a new build of vCenter 5.5 with vCO and vCloud 5.5 that is throwing this error. Thanks, Blake

[kmzimm]

I've been going through the issue with the devs via a ticket for a couple weeks now. They've collected extra logs from me via a debug build, and I'm waiting to see what they come up with. Fingers crossed.

[Stephan84]

My issue was solved by vmware support (new vcloud plugin - no official release I think). VMware request number: 13417940112

[trodemaster]

I have also been testing newer builds of the plugin. With version 5.5.0.1528081 I'm getting a java heap error still. Is this the version you tested?

[trodemaster]

Revisited this one and after restarting services again the workflow now runs without issue.

[Stephan84]

Hi, I got version 5.5.0.1528081 too, witch solved my problem.

[Slymsoft]

I still have this issue with latest vCD plugin provided by VMware support. When running Add a vDC or Add a Catalog it hangs forever at "Waiting event".

The elements are created but vCO is not able to refresh the status of creation tasks (it looks like the trigger never triggers).

My setup :

• vCO Appliance 5.5.1.1617127
• vCD plugin 5.5.0.1528081
• vCD 5.5.1.1654672

I already tried "reset current version" to reinstall plugins. Appliance has been rebooted several times.

Is there anything I can do to fix this bug ?

Help would be greatly appreciated :smileycry:

Here is a screenshot :