vCenter Crashes After Installing NetApp VASA 6.2

[IT_Jay]

I am having an issue with vCenter crashing after I try to access the NetApp VASA 6.2 plugin which is part of the NetApp Virtual Storage Console 6.2 product. The NetApp VSC was working fine for months before I needed to use the VASA provider. This is for a new SRM setup.

When vCenter crashes it's creating java dump files 'java_pid25593.hprof' in /storage/log which is filling up causing vCenter to stop logging until space is cleared

In the web client logs /var/log/vmware/vsphere-client/logs/vsphere_client_virgo.log I see Java out of memory errors javaerrors javaerrors2

Errors snippets

[ERROR] http-bio-9090-exec-2         70000003 100001 ###### com.vmware.vise.util.session.SessionUtil                          Tried to associate a session to an invalid clientId. (sessionId 100001, clientId null, the actual clientId is not null, the new clientId doesn't exist (no history data available))

[WARN ] http-bio-9090-exec-5         70000809 100001 200008 org.springframework.flex.core.DefaultExceptionLogger              The following exception occurred during request processing by the BlazeDS MessageBroker and will be serialized back to the client:  flex.messaging.MessageException: java.lang.OutOfMemoryError : PermGen space

[WARN ] data-service-pool-2237       70000809 100001 200008 com.vmware.vise.vim.commons.vcservice.impl.VcServiceImpl          Error while invoking dynamic proxy for https://test-vCenter-01.company.lan:443/sdk (guid 55df8b0a-924e-40cd-967a-942b5cea27ca, id 3000008, username null, sessionID 100001, clientId 200008) PermGen space

[2016-06-08T15:48:15.235Z] [WARN ] http-bio-9090-exec-5         70000809 100001 200008 com.vmware.vise.data.query.profiling.ProfilingUtil                The Data Service detected slow execution:

The query batch execution took too long: 19994 milliseconds.

I've trying uninstall the VSC from the windows server along with removing the plugins from vCenter  /etc/vmware/vsphere-client/vc-packages/vsphere-client-serenity rm -rf com.netapp* and reinstalling everything from scratch. Tried registering the VSC plugin with administrator@vsphere.local and other domain vpshere administrator accounts. Tried restarting all servers and appliances nothing has worked.

I have the same setup in the DR site and all of the plugins and sofware below work just fine. The only real difference is the PSC is external at the problem site HQ. There are a few KB docs that talk about external PSC's causing authentication issues. Any extra help would be greatly appreciated as this issue is preventing me setting our DR site and VMware support is not providing as much assistance  I need since this is not a production down issue. Thanks in advance.

Software versions

vCenter 6.0 Update 2 linux appliance (external PSC)

ESXi 6.0 Update 1

VMware Site Recovery Manager 6.1

NetApp Clustered Mode 8.3.1

NetApp SRA 3.0 for Clustered Mode

NetApp VASA 6.2 (issue occurs after this plugin is installed)

NetApp Virtual Storage Console 6.2P1

[Bleeder]

I've had a similar issue with other web client plugins causing the web client to crash (sometimes not immediately, but over time).  Adding RAM to vCenter may help mask the issue for a while, but I've seen bad web client plugins crash the web client in less than a day even with 32GB assigned to the vCenter server/appliance.

It might help if VMware didn't leave the creation of dump/hlog files enabled as the default, or at least performed some sort of automatic cleanup of the dump/hlog files.  As you stated, this causes /storage/log to fill and eventually bring vCenter to a halt.

PS - Hopefully you are using the 6.2 P2 release of VSC (http://mysupport.netapp.com/NOW/download/software/vsc_win/6.2P2/)

[GBartsch]

So,

To add to this:  I'm currently working at a client site and we are having similar symptoms.

- VPXD (Virtual Center Server Service) crashes

- Core dumps start appearing in the /storage/core mount point on the vCenter Appliance

- 503 Errors begin to occure attempting to connect to the Web Client or thorugh vRO

The only two extensions installed on the vCenter Appliance are the NetApp VSC 6.2.1P1 and Update Manager.

vCenter is patched to 6.0U3b.  (We have two with the issue, and one was updated to 6U3b and the other was clean installed to 6U3b.)

We do have the LogInsight and EPOPs agents installed, but they don't directly installed into the vCenter (I believe).

Has anyone else had issues with the NetApp Virtual Storage Console (VSC) and vCenter 6.0U3b?

GB