Any ideas?
vCenter is running on a Windows 2008 Std x64 box with 32GB of memory and 4vCPUs. It is a virtual machine. We are running vCenter version 5.5 1891313.
We are experiencing an issue where the VMware vSphere Profile-Driven Storage service is crashing. It started after we rebooted the vCenter server two days ago. The only change we made to the system was installing the NetApp Virtual Storage Console plug-in on the vCenter server. I've since removed that. The event log states "The VMware vSphere Profile-Driven Storage Service service terminated with service-specific error 1 (0x1)." Attempting to vmotion a server causes the error, "A general system error occurred: Invalid response code: 503 service unavailable". VMware's suggestion is to run a hotfix for Windows 2008 (non-R2), Windows6.0-KB950319-x64. This is referenced in one of VMware's KB articles. This hotfix is not applicable as is rolled up in a service pack which is already applied. If I attempt to run wrapper.exe using the wrapper.conf for the service, the output is as follows:
wrapper | --> Wrapper Started as Console
wrapper | Java Service Wrapper Professional Edition 64-bit 3.4.0
wrapper | Copyright (C) 1999-2010 Tanuki Software, Ltd. All Rights Reserved.
wrapper | http://wrapper.tanukisoftware.org
wrapper | Licensed to VMware Global, Inc. for VMware vSphere Profile-Driven Storage
wrapper |
wrapper | Launching a JVM...
jvm 1 | WrapperManager: Initializing...
jvm 1 | WrapperManager: Port 31300 already in use, using port 31301 instead.
jvm 1 | Mar 10, 2015 1:28:53 PM org.apache.catalina.core.AprLifecycleListener init
jvm 1 | INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: ../lib
jvm 1 | Mar 10, 2015 1:28:53 PM org.apache.coyote.AbstractProtocol init
jvm 1 | INFO: Initializing ProtocolHandler ["http-bio-0.0.0.0-8200"]
jvm 1 | Mar 10, 2015 1:28:53 PM org.apache.coyote.AbstractProtocol init
jvm 1 | INFO: Initializing ProtocolHandler ["http-bio-0.0.0.0-8201"]
jvm 1 | Mar 10, 2015 1:28:53 PM org.apache.catalina.core.StandardService startInternal
jvm 1 | INFO: Starting service Tomcat
jvm 1 | Mar 10, 2015 1:28:53 PM org.apache.catalina.core.StandardEngine startInternal
jvm 1 | INFO: Starting Servlet Engine: VMware vFabric tc Runtime 2.8.1.RELEASE/7.0.32.B.RELEASE
jvm 1 | Mar 10, 2015 1:28:53 PM org.apache.catalina.startup.DigesterFactory register
jvm 1 | WARNING: Could not get url for /javax/servlet/jsp/resources/jsp_2_0.xsd
jvm 1 | Mar 10, 2015 1:28:53 PM org.apache.catalina.startup.DigesterFactory register
jvm 1 | WARNING: Could not get url for /javax/servlet/jsp/resources/jsp_2_1.xsd
jvm 1 | Mar 10, 2015 1:28:53 PM org.apache.catalina.startup.DigesterFactory register
jvm 1 | WARNING: Could not get url for /javax/servlet/jsp/resources/jsp_2_2.xsd
jvm 1 | Mar 10, 2015 1:28:53 PM org.apache.catalina.startup.DigesterFactory register
jvm 1 | WARNING: Could not get url for /javax/servlet/jsp/resources/web-jsptaglibrary_1_1.dtd
jvm 1 | Mar 10, 2015 1:28:53 PM org.apache.catalina.startup.DigesterFactory register
jvm 1 | WARNING: Could not get url for /javax/servlet/jsp/resources/web-jsptaglibrary_1_2.dtd
jvm 1 | Mar 10, 2015 1:28:53 PM org.apache.catalina.startup.DigesterFactory register
jvm 1 | WARNING: Could not get url for /javax/servlet/jsp/resources/web-jsptaglibrary_2_0.xsd
jvm 1 | Mar 10, 2015 1:28:53 PM org.apache.catalina.startup.DigesterFactory register
jvm 1 | WARNING: Could not get url for /javax/servlet/jsp/resources/web-jsptaglibrary_2_1.xsd
jvm 1 | Mar 10, 2015 1:28:54 PM org.apache.coyote.AbstractProtocol start
jvm 1 | INFO: Starting ProtocolHandler ["http-bio-0.0.0.0-8200"]
jvm 1 | Mar 10, 2015 1:28:54 PM org.apache.coyote.AbstractProtocol start
jvm 1 | INFO: Starting ProtocolHandler ["http-bio-0.0.0.0-8201"]
jvm 1 | Mar 10, 2015 1:28:55 PM org.apache.coyote.AbstractProtocol init
jvm 1 | INFO: Initializing ProtocolHandler ["http-bio-0.0.0.0-32010"]
jvm 1 | Mar 10, 2015 1:28:55 PM org.apache.coyote.AbstractProtocol init
jvm 1 | INFO: Initializing ProtocolHandler ["http-bio-0.0.0.0-32100"]
jvm 1 | Mar 10, 2015 1:28:55 PM org.apache.catalina.core.StandardService startInternal
jvm 1 | INFO: Starting service Tomcat
jvm 1 | Mar 10, 2015 1:28:55 PM org.apache.catalina.core.StandardEngine startInternal
jvm 1 | INFO: Starting Servlet Engine: VMware vFabric tc Runtime 2.8.1.RELEASE/7.0.32.B.RELEASE
jvm 1 | Mar 10, 2015 1:28:55 PM org.apache.coyote.AbstractProtocol start
jvm 1 | INFO: Starting ProtocolHandler ["http-bio-0.0.0.0-32010"]
jvm 1 | Mar 10, 2015 1:28:55 PM org.apache.coyote.AbstractProtocol start
jvm 1 | INFO: Starting ProtocolHandler ["http-bio-0.0.0.0-32100"]
jvm 1 | 13:28:56 ERROR opId= - Failed to register SPS into QS:com.vmware.vim.binding.dataservice.fault.AlreadyExistsFault:
jvm 1 |
jvm 1 | inherited from com.vmware.vim.binding.dataservice.fault.AlreadyExistsFault
jvm 1 | 13:28:57 ERROR opId= - Failed to add XML document to QS:com.vmware.vim.query.client.exception.ValidationException: Got status code: 400 reason: Bad Request
jvm 1 | 13:28:57 ERROR opId= - Storage Policy Service could not be initialized: com.vmware.sps.fault.PersistenceException: Persistence failed with exception
jvm 1 | WrapperSimpleApp:
jvm 1 | WrapperSimpleApp: Encountered an error running main:
jvm 1 | WrapperSimpleApp: com.vmware.sps.fault.SpsInitializedException: Storage Policy Service could not be initialized.
jvm 1 |
jvm 1 | WrapperSimpleApp: at com.vmware.sps.SpsLocalService.run(Unknown Source)
jvm 1 | WrapperSimpleApp: at com.vmware.sps.StorageMain.main(Unknown Source)
jvm 1 | WrapperSimpleApp: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
jvm 1 | WrapperSimpleApp: at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
jvm 1 | WrapperSimpleApp: at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
jvm 1 | WrapperSimpleApp: at java.lang.reflect.Method.invoke(Unknown Source)
jvm 1 | WrapperSimpleApp: at org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:244)
jvm 1 | WrapperSimpleApp: at java.lang.Thread.run(Unknown Source)
jvm 1 | WrapperSimpleApp: Caused by: com.vmware.sps.fault.PersistenceException: Persistence failed with exception
jvm 1 | WrapperSimpleApp: at com.vmware.sps.qs.InventoryServiceImpl.persistEntityAsString(Unknown Source)
jvm 1 | WrapperSimpleApp: at com.vmware.sps.qs.InventoryServiceImpl.persistEntity(Unknown Source)
jvm 1 | WrapperSimpleApp: at com.vmware.sps.datastore.StorageCapabilityServiceImpl.createSystemLabelCapability(Unknown Source)
jvm 1 | WrapperSimpleApp: at com.vmware.sps.datastore.StorageCapabilityServiceImpl.createSystemLabelCapability(Unknown Source)
jvm 1 | WrapperSimpleApp: at com.vmware.sps.datastore.StorageCapabilityServiceImpl.initializeSystemCapability(Unknown Source)
jvm 1 | WrapperSimpleApp: at com.vmware.sps.datastore.StorageCapabilityServiceImpl.initializeService(Unknown Source)
jvm 1 | WrapperSimpleApp: ... 8 more
jvm 1 | WrapperSimpleApp: Caused by: com.vmware.sps.fault.QsQueryException: Adding XML document to query service xDB failed:
jvm 1 | WrapperSimpleApp: at com.vmware.sps.qs.SpsQsProvider.addContent(Unknown Source)
jvm 1 | WrapperSimpleApp: ... 14 more
jvm 1 | WrapperSimpleApp: Caused by: com.vmware.vim.query.client.exception.ValidationException: Got status code: 400 reason: Bad Request
jvm 1 | WrapperSimpleApp: at com.vmware.vim.query.client.impl.ValidationHelper.validateResponseCode(ValidationHelper.java:29)
jvm 1 | WrapperSimpleApp: at com.vmware.vim.query.client.impl.HttpUtil.executeWithNoResponse(HttpUtil.java:65)
jvm 1 | WrapperSimpleApp: at com.vmware.vim.query.client.provider.impl.AtomFeedPosterImpl.postFeedImpl(AtomFeedPosterImpl.java:76)
jvm 1 | WrapperSimpleApp: at com.vmware.vim.query.client.provider.impl.AtomFeedPosterImpl.access$000(AtomFeedPosterImpl.java:28)
jvm 1 | WrapperSimpleApp: at com.vmware.vim.query.client.provider.impl.AtomFeedPosterImpl$1.doAction(AtomFeedPosterImpl.java:44)
jvm 1 | WrapperSimpleApp: at com.vmware.vim.query.client.provider.impl.AtomFeedPosterImpl$1.doAction(AtomFeedPosterImpl.java:41)
jvm 1 | WrapperSimpleApp: at com.vmware.vim.query.client.impl.RetryAction.execute(RetryAction.java:33)
jvm 1 | WrapperSimpleApp: at com.vmware.vim.query.client.provider.impl.AtomFeedPosterImpl.postFeed(AtomFeedPosterImpl.java:41)
jvm 1 | WrapperSimpleApp: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
jvm 1 | WrapperSimpleApp: at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
jvm 1 | WrapperSimpleApp: at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
jvm 1 | WrapperSimpleApp: at java.lang.reflect.Method.invoke(Unknown Source)
jvm 1 | WrapperSimpleApp: at com.vmware.sps.util.impl.ServiceProxyFactory$ProxyInvocationHandler.invokeMethod(Unknown Source)
jvm 1 | WrapperSimpleApp: at com.vmware.sps.util.impl.ServiceProxyFactory$ProxyInvocationHandler.invokeMethod(Unknown Source)
jvm 1 | WrapperSimpleApp: at com.vmware.sps.util.impl.ServiceProxyFactory$ProxyInvocationHandler.invoke(Unknown Source)
jvm 1 | WrapperSimpleApp: at com.sun.proxy.$Proxy41.postFeed(Unknown Source)
jvm 1 | WrapperSimpleApp: at com.vmware.sps.qs.SpsQsProvider.createAtomFeed(Unknown Source)
jvm 1 | WrapperSimpleApp: ... 15 more
wrapper | <-- Wrapper Stopped
