We've been running the GA release for a week, today started to get the error "could not execute query - No more data to read from socket" when browsing the VCD console, e.g. right click the Provider vDC, the popup windows has the error bar with no content populated. Stack trace in vcloud-container-debug.log is attached.
Checked Oracle, the system and sysaux tablespaces are already 99% full, so added more space and rebooted Oracle, still the same error. Every time the error happens, there is an ORA-07445 on Oracle side and a trace file under udump.
Oracle is setup using default parameters. How can we fix this?
###########
/u01/app/oracle/admin/vCloudDB/udump/vclouddb_ora_21261.trc
###########
Oracle Database 10g Enterprise Edition Release 10.2.0.5.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
ORACLE_HOME = /u01/app/oracle/product/10.2.0/db_1
System name: Linux
....
2010-09-09 11:18:23.502
ACTION NAME:() 2010-09-09 11:18:23.502
MODULE NAME:(JDBC Thin Client) 2010-09-09 11:18:23.502
SERVICE NAME:(vCloudDB.xyz.com) 2010-09-09 11:18:23.502
SESSION ID:(132.402) 2010-09-09 11:18:23.502
Exception signal: 11 (SIGSEGV), code: 1 (Address not mapped to object), addr: 0x18, PC:
2010-09-09 11:18:23.511
ksedmp: internal or fatal error
ORA-07445: exception encountered: core dump [] []
Current SQL statement for this session:
/* Method: RclService.getProviderVdcSpec */ select this_.id as id172_0_, this_.cpu_allocation_mhz as cpu2_172_0_, this_.cpu_overhead_mhz as cpu3_172_0_, this_.cpu_total_mhz as cpu4_172_0_, this_.cpu_used_mhz as cpu5_172_0_, this_.mem_allocation_mb as mem6_172_0_, this_.mem_overhead_mb as mem7_172_0_, this_.mem_total_mb as mem8_172_0_, this_.mem_used_mb as mem9_172_0_, this_.storage_allocation_mb as storage10_172_0_, this_.storage_overhead_mb as storage11_172_0_, this_.storage_total_mb as storage12_172_0_, this_.storage_used_mb as storage13_172_0_ from provider_vdc_metrics_view this_ where this_.id=:1
-
Call Stack Trace -
calling call entry argument values in hex
location type point (? means dubious value)
-
-
-
-
ssd_unwind_bp: unhandled instruction at 0x2fb3962 instr=f
ksedst()+31 call ksedst1() 000000000 ? 000000001 ?
2B5429D60D50 ? 2B5429D60DB0 ?
2B5429D60CF0 ? 000000000 ?
ksedmp()+610 call ksedst() 000000000 ? 000000001 ?
2B5429D60D50 ? 2B5429D60DB0 ?
2B5429D60CF0 ? 000000000 ?
ssexhd()+629 call ksedmp() 000000003 ? 000000001 ?
2B5429D60D50 ? 2B5429D60DB0 ?
2B5429D60CF0 ? 000000000 ?
#########
vcloud-container-debug.log
#########
2010-09-09 11:18:18,135 | DEBUG | pool-jetty-22 | QueryServiceImpl | Getting count for: providerVdcQueryList |
2010-09-09 11:18:18,135 | DEBUG | pool-jetty-22 | AuthorizationMethodInterceptor | Authorizing method: public abstract void com.vmware.vcloud.management.query.GenericSecureQueryDao.checkAuthorization(com.vmware.vcloud.api.presentation.entity.query.QuerySpec). |
2010-09-09 11:18:18,136 | DEBUG | pool-jetty-22 | AuthorizationMethodInterceptor | Authorizing method: public abstract long com.vmware.vcloud.management.query.GenericQueryDao.getCount(com.vmware.vcloud.api.presentation.entity.query.QuerySpec). |
2010-09-09 11:18:18,151 | DEBUG | pool-jetty-0 | QueryServiceImpl | Result: 1 rows (com.vmware.vcloud.api.presentation.entity.query.ProviderVdcQueryListData) |
2010-09-09 11:18:21,421 | DEBUG | pool-jetty-23 | AuthorizationMethodInterceptor | Authorizing method: public abstract com.vmware.vcloud.api.presentation.entity.rcl.ProviderVdcSpec com.vmware.vcloud.api.presentation.service.RclService.getProviderVdcSpec(com.vmware.vcloud.api.presentation.entity.common.EntityRef) throws com.vmware.vcloud.api.presentation.service.PresentationException. |
2010-09-09 11:18:21,441 | DEBUG | Quartz-pool-1-thread-10 | FailoverServiceManagerImpl | checking for any failed cells from current cell vCloud.mdc.cginet (id = 1, uuid = ceaf7a20-209c-44b6-b3f8-14b495afde71) |
2010-09-09 11:18:22,979 | ERROR | pool-jetty-23 | JDBCExceptionReporter | No more data to read from socket |
2010-09-09 11:18:22,980 | DEBUG | pool-jetty-23 | BatchTask | Error executing backend call. |
org.hibernate.exception.JDBCConnectionException: could not execute query
at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:97)
at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:66)
at org.hibernate.loader.Loader.doList(Loader.java:2231)
at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2125)
at org.hibernate.loader.Loader.list(Loader.java:2120)
at org.hibernate.loader.criteria.CriteriaLoader.list(CriteriaLoader.java:118)
at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1596)
at org.hibernate.impl.CriteriaImpl.list(CriteriaImpl.java:306)
at org.hibernate.impl.CriteriaImpl.uniqueResult(CriteriaImpl.java:328)
at com.vmware.vcloud.common.persist.CriteriaProxy.uniqueResult(CriteriaProxy.java:58)
at com.vmware.ssdc.backend.dao.impl.HibernateProviderVdcDao.getProviderVdcMetrics(HibernateProviderVdcDao.java:123)
at com.vmware.ssdc.backend.services.impl.ProviderVdcManagerImpl.getProviderVdcMetricsView(ProviderVdcManagerImpl.java:220)
at com.vmware.ssdc.backend.services.impl.ProviderVdcManagerImpl.loadProviderVdcSpec(ProviderVdcManagerImpl.java:195)
at com.vmware.vcloud.rcl.impl.RclServiceImpl.getProviderVdcSpec(RclServiceImpl.java:1049)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:50)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:160)
at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:50)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:160)
at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:50)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:77)
at com.vmware.ssdc.backend.annotation.PresentationApiDiagnosticsInterceptor.aroundSecurePresentationLayerMethod(PresentationApiDiagnosticsInterceptor.java:38)
at sun.reflect.GeneratedMethodAccessor122.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:627)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:616)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:64)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:160)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy211.getProviderVdcSpec(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.vmware.vcloud.transport.batch.impl.BatchTask.call(BatchTask.java:77)
at com.vmware.vcloud.transport.batch.impl.BatchServiceImpl.invoke(BatchServiceImpl.java:109)
at sun.reflect.GeneratedMethodAccessor380.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.osgi.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:64)
at org.springframework.osgi.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:78)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at org.springframework.osgi.service.util.internal.aop.ServiceTCCLInterceptor.invokeUnprivileged(ServiceTCCLInterceptor.java:57)
at org.springframework.osgi.service.util.internal.aop.ServiceTCCLInterceptor.invoke(ServiceTCCLInterceptor.java:40)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at org.springframework.osgi.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:59)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy294.invoke(Unknown Source)
at sun.reflect.GeneratedMethodAccessor377.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at flex.messaging.services.remoting.adapters.JavaAdapter.invoke(JavaAdapter.java:406)
at flex.messaging.services.RemotingService.serviceMessage(RemotingService.java:183)
at flex.messaging.MessageBroker.routeMessageToService(MessageBroker.java:1417)
at flex.messaging.endpoints.AbstractEndpoint.serviceMessage(AbstractEndpoint.java:878)
at flex.messaging.endpoints.amf.MessageBrokerFilter.invoke(MessageBrokerFilter.java:121)
at flex.messaging.endpoints.amf.LegacyFilter.invoke(LegacyFilter.java:158)
at flex.messaging.endpoints.amf.SessionFilter.invoke(SessionFilter.java:49)
at flex.messaging.endpoints.amf.BatchProcessFilter.invoke(BatchProcessFilter.java:67)
at flex.messaging.endpoints.amf.SerializationFilter.invoke(SerializationFilter.java:146)
at flex.messaging.endpoints.BaseHTTPEndpoint.service(BaseHTTPEndpoint.java:274)
at flex.messaging.MessageBrokerServlet.service(MessageBrokerServlet.java:377)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:806)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
at org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:334)
at org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:126)
at com.vmware.vcloud.flex.AMFServlet.forward(AMFServlet.java:70)
at com.vmware.vcloud.flex.AMFServlet.doPost(AMFServlet.java:56)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:713)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:806)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1124)
at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:378)
at com.vmware.vcloud.security.filters.ValidityExceptionFilter.doFilterHttp(ValidityExceptionFilter.java:50)
at org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)
at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)
at com.vmware.vcloud.web.AuthenticationFilter$1.run(AuthenticationFilter.java:133)
at com.vmware.vcloud.web.AuthenticationFilter$1.run(AuthenticationFilter.java:121)
at com.vmware.vcloud.common.threadpool.ThreadContextExecutor.execute(ThreadContextExecutor.java:46)
at com.vmware.vcloud.web.AuthenticationFilter.doFilterHttp(AuthenticationFilter.java:145)
at org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)
at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)
at com.vmware.vcloud.security.filters.SizeValidationFilter.doFilterHttp(SizeValidationFilter.java:29)
at org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)
at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)
at org.springframework.security.util.FilterChainProxy.doFilter(FilterChainProxy.java:175)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:236)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1115)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:361)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:324)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:879)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:741)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
at org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:636)
at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:451)
Caused by: java.sql.SQLRecoverableException: No more data to read from socket
at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1200)
at oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1155)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:279)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:186)
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:521)
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:205)
at oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:861)
at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1145)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1259)
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3449)
at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3493)
at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1491)
at org.hibernate.jdbc.AbstractBatcher.getResultSet(AbstractBatcher.java:208)
at org.hibernate.loader.Loader.getResultSet(Loader.java:1808)
at org.hibernate.loader.Loader.doQuery(Loader.java:697)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:259)
at org.hibernate.loader.Loader.doList(Loader.java:2228)
... 125 more
2010-09-09 11:18:23,032 | ERROR | pool-jetty-22 | JDBCExceptionReporter | No more data to read from socket |
2010-09-09 11:18:23,032 | DEBUG | pool-jetty-22 | ConfigurationServiceImpl | Error getting configuration property restapi.security.request.size_limit |
com.vmware.vcloud.common.configuration.ConfigurationException: org.hibernate.exception.JDBCConnectionException: could not execute query
at com.vmware.vcloud.common.configuration.impl.HibernateDatabaseConfiguration.getProperty(HibernateDatabaseConfiguration.java:365)
at com.vmware.vcloud.common.configuration.impl.CombinedConfiguration.getProperty(CombinedConfiguration.java:105)
at org.apache.commons.configuration.AbstractConfiguration.resolveContainerStore(AbstractConfiguration.java:1160)
at org.apache.commons.configuration.AbstractConfiguration.getInteger(AbstractConfiguration.java:831)
at org.apache.commons.configuration.AbstractConfiguration.getInt(AbstractConfiguration.java:806)
at com.vmware.vcloud.common.configuration.impl.ConfigurationServiceImpl.getInteger(ConfigurationServiceImpl.java:219)
at sun.reflect.GeneratedMethodAccessor104.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.osgi.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:64)
at org.springframework.osgi.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:78)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at org.springframework.osgi.service.util.internal.aop.ServiceTCCLInterceptor.invokeUnprivileged(ServiceTCCLInterceptor.java:57)
at org.springframework.osgi.service.util.internal.aop.ServiceTCCLInterceptor.invoke(ServiceTCCLInterceptor.java:40)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at org.springframework.osgi.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:59)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy296.getInteger(Unknown Source)
at com.vmware.vcloud.security.filters.ValidationFilter.getRequestSizeLimit(ValidationFilter.java:124)
at com.vmware.vcloud.security.filters.SizeValidationFilter.doFilterHttp(SizeValidationFilter.java:23)
at org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)
at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)
at org.springframework.security.util.FilterChainProxy.doFilter(FilterChainProxy.java:175)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:236)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1115)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:361)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:324)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:879)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:741)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:213)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
at org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:636)
at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:451)
Caused by: org.hibernate.exception.JDBCConnectionException: could not execute query
at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:97)
at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:66)
at org.hibernate.loader.Loader.doList(Loader.java:2231)
at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2125)
at org.hibernate.loader.Loader.list(Loader.java:2120)
at org.hibernate.loader.criteria.CriteriaLoader.list(CriteriaLoader.java:118)
at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1596)
at org.hibernate.impl.CriteriaImpl.list(CriteriaImpl.java:306)
at org.hibernate.impl.CriteriaImpl.uniqueResult(CriteriaImpl.java:328)
at com.vmware.vcloud.common.configuration.impl.HibernateDatabaseConfiguration.findConfigModel(HibernateDatabaseConfiguration.java:247)
at com.vmware.vcloud.common.configuration.impl.HibernateDatabaseConfiguration.getProperty(HibernateDatabaseConfiguration.java:362)
... 49 more
Caused by: java.sql.SQLRecoverableException: No more data to read from socket
at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1200)
at oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1155)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:279)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:186)
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:521)
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:205)
at oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:861)
at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1145)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1267)
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3449)
at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3493)
at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1491)
at org.hibernate.jdbc.AbstractBatcher.getResultSet(AbstractBatcher.java:208)
at org.hibernate.loader.Loader.getResultSet(Loader.java:1808)
at org.hibernate.loader.Loader.doQuery(Loader.java:697)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:259)
at org.hibernate.loader.Loader.doList(Loader.java:2228)
... 57 more
2010-09-09 11:18:23,033 | WARN | pool-jetty-22 | ValidationFilter | Failed to get configuration value for key: restapi.security.request.size_limit return default: 524,288 |
com.vmware.vcloud.common.configuration.ConfigurationException: Error getting configuration property restapi.security.request.size_limit
at com.vmware.vcloud.common.configuration.impl.ConfigurationServiceImpl.throwConfigException(ConfigurationServiceImpl.java:123)
at com.vmware.vcloud.common.configuration.impl.ConfigurationServiceImpl.getInteger(ConfigurationServiceImpl.java:223)
at sun.reflect.GeneratedMethodAccessor104.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.osgi.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:64)
at org.springframework.osgi.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:78)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at org.springframework.osgi.service.util.internal.aop.ServiceTCCLInterceptor.invokeUnprivileged(ServiceTCCLInterceptor.java:57)
at org.springframework.osgi.service.util.internal.aop.ServiceTCCLInterceptor.invoke(ServiceTCCLInterceptor.java:40)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at org.springframework.osgi.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:59)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy296.getInteger(Unknown Source)
at com.vmware.vcloud.security.filters.ValidationFilter.getRequestSizeLimit(ValidationFilter.java:124)
at com.vmware.vcloud.security.filters.SizeValidationFilter.doFilterHttp(SizeValidationFilter.java:23)
at org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)
at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)
at org.springframework.security.util.FilterChainProxy.doFilter(FilterChainProxy.java:175)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:236)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1115)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:361)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:324)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:879)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:741)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:213)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
at org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:636)
at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:451)
Caused by: com.vmware.vcloud.common.configuration.ConfigurationException: org.hibernate.exception.JDBCConnectionException: could not execute query
at com.vmware.vcloud.common.configuration.impl.HibernateDatabaseConfiguration.getProperty(HibernateDatabaseConfiguration.java:365)
at com.vmware.vcloud.common.configuration.impl.CombinedConfiguration.getProperty(CombinedConfiguration.java:105)
at org.apache.commons.configuration.AbstractConfiguration.resolveContainerStore(AbstractConfiguration.java:1160)
at org.apache.commons.configuration.AbstractConfiguration.getInteger(AbstractConfiguration.java:831)
at org.apache.commons.configuration.AbstractConfiguration.getInt(AbstractConfiguration.java:806)
at com.vmware.vcloud.common.configuration.impl.ConfigurationServiceImpl.getInteger(ConfigurationServiceImpl.java:219)
... 44 more
Caused by: org.hibernate.exception.JDBCConnectionException: could not execute query
at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:97)
at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:66)
at org.hibernate.loader.Loader.doList(Loader.java:2231)
at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2125)
at org.hibernate.loader.Loader.list(Loader.java:2120)
at org.hibernate.loader.criteria.CriteriaLoader.list(CriteriaLoader.java:118)
at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1596)
at org.hibernate.impl.CriteriaImpl.list(CriteriaImpl.java:306)
at org.hibernate.impl.CriteriaImpl.uniqueResult(CriteriaImpl.java:328)
at com.vmware.vcloud.common.configuration.impl.HibernateDatabaseConfiguration.findConfigModel(HibernateDatabaseConfiguration.java:247)
at com.vmware.vcloud.common.configuration.impl.HibernateDatabaseConfiguration.getProperty(HibernateDatabaseConfiguration.java:362)
... 49 more
Caused by: java.sql.SQLRecoverableException: No more data to read from socket
at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1200)
at oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1155)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:279)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:186)
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:521)
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:205)
at oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:861)
at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1145)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1267)
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3449)
at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3493)
at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1491)
at org.hibernate.jdbc.AbstractBatcher.getResultSet(AbstractBatcher.java:208)
at org.hibernate.loader.Loader.getResultSet(Loader.java:1808)
at org.hibernate.loader.Loader.doQuery(Loader.java:697)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:259)
at org.hibernate.loader.Loader.doList(Loader.java:2228)
... 57 more
2010-09-09 11:18:24,760 | DEBUG | Pool-Cleaner]:Tomcat Connection Pool[1-1909277306] | PooledConnection | Unable to validate object: |
java.sql.SQLRecoverableException: No more data to read from socket
at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1200)
at oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1155)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:279)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:186)
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:521)
at oracle.jdbc.driver.T4CStatement.doOall8(T4CStatement.java:194)
at oracle.jdbc.driver.T4CStatement.executeForDescribe(T4CStatement.java:853)
at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1145)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1259)
at oracle.jdbc.driver.OracleStatement.executeInternal(OracleStatement.java:1882)
at oracle.jdbc.driver.OracleStatement.execute(OracleStatement.java:1847)
at oracle.jdbc.driver.OracleStatementWrapper.execute(OracleStatementWrapper.java:301)
at com.vmware.vcloud.common.jdbcpool.PooledConnection.validate(PooledConnection.java:383)
at com.vmware.vcloud.common.jdbcpool.PooledConnection.validate(PooledConnection.java:331)
at com.vmware.vcloud.common.jdbcpool.ConnectionPool.testAllIdle(ConnectionPool.java:1004)
at com.vmware.vcloud.common.jdbcpool.ConnectionPool$PoolCleaner.run(ConnectionPool.java:1223)
2010-09-09 11:18:24,761 | DEBUG | Pool-Cleaner]:Tomcat Connection Pool[1-1909277306] | PooledConnection | Unable to close underlying SQL connection |
java.sql.SQLRecoverableException: No more data to read from socket
at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1200)
at oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1155)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:279)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:186)
at oracle.jdbc.driver.T4C7Ocommoncall.doOLOGOFF(T4C7Ocommoncall.java:61)
at oracle.jdbc.driver.T4CConnection.logoff(T4CConnection.java:491)
at oracle.jdbc.driver.PhysicalConnection.close(PhysicalConnection.java:3754)
at com.vmware.vcloud.common.jdbcpool.PooledConnection.disconnect(PooledConnection.java:270)
at com.vmware.vcloud.common.jdbcpool.PooledConnection.release(PooledConnection.java:415)
at com.vmware.vcloud.common.jdbcpool.ConnectionPool.release(ConnectionPool.java:590)
at com.vmware.vcloud.common.jdbcpool.ConnectionPool.testAllIdle(ConnectionPool.java:1006)
at com.vmware.vcloud.common.jdbcpool.ConnectionPool$PoolCleaner.run(ConnectionPool.java:1223)
2010-09-09 11:18:34,341 | DEBUG | VC.TaskManager.NonActiveTaskCompletionsPurger | TaskManager | Marked 0 tasks for purging |
2010-09-09 11:18:34,342 | DEBUG | VC.TaskManager.NonActiveTaskCompletionsPurger | TaskManager | Purged 0 tasks. |
Bump, this is major issue for us now, can't create new vDC, not even view properties of existing vDC.
Thanks,
William
Increased SGA but no help, the VCLOUD datafile is only 44.3MB.
The udump trace has the problematic sql statement, I tried sqlplus locally here:
SQL> select * from VCLOUD.PROVIDER_VDC_METRICS_VIEW;
select * from VCLOUD.PROVIDER_VDC_METRICS_VIEW
ERROR at line 1:
ORA-03113: end-of-file on communication channel
Looks like something in Oracle is corrupted, we have only one PvDC defined, any suggestions? maybe rebuild view?
Here is from trc file, let me know if you need to the whole file:
\*** ACTION NAME:() 2010-09-10 11:40:38.105
\*** MODULE NAME:(SQL*Plus) 2010-09-10 11:40:38.105
\*** SERVICE NAME:(SYS$USERS) 2010-09-10 11:40:38.105
\*** SESSION ID:(150.265) 2010-09-10 11:40:38.105
Exception signal: 11 (SIGSEGV), code: 1 (Address not mapped to object), addr: 0x18, PC:
\*** 2010-09-10 11:40:38.112
ksedmp: internal or fatal error
ORA-07445: exception encountered: core dump [] []
Current SQL statement for this session:
select * from VCLOUD.PROVIDER_VDC_METRICS_VIEW
-
Call Stack Trace -
calling call entry argument values in hex
location type point (? means dubious value)
-
-
-
-
ssd_unwind_bp: unhandled instruction at 0x2fb3962 instr=f
ksedst()+31 call ksedst1() 000000000 ? 000000001 ?
2B64E8506D50 ? 2B64E8506DB0 ?
2B64E8506CF0 ? 000000000 ?
ksedmp()+610 call ksedst() 000000000 ? 000000001 ?
2B64E8506D50 ? 2B64E8506DB0 ?
2B64E8506CF0 ? 000000000 ?
ssexhd()+629 call ksedmp() 000000003 ? 000000001 ?
2B64E8506D50 ? 2B64E8506DB0 ?
2B64E8506CF0 ? 000000000 ?
__restore_rt()+0 call ssexhd() 00000000B ? 2B64E8507D70 ?
2B64E8507C40 ? 2B64E8506DB0 ?
2B64E8506CF0 ? 000000000 ?
kkqjpdGetSavedColSt signal __restore_rt() 2B64E884C3E0 ? 000000001 ?
atsCB()+114 2B64E884C3E0 ? 000000000 ?
00D000000 ? 2B64E87C5210 ?
qksopVisitOpns()+19 call kkqjpdGetSavedColSt 2B64E8927410 ? 000000001 ?
6 atsCB() 2B64E884C3E0 ? 000000000 ?
00D000000 ? 2B64E87C5210 ?
kkeics()+2470 call qksopVisitOpns() 2B64E8927410 ? 002C7A02A ?
000000000 ? 000000001 ?
00D000000 ? 2B64E87C5210 ?
kkoInitColStats()+5 call kkeics() 2B64E89F6D58 ? 002C7A02A ?
12 000000000 ? 000000001 ?
00D000000 ? 2B64E87C5210 ?
kkoiqb()+5937 call kkoInitColStats() 000000002 ? 002C7A02A ?
000000000 ? 000000001 ?
00D000000 ? 2B64E87C5210 ?
After a long day of debugging, finally found something:
provider_vdc_metrics_view refers to resource_pool_inv_view which in turn refers to a view called cluster_and_compute_resource which statement is:
SELECT moref, vc_id, name FROM compute_resource_inv UNION
SELECT moref, vc_id, name FROM cluster_compute_resource_inv;
compute_resource_inv table is empty in our case, so I removed it, and the provider_vdc_metrics_view script worked!
Oracle doesn't seem to like either UNION or null result of first SELECT, not sure if this is bug or configuration issue, I have limited knowledge on Oracle, can someone help?
Just to elaborate the issue a bit more
“provider_vdc_metrics_view” is a huge script and referred to a few other views, during debug, I replaced referred views with actual sql statements, was able to identify it’s the “cluster_and_compute_resource” view which caused Oracle error in our environment.
“cluster_and_compute_resource” is simple, just two lines:
SELECT moref, vc_id, name FROM compute_resource_inv UNION
SELECT moref, vc_id, name FROM cluster_compute_resource_inv;
It works fine by itself, but it causes error when used in “provider_vdc_metrics_view”, in our case compute_resource_inv table is empty while cluster_compute_resource_inv has content, so I replaced cluster_and_compute_resource with just the 2nd line of SELECT statement which is supposed to provide the same result for us, and “provider_vdc_metrics_view” was able to work.
Something is wrong with Oracle or the script itself?
I would suggest you give support a call. they should be able to help you out!
Duncan
VMware Communities User Moderator | VCDX
-
Now available: <a href="http://www.amazon.com/gp/product/1439263450?ie=UTF8&tag=yellowbricks-20&linkCode=as2&camp=1789&creative=9325&creativeASIN=1439263450">Paper - vSphere 4.0 Quick Start Guide (via amazon.com)</a> | <a href="http://www.lulu.com/product/download/vsphere-40-quick-start-guide/6169778">PDF (via lulu.com)</a>
Blogging: http://www.yellow-bricks.com | Twitter: http://www.twitter.com/DuncanYB
We're still evaluation the software, haven't purchased license and support, will support accept the call?
This is indeed an Oracle bug.
Queries where Join Predicate Push Down is possible may raise a dump on kkqjpdgetsavedcolstatscb.
Workaround:
Set _optimizer_push_pred_cost_based to false.
alter system set "_optimizer_push_pred_cost_based"=FALSE scope=both;
G.
Fix is to upgrade to Oracle 11g per Oracle support.
Bug 10019204 dump:kkqjpdgetsavedcolstatscb
This note gives a brief overview of bug 10019204.
The content was last updated on: 10-FEB-2011
Click here for details of each of the sections below.
Affects:
Product (Component) | Oracle Server (Rdbms) |
Range of versions believed to be affected | Versions >= 10.2.0.1 but BELOW 11.2 |
Versions confirmed as being affected | |
Platforms affected | Generic (all / most platforms affected) |
Fixed:
This issue is fixed in | ||||
Symptoms: | Related To: | |||
|
| |||
Description
Queries where Join Predicate Push Down is possible may raise a dump on kkqjpdgetsavedcolstatscb.
Workaround:
Set _optimizer_push_pred_cost_based to false.
Please note: The above is a summary description only. Actual symptoms can vary. Matching to any symptoms here does not confirm that you are encountering this problem. Always consult with Oracle Support for advice. |
References
Bug:10019204 (This link will only work for PUBLISHED bugs)
Note:245840.1 Information on the sections in this article
Just want to thank you for following up on your own thread and posting the workaround and the fix. I was in a jam to stand up a LAB and ran into the Oracle bug. Your workaround worked and I am doubtful I would have found it as easily on the Oracle forums.
Thanks again.
you're welcome .
ps: I was an oracle database administrator and the "support.oracle.com" was my "second house". .