VMware Cloud Community
blaircotnam
Contributor
Contributor
‎11-07-2019 11:34 AM

vCloud Director lost the connection to the vCenter Server

We are currently on vCloud Director 8.20 and vCenter 6.0 Update 3.  (Yes, we are in the process of upgrading..)

Every few days vCloud director loses the connection to the vCenter server but, it will also reconnect automatically with in 1-3 minutes.  Below is the snippet log from VCD - vcloud-contailer-info.log

-------------------------------

2019-11-03 17:03:36,718 | ERROR    | pool-jetty-47446941       | JDBCExceptionReporter          | The query has timed out. | requestId=78c7564f-dba9-4ae7-9f29-0318bf976374,request=GET https://vcd.vmcloud.ca/api/admin/vdc/96e4d057-4a74-4ec9-9116-6959f83a0b5a,requestTime=1572818616429,... (Java/1.8.0_162),accept=application/*+xml;version 20.0

2019-11-03 17:03:36,718 | ERROR    | pool-jetty-47423267       | JDBCExceptionReporter          | The query has timed out. | requestId=33717197-4bc5-4e08-b1db-2f983f8c61ed,request=GET https://vcd.vmcloud.ca/api/vAppTemplate/vm-ADAD4C2C-05BF-43D1-A879-E30D4B2818A7,requestTime=15728186...<none>,accept=application/*+xml; version 5.5

2019-11-03 17:03:36,724 | ERROR    | VC-546e9c6e-47e0-4cdd-8570-bbdf86e3f17aListener (34921915) | VcUpdateListenerImpl           | VC 546e9c6e-47e0-4cdd-8570-bbdf86e3f17a: Break on Unrecoverable error in Outer WFU Loop |

org.hibernate.exception.GenericJDBCException: could not execute query

at org.hibernate.exception.SQLStateConverter.handledNonSpecificException(SQLStateConverter.java:126)

at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:114)

at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:66)

at org.hibernate.loader.Loader.doList(Loader.java:2231)

at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2125)

at org.hibernate.loader.Loader.list(Loader.java:2120)

at org.hibernate.loader.criteria.CriteriaLoader.list(CriteriaLoader.java:118)

at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1596)

at org.hibernate.impl.CriteriaImpl.list(CriteriaImpl.java:306)

at org.hibernate.impl.CriteriaImpl.uniqueResult(CriteriaImpl.java:328)

at com.vmware.vcloud.common.persist.CriteriaProxy.uniqueResult(CriteriaProxy.java:211)

at com.vmware.vcloud.common.persist.dao.HibernateVirtualCenterDaoImpl.getVcRuntime(HibernateVirtualCenterDaoImpl.java:58)

at com.vmware.vcloud.vimproxy.internal.impl.VcUpdateListenerImpl.dispatchVcConnectionUpEvent(VcUpdateListenerImpl.java:1274)

at com.vmware.vcloud.vimproxy.internal.impl.VcUpdateListenerImpl.innerWaitForUpdatesLoop(VcUpdateListenerImpl.java:976)

at com.vmware.vcloud.vimproxy.internal.impl.VcUpdateListenerImpl.outerWaitForUpdatesLoop(VcUpdateListenerImpl.java:657)

at com.vmware.vcloud.vimproxy.internal.impl.VcUpdateListenerImpl.run(VcUpdateListenerImpl.java:367)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

Caused by: java.sql.SQLException: The query has timed out.

at net.sourceforge.jtds.jdbc.TdsCore.wait(TdsCore.java:3936)

at net.sourceforge.jtds.jdbc.TdsCore.executeSQL(TdsCore.java:1046)

at net.sourceforge.jtds.jdbc.JtdsStatement.executeSQLQuery(JtdsStatement.java:465)

at net.sourceforge.jtds.jdbc.JtdsPreparedStatement.executeQuery(JtdsPreparedStatement.java:776)

at org.hibernate.jdbc.AbstractBatcher.getResultSet(AbstractBatcher.java:208)

at org.hibernate.loader.Loader.getResultSet(Loader.java:1808)

at org.hibernate.loader.Loader.doQuery(Loader.java:697)

at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:259)

at org.hibernate.loader.Loader.doList(Loader.java:2228)

... 17 more

2019-11-03 17:03:36,732 | INFO     | VC-546e9c6e-47e0-4cdd-8570-bbdf86e3f17aListener (34921915) | VirtualCenterListener          | VC connection state event 546e9c6e-47e0-4cdd-8570-bbdf86e3f17a. Changing state to DISCONNECTED |

2019-11-03 17:03:36,732 | INFO     | VC-546e9c6e-47e0-4cdd-8570-bbdf86e3f17aListener (34921915) | VirtualCenterListener          | Disconnected from virtual center 546e9c6e-47e0-4cdd-8570-bbdf86e3f17a |

2019-11-03 17:03:36,981 | ERROR    | pool-jetty-47423262       | JDBCExceptionReporter          | The query has timed out. | requestId=6286ce62-9641-48d0-bf69-99267699488a,request=GET https://vcd.vmcloud.ca/api/query,requestTime=1572818616366,remoteAddress=x.x.x.x:63345,userAgent=<none>,accept=application/*+xml; version 5.5

2019-11-03 17:03:37,026 | INFO     | VC-546e9c6e-47e0-4cdd-8570-bbdf86e3f17aListener (34921915) | InventoryServiceImpl           | Received listener stopped for VC 546e9c6e-47e0-4cdd-8570-bbdf86e3f17a |

2019-11-03 17:03:37,061 | INFO     | pool-async-email-sender-17-thread-26 | Emailer                        | Sent email to 3 recipients. No failures |

2019-11-03 17:03:37,064 | INFO     | VimConnectionCleanup-546e9c6e-47e0-4cdd-8570-bbdf86e3f17a | VcUpdateListenerImpl           | VC 546e9c6e-47e0-4cdd-8570-bbdf86e3f17a: Ignoring exception null while unregistering filters |

-------------------------------

Looks like the DB query is timing out but, not sure why its causing a disconnect from vCloud.  Has anyone experienced the above?

Thanks for your help.

0 Kudos
1 Reply
QuickSave
Contributor
Contributor
‎10-17-2023 07:29 AM

Hi,

 

in 2023 with Cloud Director at version 10.4.2. and vCenter at 7.0.3. Build 22357613, I have the same issue with the same log entries.

After this disconnection and reconnection  my Cloud Director is really bad at handling tasks. Some tasks will take 20 minutes while they normally take 6 seconds. Not all tasks, but enough to be annoying.

My workaround at the moment is to reboot the cloud director cells.

 

Did someone knows this issue or had a solution to the described behavior?

 

kind regards

0 Kudos