vCloud Director lost the connection to the vCenter Server

[vtxops]

Hi

We have recently deployed vCloud Director 5.5.0 (build 1323688) and attached our vCenter Server Appliance 5.5.0.5101 (build 1398493) to it. We are using self-signed certificates for both vCloud director certificates, following the procedures described in the vCloud Director Installation and Upgrade Guide.

Every few days our vCloud director loses the connection to the vCenter server. As a consequence we need to manually reconnect the vCenter server. We see the following error in vcloud-container-debug.log:

2013-12-14 04:07:12,474 | DEBUG	| 7e06867c-1d10-4e63-9826-27a10d840ddfListener (240) | AbstractVlsiServiceBehavior	| VCenterVimVlsiServiceBehaviorImpl@133cf92e https://<vcenter server hostname>:443/sdk/vimService PropertyCollector:propertyCollector PropertyCollector.waitForUpdatesEx: completionStatus=false |
2013-12-14 04:07:12,475 | ERROR	| 7e06867c-1d10-4e63-9826-27a10d840ddfListener (240) | VcUpdateListenerImpl	| Break on Unrecoverable error in Outer WFU Loop |

javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated

	at sun.security.ssl.SSLSessionImpl.getPeerCertificates(Unknown Source)
	at org.apache.http.conn.ssl.AbstractVerifier.verify(AbstractVerifier.java:128)
	at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:572)
	at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:692)
	at org.apache.http.conn.scheme.SchemeSocketFactoryAdaptor.connectSocket(SchemeSocketFactoryAdaptor.java:65)
	at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
	at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:151)
	at org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:125)
	at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:641)
	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480)
	at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
	at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
	at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:111)
	at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingImpl.send(HttpProtocolBindingImpl.java:98)
	at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.sendCall(MethodInvocationHandlerImpl.java:526)
	at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.executeCall(MethodInvocationHandlerImpl.java:507)
	at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.completeCall(MethodInvocationHandlerImpl.java:295)
	at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invokeOperation(MethodInvocationHandlerImpl.java:265)
	at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invoke(MethodInvocationHandlerImpl.java:169)
	at com.sun.proxy.$Proxy1097.waitForUpdatesEx(Unknown Source)
	at com.vmware.vcloud.vimproxy.internal.impl.Vc41UpdateListener.getUpdatesFromVC(Vc41UpdateListener.java:112)
	at com.vmware.vcloud.vimproxy.internal.impl.VcUpdateListenerImpl.getUpdates(VcUpdateListenerImpl.java:1155)
	at com.vmware.vcloud.vimproxy.internal.impl.VcUpdateListenerImpl.innerWaitForUpdatesLoop(VcUpdateListenerImpl.java:906)
	at com.vmware.vcloud.vimproxy.internal.impl.VcUpdateListenerImpl.outerWaitForUpdatesLoop(VcUpdateListenerImpl.java:600)
	at com.vmware.vcloud.vimproxy.internal.impl.VcUpdateListenerImpl.run(VcUpdateListenerImpl.java:349)

The line "javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated" is indicating some problem with the SSL certificates. However, I'm unable to find out what exactly might be the problem.

In vCloud director I have _not_ checked the checkbox to verify SSL certificates (Administration => General).

Things I have tried so far:
* Re-created new SSL certificates, replacing the old ones in the keystore

* Re-configured vcloud director using /opt/vmware/vcloud-director/bin/configure -r

* Detached vCenter server from vCloud director and attached it again

So far to no avail.

Has anyone experienced that problem too? Any suggestions for a workaround or is this a known issue?

Any help is appreciated!

cheers,

- Fabian

[IamTHEvilONE]

Peer Not Authenticated is more like vCenter and vCloud Director's connection is severed or the SSL session is terminated.  One more common example is if you have a firewall with a TCP "reaper".  E.g. if a TCP session through the firewall is on for too long, it might get disconnected.

Also, this has nothing to do with vCloud Director's personal certificates (HTTPS or ConsoleProxy).

If the error is VERY consistent, try to see if there is a schedule to it.  e.g. When do backups happen?  or The disconnect email comes out about every 60 minutes.  If there is a systematic failure to it, then it'll be easier to see what might be related.

If a reconnect fixes it temporarily.  Then the certs and credentials are fine.

Most likely there is something else getting in the way.

[vtxops]

Thanks for your answer! The disconnects happen rather randomly. Sometimes I get disconnected 3 hours after last using vCD, sometimes it takes 3 days until it happens. I will now investigate together with our networking guys and will debug the connection to vCenter with tcpdump, thanks for pointing in the right direction.

cheers!

[IamTHEvilONE]

Yeah ... unless the certificate on vCenter is expired or something odd like that.  If the vCenter has been around for some time, older versions like 2.5 and 4.0 I think had only a 2 year cert.  I'm not sure if that would be a blocker in this case.

Gotta think of what's between A and B.

Can you recall a time when vCloud Director was working as expected?

[vtxops]

Well, ever since I've posted this forum entry we had not experienced a single loss of connection anymore. Previously we had a disconnect every 1-3 days ever since we installed vCloud Director. It's a new installation, so there was no time when vCloud Director was working as expected yet.

Let's see how it behaves over the next few days. I'll keep tcpdump running and will report when we have a disconnect again.

[egrigsonFS]

I know this is a very old thread but there's now a KB article about this, in case anyone else runs into it;

http://kb.vmware.com/kb/2087377‌

Regards,

Ed.