Random Reconnects to vCenter

[barnette08]

Hey guys,

I am curious if anyone else has run into this or not.  I have noticed that I will get an email notification on random nights around 3:00am to 4:00am stating:

In one environment:

vCloud Director connected to the vCenter Server Server "servername". (3:53am)

In the other environment:

vCloud Director connected to the vCenter Server Server "servername". (3:24am)

vCloud Director is trying to reconnect to the vCenter Server Server "servername". When vCloud Director reconnects, it will send another email alert. (3:14am)

This is odd to me because no changes are going on in the infrastructure that I am aware of.  The one thing that these 2 environments have in common is that they live in the same management cluster, but I have also seen this happen in our development environment (a third environment) that is hosted in its own management cluster, specific for development.  I'd be very interested to know all of your thoughts about this.  Thanks!

[IamTHEvilONE]

Do you have a firewall between the cells and the vCenter server?  We maintain an open connection at all times.  I have seen things like this happen when the Firewall has a process to kill idle TCP connections.

One test I have done is to deploy a cell in the same ESXi host as the destination vCenter (assuming it's virtual), and force the proxy over to that cell for testing (stop all other cells).  If the error goes away, it's a network fabric problem.

If you have a vcloud-container-debug.log which encapsulates the time frame (and can post it here), I should be able to take a look.

[cfor]

Contact support - we have had the same problem for a while now and our issue was found to be deadlocks in the VCD database causing the disconnects.  Support should be able to help gather logs and do some validation as you know the exact disconnect times.

[barnette08]

Since there are 4 cells in question I will probably just open a case like cfor mentioned, then report back to the thread with the resolution.  Thanks for the help guys.

[cfor]

The issue we have we have only seen on our system with several cells (we have 10 currently); our systems with two cells have not had the issue we have been seeing, so it could be the same thing.  At least worth checking out.

[IamTHEvilONE]

fair enough.  when you file the ticket, make sure to upload logs from all cells per the following:

VMware KB: Uploading diagnostic information for VMware

kb.vmware.com/kb/1008525‎

[barnette08]

will you guys have what you need with the multi-cell logs or should I just get them for each cell?

[cfor]

Always a safe bet to get the for all the cells.

/opt/vmware/vcloud-director/bin/vmware-vcd-support -l -m

That command will gather the logs from all your cells, and package up in one bundle.

(When we opened our case VMware also wanted the vCenter support bundle for the vCenter that was disconnected from - might be a good idea just to pre gather that as well)

[admin]

What version of vCloud Director are you running? vCloud Director 5.1.2 addresses an issue like this.

From the 5.1.2 Release Notes under Resolved Issues:

vCenter Server sometimes disconnects from vCloud Director after upgrading to vCloud Director 5.1

Fixed a problem that could cause vCenter Server to sometimes disconnect from vCloud Director after you upgrade vCloud Director.

vCloud Director 5.1.2 Release Notes

Upgrading to vCloud Director 5.1.x best practices