Virtual Centre 2.5 - Connections refused from remote VI client and unused established TCP connections

[warrenwalker]

Hi All,

Weve recently been seeing an issue where by remote connections to virtual centre result in connection errors. Accessing Virtual Centre from the locally installed client works without issue.

There is a firewall in between and from the client we are seeing SYN's but no SYN ACK's coming back from Virtual Centre. Ive checked the logs on both the VI Client and the Virtual Centre at the times when this problem occurs (after multiple retries you can evetually get in) and there is nothing obvious especially from VC. The client machine only reports the fact that a connection could not be established.

Anyway after performing a netstat -a to check what ports were listening from the Virtual Centre there seems to be THOUSANDS of established TCP connections from each of the clients which have the VI client installed and seem to start from 1025 up to 5000. It just appears that once the connection is established it doesnt disconnect/reset. Has anyone seen this behaviour before?

[warrenwalker]

Just to add to this netstat -b show that it is vpxd thats owning these established port connections. Either its something to do with VC or its a corrupt TCP/IP stack... Any thoughts people?

[TheVirtualNut]

Queue: tumbleweed.....

I'm getting a similar problem, with no firewall in between.

Web app trying to login to VC and returning errors.

Error on the VC is 'SOAP session count limit reached'

Maximum console connections allowed is 25, and there is maybe 4-5 connections at the time. In theory i should be able to connect another 20 clients?

Anyone have any ideas how I can manage SOAP connections? or at least see what SOAP connects are established? (ive tried Admin screen, Sessions tab, but that only shows VI Client connections)

NUTS!

[depping]

I'm facing the same problem at a customer site with VCB. same error as the one stated above. Currently talking with support, let's hope they've got a solution.

Duncan

My virtualisation blog:

If you find this information useful, please award points for "correct" or "helpful".

[RParker]

> VCB. same error as the one stated above. Currently talking with support,

A restart of the VC won't reset the connections? It's not elegant, but that should work. By reset, I mean resart of the machine for VC, not simply start / stop of the VC service.

[warrenwalker]

Yeah that does fix it but to be honest its more of a workaround and you then have further restarts looming around the corner. Ive also noticed ESX hosts appear disconnected from time to time when it gets particularly bad. Restarting hostd/ mgmt-vmware service on the ESX host makes no difference. CPU/RAM load on the physical VC server is normally very low with the odd CPU spike to around 40%.

Recently i ran a perfmon to check the handle count. After a fresh restart of VC its starts with around 800 file handles and at the time we starting to see performance loss it was over 13000! For some reason the vpxd service doesnt drop these file handles and that maybe where the problem lies. I havent digged into this but im going to check with process explorer/monitor over the next few days.

Cheers,

Warren

[depping]

That's a workaround indeed that this specific customer is using. Restarting the VC service is sufficient. But with 50+ ESX Hosts and a whole bunch of people managing the VM's that's not something you want to do on a regular base.

Duncan

My virtualisation blog:

If you find this information useful, please award points for "correct" or "helpful".