frequently exhausted sockets or buffers after upgrade to 5.1U1b

[komanek]

Hello,

about two weeks ago I upgraded my infrastructure from 5.0u2 with latest patches to 5.1u1b. Since then, every few days my vCenter stops working due to the following exception:

vpxd log:

Connection to localhost:8085 failed with error class Vmacore::SystemException(An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full. ).

vws log:

[2013-09-24 23:23:38,392 Thread-37  ERROR com.vmware.vim.health.impl.XmlUtil] Error retrieving health from url: http://localhost:8080/sms/health.xml

java.net.SocketException: No buffer space available (maximum connections reached?): connect

        at java.net.PlainSocketImpl.socketConnect(Native Method)

        at java.net.PlainSocketImpl.doConnect(Unknown Source)

        at java.net.PlainSocketImpl.connectToAddress(Unknown Source)

        at java.net.PlainSocketImpl.connect(Unknown Source)

        at java.net.SocksSocketImpl.connect(Unknown Source)

        at java.net.Socket.connect(Unknown Source)

        at java.net.Socket.connect(Unknown Source)

        at sun.net.NetworkClient.doConnect(Unknown Source)

        at sun.net.www.http.HttpClient.openServer(Unknown Source)

        at sun.net.www.http.HttpClient.openServer(Unknown Source)

        at sun.net.www.http.HttpClient.<init>(Unknown Source)

        at sun.net.www.http.HttpClient.New(Unknown Source)

        at sun.net.www.http.HttpClient.New(Unknown Source)

        at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(Unknown Source)

        at sun.net.www.protocol.http.HttpURLConnection.plainConnect(Unknown Source)

        at sun.net.www.protocol.http.HttpURLConnection.connect(Unknown Source)

        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)

        at org.apache.xerces.impl.XMLEntityManager.setupCurrentEntity(Unknown Source)

        at org.apache.xerces.impl.XMLVersionDetector.determineDocVersion(Unknown Source)

        at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)

        at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)

        at org.apache.xerces.jaxp.validation.StreamValidatorHelper.validate(Unknown Source)

        at org.apache.xerces.jaxp.validation.ValidatorImpl.validate(Unknown Source)

        at javax.xml.validation.Validator.validate(Unknown Source)

        at com.vmware.vim.health.impl.XmlUtil.getDocumentFromUrl(XmlUtil.java:96)

        at com.vmware.vim.health.impl.ComponentSpec.retrieveHealthFromUrl(ComponentSpec.java:300)

        at com.vmware.vim.health.impl.ComponentSpec.retrieveHealth(ComponentSpec.java:266)

        at com.vmware.vim.health.impl.HealthPollerImpl.retrieveHealthFromUrl(HealthPollerImpl.java:119)

        at com.vmware.vim.health.impl.HealthPollerImpl.retrieveHealth(HealthPollerImpl.java:104)

        at com.vmware.vim.health.impl.HealthPollerImpl.computeHealth(HealthPollerImpl.java:203)

        at com.vmware.vim.health.impl.HealthPollerImpl.retrieveHealth(HealthPollerImpl.java:102)

        at com.vmware.vim.health.impl.HealthPollerImpl.pollHealth(HealthPollerImpl.java:85)

        at com.vmware.vim.health.impl.HealthPollerImpl.access$100(HealthPollerImpl.java:28)

        at com.vmware.vim.health.impl.HealthPollerImpl$PollerThread.run(HealthPollerImpl.java:55)

        at java.lang.Thread.run(Unknown Source)

There is nothing about it in Windows eventlog, all processess are running, but unable to make any connection. The serever is still accessible via RDP session. Restarting vCenter and related services does not help, I have to reboot the server to resolve this.

vCenter is running on Windows 2008 R2 SP1 machine with current Windows updates, as a VM. Afd.sys is newer than the one in hotfix ( Kernel sockets leak on a multiprocessor computer that is running Windows Server 2008 R2 or Windows ...). There should be enough RAM, according to statistics.

I have one more vCenter server for a similar cluster, with the same software versions, which is not experiencing these problems. There are only two differences between these servers:

- the problematic one is using Oracle database for vCenter, the healthy one is using MS SQL Express

- the problematic one is accessed by View 5.2 broker/composer, while on the healthy one there is on such traffic

Please, is somebody here, who knows, what is going on and how to resolve this ?

Thank you in advance.

David