I have 2 linked vCenters and finally successfully updated both to 7.0.3.01400 (one was at 7.0.3.00600 and the other was at 7.0.3.01000). The one that was on 7.0.3.00600 failed a few times and had the lovely effect of requiring a revert of snapshots, causing HA issues all over the environment. I have not attempted to update hosts since prior to the revert of snapshots. Now both are current at 7.0.3.01400. I am unable to update hosts and I'm finding several issues with host management. It all seems to be certificate related.
So clearly this is looking like a host certificate issue. I am unsure how to proceed to fix the cert on the hosts, even though vCenter is showing the host certificate is valid and expires at a future date.
Thanks,
Joel
I was finally able to get my support access and get a service request started. In the logs, support found the following entry:
[2023-06-08T13:41:29.304-04:00] [ERROR] VLSI-I/O reactor-0 org.apache.http.impl.nio.client.InternalHttpAsyncClient I/O reactor terminated abnormally org.apache.http.nio.reactor.IO
ReactorException: I/O dispatch worker terminated abnormally
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:356)
at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:194)
at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapByteBuffer.<init>(http://HeapByteBuffer.java:57)
at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
at org.apache.http.nio.util.HeapByteBufferAllocator.allocate(HeapByteBufferAllocator.java:52)
at org.apache.http.nio.util.ExpandableBuffer.expandCapacity(ExpandableBuffer.java:108)
at org.apache.http.nio.util.ExpandableBuffer.expand(ExpandableBuffer.java:121)
at org.apache.http.nio.util.SimpleInputBuffer.consumeContent(SimpleInputBuffer.java:69)
at org.apache.http.nio.protocol.BasicAsyncResponseConsumer.onContentReceived(BasicAsyncResponseConsumer.java:82)
at org.apache.http.nio.protocol.AbstractAsyncResponseConsumer.consumeContent(AbstractAsyncResponseConsumer.java:147)
at com.vmware.vim.vmomi.client.http.impl.InetAddressSniffingResponseConsumer.consumeContent(InetAddressSniffingResponseConsumer.java:54)
at org.apache.http.impl.nio.client.MainClientExec.consumeContent(MainClientExec.java:329)
I was directed to the following KB on increasing the heap memory for the vsphere-ui service.
https://kb.vmware.com/s/article/2150757?lang=en_US&queryTerm=increase%20heap%20memory%20
The value it was set to was: vsphere-ui = 1458
After setting it with cloudvm-ram-size -C 2048 vsphere-ui
The value is now: vsphere-ui = 2304
I'm not sure what kind of math that is, but I can say that I am successfully able to update hosts without crashing the I/O reactor and having to restart vsphere-ui. Support did also say that they have observed others this with situation after updating to 7.0.3.001400. Of course, there are many variables involved with whether you may be affected by this, mostly around what size your vCenter was initially deployed with and how much your specific sizing.
Thanks,
Joel
Hi,
In my opinion, of course, the best option would be to involve technical support, too many things "out of place" to fend for yourself, at least I don't consider it "appropriate" in a production context. A trivial thing, which I assume you've already thought of, verify the consistency of the time in the context of your infrastructure and that if you use FQDN names there are no problems with your DNS servers.
Regards,
Ferdinando
Thank you. Yeah, I agree. This seems global enough that I wouldn't want to make any changes without input from support. I'm also finding a correlation to when I try to do updates on a host, shortly after, we get no privileges to any object and I have to restart the vsphere-ui service.
Thanks,
Joel
I was finally able to get my support access and get a service request started. In the logs, support found the following entry:
[2023-06-08T13:41:29.304-04:00] [ERROR] VLSI-I/O reactor-0 org.apache.http.impl.nio.client.InternalHttpAsyncClient I/O reactor terminated abnormally org.apache.http.nio.reactor.IO
ReactorException: I/O dispatch worker terminated abnormally
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:356)
at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:194)
at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapByteBuffer.<init>(http://HeapByteBuffer.java:57)
at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
at org.apache.http.nio.util.HeapByteBufferAllocator.allocate(HeapByteBufferAllocator.java:52)
at org.apache.http.nio.util.ExpandableBuffer.expandCapacity(ExpandableBuffer.java:108)
at org.apache.http.nio.util.ExpandableBuffer.expand(ExpandableBuffer.java:121)
at org.apache.http.nio.util.SimpleInputBuffer.consumeContent(SimpleInputBuffer.java:69)
at org.apache.http.nio.protocol.BasicAsyncResponseConsumer.onContentReceived(BasicAsyncResponseConsumer.java:82)
at org.apache.http.nio.protocol.AbstractAsyncResponseConsumer.consumeContent(AbstractAsyncResponseConsumer.java:147)
at com.vmware.vim.vmomi.client.http.impl.InetAddressSniffingResponseConsumer.consumeContent(InetAddressSniffingResponseConsumer.java:54)
at org.apache.http.impl.nio.client.MainClientExec.consumeContent(MainClientExec.java:329)
I was directed to the following KB on increasing the heap memory for the vsphere-ui service.
https://kb.vmware.com/s/article/2150757?lang=en_US&queryTerm=increase%20heap%20memory%20
The value it was set to was: vsphere-ui = 1458
After setting it with cloudvm-ram-size -C 2048 vsphere-ui
The value is now: vsphere-ui = 2304
I'm not sure what kind of math that is, but I can say that I am successfully able to update hosts without crashing the I/O reactor and having to restart vsphere-ui. Support did also say that they have observed others this with situation after updating to 7.0.3.001400. Of course, there are many variables involved with whether you may be affected by this, mostly around what size your vCenter was initially deployed with and how much your specific sizing.
Thanks,
Joel
I have updated 55 hosts since making this change and haven't encountered a single instance of issues requiring a restart of vsphere-ui.
Hi,
Interesting, so in the end "the whole mess" was the result of an insufficient memory allocation for the so-called "heap space" used by java for the UI service. From my point of view, as that the expected deployment models predetermined and unchanged in their specifications for several years now (although things have changed over time), in theory there would be no reason to "put our hands to fix things here and there" at a later time.
The remedy exposed in the KB article is effective but, reading here: "Updates of vCenter server may revert these changes, so document accordingly and as needed to repeat that process", again my very personal opinion, looks a bit like the "crawfish step".
Regards,
Ferdinando
