VMware Cloud Community
FingerJC
Enthusiast
Enthusiast
Jump to solution

Host issues after updating vCenter to 7.0.3.01400

I have 2 linked vCenters and finally successfully updated both to 7.0.3.01400 (one was at 7.0.3.00600 and the other was at 7.0.3.01000).  The one that was on 7.0.3.00600 failed a few times and had the lovely effect of requiring a revert of snapshots, causing HA issues all over the environment.  I have not attempted to update hosts since prior to the revert of snapshots.  Now both are current at 7.0.3.01400.  I am unable to update hosts and I'm finding several issues with host management.  It all seems to be certificate related.

  • VMware vSphere Lifecycle Manager had an unknown error - occurs when attempting to update a host.
  • Host certificate shows good, but "A general system error occurred: Unable to push CA certificates and CRLs to host" is logged when initiating "Refresh CA Certificates".  
  • esxupdate.log displays ERROR: Failed to verify VIB signature #2: ('VMW_bootbank_vmkusb_0.1-7vmw.703.0.50.20036589', 'Could not find a trusted signer: self signed certificate') for every VIB. 
  • Remove from inventory/add host to inventory generates "Registration/unregistration of third-party IO filter storage providers fails on a host".  iofiltervpd.log displays "IOFVPSSL_VerifySSLCertificate:238:Client certificate can't be verified".

So clearly this is looking like a host certificate issue.  I am unsure how to proceed to fix the cert on the hosts, even though vCenter is showing the host certificate is valid and expires at a future date.

Thanks,
Joel

Reply
0 Kudos
1 Solution

Accepted Solutions
FingerJC
Enthusiast
Enthusiast
Jump to solution

I was finally able to get my support access and get a service request started.  In the logs, support found the following entry:

[2023-06-08T13:41:29.304-04:00] [ERROR] VLSI-I/O reactor-0            org.apache.http.impl.nio.client.InternalHttpAsyncClient           I/O reactor terminated abnormally org.apache.http.nio.reactor.IO
ReactorException: I/O dispatch worker terminated abnormally
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:356)
        at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:194)
        at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
        at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.OutOfMemoryError: Java heap space
        at java.nio.HeapByteBuffer.<init>(http://HeapByteBuffer.java:57)
        at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
        at org.apache.http.nio.util.HeapByteBufferAllocator.allocate(HeapByteBufferAllocator.java:52)
        at org.apache.http.nio.util.ExpandableBuffer.expandCapacity(ExpandableBuffer.java:108)
        at org.apache.http.nio.util.ExpandableBuffer.expand(ExpandableBuffer.java:121)
        at org.apache.http.nio.util.SimpleInputBuffer.consumeContent(SimpleInputBuffer.java:69)
        at org.apache.http.nio.protocol.BasicAsyncResponseConsumer.onContentReceived(BasicAsyncResponseConsumer.java:82)
        at org.apache.http.nio.protocol.AbstractAsyncResponseConsumer.consumeContent(AbstractAsyncResponseConsumer.java:147)
        at com.vmware.vim.vmomi.client.http.impl.InetAddressSniffingResponseConsumer.consumeContent(InetAddressSniffingResponseConsumer.java:54)
        at org.apache.http.impl.nio.client.MainClientExec.consumeContent(MainClientExec.java:329)

I was directed to the following KB on increasing the heap memory for the vsphere-ui service. 
https://kb.vmware.com/s/article/2150757?lang=en_US&queryTerm=increase%20heap%20memory%20

The value it was set to was: vsphere-ui = 1458
After setting it with cloudvm-ram-size -C 2048 vsphere-ui
The value is now: vsphere-ui = 2304

I'm not sure what kind of math that is, but I can say that I am successfully able to update hosts without crashing the I/O reactor and having to restart vsphere-ui.  Support did also say that they have observed others this with situation after updating to 7.0.3.001400.  Of course, there are many variables involved with whether you may be affected by this, mostly around what size your vCenter was initially deployed with and how much your specific sizing. 

Thanks,
Joel

View solution in original post

Reply
0 Kudos
5 Replies
Kinnison
Expert
Expert
Jump to solution

Hi,


In my opinion, of course, the best option would be to involve technical support, too many things "out of place" to fend for yourself, at least I don't consider it "appropriate" in a production context. A trivial thing, which I assume you've already thought of, verify the consistency of the time in the context of your infrastructure and that if you use FQDN names there are no problems with your DNS servers.


Regards,
Ferdinando

Reply
0 Kudos
FingerJC
Enthusiast
Enthusiast
Jump to solution

Thank you.  Yeah, I agree.  This seems global enough that I wouldn't want to make any changes without input from support.  I'm also finding a correlation to when I try to do updates on a host, shortly after, we get no privileges to any object and I have to restart the vsphere-ui service. 

Thanks,
Joel

Reply
0 Kudos
FingerJC
Enthusiast
Enthusiast
Jump to solution

I was finally able to get my support access and get a service request started.  In the logs, support found the following entry:

[2023-06-08T13:41:29.304-04:00] [ERROR] VLSI-I/O reactor-0            org.apache.http.impl.nio.client.InternalHttpAsyncClient           I/O reactor terminated abnormally org.apache.http.nio.reactor.IO
ReactorException: I/O dispatch worker terminated abnormally
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:356)
        at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:194)
        at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
        at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.OutOfMemoryError: Java heap space
        at java.nio.HeapByteBuffer.<init>(http://HeapByteBuffer.java:57)
        at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
        at org.apache.http.nio.util.HeapByteBufferAllocator.allocate(HeapByteBufferAllocator.java:52)
        at org.apache.http.nio.util.ExpandableBuffer.expandCapacity(ExpandableBuffer.java:108)
        at org.apache.http.nio.util.ExpandableBuffer.expand(ExpandableBuffer.java:121)
        at org.apache.http.nio.util.SimpleInputBuffer.consumeContent(SimpleInputBuffer.java:69)
        at org.apache.http.nio.protocol.BasicAsyncResponseConsumer.onContentReceived(BasicAsyncResponseConsumer.java:82)
        at org.apache.http.nio.protocol.AbstractAsyncResponseConsumer.consumeContent(AbstractAsyncResponseConsumer.java:147)
        at com.vmware.vim.vmomi.client.http.impl.InetAddressSniffingResponseConsumer.consumeContent(InetAddressSniffingResponseConsumer.java:54)
        at org.apache.http.impl.nio.client.MainClientExec.consumeContent(MainClientExec.java:329)

I was directed to the following KB on increasing the heap memory for the vsphere-ui service. 
https://kb.vmware.com/s/article/2150757?lang=en_US&queryTerm=increase%20heap%20memory%20

The value it was set to was: vsphere-ui = 1458
After setting it with cloudvm-ram-size -C 2048 vsphere-ui
The value is now: vsphere-ui = 2304

I'm not sure what kind of math that is, but I can say that I am successfully able to update hosts without crashing the I/O reactor and having to restart vsphere-ui.  Support did also say that they have observed others this with situation after updating to 7.0.3.001400.  Of course, there are many variables involved with whether you may be affected by this, mostly around what size your vCenter was initially deployed with and how much your specific sizing. 

Thanks,
Joel

Reply
0 Kudos
FingerJC
Enthusiast
Enthusiast
Jump to solution

I have updated 55 hosts since making this change and haven't encountered a single instance of issues requiring a restart of vsphere-ui.

Kinnison
Expert
Expert
Jump to solution

Hi,


Interesting, so in the end "the whole mess" was the result of an insufficient memory allocation for the so-called "heap space" used by java for the UI service. From my point of view, as that the expected deployment models predetermined and unchanged in their specifications for several years now (although things have changed over time), in theory there would be no reason to "put our hands to fix things here and there" at a later time.


The remedy exposed in the KB article is effective but, reading here: "Updates of vCenter server may revert these changes, so document accordingly and as needed to repeat that process", again my very personal opinion, looks a bit like the "crawfish step".


Regards,
Ferdinando

Reply
0 Kudos