vAPI Endpoint health status alarms after upgrading to 6.5U1

[gbeekmans]

Hi all,

I upgraded our vCenter to 6.5 Update 1 (the Linux server appliance version). After upgrading, we are receiving the vAPI Endpoint health alarms. Some KB articles I read make mention of spring bean related issues which isn't happening in our case.

/var/log/vmware/vapi/endpoint/endpoint.log contains the following relevant snippets:

2017-08-23T15:21:51.294Z | WARN  | state-manager1            | ApiInterfacesFactory           | Retrieving interfaces for service 2a8ef657-f821-473f-9967-9006ffc99450\com.vmware.cis.cls.vapi has failed.

com.vmware.vapi.client.exception.ConnectionException: Connection refused

        at com.vmware.vapi.internal.protocol.client.rpc.http.ApacheHttpAsyncClientTransport$FutureCallbackImpl.failed(ApacheHttpAsyncClientTransport.java:284)

        at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:130)

        at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.connectionRequestFailed(DefaultClientExchangeHandlerImpl.java:319)

        at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.access$100(DefaultClientExchangeHandlerImpl.java:63)

        at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl$1.failed(DefaultClientExchangeHandlerImpl.java:365)

        at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:130)

        at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager$InternalPoolEntryCallback.failed(PoolingNHttpClientConnectionManager.java:474)

        at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:130)

        at org.apache.http.nio.pool.RouteSpecificPool.failed(RouteSpecificPool.java:165)

        at org.apache.http.nio.pool.AbstractNIOConnPool.requestFailed(AbstractNIOConnPool.java:527)

        at org.apache.http.nio.pool.AbstractNIOConnPool$InternalSessionRequestCallback.failed(AbstractNIOConnPool.java:775)

        at org.apache.http.impl.nio.reactor.SessionRequestImpl.failed(SessionRequestImpl.java:154)

        at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:175)

        at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:147)

        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:348)

        at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:189)

        at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.doExecute(CloseableHttpAsyncClientBase.java:67)

        at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.access$000(CloseableHttpAsyncClientBase.java:38)

        at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:57)

        at java.lang.Thread.run(Thread.java:748)

Caused by: java.net.ConnectException: Connection refused

        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)

        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)

        at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:173)

        ... 7 more

2017-08-23T15:21:51.295Z | WARN  | state-manager1            | ApiInterfacesFactory           | The cache does not contain entry for service 2a8ef657-f821-473f-9967-9006ffc99450\com.vmware.cis.cls.vapi. The interfaces will be skipped.

2017-08-23T15:21:51.295Z | INFO  | state-manager1            | HealthStatusCollectorImpl      | HEALTH YELLOW Failed to connect to 2a8ef657-f821-473f-9967-9006ffc99450\com.vmware.cis.cls.vapi vAPI provider.

2017-08-23T15:21:51.450Z | INFO  | state-manager1            | HealthStatusCollectorImpl      | HEALTH GREEN Current vApi Endpoint health status is created between 2017-08-23T15:21:50GMT and 2017-08-23T15:21:51GMT.

2017-08-23T15:21:51.450Z | INFO  | state-manager1            | HealthConfigurationEventListener | Computed health status is = YELLOW

2017-08-23T15:21:51.450Z | INFO  | state-manager1            | HealthConfigurationEventListener | HEALTH Failed to connect to 2a8ef657-f821-473f-9967-9006ffc99450\com.vmware.cis.cls.vapi vAPI provider.

No amount of rebooting the VM or restarting the service has made a difference.

What would 2a8ef657-f821-473f-9967-9006ffc99450\com.vmware.cis.cls.vapi refer to that's having issues? I wonder if this ID existed prior to the upgrade and now doesn't, or has a new ID that vAPI isn't aware of?

Besides scrapping the vCenter VM and reinstalling from scratch, is there a way to resolve this?

It doesn't seem to impact actual functionality of the system. It just shows the nuisance warnings that we can "reset to green" when it pops up and continue on.

Any thoughts and hints would be greatly appreciated.

Regards,

Gerard Beekmans

[johog26]

I have the exact same issue after upgrade from 6.5 to U1. I'm waiting for a respons from VMWare support. If it gets resolved I will post the solution.

Regards

Joakim H

[johog26 · Accepted solution]

Hello again,

I got an answer from support and it seems to work. Try this:

Connect to the vCenter Server Appliance with an SSH session and root credentials.

shell.set --enabled true
Type shell and press Enter.

cd /etc/vmware-content-library/config/
cp ts-config.properties ts-config.properties.orig
cp ts-config.properties.rpmnew ts-config.properties.rpmnew.orig
mv ts-config.properties.rpmnew ts-config.properties
service-control --stop vmware-content-library;service-control --start vmware-content-library
service-control --stop vmware-vapi-endpoint
service-control --start vmware-vapi-endpoint

Regards

Joakim H

[gbeekmans]

Hi Joakim,

That did indeed fix the issue for me as well. Looking at the two ts-config.properties files, there are only 4 lines of changes that fixed the issue:

transfer.httpClient.http.proxyHost={no-proxy}

transfer.httpClient.http.proxyPort=0

transfer.httpClient.https.proxyHost={no-proxy}

transfer.httpClient.https.proxyPort=0

I didn't do any further testing to see if it was the http or https version that ended up actually fixing it.

Thanks for following-up and sharing your findings.

Regards,

Gerard Beekmans

[gaddevamsi84]

That's great!! Worked for me. Thanks

[fieldingbr]

I have the same issue in my environment.  The issue I'm having with the steps below is I don't see the file 'ts-config.properties.rpmnew' on my vCenter in the location provided.

Thoughts?

Brandon

[Sravan_k]

Please try restarting the "vmware-vapi-endpoint" using below commands

service-control --stop vmware-vapi-endpoint

service-control --start vmware-vapi-endpoint

Regards,

Vkmr

[rajczer]

Hi

Would it be possible to confirm that this problem occurs also on version 7.0 Update 1c - 7.0.1.00200 ?

Is this same issue as https://kb.vmware.com/s/article/80409 ?
This one is mention to be resolved in vCenter Server 7.0 Update 1.

In my case /var/log/vmware/vapi/endpoint/endpoint.log contains the following entries:

2023-08-09T10:45:26.040+01:00 | INFO | state-manager1 | ApiConnectionsLsUtil | Unsupported source (metadata) type in metadata source entry com.vmware.vapi.metadata.authorization.file.certificatemanagement : /usr/lib/vmware-certificatemanagement/config/metadata/com.vmware.vcenter_authorization.json
2023-08-09T10:45:26.040+01:00 | INFO | state-manager1 | DefaultStateManager | Invoking rebuild vim-adapter-settings-builder
2023-08-09T10:45:26.052+01:00 | INFO | state-manager1 | DefaultStateManager | Invoking rebuild vapi-vcenter-servlet-builder
2023-08-09T10:45:26.052+01:00 | INFO | state-manager1 | DefaultStateManager | Invoking rebuild api-interfaces-builder
2023-08-09T10:45:26.067+01:00 | WARN | state-manager1 | ApiInterfacesFactory | Retrieving interfaces for service dd7baf7d-93c5-425d-8871-fcfc908e3f5f\com.vmware.vcenter.vlcm has failed.
com.vmware.vapi.client.exception.ConnectionException: http://localhost:12020/lcm/api invocation failed with "java.net.ConnectException: Connection refused"
at com.vmware.vapi.internal.protocol.client.rpc.http.ApacheHttpAsyncClientTransport$FutureCallbackImpl.failed(ApacheHttpAsyncClientTransport.java:210)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:137)
at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.executionFailed(DefaultClientExchangeHandlerImpl.java:101)
at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.failed(AbstractClientExchangeHandler.java:426)
at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.connectionRequestFailed(AbstractClientExchangeHandler.java:348)
at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.access$100(AbstractClientExchangeHandler.java:62)
at org.apache.http.impl.nio.client.AbstractClientExchangeHandler$1.failed(AbstractClientExchangeHandler.java:392)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:137)
at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager$1.failed(PoolingNHttpClientConnectionManager.java:316)
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:137)
at org.apache.http.nio.pool.RouteSpecificPool.failed(RouteSpecificPool.java:162)
at org.apache.http.nio.pool.AbstractNIOConnPool.requestFailed(AbstractNIOConnPool.java:613)
at org.apache.http.nio.pool.AbstractNIOConnPool$InternalSessionRequestCallback.failed(AbstractNIOConnPool.java:893)
at org.apache.http.impl.nio.reactor.SessionRequestImpl.failed(SessionRequestImpl.java:177)
at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:176)
at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:148)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:351)
at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221)
at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:714)
at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:174)
... 5 more
2023-08-09T10:45:26.068+01:00 | WARN | state-manager1 | ApiInterfacesFactory | The cache does not contain entry for service dd7baf7d-93c5-425d-8871-fcfc908e3f5f\com.vmware.vcenter.vlcm. The interfaces will be skipped.
2023-08-09T10:45:26.068+01:00 | INFO | state-manager1 | StatusInfoFactory | HEALTH YELLOW Failed to connect to dd7baf7d-93c5-425d-8871-fcfc908e3f5f\com.vmware.vcenter.vlcm vAPI provider.
2023-08-09T10:45:26.099+01:00 | INFO | state-manager1 | DefaultStateManager | Invoking rebuild vcenter-shim-service-builder
2023-08-09T10:45:26.099+01:00 | INFO | state-manager1 | DefaultStateManager | Invoking rebuild metadata-sync-builder
2023-08-09T10:45:26.099+01:00 | INFO | state-manager1 | MetadataSynchronizationBuilder | rebuild
2023-08-09T10:45:26.230+01:00 | INFO | state-manager1 | MetadataSynchronizationBuilder | rebuild finished.

Restart of the "vmware-vapi-endpoint" did not solve this issue, "vAPI Endpoint" State goes back to yellow in about 4 minutes.

Introduction of "no" proxy entries to ts-config.properties and restart of both vmware-content-library and vmware-vapi-endpoint did not help as well:

root@nac5vcsa [ /etc/vmware-content-library/config ]# ls -al
total 20
drwxr-xr-x 2 content-library cis 4096 Nov 22 2021 .
drwxr-xr-x 4 content-library cis 4096 Nov 22 2021 ..
-rw-rw---- 1 content-library cis 1011 Jan 1 2010 cls-config.properties
-rw-rw---- 1 content-library cis 1558 Jan 1 2010 ts-config.properties
-rw-rw---- 1 content-library cis 188 Jan 1 2010 vdcs-config.properties
root@nac5vcsa [ /etc/vmware-content-library/config ]# diff ts-config.properties ts-config.properties_noproxy
> # Proxy
>
> transfer.httpClient.http.proxyHost={no-proxy}
> transfer.httpClient.http.proxyPort=0
> transfer.httpClient.https.proxyHost={no-proxy}
> transfer.httpClient.https.proxyPort=0
oot@nac5vcsa [ /etc/vmware-content-library/config ]# service-control --stop vmware-content-library
Operation not cancellable. Please wait for it to finish...
Performing stop operation on service content-library...
Successfully stopped service content-library
root@nac5vcsa [ /etc/vmware-content-library/config ]# service-control --start vmware-content-library
Operation not cancellable. Please wait for it to finish...
Performing start operation on service content-library...
Successfully started service content-library
root@nac5vcsa [ /etc/vmware-content-library/config ]# service-control --stop vmware-vapi-endpoint
Operation not cancellable. Please wait for it to finish...
Performing stop operation on service vapi-endpoint...
Successfully stopped service vapi-endpoint
root@nac5vcsa [ /etc/vmware-content-library/config ]# service-control --start vmware-vapi-endpoint
Operation not cancellable. Please wait for it to finish...
Performing start operation on service vapi-endpoint...
Successfully started service vapi-endpoint
root@nac5vcsa [ /etc/vmware-content-library/config ]#

Regards
Wojciech