gbeekmans
Contributor
Contributor

vAPI Endpoint health status alarms after upgrading to 6.5U1

Jump to solution

Hi all,

I upgraded our vCenter to 6.5 Update 1 (the Linux server appliance version). After upgrading, we are receiving the vAPI Endpoint health alarms. Some KB articles I read make mention of spring bean related issues which isn't happening in our case.

/var/log/vmware/vapi/endpoint/endpoint.log contains the following relevant snippets:

2017-08-23T15:21:51.294Z | WARN  | state-manager1            | ApiInterfacesFactory           | Retrieving interfaces for service 2a8ef657-f821-473f-9967-9006ffc99450\com.vmware.cis.cls.vapi has failed.

com.vmware.vapi.client.exception.ConnectionException: Connection refused

        at com.vmware.vapi.internal.protocol.client.rpc.http.ApacheHttpAsyncClientTransport$FutureCallbackImpl.failed(ApacheHttpAsyncClientTransport.java:284)

        at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:130)

        at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.connectionRequestFailed(DefaultClientExchangeHandlerImpl.java:319)

        at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.access$100(DefaultClientExchangeHandlerImpl.java:63)

        at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl$1.failed(DefaultClientExchangeHandlerImpl.java:365)

        at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:130)

        at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager$InternalPoolEntryCallback.failed(PoolingNHttpClientConnectionManager.java:474)

        at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:130)

        at org.apache.http.nio.pool.RouteSpecificPool.failed(RouteSpecificPool.java:165)

        at org.apache.http.nio.pool.AbstractNIOConnPool.requestFailed(AbstractNIOConnPool.java:527)

        at org.apache.http.nio.pool.AbstractNIOConnPool$InternalSessionRequestCallback.failed(AbstractNIOConnPool.java:775)

        at org.apache.http.impl.nio.reactor.SessionRequestImpl.failed(SessionRequestImpl.java:154)

        at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:175)

        at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:147)

        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:348)

        at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:189)

        at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.doExecute(CloseableHttpAsyncClientBase.java:67)

        at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.access$000(CloseableHttpAsyncClientBase.java:38)

        at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:57)

        at java.lang.Thread.run(Thread.java:748)

Caused by: java.net.ConnectException: Connection refused

        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)

        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)

        at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:173)

        ... 7 more

2017-08-23T15:21:51.295Z | WARN  | state-manager1            | ApiInterfacesFactory           | The cache does not contain entry for service 2a8ef657-f821-473f-9967-9006ffc99450\com.vmware.cis.cls.vapi. The interfaces will be skipped.

2017-08-23T15:21:51.295Z | INFO  | state-manager1            | HealthStatusCollectorImpl      | HEALTH YELLOW Failed to connect to 2a8ef657-f821-473f-9967-9006ffc99450\com.vmware.cis.cls.vapi vAPI provider.

2017-08-23T15:21:51.450Z | INFO  | state-manager1            | HealthStatusCollectorImpl      | HEALTH GREEN Current vApi Endpoint health status is created between 2017-08-23T15:21:50GMT and 2017-08-23T15:21:51GMT.

2017-08-23T15:21:51.450Z | INFO  | state-manager1            | HealthConfigurationEventListener | Computed health status is = YELLOW

2017-08-23T15:21:51.450Z | INFO  | state-manager1            | HealthConfigurationEventListener | HEALTH Failed to connect to 2a8ef657-f821-473f-9967-9006ffc99450\com.vmware.cis.cls.vapi vAPI provider.

No amount of rebooting the VM or restarting the service has made a difference.

What would 2a8ef657-f821-473f-9967-9006ffc99450\com.vmware.cis.cls.vapi refer to that's having issues? I wonder if this ID existed prior to the upgrade and now doesn't, or has a new ID that vAPI isn't aware of?

Besides scrapping the vCenter VM and reinstalling from scratch, is there a way to resolve this?

It doesn't seem to impact actual functionality of the system. It just shows the nuisance warnings that we can "reset to green" when it pops up and continue on.

Any thoughts and hints would be greatly appreciated.

Regards,

Gerard Beekmans

1 Solution

Accepted Solutions
johog26
Contributor
Contributor

Hello again,

I got an answer from support and it seems to work. Try this:

Connect to the vCenter Server Appliance with an SSH session and root credentials.

shell.set --enabled true
Type shell and press Enter.

cd /etc/vmware-content-library/config/
cp ts-config.properties ts-config.properties.orig
cp ts-config.properties.rpmnew ts-config.properties.rpmnew.orig
mv ts-config.properties.rpmnew ts-config.properties
service-control --stop vmware-content-library;service-control --start vmware-content-library
service-control --stop vmware-vapi-endpoint
service-control --start vmware-vapi-endpoint

Regards

Joakim H

View solution in original post

6 Replies
johog26
Contributor
Contributor

I have the exact same issue after upgrade from 6.5 to U1. I'm waiting for a respons from VMWare support. If it gets resolved I will post the solution.

Regards

Joakim H

0 Kudos
johog26
Contributor
Contributor

Hello again,

I got an answer from support and it seems to work. Try this:

Connect to the vCenter Server Appliance with an SSH session and root credentials.

shell.set --enabled true
Type shell and press Enter.

cd /etc/vmware-content-library/config/
cp ts-config.properties ts-config.properties.orig
cp ts-config.properties.rpmnew ts-config.properties.rpmnew.orig
mv ts-config.properties.rpmnew ts-config.properties
service-control --stop vmware-content-library;service-control --start vmware-content-library
service-control --stop vmware-vapi-endpoint
service-control --start vmware-vapi-endpoint

Regards

Joakim H

View solution in original post

gbeekmans
Contributor
Contributor

Hi Joakim,

That did indeed fix the issue for me as well. Looking at the two ts-config.properties files, there are only 4 lines of changes that fixed the issue:

transfer.httpClient.http.proxyHost={no-proxy}

transfer.httpClient.http.proxyPort=0

transfer.httpClient.https.proxyHost={no-proxy}

transfer.httpClient.https.proxyPort=0

I didn't do any further testing to see if it was the http or https version that ended up actually fixing it.

Thanks for following-up and sharing your findings.

Regards,

Gerard Beekmans

gaddevamsi84
Contributor
Contributor

That's great!! Worked for me. Thanks

0 Kudos
fieldingbr
Enthusiast
Enthusiast

I have the same issue in my environment.  The issue I'm having with the steps below is I don't see the file 'ts-config.properties.rpmnew' on my vCenter in the location provided.

Thoughts?

Brandon

Sravan_k
Expert
Expert

Please try restarting the "vmware-vapi-endpoint" using below commands

service-control --stop vmware-vapi-endpoint

service-control --start vmware-vapi-endpoint

Regards,

Vkmr

0 Kudos