ralish
Enthusiast
Enthusiast

vCenter collection failed alerts post-upgrade to vRLI 8l1

Jump to solution

Since updating to vRealize Log Insight v8.1 from v8.0 I'm seeing a daily "vCenter collection failed" alert. The vCenter Server in question is running v7.0 GA (VCSA), and despite the error, data is clearly being collected as I can see it in the user interface.

If I browse to the Administration -> Integration -> vSphere pane, all the details for the connection appear correct and the Collection Status is Collecting (with a green tick). ESXi hosts are also configured and log data being received. I've tried removing the connection and re-adding it, which works fine, but the daily collection failure alert is still received.

I took a look at the appliance logs and see the following output in /var/log/loginsight/runtime.log at the time the alert is raised:

[2020-04-23 22:14:14.377+0000] ["Streaming RPC Cancellation Manager Timer"/10.10.15.51 WARN] [com.vmware.loginsight.commons.rpc.cancellation.StreamingRPCCancellationPolicy] [Q-Token expired: 18d03a097850e150]

[2020-04-23 22:14:14.377+0000] ["Streaming RPC Cancellation Manager Timer"/10.10.15.51 WARN] [com.vmware.loginsight.commons.rpc.cancellation.StreamingRPCCancellationPolicy] [Q-Token expired: 629e527ebf3d756e]

[2020-04-23 22:14:14.377+0000] ["Streaming RPC Cancellation Manager Timer"/10.10.15.51 WARN] [com.vmware.loginsight.commons.rpc.cancellation.StreamingRPCCancellationPolicy] [Q-Token expired: 255553674c96a33f]

[2020-04-23 22:14:24.465+0000] ["pool-10-thread-1"/10.10.15.51 INFO] [com.vmware.loginsight.commons.security.UrlConnectionManager] [Sending 'GET' request to URL : https://<vcsa-fqdn>/rest/appliance/system/version]

[2020-04-23 22:14:24.472+0000] ["DaemonCommands-thread-5651"/10.10.15.51 INFO] [com.sun.xml.internal.ws.monitoring] [Global client monitoring disabled. https://localhost/sdk/vimService will not be monitored]

[2020-04-23 22:14:24.474+0000] ["DaemonCommands-thread-5651"/10.10.15.51 INFO] [com.sun.metro.assembler] [MASM0002: Default [ jaxws-tubes-default.xml ] configuration file located at [ jar:file:/usr/java/jre-vmware/lib/resources.jar!/com/sun/xml/internal/ws/assembler/jaxws-tubes-default.xml ]]

[2020-04-23 22:14:24.480+0000] ["DaemonCommands-thread-5651"/10.10.15.51 INFO] [com.sun.metro.assembler] [MASM0007: No application metro.xml configuration file found.]

[2020-04-23 22:14:24.571+0000] ["pool-10-thread-1"/10.10.15.51 INFO] [com.vmware.loginsight.commons.security.UrlConnectionManager] [Response Code : 403]

[2020-04-23 22:14:24.571+0000] ["pool-10-thread-1"/10.10.15.51 INFO] [com.vmware.loginsight.commons.security.UrlConnectionManager] [Processed GET request to https://<vcsa-fqdn>/rest/appliance/system/version in 106msec]

[2020-04-23 22:14:24.571+0000] ["pool-10-thread-1"/10.10.15.51 INFO] [com.vmware.loginsight.commons.security.UrlConnectionManager] [Sending 'POST' request to URL : https://<vcsa-fqdn>/rest/com/vmware/cis/session]

[2020-04-23 22:14:24.662+0000] ["pool-9-thread-6486"/10.10.15.51 INFO] [com.vmware.vapi.provider.local.LocalProvider] [call to invokeMethod for com.vmware.loginsight.api.strata.index_messages]

[2020-04-23 22:14:24.662+0000] ["pool-9-thread-6486"/10.10.15.51 INFO] [com.vmware.loginsight.commons.rpc.clientconnpool.ClientConnectionPool] [1 pooled connections to hostname: 0.0.0.0, port: 16573, service: com.vmware.loginsight.ingestion.importer.LogImporterService$Client [10 suppressed]]

[2020-04-23 22:14:24.675+0000] ["DaemonCommands-thread-5651"/10.10.15.51 INFO] [com.sun.xml.internal.ws.monitoring] [Global client monitoring disabled. https://localhost/sdk/vimService will not be monitored]

[2020-04-23 22:14:24.677+0000] ["DaemonCommands-thread-5651"/10.10.15.51 INFO] [com.sun.metro.assembler] [MASM0002: Default [ jaxws-tubes-default.xml ] configuration file located at [ jar:file:/usr/java/jre-vmware/lib/resources.jar!/com/sun/xml/internal/ws/assembler/jaxws-tubes-default.xml ]]

[2020-04-23 22:14:24.684+0000] ["DaemonCommands-thread-5651"/10.10.15.51 INFO] [com.sun.metro.assembler] [MASM0007: No application metro.xml configuration file found.]

[2020-04-23 22:14:24.710+0000] ["pool-9-thread-6486"/10.10.15.51 INFO] [com.vmware.vapi.provider.local.LocalProvider] [call to invokeMethod for com.vmware.loginsight.api.strata.flush_index]

[2020-04-23 22:14:24.796+0000] ["DaemonCommands-thread-5651"/10.10.15.51 INFO] [com.sun.xml.internal.ws.monitoring] [Global client monitoring disabled. https://localhost/sdk/vimService will not be monitored]

[2020-04-23 22:14:24.798+0000] ["DaemonCommands-thread-5651"/10.10.15.51 INFO] [com.sun.metro.assembler] [MASM0002: Default [ jaxws-tubes-default.xml ] configuration file located at [ jar:file:/usr/java/jre-vmware/lib/resources.jar!/com/sun/xml/internal/ws/assembler/jaxws-tubes-default.xml ]]

[2020-04-23 22:14:24.805+0000] ["DaemonCommands-thread-5651"/10.10.15.51 INFO] [com.sun.metro.assembler] [MASM0007: No application metro.xml configuration file found.]

[2020-04-23 22:14:24.900+0000] ["DaemonCommands-thread-5651"/10.10.15.51 INFO] [com.vmware.loginsight.daemon.CommandManager] [configureHosts took 0.43497276 seconds]

[2020-04-23 22:14:24.934+0000] ["pool-10-thread-1"/10.10.15.51 INFO] [com.vmware.loginsight.commons.security.UrlConnectionManager] [Response Code : 200]

[2020-04-23 22:14:24.934+0000] ["pool-10-thread-1"/10.10.15.51 INFO] [com.vmware.loginsight.commons.security.UrlConnectionManager] [Processed POST request to https://<vcsa-fqdn>/rest/com/vmware/cis/session in 363msec]

[2020-04-23 22:14:24.934+0000] ["pool-10-thread-1"/10.10.15.51 INFO] [com.vmware.loginsight.commons.security.UrlConnectionManager] [Sending 'GET' request to URL : https://<vcsa-fqdn>/rest/appliance/system/version]

[2020-04-23 22:14:24.999+0000] ["pool-10-thread-1"/10.10.15.51 INFO] [com.vmware.loginsight.commons.security.UrlConnectionManager] [Response Code : 403]

[2020-04-23 22:14:24.999+0000] ["pool-10-thread-1"/10.10.15.51 INFO] [com.vmware.loginsight.commons.security.UrlConnectionManager] [Processed GET request to https://<vcsa-fqdn>/rest/appliance/system/version in 65msec]

[2020-04-23 22:14:25.519+0000] ["PersistentNotification-thread-20"/10.10.15.51 INFO] [com.vmware.loginsight.daemon.notifications.PersistentNotificationQueue] [Sending notification 'vCenter collection failed triggered at 2020-04-23T22:14:25.001Z

[2020-04-23 22:14:25.519+0000] ["PersistentNotification-thread-20"/10.10.15.51 INFO] [com.vmware.loginsight.notifications.EmailNotificationProvider] [Sending email notification]

[2020-04-23 22:14:25.521+0000] ["PersistentNotification-thread-20"/10.10.15.51 INFO] [com.vmware.loginsight.commons.email.Mailer] [Attempt to send html e-mail (given token=WH2aoI) from: 'loginsight@<domain>' to: '<alerting-email>' with subject: 'Log Insight Admin Alert: vCenter collection failed']

The HTTP 403 response from the /rest/appliance/system/version endpoint seems particularly relevant? If I make the same request manually using the same credentials as the vRLI server is using to connect to VCSA though, I get the expected response, so I'm not sure what's going on here.

Is this a known issue with vRLI 8.1? I updated it around the same time as VCSA to 7.0 (from 6.7), so possibly it's specific to VCSA 7.0 support? Any advice/ideas appreciated!

Tags (3)
1 Solution

Accepted Solutions
rwk1982
Enthusiast
Enthusiast

We also get the same "Collection Failed" after the Update to 8.1. Can you check the log "/storage/var/loginsight/plugins/vsphere/li-vsphere.log" for "Error running vSphere WCP collection" like:

[2020-02-30 25:00:00.00+0000] ["pool-10-thread-1"/127.0.0.1 ERROR] [com.vmware.loginsight.scheduled.ScheduledPluginService] [Error running vSphere WCP collection]

java.lang.Exception: vCenter API vma-vcsa02.sv-services.at is not available, response code: 403, message: {"type":"com.vmware.vapi.std.errors.unauthorized","value":{"messages":[{"args":[],"default_message":"Unable to authorize user","id":"vapi.security.authorization.invalid"}]}}

        at com.vmware.loginsight.scheduled.VSphereWCPConnector.validateVCenterVersion(VSphereWCPConnector.java:143)

        at com.vmware.loginsight.scheduled.VSphereWCPConnector.fetchEvents(VSphereWCPConnector.java:163)

        at com.vmware.loginsight.scheduled.ScheduledPluginService$ScheduledPluginServiceImpl.fetchEventsFromWCP(ScheduledPluginService.java:619)

        at com.vmware.loginsight.scheduled.ScheduledPluginService$ScheduledPluginServiceImpl.lambda$executeVsphereCollection$2(ScheduledPluginService.java:517)

        at java.util.concurrent.FutureTask.run(FutureTask.java:266)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

The WCP collection is for the collection of data from Kubernetes clusters and it is enabled per default. It can be disabled by changing the parameter "wcp-collection-enabled" from

<config>

...

  <vsphere>

    <wcp-collection-enabled value="true" />

...

</config>

to

<config>

...

  <vsphere>

    <wcp-collection-enabled value="false" />

...

</config>

in the internal configuration options from vRLI: https://kb.vmware.com/s/article/2123058

We set the paramter yesterday and got no error mail today.

Drink coffee.. Do stupid things faster with more energy...

View solution in original post

4 Replies
j_dubs
Enthusiast
Enthusiast

We get this almost daily since 8.0 on multiple vCenters.

I honestly haven't looked into it yet, but our vCenters are 6.7.  vRLI was 8.0 and now 8.1.

Will see if I can dig and see if we're getting the same in our logs.

rwk1982
Enthusiast
Enthusiast

We also get the same "Collection Failed" after the Update to 8.1. Can you check the log "/storage/var/loginsight/plugins/vsphere/li-vsphere.log" for "Error running vSphere WCP collection" like:

[2020-02-30 25:00:00.00+0000] ["pool-10-thread-1"/127.0.0.1 ERROR] [com.vmware.loginsight.scheduled.ScheduledPluginService] [Error running vSphere WCP collection]

java.lang.Exception: vCenter API vma-vcsa02.sv-services.at is not available, response code: 403, message: {"type":"com.vmware.vapi.std.errors.unauthorized","value":{"messages":[{"args":[],"default_message":"Unable to authorize user","id":"vapi.security.authorization.invalid"}]}}

        at com.vmware.loginsight.scheduled.VSphereWCPConnector.validateVCenterVersion(VSphereWCPConnector.java:143)

        at com.vmware.loginsight.scheduled.VSphereWCPConnector.fetchEvents(VSphereWCPConnector.java:163)

        at com.vmware.loginsight.scheduled.ScheduledPluginService$ScheduledPluginServiceImpl.fetchEventsFromWCP(ScheduledPluginService.java:619)

        at com.vmware.loginsight.scheduled.ScheduledPluginService$ScheduledPluginServiceImpl.lambda$executeVsphereCollection$2(ScheduledPluginService.java:517)

        at java.util.concurrent.FutureTask.run(FutureTask.java:266)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

The WCP collection is for the collection of data from Kubernetes clusters and it is enabled per default. It can be disabled by changing the parameter "wcp-collection-enabled" from

<config>

...

  <vsphere>

    <wcp-collection-enabled value="true" />

...

</config>

to

<config>

...

  <vsphere>

    <wcp-collection-enabled value="false" />

...

</config>

in the internal configuration options from vRLI: https://kb.vmware.com/s/article/2123058

We set the paramter yesterday and got no error mail today.

Drink coffee.. Do stupid things faster with more energy...
ralish
Enthusiast
Enthusiast

Great stuff Robert. I confirm we see the same "Error running vSphere WCP collection" output on our installation. We've disabled the referenced configuration setting and will advise results.

0 Kudos
ralish
Enthusiast
Enthusiast

No such error today Robert, so I think you've solved it. Hopefully VMware can improve their handling in a future point release to more elegantly handle this issue.

0 Kudos