Since updating to vRealize Log Insight v8.1 from v8.0 I'm seeing a daily "vCenter collection failed" alert. The vCenter Server in question is running v7.0 GA (VCSA), and despite the error, data is clearly being collected as I can see it in the user interface.
If I browse to the Administration -> Integration -> vSphere pane, all the details for the connection appear correct and the Collection Status is Collecting (with a green tick). ESXi hosts are also configured and log data being received. I've tried removing the connection and re-adding it, which works fine, but the daily collection failure alert is still received.
I took a look at the appliance logs and see the following output in /var/log/loginsight/runtime.log at the time the alert is raised:
[2020-04-23 22:14:14.377+0000] ["Streaming RPC Cancellation Manager Timer"/10.10.15.51 WARN] [com.vmware.loginsight.commons.rpc.cancellation.StreamingRPCCancellationPolicy] [Q-Token expired: 18d03a097850e150]
[2020-04-23 22:14:14.377+0000] ["Streaming RPC Cancellation Manager Timer"/10.10.15.51 WARN] [com.vmware.loginsight.commons.rpc.cancellation.StreamingRPCCancellationPolicy] [Q-Token expired: 629e527ebf3d756e]
[2020-04-23 22:14:14.377+0000] ["Streaming RPC Cancellation Manager Timer"/10.10.15.51 WARN] [com.vmware.loginsight.commons.rpc.cancellation.StreamingRPCCancellationPolicy] [Q-Token expired: 255553674c96a33f]
[2020-04-23 22:14:24.465+0000] ["pool-10-thread-1"/10.10.15.51 INFO] [com.vmware.loginsight.commons.security.UrlConnectionManager] [Sending 'GET' request to URL : https://<vcsa-fqdn>/rest/appliance/system/version]
[2020-04-23 22:14:24.472+0000] ["DaemonCommands-thread-5651"/10.10.15.51 INFO] [com.sun.xml.internal.ws.monitoring] [Global client monitoring disabled. https://localhost/sdk/vimService will not be monitored]
[2020-04-23 22:14:24.474+0000] ["DaemonCommands-thread-5651"/10.10.15.51 INFO] [com.sun.metro.assembler] [MASM0002: Default [ jaxws-tubes-default.xml ] configuration file located at [ jar:file:/usr/java/jre-vmware/lib/resources.jar!/com/sun/xml/internal/ws/assembler/jaxws-tubes-default.xml ]]
[2020-04-23 22:14:24.480+0000] ["DaemonCommands-thread-5651"/10.10.15.51 INFO] [com.sun.metro.assembler] [MASM0007: No application metro.xml configuration file found.]
[2020-04-23 22:14:24.571+0000] ["pool-10-thread-1"/10.10.15.51 INFO] [com.vmware.loginsight.commons.security.UrlConnectionManager] [Response Code : 403]
[2020-04-23 22:14:24.571+0000] ["pool-10-thread-1"/10.10.15.51 INFO] [com.vmware.loginsight.commons.security.UrlConnectionManager] [Processed GET request to https://<vcsa-fqdn>/rest/appliance/system/version in 106msec]
[2020-04-23 22:14:24.571+0000] ["pool-10-thread-1"/10.10.15.51 INFO] [com.vmware.loginsight.commons.security.UrlConnectionManager] [Sending 'POST' request to URL : https://<vcsa-fqdn>/rest/com/vmware/cis/session]
[2020-04-23 22:14:24.662+0000] ["pool-9-thread-6486"/10.10.15.51 INFO] [com.vmware.vapi.provider.local.LocalProvider] [call to invokeMethod for com.vmware.loginsight.api.strata.index_messages]
[2020-04-23 22:14:24.662+0000] ["pool-9-thread-6486"/10.10.15.51 INFO] [com.vmware.loginsight.commons.rpc.clientconnpool.ClientConnectionPool] [1 pooled connections to hostname: 0.0.0.0, port: 16573, service: com.vmware.loginsight.ingestion.importer.LogImporterService$Client [10 suppressed]]
[2020-04-23 22:14:24.675+0000] ["DaemonCommands-thread-5651"/10.10.15.51 INFO] [com.sun.xml.internal.ws.monitoring] [Global client monitoring disabled. https://localhost/sdk/vimService will not be monitored]
[2020-04-23 22:14:24.677+0000] ["DaemonCommands-thread-5651"/10.10.15.51 INFO] [com.sun.metro.assembler] [MASM0002: Default [ jaxws-tubes-default.xml ] configuration file located at [ jar:file:/usr/java/jre-vmware/lib/resources.jar!/com/sun/xml/internal/ws/assembler/jaxws-tubes-default.xml ]]
[2020-04-23 22:14:24.684+0000] ["DaemonCommands-thread-5651"/10.10.15.51 INFO] [com.sun.metro.assembler] [MASM0007: No application metro.xml configuration file found.]
[2020-04-23 22:14:24.710+0000] ["pool-9-thread-6486"/10.10.15.51 INFO] [com.vmware.vapi.provider.local.LocalProvider] [call to invokeMethod for com.vmware.loginsight.api.strata.flush_index]
[2020-04-23 22:14:24.796+0000] ["DaemonCommands-thread-5651"/10.10.15.51 INFO] [com.sun.xml.internal.ws.monitoring] [Global client monitoring disabled. https://localhost/sdk/vimService will not be monitored]
[2020-04-23 22:14:24.798+0000] ["DaemonCommands-thread-5651"/10.10.15.51 INFO] [com.sun.metro.assembler] [MASM0002: Default [ jaxws-tubes-default.xml ] configuration file located at [ jar:file:/usr/java/jre-vmware/lib/resources.jar!/com/sun/xml/internal/ws/assembler/jaxws-tubes-default.xml ]]
[2020-04-23 22:14:24.805+0000] ["DaemonCommands-thread-5651"/10.10.15.51 INFO] [com.sun.metro.assembler] [MASM0007: No application metro.xml configuration file found.]
[2020-04-23 22:14:24.900+0000] ["DaemonCommands-thread-5651"/10.10.15.51 INFO] [com.vmware.loginsight.daemon.CommandManager] [configureHosts took 0.43497276 seconds]
[2020-04-23 22:14:24.934+0000] ["pool-10-thread-1"/10.10.15.51 INFO] [com.vmware.loginsight.commons.security.UrlConnectionManager] [Response Code : 200]
[2020-04-23 22:14:24.934+0000] ["pool-10-thread-1"/10.10.15.51 INFO] [com.vmware.loginsight.commons.security.UrlConnectionManager] [Processed POST request to https://<vcsa-fqdn>/rest/com/vmware/cis/session in 363msec]
[2020-04-23 22:14:24.934+0000] ["pool-10-thread-1"/10.10.15.51 INFO] [com.vmware.loginsight.commons.security.UrlConnectionManager] [Sending 'GET' request to URL : https://<vcsa-fqdn>/rest/appliance/system/version]
[2020-04-23 22:14:24.999+0000] ["pool-10-thread-1"/10.10.15.51 INFO] [com.vmware.loginsight.commons.security.UrlConnectionManager] [Response Code : 403]
[2020-04-23 22:14:24.999+0000] ["pool-10-thread-1"/10.10.15.51 INFO] [com.vmware.loginsight.commons.security.UrlConnectionManager] [Processed GET request to https://<vcsa-fqdn>/rest/appliance/system/version in 65msec]
[2020-04-23 22:14:25.519+0000] ["PersistentNotification-thread-20"/10.10.15.51 INFO] [com.vmware.loginsight.daemon.notifications.PersistentNotificationQueue] [Sending notification 'vCenter collection failed triggered at 2020-04-23T22:14:25.001Z
[2020-04-23 22:14:25.519+0000] ["PersistentNotification-thread-20"/10.10.15.51 INFO] [com.vmware.loginsight.notifications.EmailNotificationProvider] [Sending email notification]
[2020-04-23 22:14:25.521+0000] ["PersistentNotification-thread-20"/10.10.15.51 INFO] [com.vmware.loginsight.commons.email.Mailer] [Attempt to send html e-mail (given token=WH2aoI) from: 'loginsight@<domain>' to: '<alerting-email>' with subject: 'Log Insight Admin Alert: vCenter collection failed']
The HTTP 403 response from the /rest/appliance/system/version endpoint seems particularly relevant? If I make the same request manually using the same credentials as the vRLI server is using to connect to VCSA though, I get the expected response, so I'm not sure what's going on here.
Is this a known issue with vRLI 8.1? I updated it around the same time as VCSA to 7.0 (from 6.7), so possibly it's specific to VCSA 7.0 support? Any advice/ideas appreciated!
We also get the same "Collection Failed" after the Update to 8.1. Can you check the log "/storage/var/loginsight/plugins/vsphere/li-vsphere.log" for "Error running vSphere WCP collection" like:
[2020-02-30 25:00:00.00+0000] ["pool-10-thread-1"/127.0.0.1 ERROR] [com.vmware.loginsight.scheduled.ScheduledPluginService] [Error running vSphere WCP collection]
java.lang.Exception: vCenter API your.vcenter.fqdn is not available, response code: 403, message: {"type":"com.vmware.vapi.std.errors.unauthorized","value":{"messages":[{"args":[],"default_message":"Unable to authorize user","id":"vapi.security.authorization.invalid"}]}}
at com.vmware.loginsight.scheduled.VSphereWCPConnector.validateVCenterVersion(VSphereWCPConnector.java:143)
at com.vmware.loginsight.scheduled.VSphereWCPConnector.fetchEvents(VSphereWCPConnector.java:163)
at com.vmware.loginsight.scheduled.ScheduledPluginService$ScheduledPluginServiceImpl.fetchEventsFromWCP(ScheduledPluginService.java:619)
at com.vmware.loginsight.scheduled.ScheduledPluginService$ScheduledPluginServiceImpl.lambda$executeVsphereCollection$2(ScheduledPluginService.java:517)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
The WCP collection is for the collection of data from Kubernetes clusters and it is enabled per default. It can be disabled by changing the parameter "wcp-collection-enabled" from
<config>
...
<vsphere>
<wcp-collection-enabled value="true" />
...
</config>
to
<config>
...
<vsphere>
<wcp-collection-enabled value="false" />
...
</config>
in the internal configuration options from vRLI: https://kb.vmware.com/s/article/2123058
We set the paramter yesterday and got no error mail today.
We get this almost daily since 8.0 on multiple vCenters.
I honestly haven't looked into it yet, but our vCenters are 6.7. vRLI was 8.0 and now 8.1.
Will see if I can dig and see if we're getting the same in our logs.
We also get the same "Collection Failed" after the Update to 8.1. Can you check the log "/storage/var/loginsight/plugins/vsphere/li-vsphere.log" for "Error running vSphere WCP collection" like:
[2020-02-30 25:00:00.00+0000] ["pool-10-thread-1"/127.0.0.1 ERROR] [com.vmware.loginsight.scheduled.ScheduledPluginService] [Error running vSphere WCP collection]
java.lang.Exception: vCenter API your.vcenter.fqdn is not available, response code: 403, message: {"type":"com.vmware.vapi.std.errors.unauthorized","value":{"messages":[{"args":[],"default_message":"Unable to authorize user","id":"vapi.security.authorization.invalid"}]}}
at com.vmware.loginsight.scheduled.VSphereWCPConnector.validateVCenterVersion(VSphereWCPConnector.java:143)
at com.vmware.loginsight.scheduled.VSphereWCPConnector.fetchEvents(VSphereWCPConnector.java:163)
at com.vmware.loginsight.scheduled.ScheduledPluginService$ScheduledPluginServiceImpl.fetchEventsFromWCP(ScheduledPluginService.java:619)
at com.vmware.loginsight.scheduled.ScheduledPluginService$ScheduledPluginServiceImpl.lambda$executeVsphereCollection$2(ScheduledPluginService.java:517)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
The WCP collection is for the collection of data from Kubernetes clusters and it is enabled per default. It can be disabled by changing the parameter "wcp-collection-enabled" from
<config>
...
<vsphere>
<wcp-collection-enabled value="true" />
...
</config>
to
<config>
...
<vsphere>
<wcp-collection-enabled value="false" />
...
</config>
in the internal configuration options from vRLI: https://kb.vmware.com/s/article/2123058
We set the paramter yesterday and got no error mail today.
Great stuff Robert. I confirm we see the same "Error running vSphere WCP collection" output on our installation. We've disabled the referenced configuration setting and will advise results.
No such error today Robert, so I think you've solved it. Hopefully VMware can improve their handling in a future point release to more elegantly handle this issue.