4 Replies Latest reply on Apr 30, 2020 5:16 PM by ralish

    vCenter collection failed alerts post-upgrade to vRLI 8l1

    ralish Enthusiast

      Since updating to vRealize Log Insight v8.1 from v8.0 I'm seeing a daily "vCenter collection failed" alert. The vCenter Server in question is running v7.0 GA (VCSA), and despite the error, data is clearly being collected as I can see it in the user interface.

       

      If I browse to the Administration -> Integration -> vSphere pane, all the details for the connection appear correct and the Collection Status is Collecting (with a green tick). ESXi hosts are also configured and log data being received. I've tried removing the connection and re-adding it, which works fine, but the daily collection failure alert is still received.

       

      I took a look at the appliance logs and see the following output in /var/log/loginsight/runtime.log at the time the alert is raised:

      [2020-04-23 22:14:14.377+0000] ["Streaming RPC Cancellation Manager Timer"/10.10.15.51 WARN] [com.vmware.loginsight.commons.rpc.cancellation.StreamingRPCCancellationPolicy] [Q-Token expired: 18d03a097850e150]
      [2020-04-23 22:14:14.377+0000] ["Streaming RPC Cancellation Manager Timer"/10.10.15.51 WARN] [com.vmware.loginsight.commons.rpc.cancellation.StreamingRPCCancellationPolicy] [Q-Token expired: 629e527ebf3d756e]
      [2020-04-23 22:14:14.377+0000] ["Streaming RPC Cancellation Manager Timer"/10.10.15.51 WARN] [com.vmware.loginsight.commons.rpc.cancellation.StreamingRPCCancellationPolicy] [Q-Token expired: 255553674c96a33f]
      [2020-04-23 22:14:24.465+0000] ["pool-10-thread-1"/10.10.15.51 INFO] [com.vmware.loginsight.commons.security.UrlConnectionManager] [Sending 'GET' request to URL : https://<vcsa-fqdn>/rest/appliance/system/version]
      [2020-04-23 22:14:24.472+0000] ["DaemonCommands-thread-5651"/10.10.15.51 INFO] [com.sun.xml.internal.ws.monitoring] [Global client monitoring disabled. https://localhost/sdk/vimService will not be monitored]
      [2020-04-23 22:14:24.474+0000] ["DaemonCommands-thread-5651"/10.10.15.51 INFO] [com.sun.metro.assembler] [MASM0002: Default [ jaxws-tubes-default.xml ] configuration file located at [ jar:file:/usr/java/jre-vmware/lib/resources.jar!/com/sun/xml/internal/ws/assembler/jaxws-tubes-default.xml ]]
      [2020-04-23 22:14:24.480+0000] ["DaemonCommands-thread-5651"/10.10.15.51 INFO] [com.sun.metro.assembler] [MASM0007: No application metro.xml configuration file found.]
      [2020-04-23 22:14:24.571+0000] ["pool-10-thread-1"/10.10.15.51 INFO] [com.vmware.loginsight.commons.security.UrlConnectionManager] [Response Code : 403]
      [2020-04-23 22:14:24.571+0000] ["pool-10-thread-1"/10.10.15.51 INFO] [com.vmware.loginsight.commons.security.UrlConnectionManager] [Processed GET request to https://<vcsa-fqdn>/rest/appliance/system/version in 106msec]
      [2020-04-23 22:14:24.571+0000] ["pool-10-thread-1"/10.10.15.51 INFO] [com.vmware.loginsight.commons.security.UrlConnectionManager] [Sending 'POST' request to URL : https://<vcsa-fqdn>/rest/com/vmware/cis/session]
      [2020-04-23 22:14:24.662+0000] ["pool-9-thread-6486"/10.10.15.51 INFO] [com.vmware.vapi.provider.local.LocalProvider] [call to invokeMethod for com.vmware.loginsight.api.strata.index_messages]
      [2020-04-23 22:14:24.662+0000] ["pool-9-thread-6486"/10.10.15.51 INFO] [com.vmware.loginsight.commons.rpc.clientconnpool.ClientConnectionPool] [1 pooled connections to hostname: 0.0.0.0, port: 16573, service: com.vmware.loginsight.ingestion.importer.LogImporterService$Client [10 suppressed]]
      [2020-04-23 22:14:24.675+0000] ["DaemonCommands-thread-5651"/10.10.15.51 INFO] [com.sun.xml.internal.ws.monitoring] [Global client monitoring disabled. https://localhost/sdk/vimService will not be monitored]
      [2020-04-23 22:14:24.677+0000] ["DaemonCommands-thread-5651"/10.10.15.51 INFO] [com.sun.metro.assembler] [MASM0002: Default [ jaxws-tubes-default.xml ] configuration file located at [ jar:file:/usr/java/jre-vmware/lib/resources.jar!/com/sun/xml/internal/ws/assembler/jaxws-tubes-default.xml ]]
      [2020-04-23 22:14:24.684+0000] ["DaemonCommands-thread-5651"/10.10.15.51 INFO] [com.sun.metro.assembler] [MASM0007: No application metro.xml configuration file found.]
      [2020-04-23 22:14:24.710+0000] ["pool-9-thread-6486"/10.10.15.51 INFO] [com.vmware.vapi.provider.local.LocalProvider] [call to invokeMethod for com.vmware.loginsight.api.strata.flush_index]
      [2020-04-23 22:14:24.796+0000] ["DaemonCommands-thread-5651"/10.10.15.51 INFO] [com.sun.xml.internal.ws.monitoring] [Global client monitoring disabled. https://localhost/sdk/vimService will not be monitored]
      [2020-04-23 22:14:24.798+0000] ["DaemonCommands-thread-5651"/10.10.15.51 INFO] [com.sun.metro.assembler] [MASM0002: Default [ jaxws-tubes-default.xml ] configuration file located at [ jar:file:/usr/java/jre-vmware/lib/resources.jar!/com/sun/xml/internal/ws/assembler/jaxws-tubes-default.xml ]]
      [2020-04-23 22:14:24.805+0000] ["DaemonCommands-thread-5651"/10.10.15.51 INFO] [com.sun.metro.assembler] [MASM0007: No application metro.xml configuration file found.]
      [2020-04-23 22:14:24.900+0000] ["DaemonCommands-thread-5651"/10.10.15.51 INFO] [com.vmware.loginsight.daemon.CommandManager] [configureHosts took 0.43497276 seconds]
      [2020-04-23 22:14:24.934+0000] ["pool-10-thread-1"/10.10.15.51 INFO] [com.vmware.loginsight.commons.security.UrlConnectionManager] [Response Code : 200]
      [2020-04-23 22:14:24.934+0000] ["pool-10-thread-1"/10.10.15.51 INFO] [com.vmware.loginsight.commons.security.UrlConnectionManager] [Processed POST request to https://<vcsa-fqdn>/rest/com/vmware/cis/session in 363msec]
      [2020-04-23 22:14:24.934+0000] ["pool-10-thread-1"/10.10.15.51 INFO] [com.vmware.loginsight.commons.security.UrlConnectionManager] [Sending 'GET' request to URL : https://<vcsa-fqdn>/rest/appliance/system/version]
      [2020-04-23 22:14:24.999+0000] ["pool-10-thread-1"/10.10.15.51 INFO] [com.vmware.loginsight.commons.security.UrlConnectionManager] [Response Code : 403]
      [2020-04-23 22:14:24.999+0000] ["pool-10-thread-1"/10.10.15.51 INFO] [com.vmware.loginsight.commons.security.UrlConnectionManager] [Processed GET request to https://<vcsa-fqdn>/rest/appliance/system/version in 65msec]
      [2020-04-23 22:14:25.519+0000] ["PersistentNotification-thread-20"/10.10.15.51 INFO] [com.vmware.loginsight.daemon.notifications.PersistentNotificationQueue] [Sending notification 'vCenter collection failed triggered at 2020-04-23T22:14:25.001Z
      [2020-04-23 22:14:25.519+0000] ["PersistentNotification-thread-20"/10.10.15.51 INFO] [com.vmware.loginsight.notifications.EmailNotificationProvider] [Sending email notification]
      [2020-04-23 22:14:25.521+0000] ["PersistentNotification-thread-20"/10.10.15.51 INFO] [com.vmware.loginsight.commons.email.Mailer] [Attempt to send html e-mail (given token=WH2aoI) from: 'loginsight@<domain>' to: '<alerting-email>' with subject: 'Log Insight Admin Alert: vCenter collection failed']
      
      

       

      The HTTP 403 response from the /rest/appliance/system/version endpoint seems particularly relevant? If I make the same request manually using the same credentials as the vRLI server is using to connect to VCSA though, I get the expected response, so I'm not sure what's going on here.

       

      Is this a known issue with vRLI 8.1? I updated it around the same time as VCSA to 7.0 (from 6.7), so possibly it's specific to VCSA 7.0 support? Any advice/ideas appreciated!

        • 1. Re: vCenter collection failed alerts post-upgrade to vRLI 8l1
          j_dubs Novice

          We get this almost daily since 8.0 on multiple vCenters.

          I honestly haven't looked into it yet, but our vCenters are 6.7.  vRLI was 8.0 and now 8.1.

          Will see if I can dig and see if we're getting the same in our logs.

          • 2. Re: vCenter collection failed alerts post-upgrade to vRLI 8l1
            Robert ¯\_ツ_/¯ Enthusiast

            We also get the same "Collection Failed" after the Update to 8.1. Can you check the log "/storage/var/loginsight/plugins/vsphere/li-vsphere.log" for "Error running vSphere WCP collection" like:

             

            [2020-02-30 25:00:00.00+0000] ["pool-10-thread-1"/127.0.0.1 ERROR] [com.vmware.loginsight.scheduled.ScheduledPluginService] [Error running vSphere WCP collection]

            java.lang.Exception: vCenter API vma-vcsa02.sv-services.at is not available, response code: 403, message: {"type":"com.vmware.vapi.std.errors.unauthorized","value":{"messages":[{"args":[],"default_message":"Unable to authorize user","id":"vapi.security.authorization.invalid"}]}}

                    at com.vmware.loginsight.scheduled.VSphereWCPConnector.validateVCenterVersion(VSphereWCPConnector.java:143)

                    at com.vmware.loginsight.scheduled.VSphereWCPConnector.fetchEvents(VSphereWCPConnector.java:163)

                    at com.vmware.loginsight.scheduled.ScheduledPluginService$ScheduledPluginServiceImpl.fetchEventsFromWCP(ScheduledPluginService.java:619)

                    at com.vmware.loginsight.scheduled.ScheduledPluginService$ScheduledPluginServiceImpl.lambda$executeVsphereCollection$2(ScheduledPluginService.java:517)

                    at java.util.concurrent.FutureTask.run(FutureTask.java:266)

                    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

                    at java.lang.Thread.run(Thread.java:748)

             

            The WCP collection is for the collection of data from Kubernetes clusters and it is enabled per default. It can be disabled by changing the parameter "wcp-collection-enabled" from

             

            <config>
            ...
              <vsphere>
                <wcp-collection-enabled value="true" />
            ...
            </config>
            

             

            to

             

            <config>
            ...
              <vsphere>
                <wcp-collection-enabled value="false" />
            ...
            </config>
            

             

            in the internal configuration options from vRLI: https://kb.vmware.com/s/article/2123058

             

            We set the paramter yesterday and got no error mail today.

            Drink coffee.. Do stupid things faster with more energy...
            • 3. Re: vCenter collection failed alerts post-upgrade to vRLI 8l1
              ralish Enthusiast

              Great stuff Robert. I confirm we see the same "Error running vSphere WCP collection" output on our installation. We've disabled the referenced configuration setting and will advise results.

              • 4. Re: vCenter collection failed alerts post-upgrade to vRLI 8l1
                ralish Enthusiast

                No such error today Robert, so I think you've solved it. Hopefully VMware can improve their handling in a future point release to more elegantly handle this issue.