VMware Cloud Community
mvelezwhiteeLea
Contributor
Contributor

Guest operation authentication failed for operation List Processes ----- What The ????

Good morning all:

I've been a long time fan of vRealize Log Insight.  I finally got a position with a company that recognizes the value and we've been running a test (eval) for pre-implementation.  Last week, we began getting messages regarding the 'ingestion' rate; along with disk space and retention time messages.  In order to alleviate the warnings, I added a 2TB disk.  I specifically did NOT try to increase the size of the existing drives.  (I seem to recall that such a move can damage the entire VM.)

Now, the system is not functioning at all.  I keep getting the message "Guest operation authentication failed for operation List Processes [name of vm]"  I've noticed also that although the new drive shows up in the "Edit Settings' screen.  I wasn't seeing any indication that the drive had been included; so I decided to reboot the system.  Now the situation is even worse because now, I can't even bring up the login web page.

Has anyone seen this before, and more importantly, does anyone know how to fix this?  Ironically, I was supposed to be giving a team demo of vRLI to my colleagues.  However, the lack of access to the program has obviously curtailed that action.  I'm hoping that my colleagues here in the community will have the solution.  Thanks in advance for any suggestions/help that you can provide.

Reply
0 Kudos
6 Replies
daphnissov
Immortal
Immortal

At this point, you may want to try and make a ticket with GSS to see, even if in evaluation, they can identify the issue.

Reply
0 Kudos
sflanders
Commander
Commander

Weird. Should work. What does the console of the VM say? What do the logs say? A script handles the LVM expansion on reboot and logs to /var/log, if LI is having issues it logs to /storage/var/loginsight. Unfortunately, there is not enough information assist with the issue.

Hope this helps! === If you find this information useful, please award points for "correct" or "helpful". ===
Reply
0 Kudos
mvelezwhiteeLea
Contributor
Contributor

I have WinScp, so I'm already connected into vRLI.  I've maneuvered to /storage/var/loginsight  and there are a myriad of logs and log directories.  Can you tell me which log files/bundles you need in order to point me in a better direction?

Thanks

Reply
0 Kudos
sflanders
Commander
Commander

/var/log/messages and runtime.log from loginsight directory at a minimum -- searching for "error" should narrow it down

Hope this helps! === If you find this information useful, please award points for "correct" or "helpful". ===
Reply
0 Kudos
mvelezwhiteeLea
Contributor
Contributor

Hi sflanders:

I'm sorry to be getting this to you so late.  I've been busy with completing some tasks requiring my learning to use the "New-WebServiceProxy" more effectively.

Anyway, I've attached the files you spoke of.  The website doesn't allow for me to send anything larger than 50 MB.  If you need me to send it to you offline, just let me know.  I received a wonderful letter from VMware Support.  Their position is that since this is an eval version, they cannot provide support.  In fact, they suggested the community.  I don't know if I'm comfortable with that answer or not.  The fact of the matter is that we do have vRealize Log Insight as it was included when our company purchased NSX; plus we were looking at expanding Log Insight to our other domains and datacenters along with our other upcoming purchases for this fiscal year.  It seemed to me that Enterprise Plus customers should be treated a bit more friendly than a dismissive email stating why they CAN'T help us.

As a matter of practice, I don't interface with their support wing unless I really have to.  We have become quite involved in using vRLI; which was at my suggestion.  It has already begun to prove itself, but this problem puts all of that in jeopardy; including VMware's lack of support.

If you are able to glean anything from the log files I've attached, I would be most grateful.  I miss not using the product as it has become such a significant tool for our monitoring and troubleshooting.

Thanks again, respectively,

Miguel

Reply
0 Kudos
sflanders
Commander
Commander

Apologies for the delay and sorry to hear about about support -- it is surprising. If you have a vCenter license OR a NSX license, you do know you can add that to LI and LI will accept it, right? This should get you support as well. Anyways, I looked at the logs and the problem is below. I suspect if you run a "mount" command on the CLI you will not see /storage/core and/or /storage/var mounted. Is this true? If so, try "service loginsight stop; mount -a" to see if you can force mount it -- it may list errors. Next thing to check is the path listed below. If you do a "ls -l /usr/lib/loginsight/application/3rd_party/" what do you get? How about "ls -l /usr/lib/loginsight/application/3rd_party/apache-tomcat-8.5.15/conf/keystore"? Finally, how about "ls -l /storage/var/loginsight/apache-tomcat/conf/keystore"?

[2018-03-09 17:49:39.940+0000] ["main"/172.18.50.66 ERROR] [com.vmware.loginsight.services.EmbeddedService] [Error when starting service com.vmware.loginsight.daemon.shared.TomcatController]

java.io.FileNotFoundException: /usr/lib/loginsight/application/3rd_party/apache-tomcat-8.5.15/conf/keystore does not exist

at com.vmware.loginsight.commons.environment.FileManager.getFileInLocation(FileManager.java:785)

at com.vmware.loginsight.commons.environment.FileManager.getFileInApplication(FileManager.java:305)

at com.vmware.loginsight.daemon.shared.TomcatController.configureTomcatServerXml(TomcatController.java:87)

at com.vmware.loginsight.daemon.shared.TomcatController.start(TomcatController.java:237)

at com.vmware.loginsight.services.EmbeddedService.startService(EmbeddedService.java:122)

at com.vmware.loginsight.daemon.LogInsightDaemon.startService(LogInsightDaemon.java:738)

at com.vmware.loginsight.daemon.LogInsightDaemon.lambda$startTomcatServer$11(LogInsightDaemon.java:565)

at com.vmware.loginsight.daemon.LogInsightDaemon.runAndTime(LogInsightDaemon.java:709)

at com.vmware.loginsight.daemon.LogInsightDaemon.startTomcatServer(LogInsightDaemon.java:561)

at com.vmware.loginsight.daemon.LogInsightDaemon.run(LogInsightDaemon.java:366)

at com.vmware.loginsight.daemon.LogInsightDaemon.main(LogInsightDaemon.java:1010)

[2018-03-09 17:49:39.940+0000] ["main"/172.18.50.66 INFO] [com.vmware.loginsight.daemon.LogInsightDaemon] [Failed to start Tomcat: /usr/lib/loginsight/application/3rd_party/apache-tomcat-8.5.15/conf/keystore does not exist]

[2018-03-09 17:49:39.940+0000] ["main"/172.18.50.66 INFO] [com.vmware.loginsight.daemon.LogInsightDaemon] [Exception during start tomcat web server]

[2018-03-09 17:49:39.940+0000] ["main"/172.18.50.66 INFO] [com.vmware.loginsight.daemon.LogInsightDaemon] [Ended start tomcat web server at 54727 ms after launch, took 34 ms]

[2018-03-09 17:49:39.947+0000] ["main"/172.18.50.66 FATAL] [com.vmware.loginsight.daemon.LogInsightDaemon] [Error starting services]

com.vmware.loginsight.daemon.LogInsightDaemon$StartupFailedException: Daemon startup failed: Failed to start Tomcat: /usr/lib/loginsight/application/3rd_party/apache-tomcat-8.5.15/conf/keystore does not exist.

at com.vmware.loginsight.daemon.LogInsightDaemon.startService(LogInsightDaemon.java:757)

at com.vmware.loginsight.daemon.LogInsightDaemon.lambda$startTomcatServer$11(LogInsightDaemon.java:565)

at com.vmware.loginsight.daemon.LogInsightDaemon.runAndTime(LogInsightDaemon.java:709)

at com.vmware.loginsight.daemon.LogInsightDaemon.startTomcatServer(LogInsightDaemon.java:561)

at com.vmware.loginsight.daemon.LogInsightDaemon.run(LogInsightDaemon.java:366)

at com.vmware.loginsight.daemon.LogInsightDaemon.main(LogInsightDaemon.java:1010)

Caused by: java.lang.Exception: /usr/lib/loginsight/application/3rd_party/apache-tomcat-8.5.15/conf/keystore does not exist

at com.vmware.loginsight.daemon.LogInsightDaemon.startService(LogInsightDaemon.java:746)

... 5 more

[2018-03-09 17:49:39.948+0000] ["main"/172.18.50.66 INFO] [com.vmware.loginsight.daemon.LogInsightDaemon] [Begin stop service API Service at 0 ms after shutdown]

[2018-03-09 17:49:39.948+0000] ["main"/172.18.50.66 INFO] [com.vmware.loginsight.daemon.shared.APIServiceController] [Stopping /]

[2018-03-09 17:49:39.949+0000] ["continue-upgrade"/172.18.50.66 ERROR] [com.vmware.loginsight.daemon.DaemonCommandsHandler] [Error continuing upgrade.]

StartupException(description:com.vmware.loginsight.daemon.LogInsightDaemon$StartupFailedException: Daemon startup failed: Failed to start Tomcat: /usr/lib/loginsight/application/3rd_party/apache-tomcat-8.5.15/conf/keystore does not exist.)

at com.vmware.loginsight.daemon.DaemonCommandsHandler.waitUntilStarted(DaemonCommandsHandler.java:425)

at com.vmware.loginsight.daemon.DaemonCommandsHandler$1.run(DaemonCommandsHandler.java:329)

at java.lang.Thread.run(Unknown Source)

[2018-03-09 17:49:39.951+0000] ["main"/172.18.50.66 ERROR] [play] [Error while stopping logger]

java.lang.ClassCastException: org.slf4j.impl.Log4jLoggerFactory cannot be cast to ch.qos.logback.classic.LoggerContext

at play.api.Logger$.shutdown(Logger.scala:275)

at play.core.server.Server$class.stop(Server.scala:93)

at play.core.server.NettyServer.stop(NettyServer.scala:206)

at com.vmware.loginsight.daemon.shared.APIServiceController.stop(APIServiceController.java:267)

at com.vmware.loginsight.services.EmbeddedService.stopService(EmbeddedService.java:146)

at com.vmware.loginsight.daemon.LogInsightDaemon.lambda$stopService$14(LogInsightDaemon.java:639)

at com.vmware.loginsight.daemon.LogInsightDaemon.runAndTime(LogInsightDaemon.java:709)

at com.vmware.loginsight.daemon.LogInsightDaemon.stopService(LogInsightDaemon.java:638)

at com.vmware.loginsight.daemon.LogInsightDaemon.shutdownServices(LogInsightDaemon.java:628)

at com.vmware.loginsight.daemon.LogInsightDaemon.main(LogInsightDaemon.java:1015)

[2018-03-09 17:49:39.954+0000] ["main"/172.18.50.66 INFO] [play] [Stopping server...]

Hope this helps! === If you find this information useful, please award points for "correct" or "helpful". ===
Reply
0 Kudos