daphnissov
Immortal
Immortal

HTTP Status 500 when pointing vRO 7.0.1 at Log Insight

Jump to solution

When configuring Log Insight integration with vRO 7.0.1 via the Control Center ("Logging Integration" menu), the following error message is seen and it doesn't matter what port or whether it is via CFAPI or SYSLOG. Could others confirm this behavior?

Pivotal tc Runtime 3.1.3.SR1/8.0.30.C.RELEASE - Error report

HTTP Status 500 - Failed to edit Log Insight Agent configuration file!

type Exception report

message Failed to edit Log Insight Agent configuration file!

description The server encountered an internal error that prevented it from fulfilling this request.

exception

java.io.IOException: Failed to edit Log Insight Agent configuration file! com.vmware.o11n.configuration.editors.LogInsightAgentConfigurationEditor.exec(LogInsightAgentConfigurationEditor.java:149) com.vmware.o11n.configuration.editors.LogInsightAgentConfigurationEditor.save(LogInsightAgentConfigurationEditor.java:100) com.vmware.o11n.configuration.logging.ConfigureLogging.commit(ConfigureLogging.java:59) com.vmware.o11n.controlcenter.logging.LogsController.acceptWizzard(LogsController.java:190) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:498) org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:222) org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:137) org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:110) org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:814) org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:737) org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:959) org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:893) org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:969) org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:871) javax.servlet.http.HttpServlet.service(HttpServlet.java:648) org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:845) javax.servlet.http.HttpServlet.service(HttpServlet.java:729) org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:316) org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:126) org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:90) org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114) org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111) org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:169) org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilterInternal(BasicAuthenticationFilter.java:213) org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) org.springframework.security.web.authentication.ui.DefaultLoginPageGeneratingFilter.doFilter(DefaultLoginPageGeneratingFilter.java:162) org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:205) org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:120) org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:64) org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:53) org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:91) org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:213) org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:176) org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346) org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262) 

note The full stack trace of the root cause is available in the Pivotal tc Runtime 3.1.3.SR1/8.0.30.C.RELEASE logs.


Pivotal tc Runtime 3.1.3.SR1/8.0.30.C.RELEASE

1 Solution

Accepted Solutions
iiliev
VMware Employee
VMware Employee

Yes, this could be a possible bug that during the upgrade the permissions are not set properly.

What you can do is to open the file /etc/sudoers, and near the end there should be the following line

vco     ALL=(root) NOPASSWD: /etc/init.d/vco-server, /etc/init.d/vco-configurator

replace it with the following line

vco     ALL=(root) NOPASSWD: /etc/init.d/vco-server, /etc/init.d/vco-configurator, /var/lib/vco/configuration/bin/config_liagent.sh

Not sure if the change will take effect immediately after file save so you may want to restart the appliance.

View solution in original post

16 Replies
iiliev
VMware Employee
VMware Employee

In the vRO log files, around the same time you got the exception, there should be a log entry looking something like:

result code: some-number

where some-number is a number value representing the status code of the executed command. Could you check the logs and tell us this number?

0 Kudos
daphnissov
Immortal
Immortal

Result code:  1

Here's the actual block from /var/log/vmware/vco/configuration/controlcenter.log

2016-04-28 06:04:41.598+0500 [http-nio-8283-exec-6] INFO  [LogInsightAgentConfigurationEditor]

2016-04-28 06:04:41.665+0500 [http-nio-8283-exec-6] ERROR [LogInsightAgentConfigurationEditor] sudo: a password is required

2016-04-28 06:04:41.666+0500 [http-nio-8283-exec-6] INFO  [LogInsightAgentConfigurationEditor] result code: 1

2016-04-28 06:07:49.867+0500 [http-nio-8283-exec-9] INFO  [LogInsightAgentConfigurationEditor]

2016-04-28 06:07:49.867+0500 [http-nio-8283-exec-9] ERROR [LogInsightAgentConfigurationEditor] sudo: a password is required

2016-04-28 06:07:49.868+0500 [http-nio-8283-exec-9] INFO  [LogInsightAgentConfigurationEditor] result code: 1



This is the full Java context failure message as seen in localhost.log


28-Apr-2016 06:04:41.686 SEVERE [http-nio-8283-exec-6] org.apache.catalina.core.StandardWrapperValve.invoke Servlet.service() for servlet [api] in context with path [/vco-controlcenter] threw exception

java.io.IOException: Failed to edit Log Insight Agent configuration file!

        at com.vmware.o11n.configuration.editors.LogInsightAgentConfigurationEditor.exec(LogInsightAgentConfigurationEditor.java:149)

        at com.vmware.o11n.configuration.editors.LogInsightAgentConfigurationEditor.save(LogInsightAgentConfigurationEditor.java:100)

        at com.vmware.o11n.configuration.logging.ConfigureLogging.commit(ConfigureLogging.java:59)

        at com.vmware.o11n.controlcenter.logging.LogsController.acceptWizzard(LogsController.java:190)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:498)

        at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:222)

        at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:137)

        at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:110)

        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:814)

        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:737)

        at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)

        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:959)

        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:893)

        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:969)

        at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:871)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:648)

        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:845)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)

        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:291)

        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)

        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)

        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:316)

        at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:126)

        at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:90)

        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)

        at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114)

        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)

        at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)

        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)

        at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:169)

        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)

        at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilterInternal(BasicAuthenticationFilter.java:213)

        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)

        at org.springframework.security.web.authentication.ui.DefaultLoginPageGeneratingFilter.doFilter(DefaultLoginPageGeneratingFilter.java:162)

        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)

        at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:205)

        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)

        at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:120)

        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)

        at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:64)

        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)

        at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:53)

        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)

        at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:91)

        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)

        at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:213)

        at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:176)

        at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)

        at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262)

        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)

        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:212)

        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)

        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141)

        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)

        at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:676)

        at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:616)

        at org.apache.catalina.authenticator.SingleSignOn.invoke(SingleSignOn.java:240)

        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)

        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:521)

        at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1096)

        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:674)

        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1500)

        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1456)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

        at java.lang.Thread.run(Thread.java:745)

0 Kudos
iiliev
VMware Employee
VMware Employee

OK, so it fails because configuration script asks for a password:


  sudo: a password is required

Which user account do you use to login to Control Center? The default 'root' user, or some other account?

0 Kudos
daphnissov
Immortal
Immortal

Always login with root to the Control Center.

0 Kudos
iiliev
VMware Employee
VMware Employee

OK.

Could you do a SSH login to vRO appliance, execute the following commands in a command terminal, and provide the output they return?

sudo -l -U root

sudo -l -U vco

Also, could you go to /usr/lib/vco/configuration/bin/ directory, and check the permissions/ownership of the file config_liagent.sh ?

0 Kudos
daphnissov
Immortal
Immortal

sudo -l -U root

ldvro01:~ # sudo -l -U root

Matching Defaults entries for root on ldvro01:

    env_keep+="LANG LANGUAGE LINGUAS LC_* _XKB_CHARSET", env_keep+=XDG_SESSION_COOKIE, targetpw

User root may run the following commands on ldvro01:

    (ALL) ALL

    (ALL) ALL



sudo -l -U vco

ldvro01:~ # sudo -l -U vco

Matching Defaults entries for vco on ldvro01:

    env_keep+="LANG LANGUAGE LINGUAS LC_* _XKB_CHARSET", env_keep+=XDG_SESSION_COOKIE, targetpw

User vco may run the following commands on ldvro01:

    (ALL) ALL

    (root) NOPASSWD: /etc/init.d/vco-server, /etc/init.d/vco-configurator



Also, could you go to /usr/lib/vco/configuration/bin/ directory, and check the permissions/ownership of the file config_liagent.sh ?

ldvro01:/usr/lib/vco/configuration/bin # ll

total 24

-rwx------ 1 vco vco  219 Apr 27 17:50 config_liagent.sh

-rwx------ 1 vco vco  230 Feb 19 15:09 controlcenter.sh

-rw-r--r-- 1 vco vco 6718 Feb 19 15:09 log4j.dtd

-rw-r--r-- 1 vco vco 3315 Feb 19 15:09 propagate.sh

-rwx------ 1 vco vco 1321 Feb 19 15:09 setenv.sh

0 Kudos
iiliev
VMware Employee
VMware Employee

Hmm, on my environment 'sudo -l -U vco' returns the following

User vco may run the following commands on sof2-vco-dhcp232:

    (ALL) ALL

    (root) NOPASSWD: /etc/init.d/vco-server, /etc/init.d/vco-configurator, /var/lib/vco/configuration/bin/config_liagent.sh

Comparing to your environment, mine can also run /var/lib/vco/configuration/bin/config_liagent.sh (LogInsight config script) without asking for a password. This looks like a good reason for failure you are seeing in your environment.

I'm not sure why your user is configured differently. Is that a clean vRO deployment or an upgrade from older vRO version?

I'll try to dig how to fix vco user permissions manually.

0 Kudos
daphnissov
Immortal
Immortal

Hmm, that's interesting. These failures (on three different setups) all had in common that they were upgraded from vRO 7.0. I suspect this might have something to do with it, so I'm deploying a fresh instance of 7.0.1 now to compare the two states.

0 Kudos
daphnissov
Immortal
Immortal

I just tested this on a new deploy of 7.0.1, and the option works as expected here and I get the same return with sudo -l -U vco as you posted. So it's apparent the permissions aren't getting set on the agent config script during an upgrade from 7.0 to 7.0.1.

0 Kudos
iiliev
VMware Employee
VMware Employee

Yes, this could be a possible bug that during the upgrade the permissions are not set properly.

What you can do is to open the file /etc/sudoers, and near the end there should be the following line

vco     ALL=(root) NOPASSWD: /etc/init.d/vco-server, /etc/init.d/vco-configurator

replace it with the following line

vco     ALL=(root) NOPASSWD: /etc/init.d/vco-server, /etc/init.d/vco-configurator, /var/lib/vco/configuration/bin/config_liagent.sh

Not sure if the change will take effect immediately after file save so you may want to restart the appliance.

View solution in original post

Hazenet
Enthusiast
Enthusiast

Just to "confirm" this.

I have a lab vRO, which was deployed as 7.0.0, and then upgraded to 7.0.1, and I am also seeing this:

vro:~ # sudo -l -U vco

Matching Defaults entries for vco on vro:

    env_keep+="LANG LANGUAGE LINGUAS LC_* _XKB_CHARSET", env_keep+=XDG_SESSION_COOKIE, targetpw

User vco may run the following commands on vro:

    (ALL) ALL

    (root) NOPASSWD: /etc/init.d/vco-server, /etc/init.d/vco-configurator

0 Kudos
daphnissov
Immortal
Immortal

Ok, changing the sudoers file by hand does fix the issue and does not seem to require a cycling of either individual services, or the appliance as a whole. I see a success message in the Control Center, and confirm by looking at the liagent.ini configuration file.

vco:~ # cat /etc/liagent.ini

[server]

hostname = loginsight.test.local

proto = cfapi

port = 9000

[logging]

[storage]

0 Kudos
iiliev
VMware Employee
VMware Employee

Nice; case solved Smiley Happy

I'll open a PR to fix the upgrade scripts for next vRO release.

0 Kudos
daphnissov
Immortal
Immortal

Yes, thanks for the help, Ilian. But it looks like I might have discovered another bug with regard to how agent configurations are getting applied on the vRO side. Would you rather I open a new thread for that, or continue it here since it kind of relates?

0 Kudos
iiliev
VMware Employee
VMware Employee

Open a new thread. Inside it, you can mention it is related to this one and provide a link.

0 Kudos
JoseMesquita
Enthusiast
Enthusiast

Thank you!

It fixed my lab environment as well.

In my case I had no integration between vRO and vR LogInsight, but I was getting the similar http 500 error when trying to change the Max file count value in vRO Configure logs settings.

Trying to integrate vRO with vR LogInsight was giving the same http 500 error as well.

I just had to follow your sudoers file modification and a vRO reboot at the end to fix both issues.

Cheers,

Jose

0 Kudos