When configuring Log Insight integration with vRO 7.0.1 via the Control Center ("Logging Integration" menu), the following error message is seen and it doesn't matter what port or whether it is via CFAPI or SYSLOG. Could others confirm this behavior?
Pivotal tc Runtime 3.1.3.SR1/8.0.30.C.RELEASE - Error report
type Exception report
message Failed to edit Log Insight Agent configuration file!
description The server encountered an internal error that prevented it from fulfilling this request.
exception
java.io.IOException: Failed to edit Log Insight Agent configuration file! com.vmware.o11n.configuration.editors.LogInsightAgentConfigurationEditor.exec(LogInsightAgentConfigurationEditor.java:149) com.vmware.o11n.configuration.editors.LogInsightAgentConfigurationEditor.save(LogInsightAgentConfigurationEditor.java:100) com.vmware.o11n.configuration.logging.ConfigureLogging.commit(ConfigureLogging.java:59) com.vmware.o11n.controlcenter.logging.LogsController.acceptWizzard(LogsController.java:190) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:498) org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:222) org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:137) org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:110) org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:814) org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:737) org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:959) org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:893) org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:969) org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:871) javax.servlet.http.HttpServlet.service(HttpServlet.java:648) org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:845) javax.servlet.http.HttpServlet.service(HttpServlet.java:729) org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:316) org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:126) org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:90) org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114) org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111) org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:169) org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilterInternal(BasicAuthenticationFilter.java:213) org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) org.springframework.security.web.authentication.ui.DefaultLoginPageGeneratingFilter.doFilter(DefaultLoginPageGeneratingFilter.java:162) org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:205) org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:120) org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:64) org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:53) org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:91) org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:213) org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:176) org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346) org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262)
note The full stack trace of the root cause is available in the Pivotal tc Runtime 3.1.3.SR1/8.0.30.C.RELEASE logs.
Yes, this could be a possible bug that during the upgrade the permissions are not set properly.
What you can do is to open the file /etc/sudoers, and near the end there should be the following line
vco ALL=(root) NOPASSWD: /etc/init.d/vco-server, /etc/init.d/vco-configurator
replace it with the following line
vco ALL=(root) NOPASSWD: /etc/init.d/vco-server, /etc/init.d/vco-configurator, /var/lib/vco/configuration/bin/config_liagent.sh
Not sure if the change will take effect immediately after file save so you may want to restart the appliance.
In the vRO log files, around the same time you got the exception, there should be a log entry looking something like:
result code: some-number
where some-number is a number value representing the status code of the executed command. Could you check the logs and tell us this number?
Result code: 1
Here's the actual block from /var/log/vmware/vco/configuration/controlcenter.log
2016-04-28 06:04:41.598+0500 [http-nio-8283-exec-6] INFO [LogInsightAgentConfigurationEditor]
2016-04-28 06:04:41.665+0500 [http-nio-8283-exec-6] ERROR [LogInsightAgentConfigurationEditor] sudo: a password is required
2016-04-28 06:04:41.666+0500 [http-nio-8283-exec-6] INFO [LogInsightAgentConfigurationEditor] result code: 1
2016-04-28 06:07:49.867+0500 [http-nio-8283-exec-9] INFO [LogInsightAgentConfigurationEditor]
2016-04-28 06:07:49.867+0500 [http-nio-8283-exec-9] ERROR [LogInsightAgentConfigurationEditor] sudo: a password is required
2016-04-28 06:07:49.868+0500 [http-nio-8283-exec-9] INFO [LogInsightAgentConfigurationEditor] result code: 1
This is the full Java context failure message as seen in localhost.log
28-Apr-2016 06:04:41.686 SEVERE [http-nio-8283-exec-6] org.apache.catalina.core.StandardWrapperValve.invoke Servlet.service() for servlet [api] in context with path [/vco-controlcenter] threw exception
java.io.IOException: Failed to edit Log Insight Agent configuration file!
at com.vmware.o11n.configuration.editors.LogInsightAgentConfigurationEditor.exec(LogInsightAgentConfigurationEditor.java:149)
at com.vmware.o11n.configuration.editors.LogInsightAgentConfigurationEditor.save(LogInsightAgentConfigurationEditor.java:100)
at com.vmware.o11n.configuration.logging.ConfigureLogging.commit(ConfigureLogging.java:59)
at com.vmware.o11n.controlcenter.logging.LogsController.acceptWizzard(LogsController.java:190)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:222)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:137)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:110)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:814)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:737)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:959)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:893)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:969)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:871)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:648)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:845)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:291)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:316)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:126)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:90)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:169)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilterInternal(BasicAuthenticationFilter.java:213)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
at org.springframework.security.web.authentication.ui.DefaultLoginPageGeneratingFilter.doFilter(DefaultLoginPageGeneratingFilter.java:162)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:205)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:120)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:64)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:53)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:91)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:213)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:176)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:212)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:676)
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:616)
at org.apache.catalina.authenticator.SingleSignOn.invoke(SingleSignOn.java:240)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:521)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1096)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:674)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1500)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1456)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
OK, so it fails because configuration script asks for a password:
sudo: a password is required
Which user account do you use to login to Control Center? The default 'root' user, or some other account?
Always login with root to the Control Center.
OK.
Could you do a SSH login to vRO appliance, execute the following commands in a command terminal, and provide the output they return?
sudo -l -U root
sudo -l -U vco
Also, could you go to /usr/lib/vco/configuration/bin/ directory, and check the permissions/ownership of the file config_liagent.sh ?
sudo -l -U root
ldvro01:~ # sudo -l -U root
Matching Defaults entries for root on ldvro01:
env_keep+="LANG LANGUAGE LINGUAS LC_* _XKB_CHARSET", env_keep+=XDG_SESSION_COOKIE, targetpw
User root may run the following commands on ldvro01:
(ALL) ALL
(ALL) ALL
sudo -l -U vco
ldvro01:~ # sudo -l -U vco
Matching Defaults entries for vco on ldvro01:
env_keep+="LANG LANGUAGE LINGUAS LC_* _XKB_CHARSET", env_keep+=XDG_SESSION_COOKIE, targetpw
User vco may run the following commands on ldvro01:
(ALL) ALL
(root) NOPASSWD: /etc/init.d/vco-server, /etc/init.d/vco-configurator
Also, could you go to /usr/lib/vco/configuration/bin/ directory, and check the permissions/ownership of the file config_liagent.sh ?
ldvro01:/usr/lib/vco/configuration/bin # ll
total 24
-rwx------ 1 vco vco 219 Apr 27 17:50 config_liagent.sh
-rwx------ 1 vco vco 230 Feb 19 15:09 controlcenter.sh
-rw-r--r-- 1 vco vco 6718 Feb 19 15:09 log4j.dtd
-rw-r--r-- 1 vco vco 3315 Feb 19 15:09 propagate.sh
-rwx------ 1 vco vco 1321 Feb 19 15:09 setenv.sh
Hmm, on my environment 'sudo -l -U vco' returns the following
User vco may run the following commands on sof2-vco-dhcp232:
(ALL) ALL
(root) NOPASSWD: /etc/init.d/vco-server, /etc/init.d/vco-configurator, /var/lib/vco/configuration/bin/config_liagent.sh
Comparing to your environment, mine can also run /var/lib/vco/configuration/bin/config_liagent.sh (LogInsight config script) without asking for a password. This looks like a good reason for failure you are seeing in your environment.
I'm not sure why your user is configured differently. Is that a clean vRO deployment or an upgrade from older vRO version?
I'll try to dig how to fix vco user permissions manually.
Hmm, that's interesting. These failures (on three different setups) all had in common that they were upgraded from vRO 7.0. I suspect this might have something to do with it, so I'm deploying a fresh instance of 7.0.1 now to compare the two states.
I just tested this on a new deploy of 7.0.1, and the option works as expected here and I get the same return with sudo -l -U vco as you posted. So it's apparent the permissions aren't getting set on the agent config script during an upgrade from 7.0 to 7.0.1.
Yes, this could be a possible bug that during the upgrade the permissions are not set properly.
What you can do is to open the file /etc/sudoers, and near the end there should be the following line
vco ALL=(root) NOPASSWD: /etc/init.d/vco-server, /etc/init.d/vco-configurator
replace it with the following line
vco ALL=(root) NOPASSWD: /etc/init.d/vco-server, /etc/init.d/vco-configurator, /var/lib/vco/configuration/bin/config_liagent.sh
Not sure if the change will take effect immediately after file save so you may want to restart the appliance.
Just to "confirm" this.
I have a lab vRO, which was deployed as 7.0.0, and then upgraded to 7.0.1, and I am also seeing this:
vro:~ # sudo -l -U vco
Matching Defaults entries for vco on vro:
env_keep+="LANG LANGUAGE LINGUAS LC_* _XKB_CHARSET", env_keep+=XDG_SESSION_COOKIE, targetpw
User vco may run the following commands on vro:
(ALL) ALL
(root) NOPASSWD: /etc/init.d/vco-server, /etc/init.d/vco-configurator
Ok, changing the sudoers file by hand does fix the issue and does not seem to require a cycling of either individual services, or the appliance as a whole. I see a success message in the Control Center, and confirm by looking at the liagent.ini configuration file.
vco:~ # cat /etc/liagent.ini
[server]
hostname = loginsight.test.local
proto = cfapi
port = 9000
[logging]
[storage]
Nice; case solved
I'll open a PR to fix the upgrade scripts for next vRO release.
Yes, thanks for the help, Ilian. But it looks like I might have discovered another bug with regard to how agent configurations are getting applied on the vRO side. Would you rather I open a new thread for that, or continue it here since it kind of relates?
Open a new thread. Inside it, you can mention it is related to this one and provide a link.
Thank you!
It fixed my lab environment as well.
In my case I had no integration between vRO and vR LogInsight, but I was getting the similar http 500 error when trying to change the Max file count value in vRO Configure logs settings.
Trying to integrate vRO with vR LogInsight was giving the same http 500 error as well.
I just had to follow your sudoers file modification and a vRO reboot at the end to fix both issues.
Cheers,
Jose