Error when trying to add Data Node

[MacVay]

Looking to see if anyone has seen this when trying to add a 2nd Data Node

Current Node is 6.0.1 with Hotfix.. New node is 6.0.1 without Hotfix.  (Tried current without hotfix as well, same error)

Tried it with Passphrase as well as part of troubleshooting.

Cheers,

[vSchroed]

I'm going to test this in my lab. I just applied the hotfix to my vROPs cluster. I have a couple of questions for you to make sure my testing is in line with your setup:

1) I'm assuming you are using the Virtual Appliance and not a Windows or Linux VM node, correct?

2) When you say adding a second data node, do you mean second-node? Or 3rd-node (ie, this is your second data-node, not counting the master-node)?

Let me know!

Jeff

[MacVay]

Thanks for the reply,

Yes, Virtual Appliance (Both)

Original was 6.0.0 then upgraded to 6.0.1 then hotfix.

It is a second node, first node is all in one.

Cheers,

[fduranti]

Do you have any adapter installed ... and specifically... do you have any 5.x adapter installed ?

I had the same problem (6.0 upgraded to 6.0.1 could not add new node to cluster) , it solved with the installation of 2 hotfix: the one in the kb article (VMware KB: Hot fix for vRealize Operations Manager 6.0.1 ) and then a second hotfix from vmware support (as specified on the kb).

I had a problem where some old 5.x adapter was crashing java while checking for compatibility.

My problem was generating some logs in the var/log/casa-logs/casa.log file , something like this:

2015-04-04 20:42:53,289 [7L000IG7] [ajp-bio-127.0.0.1-8011-exec-944] DEBUG com.vmware.vcops.casa.support.subprocess.GeneralCommand:235 - Command '/usr/bin/sudo -n /usr/bin/env python /usr/lib/vmware-vcopssuite/utilities/pakManager/bin/vcopsPakManager.py --action query_pak_detail --pak vmware-OEMAdapter-adapter-35-2209996x8664 --json --auth_token baade2a8-e0b7-4336-82fa-5ffc6d140e01::d5c48914-92a3-44a6-9319-92f8a7fa7475 --force_content_update false' completed successfully with exit code '0'.

2015-04-04 20:42:53,289 [7L000IG7] [ajp-bio-127.0.0.1-8011-exec-944] DEBUG com.vmware.vcops.casa.security.SecurityService:294 - Admin lockout status didn't change.

2015-04-04 20:42:53,289 [7L000IG7] [ajp-bio-127.0.0.1-8011-exec-944] DEBUG org.springframework.web.servlet.mvc.method.annotation.ExceptionHandlerExceptionResolver:132 - Resolving exception from handler [public com.vmware.vcops.casa.upgrade.pak.PlatformCompatibilityDoc com.vmware.vcops.casa.upgrade.pak.PakController.doGetCompatibilityAnalysis(com.vmware.vcops.casa.sysadmin.buildinfo.SuitePlatform)]: java.lang.NullPointerException

2015-04-04 20:42:53,290 [7L000IG7] [ajp-bio-127.0.0.1-8011-exec-944] DEBUG org.springframework.beans.factory.support.DefaultListableBeanFactory:246 - Returning cached instance of singleton bean 'casaControllerExceptionHandler'

2015-04-04 20:42:53,290 [7L000IG7] [ajp-bio-127.0.0.1-8011-exec-944] DEBUG org.springframework.web.servlet.mvc.method.annotation.ExceptionHandlerExceptionResolver:319 - Invoking @ExceptionHandler method: public void com.vmware.vcops.casa.exception.CasaControllerExceptionHandler.runtimeExceptionHandler(java.lang.RuntimeException,javax.servlet.http.HttpServletResponse)

2015-04-04 20:42:53,290 [7L000IG7] [ajp-bio-127.0.0.1-8011-exec-944] WARN com.vmware.vcops.casa.exception.CasaControllerExceptionHandler:148 - general runtime exception

1. java.lang.NullPointerException

at com.vmware.vcops.casa.upgrade.pak.PakService.checkPakCompatibility(PakService.java:1744)

at com.vmware.vcops.casa.upgrade.pak.PakController.doGetCompatibilityAnalysis(PakController.java:326)

at com.vmware.vcops.casa.upgrade.pak.PakController$$FastClassBySpringCGLIB$$ae3f5704.invoke(<generated>)

at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)

at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:700)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)

at com.vmware.vcops.casa.security.AdminUserLockoutStatusChangeInterceptor.invoke(AdminUserLockoutStatusChangeInterceptor.java:39)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)

at org.springframework.aop.interceptor.PerformanceMonitorInterceptor.invokeUnderTrace(PerformanceMonitorInterceptor.java:62)

at org.springframework.aop.interceptor.AbstractTraceInterceptor.invoke(AbstractTraceInterceptor.java:111)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)

at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:633)

at com.vmware.vcops.casa.upgrade.pak.PakController$$EnhancerBySpringCGLIB$$aaa08668.doGetCompatibilityAnalysis(<generated>)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:483)

        at org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:215)

at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:132)

at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104)

at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:745)

at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:685)

at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:80)

at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:919)

at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:851)

at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:953)

at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:844)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:620)

at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:829)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)

at com.vmware.vcops.casa.web.RequestInfoFilter.doFilter(RequestInfoFilter.java:62)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)

at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)

at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118)

at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)

at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)

at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)

at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)

at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:154)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)

at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)

at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:201)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)

at com.vmware.vcops.casa.security.VcopsOtpAuthenticationFilter.doFilter(VcopsOtpAuthenticationFilter.java:114)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)

at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:50)

at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:106)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)

at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)

at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192)

at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160)

at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:343)

at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:260)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)

at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)

at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)

at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)

at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)

at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)

at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)

at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)

at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)

at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:190)

at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)

at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:314)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

at java.lang.Thread.run(Thread.java:745)

[vSchroed]

fduranti might be on the right track with this. I don't have any 5.x adapters installed, adding another node worked just fine for me in the lab.

[MacVay]

I do have adapters;

Hyperic (New Version)

VIN (New Version)

But I also have the Port Adapter.....(Old Version)

I have the hotfix applied but you mention another hotfix?

[fduranti]

If you check /var/log/casa-logs/casa.log you'll find that exception just after the check for port adapter.

There's a second hotfix that is released directly from VMWare Support so you should open an SR and ask for it specifically.

[ggibson9]

I don't believe the hotfix mentioned contains the fix for this, but a up coming patch should.  In the mean time, please tro the solution below:

Updated proceedure

1. Log in to the master node as the root user.
2. Change to the $STORAGE/db/pakRepoLocal folder:

cd $STORAGE/db/pakRepoLocal

   3.    Identify any vCenter Operations Manager 5.x management packs that are installed in the vRealize Operations Manager 6.0.x environment.

grep "5.x" $STORAGE/db/pakRepoLocal/**/*.results

Note: The output will be something like the following, in this case the pack file is 'vmware-vcops-GeneralSql-adapter-218-1872479x8664'

$STORAGE/db/pakRepoLocal/vmware-vcops-GeneralSql-adapter-218-1872479x8664/vmware-vcops-GeneralSql-adapter-218-1872479x8664.results:    "pak_format": "6.x",

Note: For the rest of this document, it will be assumed that the General SQL Loader management pack is the problematic one.

    4.    Make a copy of the folder related to the problematic management pack:

cp -rfp $STORAGE/db/casa/pak/dist_pak_files/vmware-vcops-GeneralSql-adapter-218-1872479x8664.pak /tmp/vmware-vcops-GeneralSql-adapter-218-1872479x8664.$(date +%F.%H%M).

    5.    Remove the pak file from $STORAGE/db/casa/pak/dist_pak_files

      rm $STORAGE/db/casa/pak/dist_pak_files/vmware-vcops-GeneralSql-adapter-218-1872479x8664.pak

[MacVay]

Thanks,

I am in the process of trying this and will report back.

Cheers,

[thomsonac]

I had the exact same problem and VMware support nor I could get it working.  I ended up blowing away the instance and re-deploying a 2 node cluster from the start.  Also be aware that I thought I could export my policies and simply re-import them.  Nope, because I didn't add all of the adapters back (which was the suspected problem) I couldn't keep my policies.

Now with a 2 node cluster, the UI will lock up with some frequency with extremely slow logins and the dreaded "Try refreshing the UI manually" error.  According to support this is a new bug with no timeline for a fix.

Seems like another product released half-baked....

[CoryBarker]

I just went through this today.  Same error message while trying to add two additional data nodes to our vrops 6.0.1 hotfix2 cluster. 

Our problem was that we somehow we had an incompatible vcloud adapter 5.x (management pack) installed.  The correct one was installed after, but the old one was never removed, and was not showing in the UI.  However after reviewing the logs with the VMware engineer we discovered it.  We are using the vROps appliance, so in this case we had to remove the adapter from our master servers (HA pair) and remote collectors. 

1. You will need to bring each node offline as you remove the bad adapter.
2. SSH to the VM
3. Change directory to: # cd /usr/lib/vmware-vcopssuite/utilities/pakManager/bin/
4. Query installed pak files using built in python script: # ./vcopsPakManager.py --action query_pak_files
5. Command to remove pak file: # ./vcopsPakManager.py --action cleanup --remove_pak --pak vmware-bad-adapter-here-version-2160378x8664
6. Query installed pak files again using built in python script to be sure it's gone: # ./vcopsPakManager.py --action query_pak_files
7. Change directory to: # cd $STORAGE/db/casa/pak/dist_pak_files
8. List the directory to find the bad adapter here: # ls
9. Remove the bad adapter: # rm -f vmware-bad-adapter-here-version-2160378x8664.pak
10. Next you will need to edit a json, so Change directory to: # cd /storage/db/pakRepoLocal/
11. Make a copy of the following json as you will be editing the original: # cp vcopsPakManagerCommonHistory.json vcopsPakManagerCommonHistory.json.orig
12. Then edit the json: # vi vcopsPakManagerCommonHistory.json
13. I found it easier to sort this file by the version number of the adapter.  In our case we used command: /21603
14. Alright, there should be a few entries here for the bad adapter that you will need to delete.  So delete the open bracelet {and_the_contents_in_between} to the close bracelet
15. Save and quit
16. Finally use the grep command: # grep -i version_number vcopsPakManagerCommonHistory.json
17. Perform steps 1-16 on each node in your cluster, and then bring each back online.  You should now be able to add your additional node to the cluster.