6 Replies Latest reply on Jan 10, 2020 10:52 AM by ankithb

    VCSA 6.5 Update 1 - VUM Service will not start

    doggatas Novice

      After migrating from Windows 5.5 vCenter to VCSA 6.5 Update1 (intermediate step to 6.7) everything was working well. After the migration the vcenter machine cert was soon to be expired so I replaced it, though I did notice https://vc.domain.com:7444 was still using the old cert. I didn't worry about this as the old cert was not expired. A month or so passes by with vcenter being very stable. Completed host upgrades to 6.5U1 using VUM over the weekend - In the mean time the cert bound to 7444 expired, which caused problems with the Migration wizard to 6.7. Also, VUM service had stopped and wouldn't start (perhaps it was ok until I rebooted the VCSA and now VUM wont start)

       

      With the help of VMWare support we fixed this by replacing the cert in the STS_INTERNAL_SSL_CERT. This fixed the expired cert for 7444 but did not help with the VUM issues. I'm pretty well stuck, I am just waiting on VMWare support to help. In the mean time I thought I'd post here in case someone could possible help.

       

      Regards

       

      I get the following errors in the SSH session when executing service-control --start vmware-updatemgr:

       

      Error executing start on service updatemgr. Details {

          "resolution": null,

          "detail": [

              {

                  "args": [

                      "updatemgr"

                  ],

                  "id": "install.ciscommon.service.failstart",

                  "localized": "An error occurred while starting service 'updatemgr'",

                  "translatable": "An error occurred while starting service '%(0)s'"

              }

          ],

          "componentKey": null,

          "problemId": null

      }

      Service-control failed. Error {

          "resolution": null,

          "detail": [

              {

                  "args": [

                      "updatemgr"

                  ],

                  "id": "install.ciscommon.service.failstart",

                  "localized": "An error occurred while starting service 'updatemgr'",

                  "translatable": "An error occurred while starting service '%(0)s'"

              }

          ],

          "componentKey": null,

          "problemId": null

      }

       

      tail -f updatemgr-utility.log give this:

      [2018-10-10 23:29:10,255 INFO] Install Key store for Jetty

      [2018-10-10 23:29:11,579 INFO] Keystore installed successfully.

      [2018-10-10 23:29:11,953 INFO] Updating VUM extension with VC

      [2018-10-10 23:29:12,236 INFO] Updating CM service info

      [2018-10-10 23:29:12,402 ERROR] CM ReRegisterService failure. Exception is (cis.cm.fault.ComponentManagerFault) {

         dynamicType = <unset>,

         dynamicProperty = (vmodl.DynamicProperty) [],

         msg = '',

         faultCause = <unset>,

         faultMessage = (vmodl.LocalizableMessage) [],

         errorCode = 0,

         errorMessage = 'UNKNOWN'

      }

      [2018-10-10 23:29:12,402 ERROR] Unable to update CM service info

       

       

      Tail on cm.log give me this while starting service:

      2018-10-10T23:24:35.847Z [pool-2-thread-1 [] WARN  com.vmware.cis.services.cm.service.impl.LsVmomiSiteStore (f15657f3-ac72-40f0-8c90-3e948201000c)] Call to lookup service failed; uri:https://VCENTER.DoMain.int/lookupservice/sdk [(vmodl.fault.InvalidArgument) {

         faultCause = null,

         faultMessage = null,

         invalidProperty = Invalid certificate

      }]

      2018-10-10T23:24:35.847Z [pool-2-thread-1 [] ERROR com.vmware.cis.services.cm.service.ServiceManagerImplTemplate (f15657f3-ac72-40f0-8c90-3e948201000c)] reRegisterService v1: Failed to re-register c39131ca-cda5-446d-a6ac-44b51348c107 (vpxd-extension-a6f84462-5ea2-11e6-ada6-0050569777b5@vsphere.local, com.vmware.vcIntegrity/vcIntegrity 6.5.0)

      com.vmware.vim.binding.vmodl.fault.InvalidArgument: null

              at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_141]

              at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_141]

              at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_141]

              at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_141]

              at java.lang.Class.newInstance(Class.java:442) ~[?:1.8.0_141]

              at com.vmware.vim.vmomi.core.types.impl.ComplexTypeImpl.newInstance(ComplexTypeImpl.java:174) ~[vlsi-core.jar:?]

              at com.vmware.vim.vmomi.core.types.impl.DefaultDataObjectFactory.newDataObject(DefaultDataObjectFactory.java:25) ~[vlsi-core.jar:?]

              at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.ComplexStackContext.<init>(ComplexStackContext.java:30) ~[vlsi-core.jar:?]

              at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl$UnmarshallSoapFaultContext.parse(UnmarshallerImpl.java:150) ~[vlsi-core.jar:?]

              at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl$UnmarshallSoapFaultContext.unmarshall(UnmarshallerImpl.java:101) ~[vlsi-core.jar:?]

              at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl.unmarshalSoapFault(UnmarshallerImpl.java:88) ~[vlsi-core.jar:?]

              at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl.unmarshalSoapFault(UnmarshallerImpl.java:83) ~[vlsi-core.jar:?]

              at com.vmware.vim.vmomi.client.common.impl.SoapFaultStackContext.setValue(SoapFaultStackContext.java:40) ~[vlsi-client.jar:?]

              at com.vmware.vim.vmomi.client.common.impl.ResponseUnmarshaller.processNextElement(ResponseUnmarshaller.java:127) ~[vlsi-client.jar:?]

              at com.vmware.vim.vmomi.client.common.impl.ResponseUnmarshaller.unmarshal(ResponseUnmarshaller.java:70) ~[vlsi-client.jar:?]

              at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.unmarshalResponse(ResponseImpl.java:274) ~[vlsi-client.jar:?]

              at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setResponse(ResponseImpl.java:230) ~[vlsi-client.jar:?]

              at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.parseResponse(HttpExchangeBase.java:150) ~[vlsi-client.jar:?]

              at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:48) ~[vlsi-client.jar:?]

              at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingBase.executeRunnable(HttpProtocolBindingBase.java:226) ~[vlsi-client.jar:?]

              at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingImpl.send(HttpProtocolBindingImpl.java:110) ~[vlsi-client.jar:?]

              at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.sendCall(MethodInvocationHandlerImpl.java:613) ~[vlsi-client.jar:?]

              at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.executeCall(MethodInvocationHandlerImpl.java:594) ~[vlsi-client.jar:?]

              at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.completeCall(MethodInvocationHandlerImpl.java:345) ~[vlsi-client.jar:?]

              at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invokeOperation(MethodInvocationHandlerImpl.java:305) ~[vlsi-client.jar:?]

              at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invoke(MethodInvocationHandlerImpl.java:179) ~[vlsi-client.jar:?]

              at com.sun.proxy.$Proxy101.set(Unknown Source) ~[?:?]

              at com.vmware.cis.services.cm.service.impl.LsVmomiSiteStore$LsVmomiWrapper$3.execute(LsVmomiSiteStore.java:229) ~[service-cm.jar:?]

              at com.vmware.cis.services.cm.service.impl.LsVmomiSiteStore$LsVmomiWrapper$3.execute(LsVmomiSiteStore.java:226) ~[service-cm.jar:?]

              at com.vmware.cis.services.cm.service.impl.LsVmomiSiteStore$LsVmomiWrapper.callLs(LsVmomiSiteStore.java:302) ~[service-cm.jar:?]

              at com.vmware.cis.services.cm.service.impl.LsVmomiSiteStore$LsVmomiWrapper.set(LsVmomiSiteStore.java:224) ~[service-cm.jar:?]

              at com.vmware.cis.services.cm.service.impl.LsVmomiSiteStore.updateService(LsVmomiSiteStore.java:622) ~[service-cm.jar:?]

              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_141]

              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_141]

              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_141]

              at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_141]

              at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333) ~[spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]

              at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) ~[spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]

              at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]

              at com.vmware.cis.services.common.perfmon.PerfmonInterceptor.invoke(PerfmonInterceptor.java:31) ~[service-common.jar:?]

              at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]

              at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) ~[spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]

              at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]

              at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) ~[spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]

              at com.sun.proxy.$Proxy67.updateService(Unknown Source) ~[?:?]

              at com.vmware.cis.services.cm.service.ServiceManagerImplTemplate.reRegisterService(ServiceManagerImplTemplate.java:306) [service-cm.jar:?]

              at com.vmware.cis.services.cm.service.ServiceManagerImpl.reRegisterService(ServiceManagerImpl.java:291) [service-cm.jar:?]

              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_141]

              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_141]

              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_141]

              at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_141]

              at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333) [spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]

              at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) [spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]

              at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) [spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]

              at com.vmware.cis.services.common.perfmon.PerfmonInterceptor.invoke(PerfmonInterceptor.java:31) [service-common.jar:?]

              at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) [spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]

              at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) [spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]

              at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) [spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]

              at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) [spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]

              at com.sun.proxy.$Proxy68.reRegisterService(Unknown Source) [?:?]

              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_141]

              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_141]

              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_141]

              at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_141]

              at com.vmware.vim.vmomi.server.impl.InvocationTask.run(InvocationTask.java:65) [vlsi-server.jar:?]

              at com.vmware.vim.vmomi.server.common.impl.RunnableWrapper$1.run(RunnableWrapper.java:47) [vlsi-server.jar:?]

              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_141]

              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_141]

              at java.lang.Thread.run(Thread.java:748) [?:1.8.0_141]

       

       

      tcpdump on lo gives:

      Request

      POST /lookupservice/sdk HTTP/1.1

      Content-Type: text/xml; charset=utf-8

      SOAPAction: urn:lookup/2.0

      Content-Length: 21474

      Host: https://VCENTER.DoMain.int/lookupservice/sdk

      Connection: Keep-Alive

      User-Agent: VMware vim-java 1.0

      Cookie: vmware_soap_session=a7600162-79b1-4797-9e7f-4b885dde550b

      Accept-Encoding: gzip,deflate

      X-Forwarded-For: 127.0.0.1

      X-Forwarded-Proto: https

      ....CERT HASH in XML stream.........

       

      Request

      HTTP/1.1 500

      Set-Cookie: vmware_soap_session=a7600162-79b1-4797-9e7f-4b885dde550b; HttpOnly

      Content-Type: text/xml;charset=utf-8

      Content-Length: 558

      Date: Wed, 10 Oct 2018 01:45:27 GMT

      Connection: close

       

      <?xml version='1.0' encoding='UTF-8'?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><soapenv:Body><soapenv:Fault><faultcode>ServerFaultCode</faultcode><faultstring/><detail><RuntimeFaultFault xsi:type="InvalidArgument" xmlns="urn:lookup"><invalidProperty>Invalid certificate</invalidProperty></RuntimeFaultFault></detail></soapenv:Fault></soapenv:Body></soapenv:Envelope>