VMware Cloud Community
doggatas
Enthusiast
Enthusiast

VCSA 6.5 Update 1 - VUM Service will not start

After migrating from Windows 5.5 vCenter to VCSA 6.5 Update1 (intermediate step to 6.7) everything was working well. After the migration the vcenter machine cert was soon to be expired so I replaced it, though I did notice https://vc.domain.com:7444 was still using the old cert. I didn't worry about this as the old cert was not expired. A month or so passes by with vcenter being very stable. Completed host upgrades to 6.5U1 using VUM over the weekend - In the mean time the cert bound to 7444 expired, which caused problems with the Migration wizard to 6.7. Also, VUM service had stopped and wouldn't start (perhaps it was ok until I rebooted the VCSA and now VUM wont start)

With the help of VMWare support we fixed this by replacing the cert in the STS_INTERNAL_SSL_CERT. This fixed the expired cert for 7444 but did not help with the VUM issues. I'm pretty well stuck, I am just waiting on VMWare support to help. In the mean time I thought I'd post here in case someone could possible help.

Regards

I get the following errors in the SSH session when executing service-control --start vmware-updatemgr:

Error executing start on service updatemgr. Details {

    "resolution": null,

    "detail": [

        {

            "args": [

                "updatemgr"

            ],

            "id": "install.ciscommon.service.failstart",

            "localized": "An error occurred while starting service 'updatemgr'",

            "translatable": "An error occurred while starting service '%(0)s'"

        }

    ],

    "componentKey": null,

    "problemId": null

}

Service-control failed. Error {

    "resolution": null,

    "detail": [

        {

            "args": [

                "updatemgr"

            ],

            "id": "install.ciscommon.service.failstart",

            "localized": "An error occurred while starting service 'updatemgr'",

            "translatable": "An error occurred while starting service '%(0)s'"

        }

    ],

    "componentKey": null,

    "problemId": null

}

tail -f updatemgr-utility.log give this:

[2018-10-10 23:29:10,255 INFO] Install Key store for Jetty

[2018-10-10 23:29:11,579 INFO] Keystore installed successfully.

[2018-10-10 23:29:11,953 INFO] Updating VUM extension with VC

[2018-10-10 23:29:12,236 INFO] Updating CM service info

[2018-10-10 23:29:12,402 ERROR] CM ReRegisterService failure. Exception is (cis.cm.fault.ComponentManagerFault) {

   dynamicType = <unset>,

   dynamicProperty = (vmodl.DynamicProperty) [],

   msg = '',

   faultCause = <unset>,

   faultMessage = (vmodl.LocalizableMessage) [],

   errorCode = 0,

   errorMessage = 'UNKNOWN'

}

[2018-10-10 23:29:12,402 ERROR] Unable to update CM service info

Tail on cm.log give me this while starting service:

2018-10-10T23:24:35.847Z [pool-2-thread-1 [] WARN  com.vmware.cis.services.cm.service.impl.LsVmomiSiteStore (f15657f3-ac72-40f0-8c90-3e948201000c)] Call to lookup service failed; uri:https://VCENTER.DoMain.int/lookupservice/sdk [(vmodl.fault.InvalidArgument) {

   faultCause = null,

   faultMessage = null,

   invalidProperty = Invalid certificate

}]

2018-10-10T23:24:35.847Z [pool-2-thread-1 [] ERROR com.vmware.cis.services.cm.service.ServiceManagerImplTemplate (f15657f3-ac72-40f0-8c90-3e948201000c)] reRegisterService v1: Failed to re-register c39131ca-cda5-446d-a6ac-44b51348c107 (vpxd-extension-a6f84462-5ea2-11e6-ada6-0050569777b5@vsphere.local, com.vmware.vcIntegrity/vcIntegrity 6.5.0)

com.vmware.vim.binding.vmodl.fault.InvalidArgument: null

        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_141]

        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_141]

        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_141]

        at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_141]

        at java.lang.Class.newInstance(Class.java:442) ~[?:1.8.0_141]

        at com.vmware.vim.vmomi.core.types.impl.ComplexTypeImpl.newInstance(ComplexTypeImpl.java:174) ~[vlsi-core.jar:?]

        at com.vmware.vim.vmomi.core.types.impl.DefaultDataObjectFactory.newDataObject(DefaultDataObjectFactory.java:25) ~[vlsi-core.jar:?]

        at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.ComplexStackContext.<init>(ComplexStackContext.java:30) ~[vlsi-core.jar:?]

        at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl$UnmarshallSoapFaultContext.parse(UnmarshallerImpl.java:150) ~[vlsi-core.jar:?]

        at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl$UnmarshallSoapFaultContext.unmarshall(UnmarshallerImpl.java:101) ~[vlsi-core.jar:?]

        at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl.unmarshalSoapFault(UnmarshallerImpl.java:88) ~[vlsi-core.jar:?]

        at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl.unmarshalSoapFault(UnmarshallerImpl.java:83) ~[vlsi-core.jar:?]

        at com.vmware.vim.vmomi.client.common.impl.SoapFaultStackContext.setValue(SoapFaultStackContext.java:40) ~[vlsi-client.jar:?]

        at com.vmware.vim.vmomi.client.common.impl.ResponseUnmarshaller.processNextElement(ResponseUnmarshaller.java:127) ~[vlsi-client.jar:?]

        at com.vmware.vim.vmomi.client.common.impl.ResponseUnmarshaller.unmarshal(ResponseUnmarshaller.java:70) ~[vlsi-client.jar:?]

        at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.unmarshalResponse(ResponseImpl.java:274) ~[vlsi-client.jar:?]

        at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setResponse(ResponseImpl.java:230) ~[vlsi-client.jar:?]

        at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.parseResponse(HttpExchangeBase.java:150) ~[vlsi-client.jar:?]

        at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:48) ~[vlsi-client.jar:?]

        at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingBase.executeRunnable(HttpProtocolBindingBase.java:226) ~[vlsi-client.jar:?]

        at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingImpl.send(HttpProtocolBindingImpl.java:110) ~[vlsi-client.jar:?]

        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.sendCall(MethodInvocationHandlerImpl.java:613) ~[vlsi-client.jar:?]

        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.executeCall(MethodInvocationHandlerImpl.java:594) ~[vlsi-client.jar:?]

        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.completeCall(MethodInvocationHandlerImpl.java:345) ~[vlsi-client.jar:?]

        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invokeOperation(MethodInvocationHandlerImpl.java:305) ~[vlsi-client.jar:?]

        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invoke(MethodInvocationHandlerImpl.java:179) ~[vlsi-client.jar:?]

        at com.sun.proxy.$Proxy101.set(Unknown Source) ~[?:?]

        at com.vmware.cis.services.cm.service.impl.LsVmomiSiteStore$LsVmomiWrapper$3.execute(LsVmomiSiteStore.java:229) ~[service-cm.jar:?]

        at com.vmware.cis.services.cm.service.impl.LsVmomiSiteStore$LsVmomiWrapper$3.execute(LsVmomiSiteStore.java:226) ~[service-cm.jar:?]

        at com.vmware.cis.services.cm.service.impl.LsVmomiSiteStore$LsVmomiWrapper.callLs(LsVmomiSiteStore.java:302) ~[service-cm.jar:?]

        at com.vmware.cis.services.cm.service.impl.LsVmomiSiteStore$LsVmomiWrapper.set(LsVmomiSiteStore.java:224) ~[service-cm.jar:?]

        at com.vmware.cis.services.cm.service.impl.LsVmomiSiteStore.updateService(LsVmomiSiteStore.java:622) ~[service-cm.jar:?]

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_141]

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_141]

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_141]

        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_141]

        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333) ~[spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]

        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) ~[spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]

        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]

        at com.vmware.cis.services.common.perfmon.PerfmonInterceptor.invoke(PerfmonInterceptor.java:31) ~[service-common.jar:?]

        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]

        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) ~[spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]

        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]

        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) ~[spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]

        at com.sun.proxy.$Proxy67.updateService(Unknown Source) ~[?:?]

        at com.vmware.cis.services.cm.service.ServiceManagerImplTemplate.reRegisterService(ServiceManagerImplTemplate.java:306) [service-cm.jar:?]

        at com.vmware.cis.services.cm.service.ServiceManagerImpl.reRegisterService(ServiceManagerImpl.java:291) [service-cm.jar:?]

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_141]

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_141]

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_141]

        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_141]

        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333) [spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]

        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) [spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]

        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) [spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]

        at com.vmware.cis.services.common.perfmon.PerfmonInterceptor.invoke(PerfmonInterceptor.java:31) [service-common.jar:?]

        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) [spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]

        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) [spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]

        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) [spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]

        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) [spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]

        at com.sun.proxy.$Proxy68.reRegisterService(Unknown Source) [?:?]

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_141]

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_141]

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_141]

        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_141]

        at com.vmware.vim.vmomi.server.impl.InvocationTask.run(InvocationTask.java:65) [vlsi-server.jar:?]

        at com.vmware.vim.vmomi.server.common.impl.RunnableWrapper$1.run(RunnableWrapper.java:47) [vlsi-server.jar:?]

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_141]

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_141]

        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_141]

tcpdump on lo gives:

Request

POST /lookupservice/sdk HTTP/1.1

Content-Type: text/xml; charset=utf-8

SOAPAction: urn:lookup/2.0

Content-Length: 21474

Host: https://VCENTER.DoMain.int/lookupservice/sdk

Connection: Keep-Alive

User-Agent: VMware vim-java 1.0

Cookie: vmware_soap_session=a7600162-79b1-4797-9e7f-4b885dde550b

Accept-Encoding: gzip,deflate

X-Forwarded-For: 127.0.0.1

X-Forwarded-Proto: https

....CERT HASH in XML stream.........

Request

HTTP/1.1 500

Set-Cookie: vmware_soap_session=a7600162-79b1-4797-9e7f-4b885dde550b; HttpOnly

Content-Type: text/xml;charset=utf-8

Content-Length: 558

Date: Wed, 10 Oct 2018 01:45:27 GMT

Connection: close

<?xml version='1.0' encoding='UTF-8'?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><soapenv:Body><soapenv:Fault><faultcode>ServerFaultCode</faultcode><faultstring/><detail><RuntimeFaultFault xsi:type="InvalidArgument" xmlns="urn:lookup"><invalidProperty>Invalid certificate</invalidProperty></RuntimeFaultFault></detail></soapenv:Fault></soapenv:Body></soapenv:Envelope>

15 Replies
Vijay2027
Expert
Expert

ssh into vcsa

cd /usr/lib/vmware-updatemgr/bin and make sure configvalues.txt and vci-integrity.xml files aren't empty.

Reply
0 Kudos
doggatas
Enthusiast
Enthusiast

Thanks for the replies. Vmware got in there yesterday and sorted. The endpoints for VUM were trusting the wrong cert. They updated the ssltrusts with the correct cert hash and all good now.

Reply
0 Kudos
pzwahlen
Contributor
Contributor

Hi,

I think I am having the same issue as you had in this thread.

Would you be able to clarify which file was edited in order to restore the correct SSL trust ?

Regards and thanks for the update. - Patrick -

Reply
0 Kudos
Vijay2027
Expert
Expert

You will have to use ls_update_certs.py script to correct this to correct thumbprint mismatch.

I would suggest you to log a case with GSS.

Kukezh
Contributor
Contributor

In case someone stumbles here with the similar problem, the process described in this KB helped solve this problem for me: https://kb.vmware.com/s/article/2121689

(Don't forget to backup your vcenter before proceeding)

In short, the process to fix the thumbprint mismatch is as follows:

1) get the current Machine SSL certificate:

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store MACHINE_SSL_CERT --alias __MACHINE_CERT --output /tmp/machineSSL.crt

2) Get the SSL trust:

/usr/lib/vmidentity/tools/scripts/lstool.py list --url https://localhost/lookupservice/sdk --no-check-cert --ep-type com.vmware.cis.cs.identity.sso 2>/dev/null

3) Create some /tmp/newcert.crt with the content like:

-----BEGIN CERTIFICATE-----

<Insert contents of "SSL trust" field here>

-----END CERTIFICATE-----

4) get the thumbprints:

openssl x509 -in /tmp/machineSSL.crt -fingerprint -noout

openssl x509 -in /tmp/newcert.crt -fingerprint -noout

you can confirm that this is your issue here. If the thumbprints do not match, than this is indeed your issue and you need to do the following:

5)Run this script

/usr/lib/vmidentity/tools/scripts/ls_update_certs.py --url http://localhost:7080/lookupservice/sdk --fingerprint <insert fingerprint from newcert.crt> --certfile /tmp/machineSSL.crt --user administrator@vsphere.local --password <password>

Hope this helps.

ankithb
Contributor
Contributor

Thanks for the information,

Reply
0 Kudos
vmwtap
Contributor
Contributor

Kukezh​, that finally made my day after ton´s of senseless articles that all failed to gegenerate the VUM Certs !

Reply
0 Kudos
TomH201110141
Enthusiast
Enthusiast

Amazing, this helped me too to solve a not starting update-manager service on a VCSA 6.7 U3.

THANK YOU!

Reply
0 Kudos
EdwinGM
Contributor
Contributor

Thank you Kukezh

it worked for me.

comment:

if you have special characters in your password tyte it in single quotes; in my case my password was Passw0rd$$  . below command i used.

/usr/lib/vmidentity/tools/scripts/ls_update_certs.py --url http://localhost:7080/lookupservice/sdk --fingerprint 7B:B8:AC:F8:1F:D6:26:7C:3B:FD:24:0D:92:42:44:61:E8:CF:2E:A8 --certfile /tmp/machineSSL.crt --user administrator@vsphere.local --password 'Passw0rd$$'

Vmware And Replication Specialist
Reply
0 Kudos
jfbellemare
Contributor
Contributor

Hello Kukezh,

I have a VCSA 6.5 with external PSCs. That command returns nothing on the VCSA itself:

/usr/lib/vmidentity/tools/scripts/lstool.py list --url https://localhost/lookupservice/sdk --no-check-cert --ep-type com.vmware.cis.cs.identity.sso 2>/dev/null

It does return something on my two PSCs. I can see the SSL Trust for my VCSA's hostname in the list.

Can i still use your fix?

Reply
0 Kudos
Kukezh
Contributor
Contributor

Hi,

I have no experience with external PSC, but as PSC is the one who controls certificates, I assume you need to run this on the PSC itself, though I'm not sure.

Sorry, that I'm not much of a help here.

Reply
0 Kudos
marcelo_soares
Champion
Champion

After 3 days looking for a solution I found this, and it worked as expected. I believe you should include this to the KB 76298 VMware Knowledge Base

Thanks,

Marcelo

Marcelo Soares
Reply
0 Kudos
ovollmer
Contributor
Contributor

Thank you so much, Kukezh! I had this same issue and this solution here worked (after I tried several others all afternoon). 

Reply
0 Kudos
ehorher
Contributor
Contributor

Thanks a lot Kukezh. my vCenter is in 5.5 but this procedure work just fine.

Reply
0 Kudos
jorgesorianoa
Contributor
Contributor

Solution still working 😉

Thanks a lot for the clear description!

Reply
0 Kudos