topic Re: VCSA 6.5 Update 1 - VUM Service will not start in Update Manager Discussions

VCSA 6.5 Update 1 - VUM Service will not start

doggatas — Wed, 10 Oct 2018 23:45:54 GMT

After migrating from Windows 5.5 vCenter to VCSA 6.5 Update1 (intermediate step to 6.7) everything was working well. After the migration the vcenter machine cert was soon to be expired so I replaced it, though I did notice https://vc.domain.com:7444 was still using the old cert. I didn't worry about this as the old cert was not expired. A month or so passes by with vcenter being very stable. Completed host upgrades to 6.5U1 using VUM over the weekend - In the mean time the cert bound to 7444 expired, which caused problems with the Migration wizard to 6.7. Also, VUM service had stopped and wouldn't start (perhaps it was ok until I rebooted the VCSA and now VUM wont start)

With the help of VMWare support we fixed this by replacing the cert in the STS_INTERNAL_SSL_CERT. This fixed the expired cert for 7444 but did not help with the VUM issues. I'm pretty well stuck, I am just waiting on VMWare support to help. In the mean time I thought I'd post here in case someone could possible help.

Regards

I get the following errors in the SSH session when executing service-control --start vmware-updatemgr:

Error executing start on service updatemgr. Details {

"resolution": null,

"detail": [

{

"args": [

"updatemgr"

"id": "install.ciscommon.service.failstart",

"localized": "An error occurred while starting service 'updatemgr'",

"translatable": "An error occurred while starting service '%(0)s'"

}

"componentKey": null,

"problemId": null

}

Service-control failed. Error {

"resolution": null,

"detail": [

{

"args": [

"updatemgr"

"id": "install.ciscommon.service.failstart",

"localized": "An error occurred while starting service 'updatemgr'",

"translatable": "An error occurred while starting service '%(0)s'"

}

"componentKey": null,

"problemId": null

}

tail -f updatemgr-utility.log give this:

[2018-10-10 23:29:10,255 INFO] Install Key store for Jetty

[2018-10-10 23:29:11,579 INFO] Keystore installed successfully.

[2018-10-10 23:29:11,953 INFO] Updating VUM extension with VC

[2018-10-10 23:29:12,236 INFO] Updating CM service info

[2018-10-10 23:29:12,402 ERROR] CM ReRegisterService failure. Exception is (cis.cm.fault.ComponentManagerFault) {

dynamicType = <unset>,

dynamicProperty = (vmodl.DynamicProperty) [],

msg = '',

faultCause = <unset>,

faultMessage = (vmodl.LocalizableMessage) [],

errorCode = 0,

errorMessage = 'UNKNOWN'

}

[2018-10-10 23:29:12,402 ERROR] Unable to update CM service info

Tail on cm.log give me this while starting service:

2018-10-10T23:24:35.847Z [pool-2-thread-1 [] WARN com.vmware.cis.services.cm.service.impl.LsVmomiSiteStore (f15657f3-ac72-40f0-8c90-3e948201000c)] Call to lookup service failed; uri:https://VCENTER.DoMain.int/lookupservice/sdk [(vmodl.fault.InvalidArgument) {

faultCause = null,

faultMessage = null,

invalidProperty = Invalid certificate

}]

2018-10-10T23:24:35.847Z [pool-2-thread-1 [] ERROR com.vmware.cis.services.cm.service.ServiceManagerImplTemplate (f15657f3-ac72-40f0-8c90-3e948201000c)] reRegisterService v1: Failed to re-register c39131ca-cda5-446d-a6ac-44b51348c107 (vpxd-extension-a6f84462-5ea2-11e6-ada6-0050569777b5@vsphere.local, com.vmware.vcIntegrity/vcIntegrity 6.5.0)

com.vmware.vim.binding.vmodl.fault.InvalidArgument: null

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_141]

at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_141]

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_141]

at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_141]

at java.lang.Class.newInstance(Class.java:442) ~[?:1.8.0_141]

at com.vmware.vim.vmomi.core.types.impl.ComplexTypeImpl.newInstance(ComplexTypeImpl.java:174) ~[vlsi-core.jar:?]

at com.vmware.vim.vmomi.core.types.impl.DefaultDataObjectFactory.newDataObject(DefaultDataObjectFactory.java:25) ~[vlsi-core.jar:?]

at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.ComplexStackContext.<init>(ComplexStackContext.java:30) ~[vlsi-core.jar:?]

at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl$UnmarshallSoapFaultContext.parse(UnmarshallerImpl.java:150) ~[vlsi-core.jar:?]

at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl$UnmarshallSoapFaultContext.unmarshall(UnmarshallerImpl.java:101) ~[vlsi-core.jar:?]

at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl.unmarshalSoapFault(UnmarshallerImpl.java:88) ~[vlsi-core.jar:?]

at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl.unmarshalSoapFault(UnmarshallerImpl.java:83) ~[vlsi-core.jar:?]

at com.vmware.vim.vmomi.client.common.impl.SoapFaultStackContext.setValue(SoapFaultStackContext.java:40) ~[vlsi-client.jar:?]

at com.vmware.vim.vmomi.client.common.impl.ResponseUnmarshaller.processNextElement(ResponseUnmarshaller.java:127) ~[vlsi-client.jar:?]

at com.vmware.vim.vmomi.client.common.impl.ResponseUnmarshaller.unmarshal(ResponseUnmarshaller.java:70) ~[vlsi-client.jar:?]

at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.unmarshalResponse(ResponseImpl.java:274) ~[vlsi-client.jar:?]

at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setResponse(ResponseImpl.java:230) ~[vlsi-client.jar:?]

at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.parseResponse(HttpExchangeBase.java:150) ~[vlsi-client.jar:?]

at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:48) ~[vlsi-client.jar:?]

at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingBase.executeRunnable(HttpProtocolBindingBase.java:226) ~[vlsi-client.jar:?]

at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingImpl.send(HttpProtocolBindingImpl.java:110) ~[vlsi-client.jar:?]

at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.sendCall(MethodInvocationHandlerImpl.java:613) ~[vlsi-client.jar:?]

at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.executeCall(MethodInvocationHandlerImpl.java:594) ~[vlsi-client.jar:?]

at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.completeCall(MethodInvocationHandlerImpl.java:345) ~[vlsi-client.jar:?]

at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invokeOperation(MethodInvocationHandlerImpl.java:305) ~[vlsi-client.jar:?]

at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invoke(MethodInvocationHandlerImpl.java:179) ~[vlsi-client.jar:?]

at com.sun.proxy.$Proxy101.set(Unknown Source) ~[?:?]

at com.vmware.cis.services.cm.service.impl.LsVmomiSiteStore$LsVmomiWrapper$3.execute(LsVmomiSiteStore.java:229) ~[service-cm.jar:?]

at com.vmware.cis.services.cm.service.impl.LsVmomiSiteStore$LsVmomiWrapper$3.execute(LsVmomiSiteStore.java:226) ~[service-cm.jar:?]

at com.vmware.cis.services.cm.service.impl.LsVmomiSiteStore$LsVmomiWrapper.callLs(LsVmomiSiteStore.java:302) ~[service-cm.jar:?]

at com.vmware.cis.services.cm.service.impl.LsVmomiSiteStore$LsVmomiWrapper.set(LsVmomiSiteStore.java:224) ~[service-cm.jar:?]

at com.vmware.cis.services.cm.service.impl.LsVmomiSiteStore.updateService(LsVmomiSiteStore.java:622) ~[service-cm.jar:?]

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_141]

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_141]

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_141]

at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_141]

at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333) ~[spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]

at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) ~[spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]

at com.vmware.cis.services.common.perfmon.PerfmonInterceptor.invoke(PerfmonInterceptor.java:31) ~[service-common.jar:?]

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]

at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) ~[spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]

at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) ~[spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]

at com.sun.proxy.$Proxy67.updateService(Unknown Source) ~[?:?]

at com.vmware.cis.services.cm.service.ServiceManagerImplTemplate.reRegisterService(ServiceManagerImplTemplate.java:306) [service-cm.jar:?]

at com.vmware.cis.services.cm.service.ServiceManagerImpl.reRegisterService(ServiceManagerImpl.java:291) [service-cm.jar:?]

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_141]

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_141]

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_141]

at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_141]

at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333) [spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]

at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) [spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) [spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]

at com.vmware.cis.services.common.perfmon.PerfmonInterceptor.invoke(PerfmonInterceptor.java:31) [service-common.jar:?]

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) [spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]

at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) [spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) [spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]

at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) [spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]

at com.sun.proxy.$Proxy68.reRegisterService(Unknown Source) [?:?]

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_141]

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_141]

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_141]

at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_141]

at com.vmware.vim.vmomi.server.impl.InvocationTask.run(InvocationTask.java:65) [vlsi-server.jar:?]

at com.vmware.vim.vmomi.server.common.impl.RunnableWrapper$1.run(RunnableWrapper.java:47) [vlsi-server.jar:?]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_141]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_141]

at java.lang.Thread.run(Thread.java:748) [?:1.8.0_141]

tcpdump on lo gives:

Request

POST /lookupservice/sdk HTTP/1.1

Content-Type: text/xml; charset=utf-8

SOAPAction: urn:lookup/2.0

Content-Length: 21474

Host: https://VCENTER.DoMain.int/lookupservice/sdk

Connection: Keep-Alive

User-Agent: VMware vim-java 1.0

Cookie: vmware_soap_session=a7600162-79b1-4797-9e7f-4b885dde550b

Accept-Encoding: gzip,deflate

X-Forwarded-For: 127.0.0.1

X-Forwarded-Proto: https

....CERT HASH in XML stream.........

Request

HTTP/1.1 500

Set-Cookie: vmware_soap_session=a7600162-79b1-4797-9e7f-4b885dde550b; HttpOnly

Content-Type: text/xml;charset=utf-8

Content-Length: 558

Date: Wed, 10 Oct 2018 01:45:27 GMT

Connection: close

<?xml version='1.0' encoding='UTF-8'?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><soapenv:Body><soapenv:Fault><faultcode>ServerFaultCode</faultcode><faultstring/><detail><RuntimeFaultFault xsi:type="InvalidArgument" xmlns="urn:lookup"><invalidProperty>Invalid certificate</invalidProperty></RuntimeFaultFault></detail></soapenv:Fault></soapenv:Body></soapenv:Envelope>

Re: VCSA 6.5 Update 1 - VUM Service will not start

Vijay2027 — Thu, 11 Oct 2018 08:18:15 GMT

ssh into vcsa

cd /usr/lib/vmware-updatemgr/bin and make sure configvalues.txt and vci-integrity.xml files aren't empty.

Re: VCSA 6.5 Update 1 - VUM Service will not start

doggatas — Thu, 11 Oct 2018 23:26:15 GMT

Thanks for the replies. Vmware got in there yesterday and sorted. The endpoints for VUM were trusting the wrong cert. They updated the ssltrusts with the correct cert hash and all good now.

Re: VCSA 6.5 Update 1 - VUM Service will not start

pzwahlen — Sat, 04 Jan 2020 15:57:06 GMT

Hi,

I think I am having the same issue as you had in this thread.

Would you be able to clarify which file was edited in order to restore the correct SSL trust ?

Regards and thanks for the update. - Patrick -

Re: VCSA 6.5 Update 1 - VUM Service will not start

Vijay2027 — Wed, 08 Jan 2020 01:47:43 GMT

You will have to use ls_update_certs.py script to correct this to correct thumbprint mismatch.

I would suggest you to log a case with GSS.

Re: VCSA 6.5 Update 1 - VUM Service will not start

Kukezh — Thu, 09 Jan 2020 16:40:54 GMT

In case someone stumbles here with the similar problem, the process described in this KB helped solve this problem for me: https://kb.vmware.com/s/article/2121689

(Don't forget to backup your vcenter before proceeding)

In short, the process to fix the thumbprint mismatch is as follows:

1) get the current Machine SSL certificate:

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store MACHINE_SSL_CERT --alias __MACHINE_CERT --output /tmp/machineSSL.crt

2) Get the SSL trust:

/usr/lib/vmidentity/tools/scripts/lstool.py list --url https://localhost/lookupservice/sdk --no-check-cert --ep-type com.vmware.cis.cs.identity.sso 2>/dev/null

3) Create some /tmp/newcert.crt with the content like:

-----BEGIN CERTIFICATE-----

-----END CERTIFICATE-----

4) get the thumbprints:

openssl x509 -in /tmp/machineSSL.crt -fingerprint -noout

openssl x509 -in /tmp/newcert.crt -fingerprint -noout

you can confirm that this is your issue here. If the thumbprints do not match, than this is indeed your issue and you need to do the following:

5)Run this script

/usr/lib/vmidentity/tools/scripts/ls_update_certs.py --url http://localhost:7080/lookupservice/sdk --fingerprint <insert fingerprint from newcert.crt> --certfile /tmp/machineSSL.crt --user administrator@vsphere.local --password <password>

Hope this helps.

Re: VCSA 6.5 Update 1 - VUM Service will not start

ankithb — Fri, 10 Jan 2020 18:44:51 GMT

Thanks for the information,

Re: VCSA 6.5 Update 1 - VUM Service will not start

vmwtap — Sat, 25 Apr 2020 08:24:12 GMT

Kukezh, that finally made my day after ton´s of senseless articles that all failed to gegenerate the VUM Certs !

Re: VCSA 6.5 Update 1 - VUM Service will not start

TomH201110141 — Fri, 12 Jun 2020 13:43:58 GMT

Amazing, this helped me too to solve a not starting update-manager service on a VCSA 6.7 U3.

THANK YOU!

Re: VCSA 6.5 Update 1 - VUM Service will not start

EdwinGM — Thu, 25 Jun 2020 01:16:35 GMT

Thank you Kukezh

it worked for me.

comment:

if you have special characters in your password tyte it in single quotes; in my case my password was Passw0rd$$ . below command i used.

/usr/lib/vmidentity/tools/scripts/ls_update_certs.py --url http://localhost:7080/lookupservice/sdk --fingerprint 7B:B8:AC:F8:1F:D6:26:7C:3B:FD:24:0D:92:42:44:61:E8:CF:2E:A8 --certfile /tmp/machineSSL.crt --user administrator@vsphere.local --password 'Passw0rd$$'

Re: VCSA 6.5 Update 1 - VUM Service will not start

jfbellemare — Wed, 02 Sep 2020 17:36:21 GMT

Hello Kukezh,

I have a VCSA 6.5 with external PSCs. That command returns nothing on the VCSA itself:

/usr/lib/vmidentity/tools/scripts/lstool.py list --url https://localhost/lookupservice/sdk --no-check-cert --ep-type com.vmware.cis.cs.identity.sso 2>/dev/null

It does return something on my two PSCs. I can see the SSL Trust for my VCSA's hostname in the list.

Can i still use your fix?

Re: VCSA 6.5 Update 1 - VUM Service will not start

Kukezh — Fri, 11 Sep 2020 14:27:09 GMT

Hi,

I have no experience with external PSC, but as PSC is the one who controls certificates, I assume you need to run this on the PSC itself, though I'm not sure.

Sorry, that I'm not much of a help here.

Re: VCSA 6.5 Update 1 - VUM Service will not start

marcelo_soares — Wed, 14 Oct 2020 22:25:38 GMT

After 3 days looking for a solution I found this, and it worked as expected. I believe you should include this to the KB 76298 VMware Knowledge Base

Thanks,

Marcelo

Re: VCSA 6.5 Update 1 - VUM Service will not start

ovollmer — Sat, 06 Feb 2021 00:29:38 GMT

Thank you so much, Kukezh! I had this same issue and this solution here worked (after I tried several others all afternoon).

Re: VCSA 6.5 Update 1 - VUM Service will not start

ehorher — Thu, 06 Jul 2023 06:04:52 GMT

Thanks a lot Kukezh. my vCenter is in 5.5 but this procedure work just fine.

Re: VCSA 6.5 Update 1 - VUM Service will not start

jorgesorianoa — Tue, 10 Oct 2023 17:20:30 GMT

Solution still working

Thanks a lot for the clear description!