- VMware Technology Network
- :
- Cloud & SDDC
- :
- vCenter
- :
- Update Manager Discussions
- :
- Re: VCSA 6.5 with HA & 2 PSCs - VUM Service Can't ...
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
VCSA 6.5 with HA & 2 PSCs - VUM Service Can't Start
Hello All,
I am having an issue to start a VUM service based on VCSA 6.5 with HA & 2 PSCs.
Below are some info regarding the issue:
This is while I am trying to start the service.
[ ~ ]# service-control --start vmware-updatemgr
Perform start operation. vmon_profile=None, svc_names=['vmware-updatemgr'], include_coreossvcs=False, include_leafossvcs=False
2020-10-20T08:45:33.389Z Service updatemgr state STOPPED
Error executing start on service updatemgr. Details {
"resolution": null,
"detail": [
{
"args": [
"updatemgr"
],
"id": "install.ciscommon.service.failstart",
"localized": "An error occurred while starting service 'updatemgr'",
"translatable": "An error occurred while starting service '%(0)s'"
}
],
"componentKey": null,
"problemId": null
}
Service-control failed. Error {
"resolution": null,
"detail": [
{
"args": [
"updatemgr"
],
"id": "install.ciscommon.service.failstart",
"localized": "An error occurred while starting service 'updatemgr'",
"translatable": "An error occurred while starting service '%(0)s'"
}
],
"componentKey": null,
"problemId": null
}
updatemgr-utility.log
[2020-10-20 11:44:57,995 ERROR] Unable to update CM service info
[2020-10-20 11:45:33,714 INFO] Install Key store for Jetty
[2020-10-20 11:45:35,427 INFO] Keystore installed successfully.
[2020-10-20 11:45:35,767 INFO] Updating VUM extension with VC
[2020-10-20 11:45:36,175 INFO] Updating CM service info
[2020-10-20 11:45:36,310 ERROR] CM ReRegisterService failure. Exception is (cis.cm.fault.ComponentManagerFault) {
dynamicType = <unset>,
dynamicProperty = (vmodl.DynamicProperty) [],
msg = '',
faultCause = <unset>,
faultMessage = (vmodl.LocalizableMessage) [],
errorCode = 0,
errorMessage = 'UNKNOWN'
}
[2020-10-20 11:45:36,310 ERROR] Unable to update CM service info
cm.log (I have changed some specific information for security reasons)
2020-10-20T11:44:57.989+03:00 [pool-2-thread-1 [] ERROR com.vmware.cis.services.cm.service.ServiceManagerImplTemplate (cf3b9c43-81d5-4d1a-aad6-1bf3dd4b8971)] reRegisterService v1: Failed to re-register 70604e8d-0505-4ee9-ab0c-ad765d4cd1b7 (vpxd-extension-678b208e-5d5e-4c43-b100-bb0f3a079af6@domain.local, com.vmware.vcIntegrity/vcIntegrity 6.5.0)
com.vmware.vim.binding.vmodl.fault.InvalidArgument: null
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_162]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_162]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_162]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_162]
at java.lang.Class.newInstance(Class.java:442) ~[?:1.8.0_162]
at com.vmware.vim.vmomi.core.types.impl.ComplexTypeImpl.newInstance(ComplexTypeImpl.java:174) ~[vlsi-core.jar:?]
at com.vmware.vim.vmomi.core.types.impl.DefaultDataObjectFactory.newDataObject(DefaultDataObjectFactory.java:25) ~[vlsi-core.jar:?]
at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.ComplexStackContext.<init>(ComplexStackContext.java:30) ~[vlsi-core.jar:?]
at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl$UnmarshallSoapFaultContext.parse(UnmarshallerImpl.java:150) ~[vlsi-core.jar:?]
at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl$UnmarshallSoapFaultContext.unmarshall(UnmarshallerImpl.java:101) ~[vlsi-core.jar:?]
at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl.unmarshalSoapFault(UnmarshallerImpl.java:88) ~[vlsi-core.jar:?]
at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl.unmarshalSoapFault(UnmarshallerImpl.java:83) ~[vlsi-core.jar:?]
at com.vmware.vim.vmomi.client.common.impl.SoapFaultStackContext.setValue(SoapFaultStackContext.java:40) ~[vlsi-client.jar:?]
at com.vmware.vim.vmomi.client.common.impl.ResponseUnmarshaller.processNextElement(ResponseUnmarshaller.java:127) ~[vlsi-client.jar:?]
at com.vmware.vim.vmomi.client.common.impl.ResponseUnmarshaller.unmarshal(ResponseUnmarshaller.java:70) ~[vlsi-client.jar:?]
at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.unmarshalResponse(ResponseImpl.java:274) ~[vlsi-client.jar:?]
at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setResponse(ResponseImpl.java:230) ~[vlsi-client.jar:?]
at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.parseResponse(HttpExchangeBase.java:150) ~[vlsi-client.jar:?]
at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:48) ~[vlsi-client.jar:?]
at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingBase.executeRunnable(HttpProtocolBindingBase.java:226) ~[vlsi-client.jar:?]
at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingImpl.send(HttpProtocolBindingImpl.java:110) ~[vlsi-client.jar:?]
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.sendCall(MethodInvocationHandlerImpl.java:613) ~[vlsi-client.jar:?]
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.executeCall(MethodInvocationHandlerImpl.java:594) ~[vlsi-client.jar:?]
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.completeCall(MethodInvocationHandlerImpl.java:345) ~[vlsi-client.jar:?]
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invokeOperation(MethodInvocationHandlerImpl.java:305) ~[vlsi-client.jar:?]
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invoke(MethodInvocationHandlerImpl.java:179) ~[vlsi-client.jar:?]
at com.sun.proxy.$Proxy100.set(Unknown Source) ~[?:?]
at com.vmware.cis.services.cm.service.impl.LsVmomiSiteStore$LsVmomiWrapper$3.execute(LsVmomiSiteStore.java:229) ~[service-cm.jar:?]
at com.vmware.cis.services.cm.service.impl.LsVmomiSiteStore$LsVmomiWrapper$3.execute(LsVmomiSiteStore.java:226) ~[service-cm.jar:?]
at com.vmware.cis.services.cm.service.impl.LsVmomiSiteStore$LsVmomiWrapper.callLs(LsVmomiSiteStore.java:302) ~[service-cm.jar:?]
at com.vmware.cis.services.cm.service.impl.LsVmomiSiteStore$LsVmomiWrapper.set(LsVmomiSiteStore.java:224) ~[service-cm.jar:?]
at com.vmware.cis.services.cm.service.impl.LsVmomiSiteStore.updateService(LsVmomiSiteStore.java:622) ~[service-cm.jar:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_162]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_162]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_162]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162]
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333) ~[spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) ~[spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at com.vmware.cis.services.common.perfmon.PerfmonInterceptor.invoke(PerfmonInterceptor.java:31) ~[service-common.jar:?]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) ~[spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) ~[spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at com.sun.proxy.$Proxy67.updateService(Unknown Source) ~[?:?]
at com.vmware.cis.services.cm.service.ServiceManagerImplTemplate.reRegisterService(ServiceManagerImplTemplate.java:306) [service-cm.jar:?]
at com.vmware.cis.services.cm.service.ServiceManagerImpl.reRegisterService(ServiceManagerImpl.java:291) [service-cm.jar:?]
at sun.reflect.GeneratedMethodAccessor182.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_162]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162]
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333) [spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) [spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) [spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at com.vmware.cis.services.common.perfmon.PerfmonInterceptor.invoke(PerfmonInterceptor.java:31) [service-common.jar:?]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) [spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) [spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) [spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) [spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at com.sun.proxy.$Proxy68.reRegisterService(Unknown Source) [?:?]
at sun.reflect.GeneratedMethodAccessor182.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_162]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162]
at com.vmware.vim.vmomi.server.impl.InvocationTask.run(InvocationTask.java:65) [vlsi-server.jar:?]
at com.vmware.vim.vmomi.server.common.impl.RunnableWrapper$1.run(RunnableWrapper.java:47) [vlsi-server.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
2020-10-20T11:45:25.430+03:00 [pool-15-thread-4 [] INFO com.vmware.vim.lookup.client.SiteAffinityServerEndpointProvider ()] Site affinity is disabled
2020-10-20T11:45:25.505+03:00 [pool-15-thread-4 [] INFO com.vmware.vim.lookup.client.SiteAffinityServerEndpointProvider ()] Site affinity is disabled
2020-10-20T11:45:25.531+03:00 [pool-15-thread-4 [] INFO com.vmware.vim.lookup.client.SiteAffinityServerEndpointProvider ()] Site affinity is disabled
2020-10-20T11:45:35.471+03:00 [pool-15-thread-1 [] INFO com.vmware.cis.services.cm.service.ServiceManagerImplTemplate ()] search v1: Unauthorized, modifying search to only return SSO.
2020-10-20T11:45:35.472+03:00 [pool-15-thread-1 [] INFO com.vmware.vim.lookup.client.SiteAffinityServerEndpointProvider ()] Site affinity is disabled
2020-10-20T11:45:35.588+03:00 [pool-2-thread-1 [] INFO com.vmware.identity.token.impl.SamlTokenImpl (54d7ec2e-40eb-41cf-959f-7fd13f89ecf0)] SAML token for SubjectNameId [value=vpxd-extension-678b208e-5d5e-4c43-b100-bb0f3a079af6@domain.local, format=http://schemas.xmlsoap.org/claims/UPN] successfully parsed from Element
2020-10-20T11:45:35.590+03:00 [pool-2-thread-1 [] INFO com.vmware.cis.services.common.sso.LoginHelper (54d7ec2e-40eb-41cf-959f-7fd13f89ecf0)] loginByToken: Successfully authenticated 'vpxd-extension-678b208e-5d5e-4c43-b100-bb0f3a079af6@domain.local' on session 01fcfd00-95fa-4a5f-a67e-a98dc60b9560
2020-10-20T11:45:35.594+03:00 [pool-15-thread-2 [] INFO com.vmware.vim.lookup.client.SiteAffinityServerEndpointProvider ()] Site affinity is disabled
2020-10-20T11:45:36.184+03:00 [pool-2-thread-1 [] INFO com.vmware.identity.token.impl.SamlTokenImpl (6a4f8100-0b95-4429-8d8b-969c0726fdec)] SAML token for SubjectNameId [value=vpxd-extension-678b208e-5d5e-4c43-b100-bb0f3a079af6@domain.local, format=http://schemas.xmlsoap.org/claims/UPN] successfully parsed from Element
2020-10-20T11:45:36.185+03:00 [pool-2-thread-1 [] INFO com.vmware.cis.services.common.sso.LoginHelper (6a4f8100-0b95-4429-8d8b-969c0726fdec)] loginByToken: Successfully authenticated 'vpxd-extension-678b208e-5d5e-4c43-b100-bb0f3a079af6@domain.local' on session a18aa98c-07b9-4cdf-920c-4b3b9f3f2949
2020-10-20T11:45:36.189+03:00 [pool-15-thread-3 [] INFO com.vmware.vim.lookup.client.SiteAffinityServerEndpointProvider ()] Site affinity is disabled
2020-10-20T11:45:36.229+03:00 [pool-2-thread-1 [] INFO com.vmware.vim.lookup.client.SiteAffinityServerEndpointProvider (da3d8caa-7207-4af8-8782-934dca2833f0)] Site affinity is disabled
2020-10-20T11:45:36.256+03:00 [pool-2-thread-1 [] INFO com.vmware.vim.lookup.client.SiteAffinityServerEndpointProvider (da3d8caa-7207-4af8-8782-934dca2833f0)] Site affinity is disabled
2020-10-20T11:45:36.270+03:00 [pool-2-thread-1 [] INFO com.vmware.vim.lookup.client.SiteAffinityServerEndpointProvider (da3d8caa-7207-4af8-8782-934dca2833f0)] Site affinity is disabled
2020-10-20T11:45:36.305+03:00 [pool-2-thread-1 [] INFO com.vmware.vim.lookup.client.SiteAffinityServerEndpointProvider (da3d8caa-7207-4af8-8782-934dca2833f0)] Site affinity is disabled
2020-10-20T11:45:36.305+03:00 [pool-2-thread-1 [] WARN com.vmware.cis.services.cm.service.impl.LsVmomiSiteStore (da3d8caa-7207-4af8-8782-934dca2833f0)] Call to lookup service failed; uri::https://PSC01.domain.com/lookupservice/sdk [(vmodl.fault.InvalidArgument) {
faultCause = null,
faultMessage = null,
invalidProperty = Invalid certificate
}]
2020-10-20T11:45:36.305+03:00 [pool-2-thread-1 [] ERROR com.vmware.cis.services.cm.service.ServiceManagerImplTemplate (da3d8caa-7207-4af8-8782-934dca2833f0)] reRegisterService v1: Failed to re-register 70604e8d-0505-4ee9-ab0c-ad765d4cd1b7 (vpxd-extension-678b208e-5d5e-4c43-b100-bb0f3a079af6@domain.local, com.vmware.vcIntegrity/vcIntegrity 6.5.0)
com.vmware.vim.binding.vmodl.fault.InvalidArgument: null
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_162]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_162]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_162]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_162]
at java.lang.Class.newInstance(Class.java:442) ~[?:1.8.0_162]
at com.vmware.vim.vmomi.core.types.impl.ComplexTypeImpl.newInstance(ComplexTypeImpl.java:174) ~[vlsi-core.jar:?]
at com.vmware.vim.vmomi.core.types.impl.DefaultDataObjectFactory.newDataObject(DefaultDataObjectFactory.java:25) ~[vlsi-core.jar:?]
at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.ComplexStackContext.<init>(ComplexStackContext.java:30) ~[vlsi-core.jar:?]
at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl$UnmarshallSoapFaultContext.parse(UnmarshallerImpl.java:150) ~[vlsi-core.jar:?]
at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl$UnmarshallSoapFaultContext.unmarshall(UnmarshallerImpl.java:101) ~[vlsi-core.jar:?]
at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl.unmarshalSoapFault(UnmarshallerImpl.java:88) ~[vlsi-core.jar:?]
at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl.unmarshalSoapFault(UnmarshallerImpl.java:83) ~[vlsi-core.jar:?]
at com.vmware.vim.vmomi.client.common.impl.SoapFaultStackContext.setValue(SoapFaultStackContext.java:40) ~[vlsi-client.jar:?]
at com.vmware.vim.vmomi.client.common.impl.ResponseUnmarshaller.processNextElement(ResponseUnmarshaller.java:127) ~[vlsi-client.jar:?]
at com.vmware.vim.vmomi.client.common.impl.ResponseUnmarshaller.unmarshal(ResponseUnmarshaller.java:70) ~[vlsi-client.jar:?]
at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.unmarshalResponse(ResponseImpl.java:274) ~[vlsi-client.jar:?]
at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setResponse(ResponseImpl.java:230) ~[vlsi-client.jar:?]
at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.parseResponse(HttpExchangeBase.java:150) ~[vlsi-client.jar:?]
at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:48) ~[vlsi-client.jar:?]
at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingBase.executeRunnable(HttpProtocolBindingBase.java:226) ~[vlsi-client.jar:?]
at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingImpl.send(HttpProtocolBindingImpl.java:110) ~[vlsi-client.jar:?]
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.sendCall(MethodInvocationHandlerImpl.java:613) ~[vlsi-client.jar:?]
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.executeCall(MethodInvocationHandlerImpl.java:594) ~[vlsi-client.jar:?]
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.completeCall(MethodInvocationHandlerImpl.java:345) ~[vlsi-client.jar:?]
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invokeOperation(MethodInvocationHandlerImpl.java:305) ~[vlsi-client.jar:?]
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invoke(MethodInvocationHandlerImpl.java:179) ~[vlsi-client.jar:?]
at com.sun.proxy.$Proxy100.set(Unknown Source) ~[?:?]
at com.vmware.cis.services.cm.service.impl.LsVmomiSiteStore$LsVmomiWrapper$3.execute(LsVmomiSiteStore.java:229) ~[service-cm.jar:?]
at com.vmware.cis.services.cm.service.impl.LsVmomiSiteStore$LsVmomiWrapper$3.execute(LsVmomiSiteStore.java:226) ~[service-cm.jar:?]
at com.vmware.cis.services.cm.service.impl.LsVmomiSiteStore$LsVmomiWrapper.callLs(LsVmomiSiteStore.java:302) ~[service-cm.jar:?]
at com.vmware.cis.services.cm.service.impl.LsVmomiSiteStore$LsVmomiWrapper.set(LsVmomiSiteStore.java:224) ~[service-cm.jar:?]
at com.vmware.cis.services.cm.service.impl.LsVmomiSiteStore.updateService(LsVmomiSiteStore.java:622) ~[service-cm.jar:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_162]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_162]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_162]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162]
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333) ~[spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) ~[spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at com.vmware.cis.services.common.perfmon.PerfmonInterceptor.invoke(PerfmonInterceptor.java:31) ~[service-common.jar:?]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) ~[spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) ~[spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at com.sun.proxy.$Proxy67.updateService(Unknown Source) ~[?:?]
at com.vmware.cis.services.cm.service.ServiceManagerImplTemplate.reRegisterService(ServiceManagerImplTemplate.java:306) [service-cm.jar:?]
at com.vmware.cis.services.cm.service.ServiceManagerImpl.reRegisterService(ServiceManagerImpl.java:291) [service-cm.jar:?]
at sun.reflect.GeneratedMethodAccessor182.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_162]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162]
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333) [spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) [spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) [spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at com.vmware.cis.services.common.perfmon.PerfmonInterceptor.invoke(PerfmonInterceptor.java:31) [service-common.jar:?]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) [spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) [spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) [spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) [spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at com.sun.proxy.$Proxy68.reRegisterService(Unknown Source) [?:?]
at sun.reflect.GeneratedMethodAccessor182.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_162]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162]
at com.vmware.vim.vmomi.server.impl.InvocationTask.run(InvocationTask.java:65) [vlsi-server.jar:?]
at com.vmware.vim.vmomi.server.common.impl.RunnableWrapper$1.run(RunnableWrapper.java:47) [vlsi-server.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
2020-10-20T11:48:27.891+03:00 [pool-15-thread-4 [] INFO com.vmware.vim.lookup.client.SiteAffinityServerEndpointProvider ()] Site affinity is disabled
2020-10-20T11:48:27.968+03:00 [pool-15-thread-4 [] INFO com.vmware.vim.lookup.client.SiteAffinityServerEndpointProvider ()] Site affinity is disabled
2020-10-20T11:48:27.994+03:00 [pool-15-thread-4 [] INFO com.vmware.vim.lookup.client.SiteAffinityServerEndpointProvider ()] Site affinity is disabled
2020-10-20T11:48:28.239+03:00 [pool-15-thread-1 [] INFO com.vmware.vim.lookup.client.SiteAffinityServerEndpointProvider ()] Site affinity is disabled
2020-10-20T11:48:28.273+03:00 [pool-15-thread-2 [] INFO com.vmware.vim.lookup.client.SiteAffinityServerEndpointProvider ()] Site affinity is disabled
2020-10-20T11:48:35.191+03:00 [pool-2-thread-1 [] INFO com.vmware.cis.services.common.sso.LoginHelper (af898b09-76d9-4ab5-94b5-18d287f29cd0)] loginByToken: User 'vsphere-webclient-678b208e-5d5e-4c43-b100-bb0f3a079af6@domain.local' is already authenticated
2020-10-20T11:48:35.193+03:00 [pool-15-thread-3 [] INFO com.vmware.vim.lookup.client.SiteAffinityServerEndpointProvider ()] Site affinity is disabled
2020-10-20T11:48:35.204+03:00 [pool-15-thread-3 [] INFO com.vmware.vim.lookup.client.SiteAffinityServerEndpointProvider ()] Site affinity is disabled
2020-10-20T11:48:35.230+03:00 [pool-15-thread-3 [] INFO com.vmware.vim.lookup.client.SiteAffinityServerEndpointProvider ()] Site affinity is disabled
2020-10-20T11:48:35.493+03:00 [pool-2-thread-1 [] INFO com.vmware.cis.services.common.sso.LoginHelper (0feecd59-fdc2-4631-9849-e6349adf3e0a)] loginByToken: User 'vsphere-webclient-678b208e-5d5e-4c43-b100-bb0f3a079af6@domain.local' is already authenticated
2020-10-20T11:48:35.521+03:00 [pool-15-thread-4 [] INFO com.vmware.vim.lookup.client.SiteAffinityServerEndpointProvider ()] Site affinity is disabled
2020-10-20T11:48:35.639+03:00 [pool-15-thread-1 [] INFO com.vmware.vim.lookup.client.SiteAffinityServerEndpointProvider ()] Site affinity is disabled
2020-10-20T11:48:35.783+03:00 [pool-15-thread-2 [] INFO com.vmware.vim.lookup.client.SiteAffinityServerEndpointProvider ()] Site affinity is disabled
2020-10-20T11:48:35.828+03:00 [pool-15-thread-3 [] INFO com.vmware.vim.lookup.client.SiteAffinityServerEndpointProvider ()] Site affinity is disabled
2020-10-20T11:48:35.853+03:00 [pool-15-thread-4 [] INFO com.vmware.vim.lookup.client.SiteAffinityServerEndpointProvider ()] Site affinity is disabled
2020-10-20T11:48:35.882+03:00 [pool-15-thread-1 [] INFO com.vmware.vim.lookup.client.SiteAffinityServerEndpointProvider ()] Site affinity is disabled
2020-10-20T11:48:35.922+03:00 [pool-15-thread-2 [] INFO com.vmware.vim.lookup.client.SiteAffinityServerEndpointProvider ()] Site affinity is disabled
2020-10-20T11:48:35.955+03:00 [pool-15-thread-3 [] INFO com.vmware.vim.lookup.client.SiteAffinityServerEndpointProvider ()] Site affinity is disabled
2020-10-20T11:49:47.847+03:00 [pool-15-thread-4 [] INFO com.vmware.vim.lookup.client.SiteAffinityServerEndpointProvider ()] Site affinity is disabled
After I checked these details and did a quick search, I found these articles below:
https://kb.vmware.com/s/article/76298
https://communities.vmware.com/thread/598354
There is some mention regarding:
The SSL certificate of STS service cannot be verified
But I couldn't find it in my log.
This is the output from the suggest command: Openssl s_client -connect <PSC/VCSA-FQDN/IP>:7444 | less
Openssl s_client -connect psc01.domain.com:7444
CONNECTED(00000194)
depth=0 CN = psc01.domain.com, C = XX, ST = XX, L = Palo Alto, O = XX, OU = VMware Engineering
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = psc01.domain.com, C = XX, ST = XX, L = Palo Alto, O = XX, OU = VMware Engineering
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
0 s:CN = psc01.domain.com, C = XX, ST = XX, L = Palo Alto, O = XX, OU = VMware Engineering
i:CN = psc01.domain.com, DC = XX, DC = local, C = XX, ST = XX, O = psc01.domain.com, OU = VMware Engineering
---
Server certificate
-----BEGIN CERTIFICATE-----
XXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXX
-----END CERTIFICATE-----
subject=CN = psc01.domain.com, C = XX, ST = XX, L = Palo Alto, O = XX, OU = VMware Engineering
issuer=CN = psc01.domain.com, DC = XX, DC = local, C = XX, ST = XX, O = psc01.domain.com, OU = VMware Engineering
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 1592 bytes and written 451 bytes
Verification error: unable to verify the first certificate
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
Session-ID: XXXXXXXXXXXXXXXXXXXXXXXXXXXX
Session-ID-ctx:
Master-Key: XXXXXXXXXXXXXXXXXXXXXXXXXXXX
PSK identity: None
PSK identity hint: None
SRP username: None
Start Time: 1603185518
Timeout : 7200 (sec)
Verify return code: 21 (unable to verify the first certificate)
Extended master secret: yes
---
read:errno=0
I made bold some possible errors/issues. The problem looks quite similar with the one I am dealing with but still couldn't find any solution yet. Any help would be much appreciated.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hey,
Could you please confirm us that your STS certificate is not expired by following the Web Client section on this kb: VMware Knowledge Base
Also what do you mean by two PSCs? Are you pointing a single vCenter to a Load Balancer where both PSCs are connected or how is it?
Regards!
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello Lalegre,
Thank you for your reply. I can confirm that the STS certificate is valid. Regarding the two PSCs from what I am aware of they are connected directly to one VCSA instance without load balancer, there must be active/standby but I am not 100% sure.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I am suspecting issue with vcenter certificates if there is a mismatch leading to registration failure..
Note: Ensure to take all nodes cold snapshots prior to running this
You can try the steps this on the VCSA
Best Regards,
MS
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Yes I got you now, you have a secondary PSC in case something fails and do the repoint manually. I can see the extension is failing to re-register so if your STS certificates are valid please try to re-register the VUM extension manually following the next blog post: https://vwannabe.com/2018/02/21/how-to-re-register-the-embedded-vmware-update-manager-vum-to-its-vce...
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Yes, that's correct regarding the design, I have just been informed. Regarding the VUM I did unregistered it manually from the mobs and after I re-registered based on the link you sent me and after that I created this post.
Thank you
George.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
George,
Could you please share with us the contents of the file you created: vci-integrity.xml
Also could you please run a ls -ltr on the directory where the file is placed and show us the output?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello,
After investigation I think the issue must started from expired private certificates.
I rebooted both PSCs now I am getting the error below:
This is VCSA
[500] SSO error: com.vmware.vim.vmomi.client.common.UnexpectedStatusCodeException: Unexpected status code: 404
Check the vSphere Web Client server logs for details.
This is PSCs
HTTP Status 400 – Bad Request
Type Status Report
Message An error occurred while sending an authentication request to the PSC Single Sign-On server - null
Description The server cannot or will not process the request due to something that is perceived to be a client error (e.g., malformed request syntax, invalid request message framing, or deceptive request routing).
Apache Tomcat/8.5.13
# /usr/lib/vmware-vmca/bin/certificate-manager
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
| |
| *** Welcome to the vSphere 6.5 Certificate Manager *** |
| |
| -- Select Operation -- |
| |
| 1. Replace Machine SSL certificate with Custom Certificate |
| |
| 2. Replace VMCA Root certificate with Custom Signing |
| Certificate and replace all Certificates |
| |
| 3. Replace Machine SSL certificate with VMCA Certificate |
| |
| 4. Regenerate a new VMCA Root Certificate and |
| replace all certificates |
| |
| 5. Replace Solution user certificates with |
| Custom Certificate |
| |
| 6. Replace Solution user certificates with VMCA certificates |
| |
| 7. Revert last performed operation by re-publishing old |
| certificates |
| |
| 8. Reset all Certificates |
|_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _|
Note : Use Ctrl-D to exit.
Option[1 to 8]: 8
Do you wish to generate all certificates using configuration file : Option[Y/N] ? : Y
Please provide valid SSO and VC privileged user credential to perform certificate operations.
Enter username [Administrator@vsphere.local]:administrator@domain.local
Enter password:
Traceback (most recent call last):
File "/usr/lib/vmware-vmca/bin/certificate-manager", line 716, in <module>
exit(main())
File "/usr/lib/vmware-vmca/bin/certificate-manager", line 710, in main
parse_arguments()
File "/usr/lib/vmware-vmca/bin/certificate-manager", line 701, in parse_arguments
get_machine_ssl_cert_to_dir()
File "/usr/lib/vmware-vmca/bin/certificate-manager", line 573, in get_machine_ssl_cert_to_dir
vecs.get_cert_file(Constants.MACHINE_SSL_STORE, Constants.MACHINE_SSL_ALIAS, oldcert)
File "/usr/lib/vmware/site-packages/cis/certificateManagerOps.py", line 438, in get_cert_file
raise e
cis.exceptions.InvokeCommandException: {
"resolution": null,
"detail": [
{
"args": [
""
],
"id": "install.ciscommon.command.errinvoke",
"localized": "An error occurred while invoking external command : ''",
"translatable": "An error occurred while invoking external command : '%(0)s'"
},
"Error while creating backup cert file for MACHINE_SSL_CERT"
],
"componentKey": null,
"problemId": null
}
After a quick search this issue is due certificate issue possibly. I am trying now to 8. Reset all Certificates but it looks like there is some issue with the MACHINE_SSL_CERT
I found a VMware article which suggests to delete and re-create the directory for the backup/VMware certs but didn't work. https://kb.vmware.com/s/article/67660
My PSC01 has this directory with these files:
[ /var/tmp/vmware ]# ls
certool.cfg cis-license machine.cfg MACHINE_SSL_CERT.cfg root.cfg vsphere-webclient.cfg
I am trying to find a way to by pass for now the SSO/PSCs just to be able to login on the VCSA. Also need to fix the issue with the PSCs by resetting the certificates or restore them by the certificate store/backup_store if possible.
Thanks for your help.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
If your issue is with the Machine SSL certificate you should not do the Reset All but only the Machine SSL certificate using a signed one by the VMCA, could you please try with the option 3 to see how it works?
And remember that if you have any external tools consuming vCenter by its certificate you will need to trust the connection again as the thumbprint will change.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello Lalegre,
I just checked both PSCs and on PSC01 under this path: /etc/vmware/vmware-vmafd missing both: machine-ssl.crt machine-ssl.key - on the other PSC02 both of them are exist. On both of them have the ca.crt
Also from the VCSA are missing the below directories:
psc01 [ /usr/lib/vmware-vmca/bin ]# /usr/lib/vmware-vmafd/bin/vecs-cli store list
MACHINE_SSL_CERT
TRUSTED_ROOTS
TRUSTED_ROOT_CRLS
machine
vsphere-webclient
BACKUP_STORE
The think now is while I run before the wizard, I choose the number 4. and now the ca.crt has been replaced with a new one...
[ /storage/certmanager ]# /usr/lib/vmware-vmca/bin/certificate-manager
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
| |
| *** Welcome to the vSphere 6.5 Certificate Manager *** |
| |
| -- Select Operation -- |
| |
| 1. Replace Machine SSL certificate with Custom Certificate |
| |
| 2. Replace VMCA Root certificate with Custom Signing |
| Certificate and replace all Certificates |
| |
| 3. Replace Machine SSL certificate with VMCA Certificate |
| |
| 4. Regenerate a new VMCA Root Certificate and |
| replace all certificates |
| |
| 5. Replace Solution user certificates with |
| Custom Certificate |
| |
| 6. Replace Solution user certificates with VMCA certificates |
| |
| 7. Revert last performed operation by re-publishing old |
| certificates |
| |
| 8. Reset all Certificates |
|_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _|
The problem now is that the wizard is not running...
[ /storage/certmanager ]# /usr/lib/vmware-vmca/bin/certificate-manager
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
| |
| *** Welcome to the vSphere 6.5 Certificate Manager *** |
| |
| -- Select Operation -- |
| |
| 1. Replace Machine SSL certificate with Custom Certificate |
| |
| 2. Replace VMCA Root certificate with Custom Signing |
| Certificate and replace all Certificates |
| |
| 3. Replace Machine SSL certificate with VMCA Certificate |
| |
| 4. Regenerate a new VMCA Root Certificate and |
| replace all certificates |
| |
| 5. Replace Solution user certificates with |
| Custom Certificate |
| |
| 6. Replace Solution user certificates with VMCA certificates |
| |
| 7. Revert last performed operation by re-publishing old |
| certificates |
| |
| 8. Reset all Certificates |
|_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _|
Note : Use Ctrl-D to exit.
Option[1 to 8]: 3
Please provide valid SSO and VC priviledged user credential to perform certificate operations.
Enter username [Administrator@vsphere.local]:administrator@domain.local
Enter password:
Traceback (most recent call last):
File "/usr/lib/vmware-vmca/bin/certificate-manager", line 716, in <module>
exit(main())
File "/usr/lib/vmware-vmca/bin/certificate-manager", line 710, in main
parse_arguments()
File "/usr/lib/vmware-vmca/bin/certificate-manager", line 701, in parse_arguments
get_machine_ssl_cert_to_dir()
File "/usr/lib/vmware-vmca/bin/certificate-manager", line 573, in get_machine_ssl_cert_to_dir
vecs.get_cert_file(Constants.MACHINE_SSL_STORE, Constants.MACHINE_SSL_ALIAS, oldcert)
File "/usr/lib/vmware/site-packages/cis/certificateManagerOps.py", line 438, in get_cert_file
raise e
cis.exceptions.InvokeCommandException: {
"resolution": null,
"detail": [
{
"args": [
""
],
"id": "install.ciscommon.command.errinvoke",
"localized": "An error occurred while invoking external command : ''",
"translatable": "An error occurred while invoking external command : '%(0)s'"
},
"Error while creating backup cert file for MACHINE_SSL_CERT"
],
"componentKey": null,
"problemId": null
}
As I mentioned on my previous post, I followed this link below but didn't work:
I found a VMware article which suggests to delete and re-create the directory for the backup/VMware certs but didn't work. https://kb.vmware.com/s/article/67660
This is the certificate-manager.log file:
2020-10-22T09:06:08.395Z INFO certificate-manager Please provide valid SSO and VC priviledged user credential to perform certificate operations.
2020-10-22T09:06:21.643Z INFO certificate-manager Running command : ['/usr/lib/vmware-vmafd/bin/dir-cli', 'service', 'list', '--login', 'administrator@domain.local', '--password', '*****']
2020-10-22T09:06:21.734Z INFO certificate-manager Output :
1. machine-903e2ea1-2146-46c6-98dd-85d66d760b13
2. vsphere-webclient-903e2ea1-2146-46c6-98dd-85d66d760b13
3. machine-48aa987a-5cd5-47ee-bb0d-d8d577cc271b
4. vsphere-webclient-48aa987a-5cd5-47ee-bb0d-d8d577cc271b
5. machine-9c3577a7-90fd-447e-81a7-c52ddf81a218
6. vsphere-webclient-9c3577a7-90fd-447e-81a7-c52ddf81a218
7. vpxd-9c3577a7-90fd-447e-81a7-c52ddf81a218
8. vpxd-extension-9c3577a7-90fd-447e-81a7-c52ddf81a218
9. machine-3f99c86b-87ba-4108-9b15-f1ed0f5e6396
10. vsphere-webclient-3f99c86b-87ba-4108-9b15-f1ed0f5e6396
11. machine-dcbc143b-fd23-45da-a4c9-927f99a98f0c
12. vsphere-webclient-dcbc143b-fd23-45da-a4c9-927f99a98f0c
13. machine-678b208e-5d5e-4c43-b100-bb0f3a079af6
14. vsphere-webclient-678b208e-5d5e-4c43-b100-bb0f3a079af6
15. vpxd-678b208e-5d5e-4c43-b100-bb0f3a079af6
16. vpxd-extension-678b208e-5d5e-4c43-b100-bb0f3a079af6
2020-10-22T09:06:21.734Z INFO certificate-manager Authentication successful
2020-10-22T09:06:21.735Z INFO certificate-manager Running command : ['/usr/lib/vmware-vmafd/bin/dir-cli', 'service', 'list', '--login', 'administrator@domain.local', '--password', '*****']
2020-10-22T09:06:21.759Z INFO certificate-manager Output :
1. machine-903e2ea1-2146-46c6-98dd-85d66d760b13
2. vsphere-webclient-903e2ea1-2146-46c6-98dd-85d66d760b13
3. machine-48aa987a-5cd5-47ee-bb0d-d8d577cc271b
4. vsphere-webclient-48aa987a-5cd5-47ee-bb0d-d8d577cc271b
5. machine-9c3577a7-90fd-447e-81a7-c52ddf81a218
6. vsphere-webclient-9c3577a7-90fd-447e-81a7-c52ddf81a218
7. vpxd-9c3577a7-90fd-447e-81a7-c52ddf81a218
8. vpxd-extension-9c3577a7-90fd-447e-81a7-c52ddf81a218
9. machine-3f99c86b-87ba-4108-9b15-f1ed0f5e6396
10. vsphere-webclient-3f99c86b-87ba-4108-9b15-f1ed0f5e6396
11. machine-dcbc143b-fd23-45da-a4c9-927f99a98f0c
12. vsphere-webclient-dcbc143b-fd23-45da-a4c9-927f99a98f0c
13. machine-678b208e-5d5e-4c43-b100-bb0f3a079af6
14. vsphere-webclient-678b208e-5d5e-4c43-b100-bb0f3a079af6
15. vpxd-678b208e-5d5e-4c43-b100-bb0f3a079af6
16. vpxd-extension-678b208e-5d5e-4c43-b100-bb0f3a079af6
2020-10-22T09:06:21.759Z INFO certificate-manager Running command : ['/usr/lib/vmware-vmafd/bin/vecs-cli', 'store', 'list']
2020-10-22T09:06:21.767Z INFO certificate-manager Output :
MACHINE_SSL_CERT
TRUSTED_ROOTS
TRUSTED_ROOT_CRLS
machine
vsphere-webclient
BACKUP_STORE
2020-10-22T09:06:21.768Z INFO certificate-manager Running command :- service-control --start vmafdd
2020-10-22T09:06:21.768Z INFO certificate-manager please see service-control.log for service status
2020-10-22T09:06:24.99Z INFO certificate-manager Command executed successfully
2020-10-22T09:06:24.100Z INFO certificate-manager Running command :- service-control --start vmcad
2020-10-22T09:06:24.100Z INFO certificate-manager please see service-control.log for service status
2020-10-22T09:06:26.832Z INFO certificate-manager Command executed successfully
2020-10-22T09:06:26.832Z INFO certificate-manager Running command :- service-control --start vmdird
2020-10-22T09:06:26.833Z INFO certificate-manager please see service-control.log for service status
2020-10-22T09:06:29.872Z INFO certificate-manager Command executed successfully
2020-10-22T09:06:29.873Z INFO certificate-manager Performing operation on embedded setup using 'localhost' as server
2020-10-22T09:06:29.873Z INFO certificate-manager Running command :- ['/usr/lib/vmware-vmafd/bin/vecs-cli', 'entry', 'getcert', '--store', 'MACHINE_SSL_CERT', '--alias', '__MACHINE_CERT', '--output', '/var/tmp/vmware/old_machine_ssl.crt']
2020-10-22T09:06:29.885Z INFO certificate-manager Command output :-
2020-10-22T09:06:29.885Z ERROR certificate-manager
To summarize the steps now:
- Fix the PSC01 Wizard and re-create the self-signed certificate - I am looking how to fix the wizard and re-create the self-signed certificate
- Update the expired self-signed certificate on PSC02 - Need to find how to update the expired self-signed certificate
- Update the self-signed certificate from PSC01 to VCSA - Need to find how to update the self-signed certificate on VCSA
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Let's go one by one, first lets confirm that the Certificate Manager issue is this one: https://kb.vmware.com/s/article/60203. Check that on the PSC1 virtual machine to recreate the MACHINE SSL certificate.
You won't see the VECS stores inside the vCenter Server as they are stored in the PSCs and you topology is an External one.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Alright I understand regarding I am not able to see the VECS into VCSA. For some reason though our client has two sites/clusters and at the other one I am able to see them everywhere.
Regarding the MACHINE SSL certificate, I found the doc below and I managed to re-create it.
Now the PSC01 and the VCSA both are up and all running and I am able to login on the GUI!
I am still now able to run the VUM and looking on it, below is some more info:
#service-control --start vmware-updatemgr
Perform start operation. vmon_profile=None, svc_names=['vmware-updatemgr'], include_coreossvcs=False, include_leafossvcs=False
2020-10-22T11:38:05.362Z Service updatemgr state STOPPED
Error executing start on service updatemgr. Details {
"resolution": null,
"detail": [
{
"args": [
"updatemgr"
],
"id": "install.ciscommon.service.failstart",
"localized": "An error occurred while starting service 'updatemgr'",
"translatable": "An error occurred while starting service '%(0)s'"
}
],
"componentKey": null,
"problemId": null
}
Service-control failed. Error {
"resolution": null,
"detail": [
{
"args": [
"updatemgr"
],
"id": "install.ciscommon.service.failstart",
"localized": "An error occurred while starting service 'updatemgr'",
"translatable": "An error occurred while starting service '%(0)s'"
}
],
"componentKey": null,
"problemId": null
}
updatemgr-utility.log
[2020-10-22 14:33:10,135 ERROR] Unable to update CM service info
[2020-10-22 14:38:05,703 INFO] Install Key store for Jetty
[2020-10-22 14:38:07,733 INFO] Keystore installed successfully.
[2020-10-22 14:38:08,146 INFO] Updating VUM extension with VC
[2020-10-22 14:38:08,544 INFO] Updating CM service info
[2020-10-22 14:38:08,738 ERROR] CM ReRegisterService failure. Exception is (cis.cm.fault.ComponentManagerFault) {
dynamicType = <unset>,
dynamicProperty = (vmodl.DynamicProperty) [],
msg = '',
faultCause = <unset>,
faultMessage = (vmodl.LocalizableMessage) [],
errorCode = 0,
errorMessage = 'UNKNOWN'
}
[2020-10-22 14:38:08,739 ERROR] Unable to update CM service info
I will check the second PSC02 after I will fix the VUM.
Thanks,
George
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hey George,
I recommend you to follow the steps mentioned in the next community post: VCSA 6.5 Update 1 - VUM Service will not start
Check in the comments for the one that has 4 "helpful" and if you need the KB that is mentioned here which is this one: VMware Knowledge Base
Basically is that the Lookup Services does not update the certificate so this could be causing the issue.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Looks quite similar, thanks I just found it and I was about to ask about it.
Also I found this below but I don't see any issue into cm.log like this The SSL certificate of STS service cannot be verified:
https://kb.vmware.com/s/article/76298
This is the trace for the lookup service (I removed some info for security purposes):
C:\>Openssl s_client -connect PSC01:7444
CONNECTED(00000120)
Server certificate
-----BEGIN CERTIFICATE-----
XXXXXXXXXXXX
XXXXXXXXXXXX
-----END CERTIFICATE-----
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 1585 bytes and written 451 bytes
Verification error: unable to verify the first certificate
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher :
Session-ID:
Session-ID-ctx:
Master-Key:
PSK identity: None
PSK identity hint: None
SRP username: None
Start Time: 1603370614
Timeout : 7200 (sec)
Verify return code: 21 (unable to verify the first certificate)
Extended master secret: yes
---
read:errno=0
I see some verification errors, I think this should say it is valid? What is exactly the Lookup Service? Is it for SSO?
Thanks,
George.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hey George,
I suggest you to confirm everything using the steps mentioned in the KB, because you are running the openssl command but to port 7444 and that is checked using the py script for confirming the sslTrut anchor. Go step by step to confirm that is the issue you are facing because for me it seems that is the indicated KB.