Hi all,
I am facing this issue during VRSLCM deployment on top of VCF 4.1
domainmanger.log
2021-02-24T00:29:11.983+0000 ERROR [vcf_dm,03498f760f114c5c,e2b8] [c.v.e.s.c.c.GenericCertService,dm-exec-7] Error while uploading certificate to remote path /opt/vmware/vlcm/cert/server.key
2021-02-24T00:29:11.990+0000 ERROR [vcf_dm,03498f760f114c5c,e2b8] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-7] [EVV8DA] REPLACE_VRSLCM_CERTIFICATES_FAILED Replacing vRSLCM certificates failed
com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Replacing vRSLCM certificates failed
at com.vmware.evo.sddc.vrealize.vrslcm.GenerateVrslcmCertificate.uploadCertificateToVrslcm(GenerateVrslcmCertificate.java:259)
at com.vmware.evo.sddc.vrealize.vrslcm.GenerateVrslcmCertificate.execute(GenerateVrslcmCertificate.java:230)
at com.vmware.evo.sddc.vrealize.vrslcm.GenerateVrslcmCertificate.execute(GenerateVrslcmCertificate.java:41)
at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionState.lambda$static$0(FsmActionState.java:14)
at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionState.invoke(FsmActionState.java:62)
at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionPlugin.invoke(FsmActionPlugin.java:168)
at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionPlugin.invoke(FsmActionPlugin.java:153)
at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.invokeMethod(ProcessingTaskSubscriber.java:399)
at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.processTask(ProcessingTaskSubscriber.java:519)
at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.accept(ProcessingTaskSubscriber.java:123)
at sun.reflect.GeneratedMethodAccessor516.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.google.common.eventbus.Subscriber.invokeSubscriberMethod(Subscriber.java:87)
at com.google.common.eventbus.Subscriber$1.run(Subscriber.java:72)
at org.springframework.cloud.sleuth.instrument.async.TraceRunnable.run(TraceRunnable.java:67)
at org.springframework.cloud.sleuth.instrument.async.TraceRunnable.run(TraceRunnable.java:67)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.vmware.evo.sddc.common.certificateutil.GenericCertException: Error while uploading certificate to remote path /opt/vmware/vlcm/cert/server.key
at com.vmware.evo.sddc.common.certificateutil.GenericCertService.uploadCert(GenericCertService.java:143)
at com.vmware.evo.sddc.vrealize.vrslcm.GenerateVrslcmCertificate.uploadCertificateToHost(GenerateVrslcmCertificate.java:312)
at com.vmware.evo.sddc.vrealize.vrslcm.GenerateVrslcmCertificate.uploadCertificateToVrslcm(GenerateVrslcmCertificate.java:247)
... 19 common frames omitted
Caused by: com.jcraft.jsch.SftpException: java.io.IOException: inputstream is closed
at com.jcraft.jsch.ChannelSftp._put(ChannelSftp.java:697)
at com.jcraft.jsch.ChannelSftp.put(ChannelSftp.java:540)
at com.jcraft.jsch.ChannelSftp.put(ChannelSftp.java:492)
at com.vmware.evo.sddc.common.util.SshUtil.upload(SshUtil.java:393)
at com.vmware.evo.sddc.common.certificateutil.GenericCertService.uploadCert(GenericCertService.java:138)
... 21 common frames omitted
Caused by: java.io.IOException: inputstream is closed
at com.jcraft.jsch.ChannelSftp.fill(ChannelSftp.java:2911)
at com.jcraft.jsch.ChannelSftp.header(ChannelSftp.java:2935)
at com.jcraft.jsch.ChannelSftp.checkStatus(ChannelSftp.java:2473)
at com.jcraft.jsch.ChannelSftp._put(ChannelSftp.java:686)
... 25 common frames omitted
Hey,
As discussed, after a fair bit of troubleshooting, we determined this to be an MTU issue. Once that issue was resolved all seemed to work fine.
I think the failure of nginx service may be related
Have you done the standard checks and tried rebooting the appliance?
Hi Shank,
Yes, I did.
but still facing the same issue with certificate.
Are all dns entries, hostnames etc matching in case ?
Yes all matching
can I try a different version than the one in the BOM for 4.1?
I am using this one
VMware Software Install Bundle - vRealize Suite Lifecycle Manager 8.1.0-16776528
You shouldn't drift from the VCF BOM, it'll Impact upgrades. Which logs have you looked at, is there anything on operations manager or lcm?
I found these prints
domainmanger.log
2021-02-24T00:29:11.983+0000 ERROR [vcf_dm,03498f760f114c5c,e2b8] [c.v.e.s.c.c.GenericCertService,dm-exec-7] Error while uploading certificate to remote path /opt/vmware/vlcm/cert/server.key
2021-02-24T00:29:11.990+0000 ERROR [vcf_dm,03498f760f114c5c,e2b8] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-7] [EVV8DA] REPLACE_VRSLCM_CERTIFICATES_FAILED Replacing vRSLCM certificates failed
com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Replacing vRSLCM certificates failed
at com.vmware.evo.sddc.vrealize.vrslcm.GenerateVrslcmCertificate.uploadCertificateToVrslcm(GenerateVrslcmCertificate.java:259)
at com.vmware.evo.sddc.vrealize.vrslcm.GenerateVrslcmCertificate.execute(GenerateVrslcmCertificate.java:230)
at com.vmware.evo.sddc.vrealize.vrslcm.GenerateVrslcmCertificate.execute(GenerateVrslcmCertificate.java:41)
at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionState.lambda$static$0(FsmActionState.java:14)
at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionState.invoke(FsmActionState.java:62)
at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionPlugin.invoke(FsmActionPlugin.java:168)
at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionPlugin.invoke(FsmActionPlugin.java:153)
at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.invokeMethod(ProcessingTaskSubscriber.java:399)
at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.processTask(ProcessingTaskSubscriber.java:519)
at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.accept(ProcessingTaskSubscriber.java:123)
at sun.reflect.GeneratedMethodAccessor516.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.google.common.eventbus.Subscriber.invokeSubscriberMethod(Subscriber.java:87)
at com.google.common.eventbus.Subscriber$1.run(Subscriber.java:72)
at org.springframework.cloud.sleuth.instrument.async.TraceRunnable.run(TraceRunnable.java:67)
at org.springframework.cloud.sleuth.instrument.async.TraceRunnable.run(TraceRunnable.java:67)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.vmware.evo.sddc.common.certificateutil.GenericCertException: Error while uploading certificate to remote path /opt/vmware/vlcm/cert/server.key
at com.vmware.evo.sddc.common.certificateutil.GenericCertService.uploadCert(GenericCertService.java:143)
at com.vmware.evo.sddc.vrealize.vrslcm.GenerateVrslcmCertificate.uploadCertificateToHost(GenerateVrslcmCertificate.java:312)
at com.vmware.evo.sddc.vrealize.vrslcm.GenerateVrslcmCertificate.uploadCertificateToVrslcm(GenerateVrslcmCertificate.java:247)
... 19 common frames omitted
Caused by: com.jcraft.jsch.SftpException: java.io.IOException: inputstream is closed
at com.jcraft.jsch.ChannelSftp._put(ChannelSftp.java:697)
at com.jcraft.jsch.ChannelSftp.put(ChannelSftp.java:540)
at com.jcraft.jsch.ChannelSftp.put(ChannelSftp.java:492)
at com.vmware.evo.sddc.common.util.SshUtil.upload(SshUtil.java:393)
at com.vmware.evo.sddc.common.certificateutil.GenericCertService.uploadCert(GenericCertService.java:138)
... 21 common frames omitted
Caused by: java.io.IOException: inputstream is closed
at com.jcraft.jsch.ChannelSftp.fill(ChannelSftp.java:2911)
at com.jcraft.jsch.ChannelSftp.header(ChannelSftp.java:2935)
at com.jcraft.jsch.ChannelSftp.checkStatus(ChannelSftp.java:2473)
at com.jcraft.jsch.ChannelSftp._put(ChannelSftp.java:686)
... 25 common frames omitted
Does this path exist /opt/vmware/vlcm/cert/server.key?
What are the permissions on the folder?
yes, the path exists!
Any firewalls, routing issues or anything that could block comms between SDDC manager and LCM?
no firewalls
Routing should be OKay as I can ping from SDDC manager to vRSLCM and vice versa
root@sddc-manager [ ~ ]# ping 10.60.0.160
PING 10.60.0.160 (10.60.0.160) 56(84) bytes of data.
64 bytes from 10.60.0.160: icmp_seq=1 ttl=61 time=3.02 ms
64 bytes from 10.60.0.160: icmp_seq=1 ttl=61 time=3.04 ms (DUP!)
64 bytes from 10.60.0.160: icmp_seq=1 ttl=61 time=3.05 ms (DUP!)
64 bytes from 10.60.0.160: icmp_seq=1 ttl=61 time=3.05 ms (DUP!)
64 bytes from 10.60.0.160: icmp_seq=1 ttl=61 time=3.05 ms (DUP!)
64 bytes from 10.60.0.160: icmp_seq=1 ttl=61 time=3.05 ms (DUP!)
64 bytes from 10.60.0.160: icmp_seq=1 ttl=61 time=3.05 ms (DUP!)
64 bytes from 10.60.0.160: icmp_seq=1 ttl=61 time=3.06 ms (DUP!)
so it's weird 😄
Can you connect via sftp from sddc manager to lcm? It would be using a different port to ICMP.
As it is LCM you would need to go from the management subnet (SDDC MGR) to your physical network, then into NSX-T's edge and segment.
Something like this might be able to determine if there are any issues with that ?https://www.digitalocean.com/community/tutorials/how-to-use-sftp-to-securely-transfer-files-with-a-r...
Failing that, maybe a support ticket with GSS?
sftp is working fine drom SDDC >> LCM
it's a lab environment so no GSS tickets 😄
I will try again after upgrading the Environment to 4.2
Hmm extremely weird, would be curious to see what behaviour you get after the upgrade!
Hey,
As discussed, after a fair bit of troubleshooting, we determined this to be an MTU issue. Once that issue was resolved all seemed to work fine.
Hey,
I have run into this issue doing VCF 4.4 deployment. initial deployment we ran into many issues with MTU and BGP, so we opted to scrap it all and redo the deployment. However getting stuck on the exact point here on our second attempt as well. Can't determine where the MTU mismatch is occurring. Any guidance would be greatly valued.
Ali
Ensure the host where vrslcm resides can ping from its host TEP to the edge edges with large frames and no fragmentation.
Vmkping ++netstack=vxlan edgeTepIp -s 1572 -d
Hi @shank89
I faced this issue again with VCF 4.4.1 but still can't fix it as I can see MTU is working fine with jumbo frames.
any clue?