VMware Cloud Community
toffaha1
Enthusiast
Enthusiast
Jump to solution

VRSLCM deployment issue VCF 4.1

Hi all,

I am facing this issue during VRSLCM deployment on top of VCF 4.1

 

domainmanger.log

2021-02-24T00:29:11.983+0000 ERROR [vcf_dm,03498f760f114c5c,e2b8] [c.v.e.s.c.c.GenericCertService,dm-exec-7] Error while uploading certificate to remote path /opt/vmware/vlcm/cert/server.key
2021-02-24T00:29:11.990+0000 ERROR [vcf_dm,03498f760f114c5c,e2b8] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-7] [EVV8DA] REPLACE_VRSLCM_CERTIFICATES_FAILED Replacing vRSLCM certificates failed
com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Replacing vRSLCM certificates failed
at com.vmware.evo.sddc.vrealize.vrslcm.GenerateVrslcmCertificate.uploadCertificateToVrslcm(GenerateVrslcmCertificate.java:259)
at com.vmware.evo.sddc.vrealize.vrslcm.GenerateVrslcmCertificate.execute(GenerateVrslcmCertificate.java:230)
at com.vmware.evo.sddc.vrealize.vrslcm.GenerateVrslcmCertificate.execute(GenerateVrslcmCertificate.java:41)
at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionState.lambda$static$0(FsmActionState.java:14)
at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionState.invoke(FsmActionState.java:62)
at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionPlugin.invoke(FsmActionPlugin.java:168)
at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionPlugin.invoke(FsmActionPlugin.java:153)
at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.invokeMethod(ProcessingTaskSubscriber.java:399)
at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.processTask(ProcessingTaskSubscriber.java:519)
at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.accept(ProcessingTaskSubscriber.java:123)
at sun.reflect.GeneratedMethodAccessor516.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.google.common.eventbus.Subscriber.invokeSubscriberMethod(Subscriber.java:87)
at com.google.common.eventbus.Subscriber$1.run(Subscriber.java:72)
at org.springframework.cloud.sleuth.instrument.async.TraceRunnable.run(TraceRunnable.java:67)
at org.springframework.cloud.sleuth.instrument.async.TraceRunnable.run(TraceRunnable.java:67)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.vmware.evo.sddc.common.certificateutil.GenericCertException: Error while uploading certificate to remote path /opt/vmware/vlcm/cert/server.key
at com.vmware.evo.sddc.common.certificateutil.GenericCertService.uploadCert(GenericCertService.java:143)
at com.vmware.evo.sddc.vrealize.vrslcm.GenerateVrslcmCertificate.uploadCertificateToHost(GenerateVrslcmCertificate.java:312)
at com.vmware.evo.sddc.vrealize.vrslcm.GenerateVrslcmCertificate.uploadCertificateToVrslcm(GenerateVrslcmCertificate.java:247)
... 19 common frames omitted
Caused by: com.jcraft.jsch.SftpException: java.io.IOException: inputstream is closed
at com.jcraft.jsch.ChannelSftp._put(ChannelSftp.java:697)
at com.jcraft.jsch.ChannelSftp.put(ChannelSftp.java:540)
at com.jcraft.jsch.ChannelSftp.put(ChannelSftp.java:492)
at com.vmware.evo.sddc.common.util.SshUtil.upload(SshUtil.java:393)
at com.vmware.evo.sddc.common.certificateutil.GenericCertService.uploadCert(GenericCertService.java:138)
... 21 common frames omitted
Caused by: java.io.IOException: inputstream is closed
at com.jcraft.jsch.ChannelSftp.fill(ChannelSftp.java:2911)
at com.jcraft.jsch.ChannelSftp.header(ChannelSftp.java:2935)
at com.jcraft.jsch.ChannelSftp.checkStatus(ChannelSftp.java:2473)
at com.jcraft.jsch.ChannelSftp._put(ChannelSftp.java:686)
... 25 common frames omitted

Best Regards,
Muhammad Toffaha
Technical Consultant
Reply
0 Kudos
1 Solution

Accepted Solutions
shank89
Expert
Expert
Jump to solution

Hey,

 

As discussed, after a fair bit of troubleshooting, we determined this to be an MTU issue.  Once that issue was resolved all seemed to work fine.

Shashank Mohan

VCIX-NV 2022 | VCP-DCV2019 | CCNP Specialist

https://lab2prod.com.au
LinkedIn https://www.linkedin.com/in/shankmohan/
Twitter @ShankMohan
Author of NSX-T Logical Routing: https://link.springer.com/book/10.1007/978-1-4842-7458-3

View solution in original post

18 Replies
toffaha1
Enthusiast
Enthusiast
Jump to solution

I think the failure of nginx service may be related 

toffaha1_0-1614205600766.png

 

Best Regards,
Muhammad Toffaha
Technical Consultant
Reply
0 Kudos
shank89
Expert
Expert
Jump to solution

Have you done the standard checks and tried rebooting the appliance?

Shashank Mohan

VCIX-NV 2022 | VCP-DCV2019 | CCNP Specialist

https://lab2prod.com.au
LinkedIn https://www.linkedin.com/in/shankmohan/
Twitter @ShankMohan
Author of NSX-T Logical Routing: https://link.springer.com/book/10.1007/978-1-4842-7458-3
Reply
0 Kudos
toffaha1
Enthusiast
Enthusiast
Jump to solution

Hi Shank,

Yes, I did.

but still facing the same issue with certificate.

Best Regards,
Muhammad Toffaha
Technical Consultant
Reply
0 Kudos
shank89
Expert
Expert
Jump to solution

Are all dns entries, hostnames etc matching in case ?

Shashank Mohan

VCIX-NV 2022 | VCP-DCV2019 | CCNP Specialist

https://lab2prod.com.au
LinkedIn https://www.linkedin.com/in/shankmohan/
Twitter @ShankMohan
Author of NSX-T Logical Routing: https://link.springer.com/book/10.1007/978-1-4842-7458-3
Reply
0 Kudos
toffaha1
Enthusiast
Enthusiast
Jump to solution

Yes all matching

can I try a different version than the one in the BOM for 4.1?

I am using this one 

VMware Software Install Bundle - vRealize Suite Lifecycle Manager 8.1.0-16776528

Best Regards,
Muhammad Toffaha
Technical Consultant
Reply
0 Kudos
shank89
Expert
Expert
Jump to solution

You shouldn't drift from the VCF BOM, it'll Impact upgrades. Which logs have you looked at, is there anything on operations manager or lcm?

Shashank Mohan

VCIX-NV 2022 | VCP-DCV2019 | CCNP Specialist

https://lab2prod.com.au
LinkedIn https://www.linkedin.com/in/shankmohan/
Twitter @ShankMohan
Author of NSX-T Logical Routing: https://link.springer.com/book/10.1007/978-1-4842-7458-3
Reply
0 Kudos
toffaha1
Enthusiast
Enthusiast
Jump to solution

I found these prints 

domainmanger.log

2021-02-24T00:29:11.983+0000 ERROR [vcf_dm,03498f760f114c5c,e2b8] [c.v.e.s.c.c.GenericCertService,dm-exec-7] Error while uploading certificate to remote path /opt/vmware/vlcm/cert/server.key
2021-02-24T00:29:11.990+0000 ERROR [vcf_dm,03498f760f114c5c,e2b8] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-7] [EVV8DA] REPLACE_VRSLCM_CERTIFICATES_FAILED Replacing vRSLCM certificates failed
com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Replacing vRSLCM certificates failed
at com.vmware.evo.sddc.vrealize.vrslcm.GenerateVrslcmCertificate.uploadCertificateToVrslcm(GenerateVrslcmCertificate.java:259)
at com.vmware.evo.sddc.vrealize.vrslcm.GenerateVrslcmCertificate.execute(GenerateVrslcmCertificate.java:230)
at com.vmware.evo.sddc.vrealize.vrslcm.GenerateVrslcmCertificate.execute(GenerateVrslcmCertificate.java:41)
at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionState.lambda$static$0(FsmActionState.java:14)
at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionState.invoke(FsmActionState.java:62)
at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionPlugin.invoke(FsmActionPlugin.java:168)
at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionPlugin.invoke(FsmActionPlugin.java:153)
at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.invokeMethod(ProcessingTaskSubscriber.java:399)
at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.processTask(ProcessingTaskSubscriber.java:519)
at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.accept(ProcessingTaskSubscriber.java:123)
at sun.reflect.GeneratedMethodAccessor516.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.google.common.eventbus.Subscriber.invokeSubscriberMethod(Subscriber.java:87)
at com.google.common.eventbus.Subscriber$1.run(Subscriber.java:72)
at org.springframework.cloud.sleuth.instrument.async.TraceRunnable.run(TraceRunnable.java:67)
at org.springframework.cloud.sleuth.instrument.async.TraceRunnable.run(TraceRunnable.java:67)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.vmware.evo.sddc.common.certificateutil.GenericCertException: Error while uploading certificate to remote path /opt/vmware/vlcm/cert/server.key
at com.vmware.evo.sddc.common.certificateutil.GenericCertService.uploadCert(GenericCertService.java:143)
at com.vmware.evo.sddc.vrealize.vrslcm.GenerateVrslcmCertificate.uploadCertificateToHost(GenerateVrslcmCertificate.java:312)
at com.vmware.evo.sddc.vrealize.vrslcm.GenerateVrslcmCertificate.uploadCertificateToVrslcm(GenerateVrslcmCertificate.java:247)
... 19 common frames omitted
Caused by: com.jcraft.jsch.SftpException: java.io.IOException: inputstream is closed
at com.jcraft.jsch.ChannelSftp._put(ChannelSftp.java:697)
at com.jcraft.jsch.ChannelSftp.put(ChannelSftp.java:540)
at com.jcraft.jsch.ChannelSftp.put(ChannelSftp.java:492)
at com.vmware.evo.sddc.common.util.SshUtil.upload(SshUtil.java:393)
at com.vmware.evo.sddc.common.certificateutil.GenericCertService.uploadCert(GenericCertService.java:138)
... 21 common frames omitted
Caused by: java.io.IOException: inputstream is closed
at com.jcraft.jsch.ChannelSftp.fill(ChannelSftp.java:2911)
at com.jcraft.jsch.ChannelSftp.header(ChannelSftp.java:2935)
at com.jcraft.jsch.ChannelSftp.checkStatus(ChannelSftp.java:2473)
at com.jcraft.jsch.ChannelSftp._put(ChannelSftp.java:686)
... 25 common frames omitted

 

 

 

Best Regards,
Muhammad Toffaha
Technical Consultant
Reply
0 Kudos
shank89
Expert
Expert
Jump to solution

Does this path exist /opt/vmware/vlcm/cert/server.key?

What are the permissions on the folder?

Shashank Mohan

VCIX-NV 2022 | VCP-DCV2019 | CCNP Specialist

https://lab2prod.com.au
LinkedIn https://www.linkedin.com/in/shankmohan/
Twitter @ShankMohan
Author of NSX-T Logical Routing: https://link.springer.com/book/10.1007/978-1-4842-7458-3
toffaha1
Enthusiast
Enthusiast
Jump to solution

yes, the path exists!

toffaha1_0-1614286448898.png

 

 

Best Regards,
Muhammad Toffaha
Technical Consultant
Reply
0 Kudos
shank89
Expert
Expert
Jump to solution

Any firewalls, routing issues or anything that could block comms between SDDC manager and LCM?

Shashank Mohan

VCIX-NV 2022 | VCP-DCV2019 | CCNP Specialist

https://lab2prod.com.au
LinkedIn https://www.linkedin.com/in/shankmohan/
Twitter @ShankMohan
Author of NSX-T Logical Routing: https://link.springer.com/book/10.1007/978-1-4842-7458-3
Reply
0 Kudos
toffaha1
Enthusiast
Enthusiast
Jump to solution

no firewalls

Routing should be OKay as I can ping from SDDC manager to vRSLCM and vice versa

root@sddc-manager [ ~ ]# ping 10.60.0.160
PING 10.60.0.160 (10.60.0.160) 56(84) bytes of data.
64 bytes from 10.60.0.160: icmp_seq=1 ttl=61 time=3.02 ms
64 bytes from 10.60.0.160: icmp_seq=1 ttl=61 time=3.04 ms (DUP!)
64 bytes from 10.60.0.160: icmp_seq=1 ttl=61 time=3.05 ms (DUP!)
64 bytes from 10.60.0.160: icmp_seq=1 ttl=61 time=3.05 ms (DUP!)
64 bytes from 10.60.0.160: icmp_seq=1 ttl=61 time=3.05 ms (DUP!)
64 bytes from 10.60.0.160: icmp_seq=1 ttl=61 time=3.05 ms (DUP!)
64 bytes from 10.60.0.160: icmp_seq=1 ttl=61 time=3.05 ms (DUP!)
64 bytes from 10.60.0.160: icmp_seq=1 ttl=61 time=3.06 ms (DUP!)

so it's weird 😄 

Best Regards,
Muhammad Toffaha
Technical Consultant
Reply
0 Kudos
shank89
Expert
Expert
Jump to solution

Can you connect via sftp from sddc manager to lcm? It would be using a different port to ICMP.

As it is LCM you would need to go from the management subnet (SDDC MGR) to your physical network, then into NSX-T's edge and segment.

Something like this might be able to determine if there are any issues with that ?https://www.digitalocean.com/community/tutorials/how-to-use-sftp-to-securely-transfer-files-with-a-r...

 

Failing that, maybe a support ticket with GSS?

Shashank Mohan

VCIX-NV 2022 | VCP-DCV2019 | CCNP Specialist

https://lab2prod.com.au
LinkedIn https://www.linkedin.com/in/shankmohan/
Twitter @ShankMohan
Author of NSX-T Logical Routing: https://link.springer.com/book/10.1007/978-1-4842-7458-3
toffaha1
Enthusiast
Enthusiast
Jump to solution

sftp is working fine drom SDDC >> LCM

it's a lab environment so no GSS tickets 😄 

I will try again after upgrading the Environment to 4.2

Best Regards,
Muhammad Toffaha
Technical Consultant
Reply
0 Kudos
shank89
Expert
Expert
Jump to solution

Hmm extremely weird, would be curious to see what behaviour you get after the upgrade!

Shashank Mohan

VCIX-NV 2022 | VCP-DCV2019 | CCNP Specialist

https://lab2prod.com.au
LinkedIn https://www.linkedin.com/in/shankmohan/
Twitter @ShankMohan
Author of NSX-T Logical Routing: https://link.springer.com/book/10.1007/978-1-4842-7458-3
Reply
0 Kudos
shank89
Expert
Expert
Jump to solution

Hey,

 

As discussed, after a fair bit of troubleshooting, we determined this to be an MTU issue.  Once that issue was resolved all seemed to work fine.

Shashank Mohan

VCIX-NV 2022 | VCP-DCV2019 | CCNP Specialist

https://lab2prod.com.au
LinkedIn https://www.linkedin.com/in/shankmohan/
Twitter @ShankMohan
Author of NSX-T Logical Routing: https://link.springer.com/book/10.1007/978-1-4842-7458-3
easiddiqui
Contributor
Contributor
Jump to solution

Hey,

I have run into this issue doing VCF 4.4 deployment. initial deployment we ran into many issues with MTU and BGP, so we opted to scrap it all and redo the deployment. However getting stuck on the exact point here on our second attempt as well. Can't determine where the MTU mismatch is occurring. Any guidance would be greatly valued.

Ali

Reply
0 Kudos
shank89
Expert
Expert
Jump to solution

Ensure the host where vrslcm resides can ping from its host TEP to the edge edges with large frames and no fragmentation.

 

Vmkping ++netstack=vxlan edgeTepIp -s 1572 -d

Shashank Mohan

VCIX-NV 2022 | VCP-DCV2019 | CCNP Specialist

https://lab2prod.com.au
LinkedIn https://www.linkedin.com/in/shankmohan/
Twitter @ShankMohan
Author of NSX-T Logical Routing: https://link.springer.com/book/10.1007/978-1-4842-7458-3
Reply
0 Kudos
toffaha1
Enthusiast
Enthusiast
Jump to solution

Hi @shank89 

I faced this issue again with VCF 4.4.1 but still can't fix it as I can see MTU is working fine with jumbo frames.

any clue?

Best Regards,
Muhammad Toffaha
Technical Consultant
Reply
0 Kudos