Re: Catalog Sync - Known Limitations + javax.net.s...

roberttosulliva · ‎02-23-2015

Is anyone using catalog sync to sync between tow nodes in different datacenters from East to West Coast US/EMEA to US or other similar long distance?

What is the typical vApp size and number of vApps being transferred?

We are trying to synchronize out catalogs from an EMEA vcloud Datacenter to an American datacenter in California. Latency is approx 160ms we have a VPN connection to the site and transfer rate varies from approx 200kb/s to 1.5M/s. I am not convinced this is a network issues as we are able to transfer large volumes of data using https or scp with vCloud Connector, wget, etc across the link that we have in place. We have an SR open with VMware in relation to this (15599956202)

Typically when the vApp is being synchronized is over 10Gb the transfer will fail. Previously we had used vCloud Connector but that does not retain network and MAC settings that we require and has several other known issues which means we need to reconfigure a vApp every time a vApp is transferred - which is not sustainable for large numbers of vApps or if our vApps start to be updated and need re-synchronization. We we were hoping Catalog Sync would slove all these headaches!

Testing Conducted

User connect on a daily basis to vCloud cells using https and are not being disrupted

Large vApps transferred using 200Gb vCloud Connector/HTTPS from EMEA to California – completed/no network issue

Large file 20Gb SCP from EMEA to California – completed/no network issue

vCloud jetty https direct download (10.7Gb) from EMEA to California management server using download manager - completed/no network issue

vCloud jetty https direct download (10.7Gb) from EMEA to California using wget – completed/no network issue

Files copied to /opt/vmware/vcloud-director/tmp/jetty-cloud4544970408731909191.osgi to be available via URL

Catalog sync has worked successfully from EMEA to California with approx 20 vApps so far we have a total of 125 more vApps to sync

Catalog sync from EMEA to our Lab environment also in EMEA but different site (40ms latency) for 49Gb data completed successfully

Our largest current vApp transfer is approx 160Gb largest individual vmdk is 30Gb. The average template transfer will be 80Gb-120Gb - vApps that have copied all have raw data of less than 10Gb and tend to be single VM building blocks for creating custom vApps - not more complex multi-vm vApps.

I have searched online and tried forums/google the links that seem to match are in relation to the SSL versions or encryption implemented on the servers which in this case are both vCloud director cells - https://bugs.openjdk.java.net/browse/JDK-4615819

http://www.coderanch.com/t/430224/sockets/java/javax-net-ssl-SSLException-bad

Alternatively a buffer issue may be at fault within java may be contributing to problems during data transfer - https://community.oracle.com/thread/1535204

Log Extract:

2015-02-20 19:02:03,033 | ERROR | pool-TransferSession-29590-thread-1 | TransferService | Error transferring file from https://api-ed

c2.productcloud.trendmicro.com/vcsp/lib/729dd4f1-0aab-4e23-ae16-78d64450dc77/item/e27d547e-60b1-4165-89aa-e12f5326a5be/file/vm-7791ec65-5aca-48a1-a02c-

5bff448eab0e-disk-0.vmdk to /opt/vmware/vcloud-director/data/transfer/d59b76e1-c384-43c6-ba9e-e0ebb0200cd0/vm-7791ec65-5aca-48a1-a02c-5bff448eab0e-disk

-0.vmdk |

javax.net.ssl.SSLException: bad record MAC

at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)

at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1884)

at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:974)

at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884)

at sun.security.ssl.AppInputStream.read(AppInputStream.java:102)

at org.apache.http.impl.io.AbstractSessionInputBuffer.read(AbstractSessionInputBuffer.java:204)

at org.apache.http.impl.io.ContentLengthInputStream.read(ContentLengthInputStream.java:182)

at org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:138)

at com.vmware.ts.impl.TransferEndpointImpl.storeFile(TransferEndpointImpl.java:365)

at com.vmware.ts.impl.TransferEndpointImpl.attemptDownload(TransferEndpointImpl.java:327)

at com.vmware.ts.impl.TransferEndpointImpl.downloadTo(TransferEndpointImpl.java:227)

at com.vmware.ts.impl.TransferItemImpl.run(TransferItemImpl.java:293)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

at java.util.concurrent.FutureTask.run(FutureTask.java:262)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:745)

Caused by: javax.crypto.BadPaddingException: bad record MAC

at sun.security.ssl.InputRecord.decrypt(InputRecord.java:197)

at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:969)

... 16 more

2015-02-20 19:02:03,033 | DEBUG | pool-TransferSession-29590-thread-1 | TransferService | Transfer endpoint com.vmware.ts.impl.Transf

erEndpointImpl@20184fdc has error, skipping transfer |

2015-02-20 19:02:04,762 | ERROR | backend-activity-pool-2560 | FutureUtil | Transfer session: d59b76e1-c384-43c6-ba9e-e0ebb0200cd0, vApp/template: TPLW - Windows 7 Enterprise SP1 - 64 bit - DE. Unable to retrieve OVF descriptor. | activity=urn:uuid:8c314c9c-d133-491c-be92-3929f718c8cd vcd=de075bc5-819d-479e-92ac-283b71d17d6c,task=8c314c9c-d133-491c-be92-3929f718c8cd activity=urn:uuid:8c314c9c-d133-491c-be92-3929f718c8cd vcd=de075bc5-819d-479e-92ac-283b71d17d6c,task=73fd8b0c-3e2a-4062-885b-b6cbd197fcf2 activity=urn:uuid:73fd8b0c-3e2a-4062-885b-b6cbd197fcf2

com.vmware.vcloud.api.presentation.service.InternalServerErrorException: Catalog item file transfer failed due to SSL error.

at com.vmware.vcloud.fabric.transfer.impl.ReceivingSocketImpl$FileImportResourceFuture.isDone(ReceivingSocketImpl.java:126)

at com.vmware.vcloud.fabric.foundation.futures.FutureUtil.waitForFutureOrCancel(FutureUtil.java:124)

at com.vmware.ssdc.backend.services.impl.VAppUploadManagerImpl.handleContentUploaded(VAppUploadManagerImpl.java:2776)

at com.vmware.ssdc.backend.services.impl.VAppUploadManagerImpl.handleUploadBody(VAppUploadManagerImpl.java:1878)

at com.vmware.ssdc.backend.services.impl.VAppUploadManagerImpl.handleUpload(VAppUploadManagerImpl.java:1794)

at com.vmware.ssdc.backend.services.impl.VAppUploadManagerImpl.executeTask(VAppUploadManagerImpl.java:4234)

at com.vmware.vcloud.backendbase.management.system.TaskActivity$ExecutePhase$2.doInSecurityContext(TaskActivity.java:427)

at com.vmware.vcloud.backendbase.management.system.TaskActivity$ExecutePhase$2.doInSecurityContext(TaskActivity.java:421)

at com.vmware.vcloud.backendbase.management.system.TaskActivity$SecurityContextTemplate.executeForOrgAndUser(TaskActivity.java:674)

at com.vmware.vcloud.backendbase.management.system.TaskActivity$ExecutePhase.execute(TaskActivity.java:421)

at com.vmware.vcloud.backendbase.management.system.TaskActivity$ExecutePhase.invoke(TaskActivity.java:354)

at com.vmware.vcloud.activity.executors.ActivityRunner.run(ActivityRunner.java:109)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

at java.util.concurrent.FutureTask.run(FutureTask.java:262)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:745)

2015-02-20 19:02:04,763 | WARN | backend-activity-pool-2560 | VAppUploadManagerImpl | Transfer session: d59b76e1-c384-43c6-ba9e-e0ebb0200cd0. Error during content upload: Catalog item file transfer failed due to SSL error. | activity=urn:uuid:8c314c9c-d133-491c-be92-3929f718c8cd vcd=de075bc5-819d-479e-92ac-283b71d17d6c,task=8c314c9c-d133-491c-be92-3929f718c8cd activity=urn:uuid:8c314c9c-d133-491c-be92-3929f718c8cd vcd=de075bc5-819d-479e-92ac-283b71d17d6c,task=73fd8b0c-3e2a-4062-885b-b6cbd197fcf2 activity=urn:uuid:73fd8b0c-3e2a-4062-885b-b6cbd197fcf2