Hi all,
We are currently using VCD 5.5 and we are experiencing issues with deploying large vApps.
I've noticed that when the creation task takes more than 5 Min it fails with the following error:
- Internal Server Error
- java.net.SocketTimeoutException: Read timed out
I'll appreciate your assistance
Please forgive typos
This is the Debug:
com.vmware.ssdc.library.exceptions.MultipleLMException: One or more exceptions have occurred - Multiple Exceptions follow: [com.vmware.vcloud.api.presentation.service.InternalServerErrorException: Internal Server Error, com.vmware.vcloud.api.presentation.service.InternalServerErrorException: Internal Server Error]
at com.vmware.ssdc.library.ExceptionFactory.createFromMultiple(ExceptionFactory.java:35)
at com.vmware.vcloud.vdc.impl.AbstractVdcService.waitForFabricVmFutures(AbstractVdcService.java:902)
at com.vmware.vcloud.vdc.impl.AbstractVdcService.copyVAppContents(AbstractVdcService.java:638)
at com.vmware.vcloud.vdc.impl.AbstractVdcService.copyVAppContents(AbstractVdcService.java:291)
at com.vmware.vcloud.vdc.impl.ResourceEntityLifeCycleImpl.instantiateVAppTask(ResourceEntityLifeCycleImpl.java:1332)
at com.vmware.vcloud.vdc.impl.ResourceEntityLifeCycleImpl.executeTask(ResourceEntityLifeCycleImpl.java:1769)
at com.vmware.vcloud.backendbase.management.system.TaskActivity$ExecutePhase$2.doInSecurityContext(TaskActivity.java:427)
at com.vmware.vcloud.backendbase.management.system.TaskActivity$ExecutePhase$2.doInSecurityContext(TaskActivity.java:421)
at com.vmware.vcloud.backendbase.management.system.TaskActivity$SecurityContextTemplate.executeForOrgAndUser(TaskActivity.java:674)
at com.vmware.vcloud.backendbase.management.system.TaskActivity$ExecutePhase.execute(TaskActivity.java:421)
at com.vmware.vcloud.backendbase.management.system.TaskActivity$ExecutePhase.invoke(TaskActivity.java:354)
at com.vmware.vcloud.activity.executors.ActivityRunner.run(ActivityRunner.java:109)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Subexception: com.vmware.vcloud.api.presentation.service.InternalServerErrorException: Internal Server Error
at com.vmware.vcloud.fabric.foundation.futures.FutureUtil.convertExecutionException(FutureUtil.java:198)
at com.vmware.vcloud.vdc.impl.AbstractVdcService.waitForFabricVmFutures(AbstractVdcService.java:837)
at com.vmware.vcloud.vdc.impl.AbstractVdcService.copyVAppContents(AbstractVdcService.java:638)
at com.vmware.vcloud.vdc.impl.AbstractVdcService.copyVAppContents(AbstractVdcService.java:291)
at com.vmware.vcloud.vdc.impl.ResourceEntityLifeCycleImpl.instantiateVAppTask(ResourceEntityLifeCycleImpl.java:1332)
at com.vmware.vcloud.vdc.impl.ResourceEntityLifeCycleImpl.executeTask(ResourceEntityLifeCycleImpl.java:1769)
at com.vmware.vcloud.backendbase.management.system.TaskActivity$ExecutePhase$2.doInSecurityContext(TaskActivity.java:427)
at com.vmware.vcloud.backendbase.management.system.TaskActivity$ExecutePhase$2.doInSecurityContext(TaskActivity.java:421)
at com.vmware.vcloud.backendbase.management.system.TaskActivity$SecurityContextTemplate.executeForOrgAndUser(TaskActivity.java:674)
at com.vmware.vcloud.backendbase.management.system.TaskActivity$ExecutePhase.execute(TaskActivity.java:421)
at com.vmware.vcloud.backendbase.management.system.TaskActivity$ExecutePhase.invoke(TaskActivity.java:354)
at com.vmware.vcloud.activity.executors.ActivityRunner.run(ActivityRunner.java:109)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: com.vmware.vim.vmomi.client.exception.ConnectionException: java.net.SocketTimeoutException: Read timed out
at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setError(ResponseImpl.java:224)
at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:131)
at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingImpl.send(HttpProtocolBindingImpl.java:98)
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.sendCall(MethodInvocationHandlerImpl.java:526)
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.executeCall(MethodInvocationHandlerImpl.java:507)
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.completeCall(MethodInvocationHandlerImpl.java:295)
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invokeOperation(MethodInvocationHandlerImpl.java:265)
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invoke(MethodInvocationHandlerImpl.java:169)
at com.sun.proxy.$Proxy1146.recommendDatastores(Unknown Source)
at com.vmware.vcloud.fabric.storage.placement.impl.StoragePodPlacementManagerImpl.getDatastoreFromSDRS(StoragePodPlacementManagerImpl.java:266)
at com.vmware.vcloud.fabric.storage.placement.impl.StoragePodPlacementManagerImpl.recommendDatastore(StoragePodPlacementManagerImpl.java:124)
at com.vmware.vcloud.fabric.storage.placement.impl.PlacementManagerImpl.selectDatastoreInPod(PlacementManagerImpl.java:379)
at com.vmware.vcloud.fabric.storage.placement.impl.PlacementManagerImpl.selectDatastore(PlacementManagerImpl.java:168)
at com.vmware.vcloud.fabric.storage.storedVm.impl.CreateFromExistingStoredVmActivity$CreateStoredVmInVcPhase.selectDatastore(CreateFromExistingStoredVmActivity.java:769)
at com.vmware.vcloud.fabric.storage.storedVm.impl.CreateFromExistingStoredVmActivity$CreateStoredVmInVcPhase.findOrCreateSourceVm(CreateFromExistingStoredVmActivity.java:631)
at com.vmware.vcloud.fabric.storage.storedVm.impl.CreateFromExistingStoredVmActivity$CreateStoredVmInVcPhase.copyVmWithLinkedCloneFromTemplate(CreateFromExistingStoredVmActivity.java:579)
at com.vmware.vcloud.fabric.storage.storedVm.impl.CreateFromExistingStoredVmActivity$CreateStoredVmInVcPhase.invoke(CreateFromExistingStoredVmActivity.java:265)
... 7 more
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at sun.security.ssl.InputRecord.readFully(Unknown Source)
at sun.security.ssl.InputRecord.read(Unknown Source)
at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)
at sun.security.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
at sun.security.ssl.AppInputStream.read(Unknown Source)
at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:281)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:92)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:62)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:254)
at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:289)
at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:252)
at org.apache.http.impl.conn.AbstractClientConnAdapter.receiveResponseHeader(AbstractClientConnAdapter.java:219)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:300)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:127)
at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:713)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:518)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:111)
... 22 more
Subexception: com.vmware.vcloud.api.presentation.service.InternalServerErrorException: Internal Server Error
at com.vmware.vcloud.fabric.foundation.futures.FutureUtil.convertExecutionException(FutureUtil.java:198)
at com.vmware.vcloud.vdc.impl.AbstractVdcService.waitForFabricVmFutures(AbstractVdcService.java:837)
at com.vmware.vcloud.vdc.impl.AbstractVdcService.copyVAppContents(AbstractVdcService.java:638)
at com.vmware.vcloud.vdc.impl.AbstractVdcService.copyVAppContents(AbstractVdcService.java:291)
at com.vmware.vcloud.vdc.impl.ResourceEntityLifeCycleImpl.instantiateVAppTask(ResourceEntityLifeCycleImpl.java:1332)
at com.vmware.vcloud.vdc.impl.ResourceEntityLifeCycleImpl.executeTask(ResourceEntityLifeCycleImpl.java:1769)
at com.vmware.vcloud.backendbase.management.system.TaskActivity$ExecutePhase$2.doInSecurityContext(TaskActivity.java:427)
at com.vmware.vcloud.backendbase.management.system.TaskActivity$ExecutePhase$2.doInSecurityContext(TaskActivity.java:421)
at com.vmware.vcloud.backendbase.management.system.TaskActivity$SecurityContextTemplate.executeForOrgAndUser(TaskActivity.java:674)
at com.vmware.vcloud.backendbase.management.system.TaskActivity$ExecutePhase.execute(TaskActivity.java:421)
at com.vmware.vcloud.backendbase.management.system.TaskActivity$ExecutePhase.invoke(TaskActivity.java:354)
at com.vmware.vcloud.activity.executors.ActivityRunner.run(ActivityRunner.java:109)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: com.vmware.vim.vmomi.client.exception.ConnectionException: java.net.SocketTimeoutException: Read timed out
at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setError(ResponseImpl.java:224)
at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:131)
at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingImpl.send(HttpProtocolBindingImpl.java:98)
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.sendCall(MethodInvocationHandlerImpl.java:526)
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.executeCall(MethodInvocationHandlerImpl.java:507)
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.completeCall(MethodInvocationHandlerImpl.java:295)
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invokeOperation(MethodInvocationHandlerImpl.java:265)
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invoke(MethodInvocationHandlerImpl.java:169)
at com.sun.proxy.$Proxy1112.recommendDatastores(Unknown Source)
at com.vmware.vcloud.fabric.storage.placement.impl.StoragePodPlacementManagerImpl.getDatastoreFromSDRS(StoragePodPlacementManagerImpl.java:266)
at com.vmware.vcloud.fabric.storage.placement.impl.StoragePodPlacementManagerImpl.recommendDatastore(StoragePodPlacementManagerImpl.java:124)
at com.vmware.vcloud.fabric.storage.placement.impl.PlacementManagerImpl.selectDatastoreInPod(PlacementManagerImpl.java:379)
at com.vmware.vcloud.fabric.storage.placement.impl.PlacementManagerImpl.selectDatastore(PlacementManagerImpl.java:168)
at com.vmware.vcloud.fabric.storage.storedVm.impl.CreateFromExistingStoredVmActivity$CreateStoredVmInVcPhase.selectDatastore(CreateFromExistingStoredVmActivity.java:769)
at com.vmware.vcloud.fabric.storage.storedVm.impl.CreateFromExistingStoredVmActivity$CreateStoredVmInVcPhase.findOrCreateSourceVm(CreateFromExistingStoredVmActivity.java:631)
at com.vmware.vcloud.fabric.storage.storedVm.impl.CreateFromExistingStoredVmActivity$CreateStoredVmInVcPhase.copyVmWithLinkedCloneFromTemplate(CreateFromExistingStoredVmActivity.java:579)
at com.vmware.vcloud.fabric.storage.storedVm.impl.CreateFromExistingStoredVmActivity$CreateStoredVmInVcPhase.invoke(CreateFromExistingStoredVmActivity.java:265)
... 7 more
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at sun.security.ssl.InputRecord.readFully(Unknown Source)
at sun.security.ssl.InputRecord.read(Unknown Source)
at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)
at sun.security.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
at sun.security.ssl.AppInputStream.read(Unknown Source)
at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:281)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:92)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:62)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:254)
at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:289)
at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:252)
at org.apache.http.impl.conn.AbstractClientConnAdapter.receiveResponseHeader(AbstractClientConnAdapter.java:219)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:300)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:127)
at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:713)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:518)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:111)
The critical point is here:
at com.sun.proxy.$Proxy1146.recommendDatastores(Unknown Source)
at com.vmware.vcloud.fabric.storage.placement.impl.StoragePodPlacementManagerImpl.getDatastoreFromSDRS(StoragePodPlacementManagerImpl.java:266)
Turn of DRS in vCenter as a temporary fix. If you can deploy then, it's definitely Storage DRS related.
Newer versions of vCenter have improvements to speed up the returned result of SDRS recommending a datastore for VM placement.
The critical point is here:
at com.sun.proxy.$Proxy1146.recommendDatastores(Unknown Source)
at com.vmware.vcloud.fabric.storage.placement.impl.StoragePodPlacementManagerImpl.getDatastoreFromSDRS(StoragePodPlacementManagerImpl.java:266)
Turn of DRS in vCenter as a temporary fix. If you can deploy then, it's definitely Storage DRS related.
Newer versions of vCenter have improvements to speed up the returned result of SDRS recommending a datastore for VM placement.
Thanks for your help.. it actually working now (as a workaround)
In case upgrading to the latest update of VC 5.5 (we are currently running 5.5.0 build 2183111) wont help, what would you recommend us to do?
The call that is failing is vCenter code in Storage DRS. it's taking too long. if you can't upgrade, then the workaround will be that vCD Determines initial placement of VMs. Then there is no load balancing after the fact (since SDRS is off).
If you need more concise evaluation, then you might want to file a support request for vCenter and submit logs from vCenter with a window for failure (deploy time and failure time in vCD).
I will upgrade the VC to the latest update, hopefully it will fix the issue.
Thanks again for your help.
Hi,I want to use vcloud restful api to create two VMs in one vApp at the same time,How to do this?
I'll appreciate your assistance
Thank you !