Golan_M
Contributor
Contributor

vCloud Director 5.5 creating large (more than 8 VMs) vApp from a vApp template fails

Jump to solution

Hi all,

We are currently using VCD 5.5 and we are experiencing issues with deploying large vApps.

I've noticed that when the creation task takes more than 5 Min it fails with the following error:

- Internal Server Error

- java.net.SocketTimeoutException: Read timed out



I'll appreciate your assistance Smiley Happy

Please forgive typos Smiley Happy



This is the Debug:


com.vmware.ssdc.library.exceptions.MultipleLMException: One or more exceptions have occurred - Multiple Exceptions follow: [com.vmware.vcloud.api.presentation.service.InternalServerErrorException: Internal Server Error, com.vmware.vcloud.api.presentation.service.InternalServerErrorException: Internal Server Error]

  at com.vmware.ssdc.library.ExceptionFactory.createFromMultiple(ExceptionFactory.java:35)

  at com.vmware.vcloud.vdc.impl.AbstractVdcService.waitForFabricVmFutures(AbstractVdcService.java:902)

  at com.vmware.vcloud.vdc.impl.AbstractVdcService.copyVAppContents(AbstractVdcService.java:638)

  at com.vmware.vcloud.vdc.impl.AbstractVdcService.copyVAppContents(AbstractVdcService.java:291)

  at com.vmware.vcloud.vdc.impl.ResourceEntityLifeCycleImpl.instantiateVAppTask(ResourceEntityLifeCycleImpl.java:1332)

  at com.vmware.vcloud.vdc.impl.ResourceEntityLifeCycleImpl.executeTask(ResourceEntityLifeCycleImpl.java:1769)

  at com.vmware.vcloud.backendbase.management.system.TaskActivity$ExecutePhase$2.doInSecurityContext(TaskActivity.java:427)

  at com.vmware.vcloud.backendbase.management.system.TaskActivity$ExecutePhase$2.doInSecurityContext(TaskActivity.java:421)

  at com.vmware.vcloud.backendbase.management.system.TaskActivity$SecurityContextTemplate.executeForOrgAndUser(TaskActivity.java:674)

  at com.vmware.vcloud.backendbase.management.system.TaskActivity$ExecutePhase.execute(TaskActivity.java:421)

  at com.vmware.vcloud.backendbase.management.system.TaskActivity$ExecutePhase.invoke(TaskActivity.java:354)

  at com.vmware.vcloud.activity.executors.ActivityRunner.run(ActivityRunner.java:109)

  at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)

  at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)

  at java.util.concurrent.FutureTask.run(Unknown Source)

  at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

  at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

  at java.lang.Thread.run(Unknown Source)

Subexception: com.vmware.vcloud.api.presentation.service.InternalServerErrorException: Internal Server Error

  at com.vmware.vcloud.fabric.foundation.futures.FutureUtil.convertExecutionException(FutureUtil.java:198)

  at com.vmware.vcloud.vdc.impl.AbstractVdcService.waitForFabricVmFutures(AbstractVdcService.java:837)

  at com.vmware.vcloud.vdc.impl.AbstractVdcService.copyVAppContents(AbstractVdcService.java:638)

  at com.vmware.vcloud.vdc.impl.AbstractVdcService.copyVAppContents(AbstractVdcService.java:291)

  at com.vmware.vcloud.vdc.impl.ResourceEntityLifeCycleImpl.instantiateVAppTask(ResourceEntityLifeCycleImpl.java:1332)

  at com.vmware.vcloud.vdc.impl.ResourceEntityLifeCycleImpl.executeTask(ResourceEntityLifeCycleImpl.java:1769)

  at com.vmware.vcloud.backendbase.management.system.TaskActivity$ExecutePhase$2.doInSecurityContext(TaskActivity.java:427)

  at com.vmware.vcloud.backendbase.management.system.TaskActivity$ExecutePhase$2.doInSecurityContext(TaskActivity.java:421)

  at com.vmware.vcloud.backendbase.management.system.TaskActivity$SecurityContextTemplate.executeForOrgAndUser(TaskActivity.java:674)

  at com.vmware.vcloud.backendbase.management.system.TaskActivity$ExecutePhase.execute(TaskActivity.java:421)

  at com.vmware.vcloud.backendbase.management.system.TaskActivity$ExecutePhase.invoke(TaskActivity.java:354)

  at com.vmware.vcloud.activity.executors.ActivityRunner.run(ActivityRunner.java:109)

  at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)

  at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)

  at java.util.concurrent.FutureTask.run(Unknown Source)

  at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

  at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

  at java.lang.Thread.run(Unknown Source)

Caused by: com.vmware.vim.vmomi.client.exception.ConnectionException: java.net.SocketTimeoutException: Read timed out

  at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setError(ResponseImpl.java:224)

  at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:131)

  at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingImpl.send(HttpProtocolBindingImpl.java:98)

  at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.sendCall(MethodInvocationHandlerImpl.java:526)

  at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.executeCall(MethodInvocationHandlerImpl.java:507)

  at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.completeCall(MethodInvocationHandlerImpl.java:295)

  at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invokeOperation(MethodInvocationHandlerImpl.java:265)

  at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invoke(MethodInvocationHandlerImpl.java:169)

  at com.sun.proxy.$Proxy1146.recommendDatastores(Unknown Source)

  at com.vmware.vcloud.fabric.storage.placement.impl.StoragePodPlacementManagerImpl.getDatastoreFromSDRS(StoragePodPlacementManagerImpl.java:266)

  at com.vmware.vcloud.fabric.storage.placement.impl.StoragePodPlacementManagerImpl.recommendDatastore(StoragePodPlacementManagerImpl.java:124)

  at com.vmware.vcloud.fabric.storage.placement.impl.PlacementManagerImpl.selectDatastoreInPod(PlacementManagerImpl.java:379)

  at com.vmware.vcloud.fabric.storage.placement.impl.PlacementManagerImpl.selectDatastore(PlacementManagerImpl.java:168)

  at com.vmware.vcloud.fabric.storage.storedVm.impl.CreateFromExistingStoredVmActivity$CreateStoredVmInVcPhase.selectDatastore(CreateFromExistingStoredVmActivity.java:769)

  at com.vmware.vcloud.fabric.storage.storedVm.impl.CreateFromExistingStoredVmActivity$CreateStoredVmInVcPhase.findOrCreateSourceVm(CreateFromExistingStoredVmActivity.java:631)

  at com.vmware.vcloud.fabric.storage.storedVm.impl.CreateFromExistingStoredVmActivity$CreateStoredVmInVcPhase.copyVmWithLinkedCloneFromTemplate(CreateFromExistingStoredVmActivity.java:579)

  at com.vmware.vcloud.fabric.storage.storedVm.impl.CreateFromExistingStoredVmActivity$CreateStoredVmInVcPhase.invoke(CreateFromExistingStoredVmActivity.java:265)

  ... 7 more

Caused by: java.net.SocketTimeoutException: Read timed out

  at java.net.SocketInputStream.socketRead0(Native Method)

  at java.net.SocketInputStream.read(Unknown Source)

  at java.net.SocketInputStream.read(Unknown Source)

  at sun.security.ssl.InputRecord.readFully(Unknown Source)

  at sun.security.ssl.InputRecord.read(Unknown Source)

  at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)

  at sun.security.ssl.SSLSocketImpl.readDataRecord(Unknown Source)

  at sun.security.ssl.AppInputStream.read(Unknown Source)

  at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)

  at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)

  at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:281)

  at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:92)

  at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:62)

  at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:254)

  at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:289)

  at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:252)

  at org.apache.http.impl.conn.AbstractClientConnAdapter.receiveResponseHeader(AbstractClientConnAdapter.java:219)

  at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:300)

  at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:127)

  at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:713)

  at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:518)

  at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)

  at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)

  at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:111)

  ... 22 more

Subexception: com.vmware.vcloud.api.presentation.service.InternalServerErrorException: Internal Server Error

  at com.vmware.vcloud.fabric.foundation.futures.FutureUtil.convertExecutionException(FutureUtil.java:198)

  at com.vmware.vcloud.vdc.impl.AbstractVdcService.waitForFabricVmFutures(AbstractVdcService.java:837)

  at com.vmware.vcloud.vdc.impl.AbstractVdcService.copyVAppContents(AbstractVdcService.java:638)

  at com.vmware.vcloud.vdc.impl.AbstractVdcService.copyVAppContents(AbstractVdcService.java:291)

  at com.vmware.vcloud.vdc.impl.ResourceEntityLifeCycleImpl.instantiateVAppTask(ResourceEntityLifeCycleImpl.java:1332)

  at com.vmware.vcloud.vdc.impl.ResourceEntityLifeCycleImpl.executeTask(ResourceEntityLifeCycleImpl.java:1769)

  at com.vmware.vcloud.backendbase.management.system.TaskActivity$ExecutePhase$2.doInSecurityContext(TaskActivity.java:427)

  at com.vmware.vcloud.backendbase.management.system.TaskActivity$ExecutePhase$2.doInSecurityContext(TaskActivity.java:421)

  at com.vmware.vcloud.backendbase.management.system.TaskActivity$SecurityContextTemplate.executeForOrgAndUser(TaskActivity.java:674)

  at com.vmware.vcloud.backendbase.management.system.TaskActivity$ExecutePhase.execute(TaskActivity.java:421)

  at com.vmware.vcloud.backendbase.management.system.TaskActivity$ExecutePhase.invoke(TaskActivity.java:354)

  at com.vmware.vcloud.activity.executors.ActivityRunner.run(ActivityRunner.java:109)

  at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)

  at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)

  at java.util.concurrent.FutureTask.run(Unknown Source)

  at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

  at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

  at java.lang.Thread.run(Unknown Source)

Caused by: com.vmware.vim.vmomi.client.exception.ConnectionException: java.net.SocketTimeoutException: Read timed out

  at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setError(ResponseImpl.java:224)

  at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:131)

  at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingImpl.send(HttpProtocolBindingImpl.java:98)

  at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.sendCall(MethodInvocationHandlerImpl.java:526)

  at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.executeCall(MethodInvocationHandlerImpl.java:507)

  at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.completeCall(MethodInvocationHandlerImpl.java:295)

  at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invokeOperation(MethodInvocationHandlerImpl.java:265)

  at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invoke(MethodInvocationHandlerImpl.java:169)

  at com.sun.proxy.$Proxy1112.recommendDatastores(Unknown Source)

  at com.vmware.vcloud.fabric.storage.placement.impl.StoragePodPlacementManagerImpl.getDatastoreFromSDRS(StoragePodPlacementManagerImpl.java:266)

  at com.vmware.vcloud.fabric.storage.placement.impl.StoragePodPlacementManagerImpl.recommendDatastore(StoragePodPlacementManagerImpl.java:124)

  at com.vmware.vcloud.fabric.storage.placement.impl.PlacementManagerImpl.selectDatastoreInPod(PlacementManagerImpl.java:379)

  at com.vmware.vcloud.fabric.storage.placement.impl.PlacementManagerImpl.selectDatastore(PlacementManagerImpl.java:168)

  at com.vmware.vcloud.fabric.storage.storedVm.impl.CreateFromExistingStoredVmActivity$CreateStoredVmInVcPhase.selectDatastore(CreateFromExistingStoredVmActivity.java:769)

  at com.vmware.vcloud.fabric.storage.storedVm.impl.CreateFromExistingStoredVmActivity$CreateStoredVmInVcPhase.findOrCreateSourceVm(CreateFromExistingStoredVmActivity.java:631)

  at com.vmware.vcloud.fabric.storage.storedVm.impl.CreateFromExistingStoredVmActivity$CreateStoredVmInVcPhase.copyVmWithLinkedCloneFromTemplate(CreateFromExistingStoredVmActivity.java:579)

  at com.vmware.vcloud.fabric.storage.storedVm.impl.CreateFromExistingStoredVmActivity$CreateStoredVmInVcPhase.invoke(CreateFromExistingStoredVmActivity.java:265)

  ... 7 more

Caused by: java.net.SocketTimeoutException: Read timed out

  at java.net.SocketInputStream.socketRead0(Native Method)

  at java.net.SocketInputStream.read(Unknown Source)

  at java.net.SocketInputStream.read(Unknown Source)

  at sun.security.ssl.InputRecord.readFully(Unknown Source)

  at sun.security.ssl.InputRecord.read(Unknown Source)

  at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)

  at sun.security.ssl.SSLSocketImpl.readDataRecord(Unknown Source)

  at sun.security.ssl.AppInputStream.read(Unknown Source)

  at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)

  at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)

  at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:281)

  at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:92)

  at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:62)

  at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:254)

  at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:289)

  at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:252)

  at org.apache.http.impl.conn.AbstractClientConnAdapter.receiveResponseHeader(AbstractClientConnAdapter.java:219)

  at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:300)

  at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:127)

  at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:713)

  at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:518)

  at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)

  at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)

  at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:111)

1 Solution

Accepted Solutions
IamTHEvilONE
Immortal
Immortal


The critical point is here:


  at com.sun.proxy.$Proxy1146.recommendDatastores(Unknown Source)

  at com.vmware.vcloud.fabric.storage.placement.impl.StoragePodPlacementManagerImpl.getDatastoreFromSDRS(StoragePodPlacementManagerImpl.java:266)

Turn of DRS in vCenter as a temporary fix.  If you can deploy then, it's definitely Storage DRS related.

Newer versions of vCenter have improvements to speed up the returned result of SDRS recommending a datastore for VM placement.

View solution in original post

5 Replies
IamTHEvilONE
Immortal
Immortal


The critical point is here:


  at com.sun.proxy.$Proxy1146.recommendDatastores(Unknown Source)

  at com.vmware.vcloud.fabric.storage.placement.impl.StoragePodPlacementManagerImpl.getDatastoreFromSDRS(StoragePodPlacementManagerImpl.java:266)

Turn of DRS in vCenter as a temporary fix.  If you can deploy then, it's definitely Storage DRS related.

Newer versions of vCenter have improvements to speed up the returned result of SDRS recommending a datastore for VM placement.

Golan_M
Contributor
Contributor

Thanks for your help.. it actually working now (as a workaround) Smiley HappySmiley Happy

In case upgrading to the latest update of VC 5.5  (we are currently running 5.5.0 build 2183111) wont help, what would you recommend us to do?

0 Kudos
IamTHEvilONE
Immortal
Immortal

The call that is failing is vCenter code in Storage DRS.  it's taking too long.  if you can't upgrade, then the workaround will be that vCD Determines initial placement of VMs.  Then there is no load balancing after the fact (since SDRS is off).

If you need more concise evaluation, then you might want to file a support request for vCenter and submit logs from vCenter with a window for failure (deploy time and failure time in vCD).

Golan_M
Contributor
Contributor

I will upgrade the VC to the latest update, hopefully it will fix the issue.

Thanks again for your help.

Smiley Happy

0 Kudos
Mlitsn
Contributor
Contributor

Hi,I want to use vcloud restful api to create two VMs in one vApp at the same time,How to do this?

I'll appreciate your assistance


Thank you ! Smiley Happy

0 Kudos