Internal server error when importing VM or vApp from vSphere

[NitsanReznik]

Hi,

Sorry for the long discussion here, just thought to provide as much as details as I can in order for you guys to assist. Any help would be highly grateful!

Recently, we made an upgrade to our vCloud Director which was on version 9.0 to version 9.5 and then to 9.7 according to VMware best practice.

Everything was working properly after above upgrade until someone from our IT Team mistakenly turned off DRS on the VCenter where vCD ESXi hosts are presented. We re-enabled DRS again and from then, we have been experiencing some really strange error. The first error we had was when we tried updating our Org VDCs then, after some research and creation of a new Org VDC, we realized that the resource pools were not existed on the same Org VDCs (Which we had the update error mentioned above) so, we had to re-create the Org VDCs then, resource pools do show up and Org VDC are successfully updated.

The second and more crucial error we've been facing now is when we try to import VM or vApp from vSphere, we keep getting the following error: Internal server error (No other details apart from that). When we create a new vApp via flash or via the tenant (in HTML), there's no problem and both vApp and VM are successfully created. The problem seems to be only when we want to import the VM from vSphere. The main reason we need this import to work is due to our Backup product (We use Netbackup) and it follows and applies the same process of importing a VM when a restore is needed. It's also worth mentioning that Backup vCD VMs through Netbackup works well with no issues, the problem exists only when we try to restore vCD VMs (Either to the same location or to an alternate location) as like mentioned above, it follows an import from VSphere first as well.

Now, below are the steps we tried (With no luck) to troubleshoot:

1. Properly reboot vCD and vSphere environments.

2. Make sure there are communication between vSphere and vCD.

3. Make sure ESXi hosts with the cluster are able to see the NFS Datastore.

4. Make sure vCD are not showing any errors on NFS, ESXi hosts including network pools.

5. Make sure all vCD components are not showing any errors and in green state.

6. Create new Org, allocate the correct resources, network pool, gives the same error.

7. Disable "VM Discovery Enable" didn't make any difference.

Finally, we have had to keep monitoring the "vcloud-container-debug.log" file to view any debug errors, (We obviously first enabled debug) and below (Highlighted in blue) is what we have found repeating itself couple of times which matches the same amount of attempts we tried the import. I have also attached the vcloud-container-debug.log file to see if anyone of you can help us. If you also think we need to analyse any other logs, we would appreciate if you could let us know.

Below are the details of our vCD and vSphere environments (highlighted in green):

vCenter: 6.5

2 ESXi hosts within a cluster 6.5

DataStore NFS 3

vCD 9.7.0.1

NSX Manager 6.4.1

Please let me know if any other information is needed.

| Backend-activity-pool-949040 | CreateStoredVmActivity         | [Activity Execution] Uncaught Exception during Activity execution. Recent phase: com.vmware.vcloud.vdc.impl.CreateStoredVmActivity$InvokeStorageFabricPhase@6af50845 - Handle: urn:uuid:82837f74-8a6a-406c-b351-bbbee1709e6a, Current Phase: CreateStoredVmActivity$InvokeStorageFabricPhase | requestId=0b2c5585-a7cc-415b-bbd7-39dc7e170f27,request=POST https://demovcd1.biomnidemo.com/cloud/amfsecure,requestTime=1571923970431,remoteAddress=10.11.249.35... (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 ...,accept=*/* method=importService.importIntoExistingVApp vcd=0e728462-edc6-496c-bec6-275a679ec2b9,task=81f8e6b0-5b3e-46b0-8c90-2c385c8e5aef activity=(com.vmware.vcloud.backendbase.management.system.TaskActivity,urn:uuid:81f8e6b0-5b3e-46b0-8c90-2c385c8e5aef) activity=(com.vmware.vcloud.vdc.impl.ImportVmIntoExistingVAppActivity,urn:uuid:717bcd2a-427d-4ad2-9e6d-ca0c67089dca) activity=(com.vmware.vcloud.vdc.impl.CreateStoredVmActivity,urn:uuid:82837f74-8a6a-406c-b351-bbbee1709e6a)

java.util.concurrent.ExecutionException: com.vmware.vcloud.api.presentation.service.InternalServerErrorException: Internal Server Error

at com.vmware.vcloud.activity.executors.ActivityRunner.run(ActivityRunner.java:136)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

Caused by: com.vmware.vcloud.api.presentation.service.InternalServerErrorException: Internal Server Error

at com.vmware.vcloud.common.future.FutureUtil.convertExecutionException(FutureUtil.java:219)

at com.vmware.vcloud.common.activity.AbstractActivity$RetryablePhase.invokeCancelableOperation(AbstractActivity.java:285)

at com.vmware.vcloud.common.activity.AbstractActivity$CancelablePhase.invoke(AbstractActivity.java:365)

at com.vmware.vcloud.activity.executors.ActivityRunner.runPhase(ActivityRunner.java:175)

at com.vmware.vcloud.activity.executors.ActivityRunner.run(ActivityRunner.java:112)

... 5 more

Caused by: java.lang.NullPointerException

at com.vmware.vcloud.fabric.storage.storedVm.impl.CreateFromVCenterVmActivity$AddVmToVmGroupPhaseForImportFromVc.invoke(CreateFromVCenterVmActivity.java:282)

[paluszekd]

Are your storage policies showing up correctly?

I suggest logging an SR - I have a feeling this is due to the recreated resource pool structure and vCD struggling with this recreation. Unfortunately, destruction of the RP hierarchy can cause a multitude of issues.