Quark998
Contributor
Contributor

Unable to deploy or create any vApp

I had everything working nicely, and then from one day to the next I am suddenly unable to either deploy a vApp for a template, or to create a new one. Copying does not work either. All I get is an error message "Failed to Create" on the vApp I am trying to deploy, with a red "Cannot create", clicking on which reveals, helpfully:

One or more exceptions have occurred
- Error
- Error
- Error
- Error

The vCenter client shows that the (linked) clones are created successfully, but after that, nothing happens - no vShield is deployed, ports configured etc. like normally would be the case. I have one existing, deployed vApp that I can start and stop without issues, it just anything new that seems to fail.

I did have a few issues with the time on my vCloud Director appliance, but that has since been fixed - but it is roughly when this started.

The only place I know of that could contain information is the vcloud-container-info.log. It shows for the attempt to deploy:

2012-11-09 15:43:30,497 | INFO     | sf-activity-pool-4        | VC20VirtualServer              | Done with CloneVM_Task for vm [vcId=d11ba018-59c0-4bb0-92b
3-8320b15ccab9, moref=vm-1534]. New vm moref = vm-1660 | vcd=07e29ffd-c937-42d3-8cb7-0ed7cb0a7158,task=7a11f64f-48d8-4969-ad2f-3f0e4de26859 activity=urn:uui
d:1c1a84a6-a010-49aa-af33-a51c303c5c53 activity=urn:uuid:1c1a84a6-a010-49aa-af33-a51c303c5c53
2012-11-09 15:43:30,497 | INFO     | sf-activity-pool-4        | VC20VirtualServer              | Cloned VM. valref = [vcId=d11ba018-59c0-4bb0-92b3-8320b15c
cab9, moref=vm-1660] | vcd=07e29ffd-c937-42d3-8cb7-0ed7cb0a7158,task=7a11f64f-48d8-4969-ad2f-3f0e4de26859 activity=urn:uuid:1c1a84a6-a010-49aa-af33-a51c303c
5c53 activity=urn:uuid:1c1a84a6-a010-49aa-af33-a51c303c5c53
2012-11-09 15:43:30,742 | ERROR    | sf-activity-pool-2        | SpbmServiceImpl                | SPS UnexpectedStatusCodeException failure of 1 attempts to
make SPS call at VC d11ba018-59c0-4bb0-92b3-8320b15ccab9. | vcd=07e29ffd-c937-42d3-8cb7-0ed7cb0a7158,task=7a11f64f-48d8-4969-ad2f-3f0e4de26859 activity=urn
:uuid:57e9a2cb-5b07-4fde-aed3-fe19a410e1cd activity=urn:uuid:57e9a2cb-5b07-4fde-aed3-fe19a410e1cd

...

2012-11-09 15:43:37,413 | ERROR    | sf-activity-pool-2        | SpbmServiceImpl                | SPS UnexpectedStatusCodeException failure of 5 attempts to
make SPS call at VC d11ba018-59c0-4bb0-92b3-8320b15ccab9. | vcd=07e29ffd-c937-42d3-8cb7-0ed7cb0a7158,task=7a11f64f-48d8-4969-ad2f-3f0e4de26859 activity=urn
:uuid:57e9a2cb-5b07-4fde-aed3-fe19a410e1cd activity=urn:uuid:57e9a2cb-5b07-4fde-aed3-fe19a410e1cd
2012-11-09 15:43:37,443 | ERROR    | sf-activity-pool-2        | VmCopyServiceImpl              | partial failure while copying VM with name "BE2012A (57e9a
2cb-5b07-4fde-aed3-fe19a410e1cd)", db ID: a3cb43fe-f0d2-48f5-a0bc-73fdd3d84a89, valref: [vcId=d11ba018-59c0-4bb0-92b3-8320b15ccab9, moref=vm-1661] | vcd=07e
29ffd-c937-42d3-8cb7-0ed7cb0a7158,task=7a11f64f-48d8-4969-ad2f-3f0e4de26859 activity=urn:uuid:57e9a2cb-5b07-4fde-aed3-fe19a410e1cd activity=urn:uuid:57e9a2c
b-5b07-4fde-aed3-fe19a410e1cd
2012-11-09 15:43:37,468 | ERROR    | sf-activity-pool-2        | CreateFromExistingStoredVmActivity | [Activity Execution] Handle: urn:uuid:57e9a2cb-5b07-4f
de-aed3-fe19a410e1cd, Current Phase: com.vmware.vcloud.fabric.storage.storedVm.impl.CreateFromExistingStoredVmActivity$CreateStoredVmInVcPhase, ActivityExec
utionState Parameter Names: [NDC, CREATION_PARAMS, com.vmware.activityEntityRecord.EntityId, ORIGIN_PARAMS] - Failed VM cloning with partial error | vcd=07e
29ffd-c937-42d3-8cb7-0ed7cb0a7158,task=7a11f64f-48d8-4969-ad2f-3f0e4de26859 activity=urn:uuid:57e9a2cb-5b07-4fde-aed3-fe19a410e1cd activity=urn:uuid:57e9a2c
b-5b07-4fde-aed3-fe19a410e1cd
com.vmware.ssdc.util.LMException: Error
at com.vmware.ssdc.util.LMException.wrap(LMException.java:129)
at com.vmware.vcloud.spbm.impl.SpbmServiceImpl.submitSpsRequest(SpbmServiceImpl.java:888)
at com.vmware.vcloud.spbm.impl.SpbmServiceImpl.associateVmAndDisksWithStorageProfile(SpbmServiceImpl.java:679)
at com.vmware.vcloud.val.internal.impl.VC20VirtualServer.associateVmAndDisksWithStorageProfile(VC20VirtualServer.java:3631)
at com.vmware.vcloud.val.internal.impl.VC20VirtualServer.clone(VC20VirtualServer.java:2581)
at com.vmware.vcloud.fabric.storage.copy.impl.VmCopyServiceImpl.intraVcCopyVm(VmCopyServiceImpl.java:717)
at com.vmware.vcloud.fabric.storage.copy.impl.VmCopyServiceImpl.copyVm(VmCopyServiceImpl.java:340)
at com.vmware.vcloud.fabric.storage.copy.impl.VmCopyServiceImpl.copyVms(VmCopyServiceImpl.java:283)
at com.vmware.vcloud.fabric.storage.copy.impl.VmCopyServiceImpl.copyVm(VmCopyServiceImpl.java:251)
at com.vmware.vcloud.fabric.storage.storedVm.impl.AbstractCreateStoredVmActivity.copyVmInVc(AbstractCreateStoredVmActivity.java:160)
at com.vmware.vcloud.fabric.storage.storedVm.impl.CreateFromExistingStoredVmActivity$CreateStoredVmInVcPhase.copyVmWithLinkedCloneFromTemplate(Creat
eFromExistingStoredVmActivity.java:472)
at com.vmware.vcloud.fabric.storage.storedVm.impl.CreateFromExistingStoredVmActivity$CreateStoredVmInVcPhase.invoke(CreateFromExistingStoredVmActivi
ty.java:267)
at com.vmware.vcloud.fabric.foundation.activity.executors.ActivityRunner.run(ActivityRunner.java:123)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
2012-11-09 15:43:41,407 | ERROR    | sf-activity-pool-4        | SpbmServiceImpl                | SPS UnexpectedStatusCodeException failure of 5 attempts to
make SPS call at VC d11ba018-59c0-4bb0-92b3-8320b15ccab9. | vcd=07e29ffd-c937-42d3-8cb7-0ed7cb0a7158,task=7a11f64f-48d8-4969-ad2f-3f0e4de26859 activity=urn
:uuid:1c1a84a6-a010-49aa-af33-a51c303c5c53 activity=urn:uuid:1c1a84a6-a010-49aa-af33-a51c303c5c53
2012-11-09 15:43:41,438 | ERROR    | sf-activity-pool-4        | VmCopyServiceImpl              | partial failure while copying VM with name "DCEXCH1 (1c1a8
4a6-a010-49aa-af33-a51c303c5c53)", db ID: 2565015c-e95c-4398-9d33-123b88dec58b, valref: [vcId=d11ba018-59c0-4bb0-92b3-8320b15ccab9, moref=vm-1660] | vcd=07e
29ffd-c937-42d3-8cb7-0ed7cb0a7158,task=7a11f64f-48d8-4969-ad2f-3f0e4de26859 activity=urn:uuid:1c1a84a6-a010-49aa-af33-a51c303c5c53 activity=urn:uuid:1c1a84a
6-a010-49aa-af33-a51c303c5c53
2012-11-09 15:43:41,461 | ERROR    | sf-activity-pool-4        | CreateFromExistingStoredVmActivity | [Activity Execution] Handle: urn:uuid:1c1a84a6-a010-49
aa-af33-a51c303c5c53, Current Phase: com.vmware.vcloud.fabric.storage.storedVm.impl.CreateFromExistingStoredVmActivity$CreateStoredVmInVcPhase, ActivityExec
utionState Parameter Names: [NDC, CREATION_PARAMS, com.vmware.activityEntityRecord.EntityId, ORIGIN_PARAMS] - Failed VM cloning with partial error | vcd=07e
29ffd-c937-42d3-8cb7-0ed7cb0a7158,task=7a11f64f-48d8-4969-ad2f-3f0e4de26859 activity=urn:uuid:1c1a84a6-a010-49aa-af33-a51c303c5c53 activity=urn:uuid:1c1a84a
6-a010-49aa-af33-a51c303c5c53
com.vmware.ssdc.util.LMException: Error
at com.vmware.ssdc.util.LMException.wrap(LMException.java:129)
at com.vmware.vcloud.spbm.impl.SpbmServiceImpl.submitSpsRequest(SpbmServiceImpl.java:888)
at com.vmware.vcloud.spbm.impl.SpbmServiceImpl.associateVmAndDisksWithStorageProfile(SpbmServiceImpl.java:679)
at com.vmware.vcloud.val.internal.impl.VC20VirtualServer.associateVmAndDisksWithStorageProfile(VC20VirtualServer.java:3631)
at com.vmware.vcloud.val.internal.impl.VC20VirtualServer.clone(VC20VirtualServer.java:2581)
at com.vmware.vcloud.fabric.storage.copy.impl.VmCopyServiceImpl.intraVcCopyVm(VmCopyServiceImpl.java:717)
at com.vmware.vcloud.fabric.storage.copy.impl.VmCopyServiceImpl.copyVm(VmCopyServiceImpl.java:340)
at com.vmware.vcloud.fabric.storage.copy.impl.VmCopyServiceImpl.copyVms(VmCopyServiceImpl.java:283)
at com.vmware.vcloud.fabric.storage.copy.impl.VmCopyServiceImpl.copyVm(VmCopyServiceImpl.java:251)
at com.vmware.vcloud.fabric.storage.storedVm.impl.AbstractCreateStoredVmActivity.copyVmInVc(AbstractCreateStoredVmActivity.java:160)
at com.vmware.vcloud.fabric.storage.storedVm.impl.CreateFromExistingStoredVmActivity$CreateStoredVmInVcPhase.copyVmWithLinkedCloneFromTemplate(Creat
eFromExistingStoredVmActivity.java:472)
at com.vmware.vcloud.fabric.storage.storedVm.impl.CreateFromExistingStoredVmActivity$CreateStoredVmInVcPhase.invoke(CreateFromExistingStoredVmActivi
ty.java:267)
at com.vmware.vcloud.fabric.foundation.activity.executors.ActivityRunner.run(ActivityRunner.java:123)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)

etc. Help!

0 Kudos
3 Replies
Michael_Ottati
Enthusiast
Enthusiast

I just had exactly the same issue with a new installation of vCloud that I was using to set up a lab. I also found the error spectacularly uninformative. The fix was, to say the least, obscure.

In my case the root cause had to do with storage profiles, and how they had been created. Because this vCloud installation was being created in parallel with an existing Lab Manager installation we now use, I was limited in the resources I had access to. I only had a few storage volumes and a few hosts that I could move from the old Lab Manager installation to the new vCloud installation. I decided to create storage profiles because the default installation of vCloud created its own storage profile of "All", and I was unhappy with the placement of the VMs when this default storage profile was used.

It turned out that it was not enough to simply create storage profiles, these profiles had to be associated with hosts bac in vCenter. The mechanism for doing this is to:

  1. Login to vCenter
  2. Navigate to VM Storage Profiles (Home -> Management -> VM Storage Profiles)
  3. Select the Datastore(s) you are using in vCloud
  4. Click on Enable VM Storage Profiles

This brings up a dialog box where you can asociate the stoarge profile with a given host.

I have tried to attach an illistration showing this. I wanted to put it in the message itself however at the moment the InsertImage button is greyed out. Look at the attached StorageProfile.PNG file for an illistration of what I wrote above.

Once we associated the storage profile with the host being used by vCloud, the error you descibe cleared. Hopefully this is your issue as well. If so the fix is quick and you should be on your way soon after trying this.

Quark998
Contributor
Contributor

Thanks for that - I had thought about the Storage Profiles, because I have had a pretty similar error when I didn't have Storage Profiles enabled/licensed on the individual hosts.

So you mentioning them has sent me on the right track - I tried to double-check them in Management -> VM Storage Profiles and got greeted by this error:

http://communities.vmware.com/thread/338847

I then checked my vCenter Appliance, and even though it was generally working it was showing as "stopped" - so I assume that one or more services had gotten their knickers in a twist. I clicked "start" and it went back to "running", and at that point the VM Storage Profiles tab started working in vCenter again - and lo and behold, vCloud Director as well.

So it looks like my error was a non-started storage profile service (not sure which service exactly is the one is responsible for them), and I am keeping my fingers crossed that it will stay up because I have no idea why it was stopped in the first place.

It also goes to show that mysterious non-descript errors in vCloud Director are probably related to problems with storage profiles, even though the reasons might be slightly different.

Thanks!

0 Kudos
DougBaer
VMware Employee
VMware Employee

I saw this happen recently when my Inventory Service on the vCenter stopped due to insufficient disk space.

The first clue beyond the vCD deployment failures was the stopped Inventory Service reported in the vCenter UI, but looking at the logs on the vCD cell during the attempt to restart the Inventory Service showed multiple errors that vCenter was unable to write to its journal due to lack of disk space.

You'll want to look in /opt/vmware/vcloud-director/logs for the logs on the cell.

Doug Baer, Staff Architect, Sr. Manager of vPod Architecture team for the VMware Hands-on Labs | VCDX #019, vExpert 2012-20 | @dobaer
0 Kudos