vCAC 6 Blueprint fails when using vCNS Transport Zone (network scope)

[bugblatterbeast]

Hi All

I currently have vCAC 6 setup with vCNS, the integration works and I can see my Transport Zone (which is actually the vCNS VXLAN Network Scope).  I then created a basic blueprint which uses the Transport Zone to create a simple private network.

When I deploy the blueprint from the catalogue I see the new Portgroup get created in vCenter, a new virtual wire gets created and associated with the Network Scope in vCNS but then it stops with the following error (in vCAC):-

Request Failed: Machine DVT-1-014: Failed to assign one or more network and security settings. Error: " Required parameter uuid is missing while parsing call information for method FindByUuid at line 1, column 218 while parsing SOAP body at line 1...

The DEM Worker log has a bit more info:-

[UTC:2014-03-18 17:07:40 Local:2014-03-18 17:07:40] [Trace]: VCNSModel: The CreateVirtualWireNetwork activity finished for virtual wire vxlan-test-private-network-92e82ee2-ac70-40a3-860f-bda26c97cdaf on network scope vCAC-transport-zone and tenant .

[UTC:2014-03-18 17:07:43 Local:2014-03-18 17:07:43] [Trace]: VCNSModel: The VCNSLogOff activity started.

[UTC:2014-03-18 17:07:43 Local:2014-03-18 17:07:43] [Trace]: VCNSModel: The VCNSLogOff activity finished.

[UTC:2014-03-18 17:07:43 Local:2014-03-18 17:07:43] [Error]: <?xml version="1.0" encoding="utf-16"?>

<boolean>false</boolean>

An error occurred during the execution of the workflow. Please look at the inner exception for more details.

System.Web.Services.Protocols.SoapException:

Required parameter uuid is missing

while parsing call information for method FindByUuid

at line 1, column 218

while parsing SOAP body

at line 1, column 207

while parsing SOAP envelope

at line 1, column 38

while parsing HTTP request for method findByUuid

on object of type vim.SearchIndex

at line 1, column 0

Any ideas folks?

Danny

[bugblatterbeast]

So this finally got sorted.  First off, after trying a lot of different approaches (with VMware Support) I ended up rebuilding the entire vCAC environment (ID, vCAC and IaaS servers) but I kept the original vCenter and vShield servers.

When I tried a MM blueprint deploy on the new environment I got errors again but this time they were different. From the DEM I was getting this:-

[UTC:2014-04-10 16:08:47 Local:2014-04-10 17:08:47] [Trace]: VCNSModel: The CreateVirtualWireNetwork activity finished for virtual wire PRIVATE_IP_Pool_1-369141d0-b32b-45db-92fa-cae8d56b4fbf on network scope vCAC-transport-zone and tenant .

[UTC:2014-04-10 16:08:48 Local:2014-04-10 17:08:48] [Trace]: Worker Controller: QueryTimerEventHandler

[UTC:2014-04-10 16:08:48 Local:2014-04-10 17:08:48] [Trace]: Worker Controller: GetWorkflowInstances

[UTC:2014-04-10 16:08:48 Local:2014-04-10 17:08:48] [Debug]: Worker Controller.GetWorkflowInstances: Executing:1 Requested:14 Received:0

[UTC:2014-04-10 16:08:48 Local:2014-04-10 17:08:48] [Trace]: Worker Controller: WritePendingCompletions

[UTC:2014-04-10 16:08:52 Local:2014-04-10 17:08:52] [Trace]: VCNSModel: The AddHostNics activity started.

[UTC:2014-04-10 16:08:53 Local:2014-04-10 17:08:53] [Trace]: VCNSModel: The AddHostNics activity finished.

[UTC:2014-04-10 16:08:53 Local:2014-04-10 17:08:53] [Trace]: VCNSModel: The AssignNetworkToVMNic activity started for machine [None].

[UTC:2014-04-10 16:08:53 Local:2014-04-10 17:08:53] [Trace]: VCNSModel: The AssignNetworkToVMNic activity finished for machine [None].

[UTC:2014-04-10 16:08:53 Local:2014-04-10 17:08:53] [Trace]: VCNSModel: The AssignNetworkToVMNic activity started for machine DVOPS1-008.

[UTC:2014-04-10 16:08:53 Local:2014-04-10 17:08:53] [Trace]: VCNSModel: The AssignNetworkToVMNic activity finished for machine DVOPS1-008.

[UTC:2014-04-10 16:08:53 Local:2014-04-10 17:08:53] [Trace]: VCNSModel: The ConfigureVirtualWireNetworks activity finished for multi-machine 369141d0-b32b-45db-92fa-cae8d56b4fbf.

[UTC:2014-04-10 16:08:53 Local:2014-04-10 17:08:53] [Trace]: VCNSModel: The ConfigureEdge activity started for multi-machine 369141d0-b32b-45db-92fa-cae8d56b4fbf.

[UTC:2014-04-10 16:08:53 Local:2014-04-10 17:08:53] [Trace]: VCNSModel: The CreateEdge activity started for multi-machine 369141d0-b32b-45db-92fa-cae8d56b4fbf.

[UTC:2014-04-10 16:08:58 Local:2014-04-10 17:08:58] [Trace]: VCNSModel: The VCNSLogOff activity started.

[UTC:2014-04-10 16:08:58 Local:2014-04-10 17:08:58] [Trace]: VCNSModel: The VCNSLogOff activity finished.

[UTC:2014-04-10 16:08:58 Local:2014-04-10 17:08:58] [Error]: <?xml version="1.0" encoding="utf-16"?> <boolean>false</boolean> An error occurred during the execution of the workflow. Please look at the inner exception for more details.

1. System.Net.Http.HttpRequestException: Response status code indicates failure: Bad Request. Response message: <?xml version="1.0" encoding="UTF-8"?> <error><details>Failed to publish configuration on vShield Edge. Failed to deploy edge appliance.</details><errorCode>10105</errorCode><rootCauseString>Operation failed in VC. Type of ManagedObjectReference was not correct.</rootCauseString><moduleName>vShield Edge</moduleName></error>.

   at System.Activities.WorkflowApplication.InvokeAsyncResult..ctor(Activity activity, IDictionary`2 inputs, WorkflowInstanceExtensionManager extensions, TimeSpan timeout, SynchronizationContext syncContext, AsyncInvokeContext invokeContext, AsyncCallback callback, Object state)

   at System.Activities.WorkflowInvoker.InternalInvokeAsync(IDictionary`2 inputs, TimeSpan timeout, Object userState)

[UTC:2014-04-10 16:08:59 Local:2014-04-10 17:08:59] [Trace]: Worker Controller: WorkflowApplication_Completed

[UTC:2014-04-10 16:08:59 Local:2014-04-10 17:08:59] [Trace]: Worker Controller: ProcessWorkflowCompleted

[UTC:2014-04-10 16:08:59 Local:2014-04-10 17:08:59] [Debug]: Workflow Complete:  186 - Failed: An error occurred during the execution of the workflow. Please look at the inner exception for more details.

[UTC:2014-04-10 16:08:59 Local:2014-04-10 17:08:59] [Trace]: Worker Controller: WriteCompletedWorkflow

[UTC:2014-04-10 16:08:59 Local:2014-04-10 17:08:59] [Error]: <?xml version="1.0" encoding="utf-16"?> <boolean>false</boolean> Workflow "AppServiceWaitComponents" failed with the following exception:

1. System.InvalidOperationException: An error occurred during the execution of the workflow. Please look at the inner exception for more details. ---> System.Net.Http.HttpRequestException: Response status code indicates failure: Bad Request. Response message: <?xml version="1.0" encoding="UTF-8"?> <error><details>Failed to publish configuration on vShield Edge. Failed to deploy edge appliance.</details><errorCode>10105</errorCode><rootCauseString>Operation failed in VC. Type of ManagedObjectReference was not correct.</rootCauseString><moduleName>vShield Edge</moduleName></error>.

   at System.Activities.WorkflowApplication.InvokeAsyncResult..ctor(Activity activity, IDictionary`2 inputs, WorkflowInstanceExtensionManager extensions, TimeSpan timeout, SynchronizationContext syncContext, AsyncInvokeContext invokeContext, AsyncCallback callback, Object state)

   at System.Activities.WorkflowInvoker.InternalInvokeAsync(IDictionary`2 inputs, TimeSpan timeout, Object userState)

   --- End of inner exception stack trace ---

   at DynamicOps.Repository.Activities.InvokeRepositoryWorkflow.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)

   at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager) INNER EXCEPTION: System.Net.Http.HttpRequestException: Response status code indicates failure: Bad Request. Response message: <?xml version="1.0" encoding="UTF-8"?> <error><details>Failed to publish configuration on vShield Edge. Failed to deploy edge appliance.</details><errorCode>10105</errorCode><rootCauseString>Operation failed in VC. Type of ManagedObjectReference was not correct.</rootCauseString><moduleName>vShield Edge</moduleName></error>.

   at System.Activities.WorkflowApplication.InvokeAsyncResult..ctor(Activity activity, IDictionary`2 inputs, WorkflowInstanceExtensionManager extensions, TimeSpan timeout, SynchronizationContext syncContext, AsyncInvokeContext invokeContext, AsyncCallback callback, Object state)

So then I revisited this post:-

http://grantorchard.com/vcac/operations/vcac-vcns-datastore-clusters-gotcha/

Sure enough, when I removed the Datastore Cluster from the Reservation and replaced it with the individual LUNs everything started working.  I had tried this on my first environment but it made no difference, so there must have been something else wrong originally.

So the problem has been fixed but the root cause of the original issue is still unknown.  VMware Support are logging a PR for the Datastore Cluster issue so hopefully this will get sorted on a future release.

[GrantOrchardVMw]

Glad to hear that helped. The original issue *may* have been caused if you had a cluster (computer resource) that was then removed. The null object thing often seems to refer to a db entry that hasn't been cleaned up properly after an object removal.

Grant

[bugblatterbeast]

Yep, without your blog post I may have just given up on it entirely.  Hopefully this will be resolved in the next release of vCAC.

As far as I recall I do not believe I removed any cluster resource from vCAC  i'm just glad it's now working

[ovdkruijf]

I'm having the exact same problem as described in the opening post.

Does this mean a rebuild of the VCAC environment is the only option?

[Jonesie946]

I am also experiencing the issue described in the original post. Has anyone been able to recover without redeploying vCAC? That's really not an option.

I am not using datastore clusters, so that isn't the issue.

Thanks - David

[bragazzi]

I've had the same problem.  Had erroneously included a datastore cluster in a reservation and ever after adjusting the reservation, it would not provision a multi-machine blueprint successfully.  It's not a good solution, but I just rebuilt the IaaS server and used a new SQL database.  I was able to keep the SSO and vCAC Server Appliance intact, but have had to redo the endpoints, fabric groups, business groups, reservations, customized stub workflows and entitlements.   I really hope this bug is corrected soon.

[GrantOrchardVMw]

For anyone who reads this thread and has this problem LOG A GSS CASE.

This can be resolved by making some SQL edits, but it does require some guidance.

Grant

[Jonesie946]

Grant is correct. I was able to get my vCAC install back up and running with the help of a great VMware Support Engineer.

Apparently, I had renamed a cluster in vCenter, but the IaaS SQL DB was not updated, despite running several Inventory collections in vCAC. I was told that this SHOULD be fixed in 6.1. I can only hope...

David