Upload OVF in vCloud Director failing

[peterdabr]

I've been investing issue with OVF upload via vCD console and I'm hoping that somebody could help me solve this problem.

Here's what I have:

- ESXi 5 hosts in the cluster

- 2 vCD cells behind LB (VMs)

- vCenter5 (VM)

- only 1 cell is active (the other one is currently disabled using cell-management-tool to make troubleshooting as simple as possible)

- /opt/vmware/vcloud-director/data/transfer is mounted on both cells

- http and httpconsole are active on 192.168.35.x IPs and web UI from which I'm trying to upload ovf is open on a separate server located the same network (this is to make sure there's no ACLs contributing to the problem)

-  I'm using vMA ovf downloaded from vmware.com as my test VA for upload

Problem:

When running 'Upload' in 'Catalogs'->'vApp Templates' the transfer succeedes in the transfer pop-up window box but when I refresh 'vApp Templates' tab the same vapp shows up with 'Failed to create.  Cannot upload' error in the Status column. Details for that error are very generic:

"The object or item referred to could not be found.

- The object or item referred to could not be found."

I know that the problem is not the upload process itself or whether ovf is written to the destination as I can see the transfer progress bar gradually increasing and data being written to the disk succesfully.

My suspicion is that the problem has something to do with the networking as the corresponding error message in vCenter has the same wording as in vCD but  'Error stack' section points to dVS (see attached screenshot):

"Deploy OVF template: The object or item referred to could not be found

Error stack: DVS ce f2 34 50 1a 5e 51 df-b3 80 85 b8 4c f2 9c 02 cannot be found"

There are networks (with dvs and network pools presence in vCD) in Org vDC to which I attempt to upload ovf but I'm not sure why it is refering to that DVS. I'm not sure whether or not existing DVS has matching guid (I'm guessing the numbering in the error is either DVS's guid or port group guid)

I'm also attaching portion of vcloud-container-debug.log representing time period from start to finish of failed upload task.

Any help will be much appreciated.

PD

[peterdabr]

So, I haven't gotten a single response to my question...

I decided to pick up where I left off last time and dig in a little bit deeper and these are my findings that may shed some light and trigger somebody to reply.

In this new vCenter/vCD environment I'm working on for one of my customers, there are two datacenter objects in vCenter, each with a single esx cluster representing different client. Both clients will access their environments through the same vCD portal. The relevent configuration looks like this:

client1:

vcenter1 -> 'datacenter1' object -> esxcluster1-> dvSwitch1 (dvs uuid: ab....a1)

client2:

vcenter1 -> 'datacenter2' object -> esxcluster2-> dvSwitch2 (dvs uuid: bc..b7)

For some reason, when I try to upload any vApp to the Catalog on client1, it fails with a generic error in vCD, but when I look at corresponding error message in vcenter, it refers to dvSwitch2 with the following message:

"DVS  bc..b7 cannot be found"

(see screenshot attached)

Why would it refer to a distributed switch that's not even part of the cluster I'm trying to upload ovf to? I'm guessing it errors out with "cannot be found" because it is not in use on client1's datacenter/cluster...but it doesn't make sense.

Can't I have private cloud clients (each represented by a single ded esx cluster -> pvDC/ovDC) in multitenant vCenter separated by Datacenter objects if I wanted to?

Any help with above questions and leads to solving the problem would be greately appreciated.

Peter

[peterdabr]

This is actually a bug introduced in 1.5.1.

Temporary workaround (not elegant) is to add all hosts to all dvswitches (but not attaching any physical nics to them). Per VMware support this is fixed in 5.1

Peter