Hey Folks,
After performing an upgrade of vCloud Director to 1.5, and subsequently vShield Manger to 5.0, we've been unable to successfully "upgrade" any of the existing vShield Edge devices we have.
What's odd is that whenever we issue the command (via vCD UI) to "reset" a network (or make new ones), the appropriate vShield Edge device looks to be created and initialized properly, as I can ping from a VM behind that Edge device out to the outside world.
But after 3 minutes (180 seconds) vCloud apparently tells vCenter to delete the edge device, and the following error shows up in vCloud Director:
*Scrubbed for confidentiality*
It seems like this is just some sort of timeout, but I'm unsure why vCD keeps thinking the Edge appliance _isn't_ ready for initialization.
Could this be a licensing issue? (As in, do we need to re-apply a license key for the newer version of vShield Manager?)
I'll also be submitting this to VMware support as well, but I just wanted to see if anyone ran into this same issue.
Not a licensing problem. File the SR and attach the cell and VSM logs so we can take a look
Did you install a vShield Edge 5.0 License? It is not the same as the old license, so to answer your question you DO need to install a new license key for vShield 5.0.
We didn't have the upgraded Edge license at the time of the vCD 1.5 upgrade or the vSM 5 upgrade, but we did import the upgraded version of our old key in afterwards. That didn't seem to improve the situation.
I do have an SR in (and they've been super helpful so far), but in case someone...ANYONE has seen this same error, in the vcloud-container-debug.log (from the vCD Cell), I see the following (Important part in bold):
* Items in square brackets were changed due to confidential information *
I do have a valid CA backed SSL cert on VSM that I added after the upgrade of VSM, but that didn't seem to help.
I also think it is somewhat odd that vCloud is attempting to hit VSM to check for the Edge device being ready via VSM's IP address, not hostname. Maybe that's by design I guess.
I just hit this same issue working with someone and we isolated the issue to a firewall between the vCD Cell and the vShield Manager. They had configured a packet inspection rule which reaps idle connections. We saw while tailing the log that VSM tried to send a packet (443/TCP) to VCD which was dropped. Once the rule was bypassed the Edge deployments completed successfully in VCD. I hope that helps.
Guys, I found out what the issue was here. This is actually a bug in the upgrade process in 1.0.x to 1.5. When the vCD database gets updated, sometimes the table that holds the vSE version number is not updated correctly, for instance 4.1 to 5.
Anyway, this bug has a very easy workaround. If you receive this error, simply log in to vCD as an administrator, go to the Manage & Monitor tab, highlight the vCenter server in question, right click, and choose Properties. From there, choose the vShield Manager tab and re-enter only the username that is specified. Once you clear out and re-enter only the username, click OK.
This instantly solved the issues I was having and was definitely confirmed as a bug with a 1.0.x to 1.5 upgrade.
Hope this helps!
Just an FYI - I ran in to the same issue this weekend while upgrading a large pod from 1.0.1 to 1.5. Same issue here, resetting username in vCD administrattion interface cleared it up.
I also created a quick blog entry about this, if anyone is interested:
I'm having this exact issue, but with the new 1.5 installation. Can anyone point me in the right direction? I have a call open with VMware support, but they're not helping matters much.
Just to update my situation for anyone who may come across it...
We have fixed the issue with the technique described in this thread. However the underlying cause of the problem wasn't the upgrade to vCD 1.5 as we did a new installation of 1.5.
Our issue came from the fact that we were running vShield 4.1 and upgraded it to v 5.0 after the installation of vCD 1.5.
It effectively results in the same situation, the vCD database sees the wrong version of vShield.
So, removing and re-adding the admin user in the PvDC fixed the problem.