vShield Edge 1.5.0 appliances failing to upgrade to 5.1

[tekemclain]

Just upgraded my lab to vSphere Suite 5.1 and I'm having some major pains upgrading the vShield Edge firewalls that were deployed before the upgrade.  To add to it, the appliances do not appear in the vShield Manager interface but do appear in the vCloud Director interface.  vShield manager was upgraded using an upgrade bundle.

The error I receive when trying to re-deploy, modify, or delete an existing appliance from vCloud Director is below.  Any body have any suggestions on how to fix this?  What I have tried so far is to reconnect vShield Manager to vCloud Director AND vSphere.  All integration pieces are working as expected otherwise and I can deploy new organizational networks and vSphere Edge firewalls without any problems.

Thanks,

Ed

Cannot redeploy edge gateway Admin_Internal_Org_net (urn:uuid:cb7c96bc-df12-4b2c-8679-dcbf33d11e5c)

org.springframework.web.client.RestClientException: Redeploy failed: Edge connected to 'dvportgroup-66' failed to upgrade.

The error on the vcd cell is:

2012-09-14 10:41:17,114 | WARN     | nf-activity-pool-3        | LoggingRestTemplate            | POST request for "https://172.25.0.84:443/api/2.0/networks/dvportgroup-66/edge/upgrade" resulted in 404 (Not Found); invoking error handler |

2012-09-14 10:41:17,117 | ERROR    | nf-activity-pool-3        | NetworkSecurityErrorHandler    | Response error xml : <?xml version="1.0" encoding="UTF-8" standalone="yes"?><Errors><Error><code>70001</code><description>vShield Edge not installed for given networkID. Cannot proceed with the operation</description></Error></Errors> |

2012-09-14 10:41:17,298 | ERROR    | nf-activity-pool-3        | DeployGatewayActivity          | [Activity Execution] Handle: urn:uuid:cb7c96bc-df12-4b2c-8679-dcbf33d11e5c, Current Phase: com.vmware.vcloud.fabric.net.activities.gateway.DeployGatewayActivity$GenerateBacking, ActivityExecutionState Parameter Names: [BACKING_SPEC, NDC, activitySupervisionRequest, com.vmware.activityEntityRecord.EntityId, REDEPLOY, DEPLOY_PARAMS] - Could not deploy gateway Admin_Internal_Org_net |

org.springframework.web.client.RestClientException: Redeploy failed: Edge connected to 'dvportgroup-66' failed to upgrade.

        at com.vmware.vcloud.fabric.nsm.services.spock.EdgeManagerSpock.redeployEdge(EdgeManagerSpock.java:734)

        at com.vmware.vcloud.fabric.net.activities.gateway.DeployGatewayActivity$GenerateBacking.invoke(DeployGatewayActivity.java:347)

        at com.vmware.vcloud.fabric.foundation.activity.executors.ActivityRunner.run(ActivityRunner.java:123)

        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)

        at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)

        at java.util.concurrent.FutureTask.run(Unknown Source)

        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

        at java.lang.Thread.run(Unknown Source)

2012-09-14 10:41:17,349 | ERROR    | LocalTaskScheduler-Pool-3 | GatewayManagerImpl             | Could not redeploy EdgeGateway Admin_Internal_Org_net (urn:uuid:cb7c96bc-df12-4b2c-8679-dcbf33d11e5c) | vcd=a216997e-506d-4e43-a2cf-92b4659b1e1a,task=feb14e0f-bf93-4026-abc2-ed8bdbf398dd

java.util.concurrent.ExecutionException: org.springframework.web.client.RestClientException: Redeploy failed: Edge connected to 'dvportgroup-66' failed to upgrade.

        at com.vmware.vcloud.fabric.foundation.activity.toolkit.AbstractActivity$BasePhase.finish(AbstractActivity.java:202)

        at com.vmware.vcloud.fabric.net.activities.gateway.DeployGatewayActivity$GenerateBacking.invoke(DeployGatewayActivity.java:360)

        at com.vmware.vcloud.fabric.foundation.activity.executors.ActivityRunner.run(ActivityRunner.java:123)

        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)

        at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)

        at java.util.concurrent.FutureTask.run(Unknown Source)

        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

        at java.lang.Thread.run(Unknown Source)

Caused by: org.springframework.web.client.RestClientException: Redeploy failed: Edge connected to 'dvportgroup-66' failed to upgrade.

        at com.vmware.vcloud.fabric.nsm.services.spock.EdgeManagerSpock.redeployEdge(EdgeManagerSpock.java:734)

        at com.vmware.vcloud.fabric.net.activities.gateway.DeployGatewayActivity$GenerateBacking.invoke(DeployGatewayActivity.java:347)

        ... 7 more

2012-09-14 10:41:17,363 | ERROR    | LocalTaskScheduler-Pool-3 | TaskServiceImpl                | Unable to execute task NETWORK_GATEWAY_REDEPLOY(com.vmware.vcloud.entity.task:feb14e0f-bf93-4026-abc2-ed8bdbf398dd) | vcd=a216997e-506d-4e43-a2cf-92b4659b1e1a,task=feb14e0f-bf93-4026-abc2-ed8bdbf398dd

com.vmware.vcloud.dl.net.exception.NetworkInternalServerException: Cannot redeploy edge gateway Admin_Internal_Org_net (urn:uuid:cb7c96bc-df12-4b2c-8679-dcbf33d11e5c)

[sorabhk5]

Please collect the tech support logs from vShield Manager.

Restart the vShield Manager and try upgrading the edge again if it still fails - please upload the VSM & VCD Logs.

(or Log a ticket with Support)

[brsockel]

Was there any resolution to this. I am having a similar problem.  This is a clean install of Vcloud Director/Vshields 5.1.

Thanks

[tekemclain]

I have a case open with vmware on it and they are still troubleshooting it out.  There appears to be a bug in the upgrade of the database but until mine is fixed and a clear answer is given that is all I know.

[ldelloca]

We have an open case too.

The upgrade process of the manager went well, but we were not able to upgarde the existing EDGEs.

To our understanding, the existing EDGEs where kinda removed from the database, so the upgrade process is stuck since it cannot even remove them because they are not officially existing in the database. We also tried to delete them directly from the underlying vCenter but hade no luck.

No problem instead on deploying new appliances...

[JasonBurrell]

I am also experiencing the same thing.  Sounds like people should not upgrade vSM to 5.1 until this can be resolved.

[tekemclain]

Thanks for reminding me of this thread Jason.

Our issue has since been fixed.  The end fix was to open a case with VMWare and send them an ovf of our vshield appliance.  They then took it, fixed it, and sent us back a new one.  Once that one was imported back into VC and re-configured (ip's were still configured for vmwares lab), we were able to upgrade all of our old appliances.  According to the message I got back from vmware support they are told me to hold off on upgrading any of our production systems until they could get a maintenance build out to address the issue.

Hope that helps.

Ed.

[vogtmatt]

For anyone that comes in after, this is how I solved it without sending my OVF to VMware

I rolled back to the snapshot I took of vShield Manager (was on 5.0.1) after vCloud Director upgrade, before vShield upgrade. I then went through the steps again in this VMware KB: Upgrading to vCloud Networking and Security 5.1.2a best practices with a couple differences. Even though I had enough space, I ran the VMware-vShield-Manager-upgrade-bundle-maintenance-5.0-939118.tar.gz bundle anyway. After that finished, I ran the VMware-vShield-Manager-upgrade-bundle-5.1.2-943471.tar.gz bundle upgrade.

Before I did the backup, restore, maintenance bundle upgrade, I went through and did an upgrade of each edge gateway (under the Edges dropdown). Then, I installed the VMware-vShield-Manager-upgrade-bundle-maintenance-5.1.2-997359.tar.gz bundle. After that was all booted back up and stable, I stopped vCloud Director, took a backup of the vSM, deployed the new vSM OVF, installed the VMware-vShield-Manager-upgrade-bundle-maintenance-5.1.2-997359.tar.gz bundle to that, restored the backup, re-registered vSM with vCenter, started vCD, re-registered vCD with vSM. And now we're fully functional again.