8 Replies Latest reply on Dec 8, 2011 3:42 PM by gaadmin

    Unable to create or reset networks in vCD 1.5 after upgrade from 1.0

    b0rbb Enthusiast

      Hey Folks,

       

      After performing an upgrade of vCloud Director to 1.5, and subsequently vShield Manger to 5.0, we've been unable to successfully "upgrade" any of the existing vShield Edge devices we have.

       

      What's odd is that whenever we issue the command (via vCD UI) to "reset" a network (or make new ones), the appropriate vShield Edge device looks to be created and initialized properly, as I can ping from a VM behind that Edge device out to the outside world.

       

      But after 3 minutes (180 seconds) vCloud apparently tells vCenter to delete the edge device, and the following error shows up in vCloud Director:

       

      *Scrubbed for confidentiality*

       

      Cannot create vShield Edge Device for network: <Org-Network-Name>[Unique ID Number].
      - edge error: Creating/configuring the VR failed: vShield Edge Device on network: <Org-Network-Name>[Unique ID Number] is not ready for initialization after 180 seconds.

       

      It seems like this is just some sort of timeout, but I'm unsure why vCD keeps thinking the Edge appliance _isn't_ ready for initialization.

       

      Could this be a licensing issue?  (As in, do we need to re-apply a license key for the newer version of vShield Manager?)

       

      I'll also be submitting this to VMware support as well, but I just wanted to see if anyone ran into this same issue.

        • 1. Re: Unable to create or reset networks in vCD 1.5 after upgrade from 1.0
          _morpheus_ Expert
          VMware Employees

          Not a licensing problem. File the SR and attach the cell and VSM logs so we can take a look

          • 2. Re: Unable to create or reset networks in vCD 1.5 after upgrade from 1.0
            Hot Shot

            Did you install a vShield Edge 5.0 License?  It is not the same as the old license, so to answer your question you DO need to install a new license key for vShield 5.0.

            • 3. Re: Unable to create or reset networks in vCD 1.5 after upgrade from 1.0
              b0rbb Enthusiast

              We didn't have the upgraded Edge license at the time of the vCD 1.5 upgrade or the vSM 5 upgrade, but we did import the upgraded version of our old key in afterwards.  That didn't seem to improve the situation.

               

              I do have an SR in (and they've been super helpful so far), but in case someone...ANYONE has seen this same error, in the vcloud-container-debug.log (from the vCD Cell), I see the following (Important part in bold):

               

              * Items in square brackets were changed due to confidential information *

               

              | Added [NETWORK (id = c38b3242-af22-4e29-b01b-4b2f6c8b8f28)][[vcId=af805138-5382-4c09-aff1-59869fcdf4aa, moref=dvportgroup-1857]] validator to validation queue |
              2011-09-28 15:39:18,039 | DEBUG    | VSMCLIENT HeartbeatTimer  | RestFulCallManager             | VSMCLIENT-2.0.0 trustAllHttpsCertificates; Message : User has NOT set the SSL context, using the default. |
              2011-09-28 15:39:18,095 | DEBUG    | VSMCLIENT HeartbeatTimer  | RestFulCallManager             | VSMCLIENT-2.0.0 verify; Message : Returning from hostname verification. |
              2011-09-28 15:39:18,133 | INFO     | VSMCLIENT HeartbeatTimer  | RestFulCallManager             | VSMCLIENT-2.0.0 checkVSMReachability; Response :HTTP/1.1 200 OK (Operation Succeeded) |
              2011-09-28 15:39:18,134 | DEBUG    | VSMCLIENT HeartbeatTimer  | RestFulCallManager             | VSMCLIENT-2.0.0 trustAllHttpsCertificates; Message : User has NOT set the SSL context, using the default. |
              2011-09-28 15:39:18,177 | DEBUG    | pool-validationservice-14-thread-1889 | ObjectConditionCollection      | Setting condition (hostdisconnectedfromdvswitch) on object: (MANAGED_SERVER-7bd8af38-5241-4ff5-b786-2689367eab67) |
              2011-09-28 15:39:18,189 | DEBUG    | VSMCLIENT HeartbeatTimer  | RestFulCallManager             | VSMCLIENT-2.0.0 verify; Message : Returning from hostname verification. |
              2011-09-28 15:39:18,202 | INFO     | VSMCLIENT HeartbeatTimer  | RestFulCallManager             | VSMCLIENT-2.0.0 checkVSMReachability; Response :HTTP/1.1 200 OK (Operation Succeeded) |
              2011-09-28 15:39:19,008 | DEBUG    | Quartz-pool-1-thread-2239 | ShieldNetworkApplianceFactory  | Waiting for vShield Edge Device(vm-1859) to be ready on network: c38b3242-af22-4e29-b01b-4b2f6c8b8f28/dvportgroup-1857...  | vcd=424399200f4c433db88c876dc8a0d185,task=d1a24062c8f44cd0afd4247d33b6f4f1
              2011-09-28 15:39:19,008 | INFO     | Quartz-pool-1-thread-2239 | FirewallManager                | VSMCLIENT-2.0.0 isVSEUp; URI :https://[IP_ADDRESS_OF_VSM]:443/api/1.0/network/isVSEReady/vm-1859 | vcd=424399200f4c433db88c876dc8a0d185,task=d1a24062c8f44cd0afd4247d33b6f4f1
              2011-09-28 15:39:19,008 | DEBUG    | Quartz-pool-1-thread-2239 | RestFulCallManager             | VSMCLIENT-2.0.0 trustAllHttpsCertificates; Message : User has NOT set the SSL context, using the default. | vcd=424399200f4c433db88c876dc8a0d185,task=d1a24062c8f44cd0afd4247d33b6f4f1
              2011-09-28 15:39:19,063 | DEBUG    | Quartz-pool-1-thread-2239 | RestFulCallManager             | VSMCLIENT-2.0.0 verify; Message : Returning from hostname verification. | vcd=424399200f4c433db88c876dc8a0d185,task=d1a24062c8f44cd0afd4247d33b6f4f1
              2011-09-28 15:39:19,104 | DEBUG    | Quartz-pool-1-thread-2239 | RestFulCallManager             | VSMCLIENT-2.0.0 isVSEUp; Response : HTTP/1.1 404 Not Found<?xml version="1.0" encoding="UTF-8" standalone="yes"?><Errors><Error><code>70001</code><description>vShield Edge not installed for given networkID. Cannot proceed with the operation</description></Error></Errors> | vcd=424399200f4c433db88c876dc8a0d185,task=d1a24062c8f44cd0afd4247d33b6f4f1
              2011-09-28 15:39:19,105 | DEBUG    | Quartz-pool-1-thread-2239 | ShieldNetworkApplianceFactory  | vse:vm-1859, net:c38b3242-af22-4e29-b01b-4b2f6c8b8f28/dvportgroup-1857 is not up yet. | vcd=424399200f4c433db88c876dc8a0d185,task=d1a24062c8f44cd0afd4247d33b6f4f1

               

              I do have a valid CA backed SSL cert on VSM that I added after the upgrade of VSM, but that didn't seem to help.

               

              I also think it is somewhat odd that vCloud is attempting to hit VSM to check for the Edge device being ready via VSM's IP address, not hostname. Maybe that's by design I guess.

              • 4. Re: Unable to create or reset networks in vCD 1.5 after upgrade from 1.0
                d_olson Novice
                VMware Employees

                I just hit this same issue working with someone and we isolated the issue to a firewall between the vCD Cell and the vShield Manager. They had configured a packet inspection rule which reaps idle connections. We saw while tailing the log that VSM tried to send a packet (443/TCP) to VCD which was dropped. Once the rule was bypassed the Edge deployments completed successfully in VCD. I hope that helps.

                • 5. Re: Unable to create or reset networks in vCD 1.5 after upgrade from 1.0
                  jgiland Enthusiast
                  vExpert

                  Guys, I found out what the issue was here.  This is actually a bug in the upgrade process in 1.0.x to 1.5.  When the vCD database gets updated, sometimes the table that holds the vSE version number is not updated correctly, for instance 4.1 to 5.

                   

                  Anyway, this bug has a very easy workaround.  If you receive this error, simply log in to vCD as an administrator, go to the Manage & Monitor tab, highlight the vCenter server in question, right click, and choose Properties.  From there, choose the vShield Manager tab and re-enter only the username that is specified.  Once you clear out and re-enter only the username, click OK.

                   

                  This instantly solved the issues I was having and was definitely confirmed as a bug with a 1.0.x to 1.5 upgrade.

                   

                  Hope this helps!

                  • 6. Re: Unable to create or reset networks in vCD 1.5 after upgrade from 1.0
                    jgiland Enthusiast
                    vExpert

                    Just an FYI - I ran in to the same issue this weekend while upgrading a large pod from 1.0.1 to 1.5.  Same issue here, resetting username in vCD administrattion interface cleared it up.

                     

                    I also created a quick blog entry about this, if anyone is interested:

                     

                    http://virtual-grind.com/2011/10/03/unable-to-create-or-restart-networks-after-vcloud-director-upgrade-to-1-5/

                    • 7. Re: Unable to create or reset networks in vCD 1.5 after upgrade from 1.0
                      gaadmin Enthusiast

                      I'm having this exact issue, but with the new 1.5 installation. Can anyone point me in the right direction? I have a call open with VMware support, but they're not helping matters much.

                      • 8. Re: Unable to create or reset networks in vCD 1.5 after upgrade from 1.0
                        gaadmin Enthusiast

                        Just to update my situation for anyone who may come across it...

                         

                        We have fixed the issue with the technique described in this thread. However the underlying cause of the problem wasn't the upgrade to vCD 1.5 as we did a new installation of 1.5.

                         

                        Our issue came from the fact that we were running vShield 4.1 and upgraded it to v 5.0 after the installation of vCD 1.5.

                         

                        It effectively results in the same situation, the vCD database sees the wrong version of vShield.

                         

                        So, removing and re-adding the admin user in the PvDC fixed the problem.