CSIEnvironments
Enthusiast
Enthusiast

Bug in vCloud Director 5.1.1 and 5.1 with vCDNI NetPools

Hi,

We have after lots of trial an error, discovered a bug in vCloud Director 5.1 and 5.1.1. It relates to vCDNI Pools and connectivity. We re-created and experienced the bug both times in a vCD manual installer environment & using the Appliances (VC/VCD/vShield) and the issue persists so it is not a configuration issue on our side.

From a fresh install we created various NAT’d vApps using isolated-backed networks, all vApps deploy correctly and connectivity is fine. If we stop all the vApps and delete ALL the NetPools, create a new NetPool whether it’s with the original VLAN tagging or a new VLAN tag, any vApp we start after that has connectivity issues and we are unable to connect to the vApps. To resolve the problem all hosts have to be re-prepared within vCD, and the connectivity is restored, without doing anything else to the currently running vApps.

While everything is working we are able to delete and create as many NetPools as we like as long as that at any point we always have at least 1 NetPool configured. The bug only occurs if all NetPools are deletes. Connectivity between VM’s and the Default Gateway is broken, even if the affected VM’s are on the same physical host, attached to the same portgroup on the same vDS.

Setup:

3 ESXi Hosts in Clusters.

2 in Maintenance mode to isolate that it's not a cross host issue.

To reproduce:

Deploy a vApps – isolated-backed network and NAT

Confirm connectivity

Stop the vApp

Using the Provider vDC set "Networks & Pool" to none.

Delete all the Pools except vDC-VXLAN-NP as it can’t be deleted.

Create a new NetPool with any VLAN tag

Start the vApp and at this point connectivity will be broken

Re-Prepare the hosts from vCD. As soon as the vApps VM’s are migrated to a new prepared host connectivity to those VM’s is restored.

Attached is the pic of our setup. Logged a support request with VMWare too...

If you have a spare lab please attempt to reproduce and confirm.

Regards,

Dean

0 Kudos
14 Replies
_morpheus_
Expert
Expert

You're right. Please file an SR and give me the SR number so we can prioritize the fix and get it into a patch release.

CSIEnvironments
Enthusiast
Enthusiast

SR has already been opened: 12238384810

0 Kudos
JohannStander
Enthusiast
Enthusiast

life saver!

i got exactly the same problem, and so glad i found this post...thank you.

was about to pull my hair out to figure out what was going on..

I am on vcloud director 5.1.1.868405  with latest vshield manager Release 5.1.1-848085 and got the exact same problem.

Just wasted 2 full days of my life!

hope we get this fixed asap.

Cheers

0 Kudos
CSIEnvironments
Enthusiast
Enthusiast

So looks like I've come across a related issue. We had a power failure in Decemeber and a few days ago I was deploying vApps in vCloud Director and every virtual machine's network adapter would not connect. It showed as unplugged for every machine in the Windows OS and if you edit settings of the VM's in the Virtual Ceneter you will see the network adapter is disconnected. If you check the box, click ok it does not connect the adapter. There were no other vApps deployed and I had many switch ports avaliable. I stopped and started the vApps added to the cloud from Catelog  multiple times, I build new vApps and imported fresh VM's but the issue persisted.

Being suspicious about the previous issue, I stopped all vApps, reprepared the host, started all the vApps and networking was fine. I changed absolutely nothing except repreparing the host. This issue seems very similar to the 1st bug I experienced.

How to reproduce this I have no clue, but it's strange network issues are resolved with repreparing the host.

I have logged a new SR for this issue. The case number is 13270982401.

Do you have any feedback on the orignial case and input on this issue _morpheus_?

0 Kudos
_morpheus_
Expert
Expert

We've decided not to fix this issue in the patch release. VCDNI is going away in the next major release, and my advice for anyone using VCD 5.1.x is to transition to VXLAN.

Do you have any feedback on the orignial case and input on this issue _morpheus_?

CSIEnvironments
Enthusiast
Enthusiast

Unfortunately we are not able to simply transition to VXLAN, we are in the final stages of our Migration process from LabManager and are about to take this live in the next few months and have done all our development around VCDNi. We may need to hold off on the migration until the next major release is out.

Would you be able to give me an ETA on when the next major release is planned for vCloud Director?

Also when will the closed beta or private testing start? I ask because we normally get access to alpha's/beta's so would be nice to get it asap.

Thanks!

0 Kudos
_morpheus_
Expert
Expert

I can't give out any information on upcoming release dates. You'll have to follow up with your account team for that.

As for beta, you'll need to get your account team to nominate you for the beta. You can send me a private message if you're having any issue with getting access to the beta

0 Kudos
NexusNetworks
Enthusiast
Enthusiast

Reading this has me a little on edge. I have vCD 5.1.1 in production with a network pool supporting many clients. From what I gather this only would happen if I delete the network pool, which I can't fathom doing at the moment. I am planning a migration to shut down the infrastructure and move it from a cabinet to a cage in the datacenter soon; would that affect it and cause this connectivity issue to show it's face?

0 Kudos
BhaskarSA
Contributor
Contributor

Hi,

Deleting the network pools is not a common operation in vCloud Director.

Please see the following KB article which describes this issue and provides a workaround.

http://kb.vmware.com/kb/2043526

-Bhaskar

0 Kudos
NexusNetworks
Enthusiast
Enthusiast

So if going with VXLANs is now the way to go. Is there an easy way to migrate a client from vCDNI to VXLAN with no downtime or reconfiguring org networks? I am going to create a new dvSwitch and create a VXLAN on it and migrate away from vCDNI.

0 Kudos
_morpheus_
Expert
Expert

There isn't right now but we're working on something to do this.

Billy Lucas wrote:

So if going with VXLANs is now the way to go. Is there an easy way to migrate a client from vCDNI to VXLAN with no downtime or reconfiguring org networks? I am going to create a new dvSwitch and create a VXLAN on it and migrate away from vCDNI.

0 Kudos
NexusNetworks
Enthusiast
Enthusiast

any advice for migrating from vCDNI to VXLAN? anything is appreciated. I can only imagine what cool stuff the next version of vCD holds.

0 Kudos
_morpheus_
Expert
Expert

We are working on a hot-migration for VCDNI to VXLAN. It will be released at some point before VCDNI goes away

0 Kudos
JayhawkEric
Expert
Expert

I just had the same issue when upgrading from 5.1.1 to 5.1.2.  Have to make sure and unprepare/prepare all hosts.

VCP5-DV twitter - @ericblee6 blog - http://vEric.me
0 Kudos