vShield Manager, VXLAN virtual wires, VDS uplink changes

[MillardJK]

Need some input on how to fix this situation without rebuilding the whole damn cluster...

I have vSM providing VXLAN virtual wires/networks for vCloud Director. As a part of some host updates--migrating from all 1Gbps to 1/10Gbps mixed--I was able to rearrange the NICs on the hosts and add additional uplinks to the total available on the DVS switch that VXLAN is "riding on." In the course of making these updates, I also renamed the uplinks, which resulted in errors and discovery of the root problem.

The errors were popping up when trying to instantiate a new VXLAN network. The vCloud error was pretty inscrutable (as usual), but trying to manually create a new network in vSM provided useful information: the error was a failure to set the teaming mode for the new port group, and it referenced one of the former uplink names.

After I added additional "dummy" uplinks to the DVS and renamed them so that the old names were included, the virtual wires could be built just fine. However, in reviewing the new portgroups, it was clear that vShield was building them using the original, pre-reconfiguration uplink names, ignoring all the new uplinks: the new uplinks were set as "unused" in the teaming properties, and the "active" were the dummy uplinks that had no physical adapters associated with them!

I've restarted vSM, re-entered the vCenter credentials to try and get it to re-sync with the network configuration, but to no avail.

I need some way to force vSM to re-enumerate that DVS that VXLAN is using so that it'll end up with the correct uplinks. To date, my Google-fu has failed me, so I'm hoping someone on the forums might have a clue. Heck, for all I know, this is a defect that I've just uncovered...

[IamTHEvilONE]

This might not answer your question, but at least explain what's going on from vCloud Director's perspective.

1. in vCD a new isolated network is requested
2. the network pool for the org vDC is VXLAN
3. vCD sends the new vWire request to vShield/NSX to provision
4. vCD waits to see that a new port group is created with the expected friendly name value
5. Once vCD sees the new port group, record the unique reference in the vCD DB
6. If an Edge is required, deploy an Edge to protect the network and assign an uplink.

The uplinks (quantity and order) for the purposes of VXLAN should be rather abstracted from vCD's view.  If this were vCNI, vCD would definitely care about created the Port Group directly.

What I wonder, is if vShield Manager records the DVS and uplink data from the host when you prepare the host/cluster for VXLAN (which is when you pick a DvSwitch and set MTU/etc).

[SRoland · Accepted solution]

Assuming you are using "failover order" configured for your VXLAN, you can try to use REST API call to change the uplink names in vCNS manager (almost similar to what you can see in KB 2093324).

Note :  I recommend to backup or snapshot the Manager first! Just in case ....

Headers required as:

Accept: application/xml

Content-Type : application/xml

Basic Auth

1./

Use GET on the following to the prepared VDS':
http://<manager>/api/2.0/vdn/switches

( obviously replace the <manager> part with the name or IP of your vShield/vCNS manager )

You will get something like this (this is just a part of it if you have more than one vDS) :

<vdsContext>

<switch>

<objectId>dvs-18</objectId>

<type>

<typeName>VmwareDistributedVirtualSwitch</typeName>

</type>

<name>DSwitch</name>

<revision>16</revision>

<objectTypeName>VmwareDistributedVirtualSwitch</objectTypeName>

<scope>

<id>datacenter-2</id>

<objectTypeName>Datacenter</objectTypeName>

<name>cloud</name>

</scope>

<extendedAttributes/>

</switch>

<mtu>1600</mtu>

<teaming>FAILOVER_ORDER</teaming>

<uplinkPortName>Uplink 2</uplinkPortName>

<uplinkPortName>Uplink 1</uplinkPortName>

<promiscuousMode>false</promiscuousMode>

</vdsContext>

2./

Modify the <uplinkPortName> parts as you need. For example:

...

<uplinkPortName>NewLink1</uplinkPortName>

<uplinkPortName>NewLink2</uplinkPortName>

...
Leave the rest as it is.

3./

Then use execute a "PUT" for the URL below in the REST client containing the above ( in step 2./ ) modified body (and again: assuming you are using failover order).
http://<manager>/api/2.0/vdn/switches/dvs-18

Note: Replace the "dvs-18" with the id in between <objectId> and </objectId> to what you got in the GET query ( again in bold and red ),

You should get a HTTP 200 code if all is OK. See : vShield API Guide around page 154, but personally I think the "Edit Teaming Policy" part is not correct.

This will not change any existing port-group setting in vCenter. You will need to edit them manually. This change is only for any further VXLAN v-wire creation.

HTH

Roland

P.s: I did my best to test and try the above example, but no guarantee and no support provided. For support please open a service request with VMware.

[MillardJK]

Bingo. That worked. Time for a blog post documenting it all for posterity...