Re: Auto Deploy and VCD Network Isolation - Not Wo...

Jackal830 · ‎05-20-2015

Hello,

I have opened a support case about this issue, but the case is moving a lot slower than I would like to see. I thought I'd post here to see if anyone has ran into anything similar.

We recently switched our 24-host vCloud cluster (more info about this cluster below) over to Auto-Deploy (was using boot from USB). After the switch, we had a few complaints from customers that use isolated networks. It turns out that only 4 of our 24 hosts were working properly with isolated networks. All 24-hosts are using the same auto-deploy image.

When the hosts boot up, the vCloud Director web interface shows the following with green check marks: "Status", "Enabled", "Ready", "Available", and "VCD Network Isolation Capable".

To get my other 20 hosts working, I disabled all 20 hosts in the vCloud interface, unprepared them, and prepared them. After this, network isolation worked for those 20 hosts. When I reboot one of the hosts, the host still looks like it's good in the interface, but network isolation doesn't work until I disable, unprepare, and prepare.

Here is some more information about our environment:

vCenter build 2001466

ESXi Host Build 2702864

VCD Build 5.5.2.2000523

vShield Build 5.5.3

Here is the Deploy Rule that the hosts are using:

Name : UCS-2.2.1-pcloud.5.5.aln

PatternList : {oemstring=$SPT:ESX_PublicCloud_autod.2.2.x}

ItemList : {ESXi-5.5.0-autod-2.2.1.vshield}

I have confirmed that this is indeed the profile being used by looking at the Summary tab for the hosts and seeing the listed Image Profile.

And here is what's in that item:

PowerCLI D:\ImageBuild> Get-EsxSoftwarePackage

Name Version Vendor Creation Date

---- ------- ------ -------------

misc-cnic-register 1.72.1.v50.1i-1vmw.550.0.0.... VMware 9/19/2013 6:0...

scsi-lpfc820 8.2.3.1-129vmw.550.0.0.1331820 VMware 9/19/2013 6:0...

scsi-megaraid-mbox 2.20.5.1-6vmw.550.0.0.1331820 VMware 9/19/2013 6:0...

esx-xlibs 5.5.0-0.0.1331820 VMware 9/19/2013 6:0...

lpfc 10.0.100.1-1vmw.550.0.0.133... VMware 9/19/2013 6:0...

mtip32xx-native 3.3.4-1vmw.550.1.15.1623387 VMware 2/22/2014 1:1...

net-nx-nic 5.0.621-1vmw.550.0.0.1331820 VMware 9/19/2013 6:0...

block-cciss 3.6.14-10vmw.550.0.0.1331820 VMware 9/19/2013 6:0...

scsi-fnic 1.6.0.5-1OEM.500.0.0.472560 cisco 9/30/2013 11:...

net-enic 2.1.2.42-1OEM.500.0.0.472560 Cisco 9/5/2013 8:30...

sata-sata-sil24 1.1-1vmw.550.0.0.1331820 VMware 9/19/2013 6:0...

rste 2.0.2.0088-4vmw.550.1.15.16... VMware 2/22/2014 1:1...

elxnet 10.0.100.0v-1vmw.550.0.0.13... VMware 9/19/2013 6:0...

scsi-aacraid 1.1.5.1-9vmw.550.0.0.1331820 VMware 9/19/2013 6:0...

ata-pata-cmd64x 0.2.5-3vmw.550.0.0.1331820 VMware 9/19/2013 6:0...

net-cnic 1.72.52.v55.1-1vmw.550.0.0.... VMware 9/19/2013 6:0...

scsi-adp94xx 1.0.8.12-6vmw.550.0.0.1331820 VMware 9/19/2013 6:0...

net-be2net 4.6.100.0v-1vmw.550.0.0.133... VMware 9/19/2013 6:0...

net-ixgbe 3.7.13.7.14iov-12vmw.550.2.... VMware 4/29/2015 6:4...

net-igb 5.0.5.1.1-1vmw.550.2.54.240... VMware 1/1/2015 8:00...

epsec-mux 5.1.0-01814505 VMware 5/13/2014 4:3...

esx-base 5.5.0-2.62.2702864 VMware 4/29/2015 6:4...

ata-pata-sil680 0.4.8-3vmw.550.0.0.1331820 VMware 9/19/2013 6:0...

ipmi-ipmi-msghandler 39.1-4vmw.550.0.0.1331820 VMware 9/19/2013 6:0...

ata-pata-hpt3x2n 0.3.4-3vmw.550.0.0.1331820 VMware 9/19/2013 6:0...

scsi-megaraid-sas 5.34-9vmw.550.2.33.2068190 VMware 8/23/2014 1:5...

scsi-mptsas 4.23.01.00-9vmw.550.0.0.133... VMware 9/19/2013 6:0...

net-bnx2 2.2.3d.v55.2-1vmw.550.0.0.1... VMware 9/19/2013 6:0...

ata-pata-via 0.3.3-2vmw.550.0.0.1331820 VMware 9/19/2013 6:0...

ima-qla4xxx 2.01.31-1vmw.550.0.0.1331820 VMware 9/19/2013 6:0...

ata-pata-amd 0.3.10-3vmw.550.0.0.1331820 VMware 9/19/2013 6:0...

ata-pata-serverworks 0.4.3-3vmw.550.0.0.1331820 VMware 9/19/2013 6:0...

sata-sata-promise 2.12-3vmw.550.0.0.1331820 VMware 9/19/2013 6:0...

sata-sata-nv 3.5-4vmw.550.0.0.1331820 VMware 9/19/2013 6:0...

ipmi-ipmi-devintf 39.1-4vmw.550.0.0.1331820 VMware 9/19/2013 6:0...

scsi-ips 7.12.05-4vmw.550.0.0.1331820 VMware 9/19/2013 6:0...

sata-sata-svw 2.3-3vmw.550.0.0.1331820 VMware 9/19/2013 6:0...

scsi-mptspi 4.23.01.00-9vmw.550.0.0.133... VMware 9/19/2013 6:0...

net-e1000e 1.1.2-4vmw.550.1.15.1623387 VMware 2/22/2014 1:1...

esx-xserver 5.5.0-0.0.1331820 VMware 9/19/2013 6:0...

net-tg3 3.123c.v55.5-1vmw.550.2.33.... VMware 8/23/2014 1:5...

net-forcedeth 0.61-2vmw.550.0.0.1331820 VMware 9/19/2013 6:0...

scsi-aic79xx 3.1-5vmw.550.0.0.1331820 VMware 9/19/2013 6:0...

sata-ata-piix 2.12-10vmw.550.2.33.2068190 VMware 8/23/2014 1:5...

scsi-bnx2i 2.72.11.v55.4-1vmw.550.0.0.... VMware 9/19/2013 6:0...

ohci-usb-ohci 1.0-3vmw.550.0.0.1331820 VMware 9/19/2013 6:0...

scsi-qla4xxx 5.01.03.2-6vmw.550.0.0.1331820 VMware 9/19/2013 6:0...

scsi-qla2xxx 902.k1.1-9vmw.550.0.0.1331820 VMware 9/19/2013 6:0...

net-mlx4-core 1.9.7.0-1vmw.550.0.0.1331820 VMware 9/19/2013 6:0...

ata-pata-atiixp 0.4.6-4vmw.550.0.0.1331820 VMware 9/19/2013 6:0...

misc-drivers 5.5.0-2.62.2702864 VMware 4/29/2015 6:4...

esx-dvfilter-generic-... 5.5.0-0.0.1331820 VMware 9/19/2013 6:0...

ata-pata-pdc2027x 1.0-3vmw.550.0.0.1331820 VMware 9/19/2013 6:0...

qlnativefc 1.0.12.0-1vmw.550.0.0.1331820 VMware 9/19/2013 6:0...

lsi-mr3 0.255.03.01-2vmw.550.1.16.1... VMware 4/15/2014 9:0...

vshield 5.5.3-2172759 VMware 9/30/2014 2:3...

net-vmxnet3 1.1.3.0-3vmw.550.2.39.2143827 VMware 9/18/2014 11:...

scsi-hpsa 5.5.0-44vmw.550.0.0.1331820 VMware 9/19/2013 6:0...

tools-light 5.5.0-2.62.2702864 VMware 4/29/2015 6:4...

scsi-mpt2sas 14.00.00.00-3vmw.550.1.15.1... VMware 2/22/2014 1:1...

scsi-bnx2fc 1.72.53.v55.1-1vmw.550.0.0.... VMware 9/19/2013 6:0...

lsi-msgpt3 00.255.03.03-1vmw.550.1.15.... VMware 2/22/2014 1:1...

net-e1000 8.0.3.1-3vmw.550.0.0.1331820 VMware 9/19/2013 6:0...

xhci-xhci 1.0-2vmw.550.2.39.2143827 VMware 9/18/2014 11:...

ipmi-ipmi-si-drv 39.1-4vmw.550.0.0.1331820 VMware 9/19/2013 6:0...

sata-ahci 3.0-21vmw.550.2.54.2403361 VMware 1/1/2015 8:00...

net-bnx2x 1.72.56.v55.2-1vmw.550.0.0.... VMware 9/19/2013 6:0...

scsi-megaraid2 2.00.4-9vmw.550.0.0.1331820 VMware 9/19/2013 6:0...

ehci-ehci-hcd 1.0-3vmw.550.0.0.1331820 VMware 9/19/2013 6:0...

esx-tboot 5.5.0-2.33.2068190 VMware 8/23/2014 1:5...

uhci-usb-uhci 1.0-3vmw.550.0.0.1331820 VMware 9/19/2013 6:0...

sata-sata-sil 2.3-4vmw.550.0.0.1331820 VMware 9/19/2013 6:0...

vcloud-agent 5.5.0-1280396 VMware 8/17/2013 4:0...

net-mlx4-en 1.9.7.0-1vmw.550.0.0.1331820 VMware 9/19/2013 6:0...

And here is the output of testing the deploy rule compliance:

PowerCLI D:\ImageBuild> Get-Cluster 0000000-ESXVCLOUDCL1.ALN | Get-VMHost |Test-

DeployRuleSetCompliance

VMHost ItemList

------ --------

esx142269.vm.seo.... {}

esx140622.vm.seo.... {}

esx139784.vm.seo.... {}

esx140617.vm.seo.... {}

esx138793.vm.seo.... {}

esx135523.vm.seo.... {}

esx138945.vm.seo.... {}

esx138794.vm.seo.... {}

esx139783.vm.seo.... {}

esx140309.vm.seo.... {}

esx140310.vm.seo.... {}

esx140311.vm.seo.... {}

esx140313.vm.seo.... {}

esx140339.vm.seo.... {}

esx140614.vm.seo.... {}

esx140615.vm.seo.... {}

esx140616.vm.seo.... {}

esx140618.vm.seo.... {}

esx140619.vm.seo.... {}

esx140621.vm.seo.... {}

esx141947.vm.seo.... {}

esx141945.vm.seo.... {}

esx142271.vm.seo.... {}

esx142270.vm.seo.... {}

Here is another vib list from a host:

~ # esxcli software vib list

Name Version Vendor Acceptance Level Install Date

----------------------------- ------------------------------------- ------ ---------------- ------------

net-enic 2.1.2.42-1OEM.500.0.0.472560 Cisco VMwareCertified -

ata-pata-amd 0.3.10-3vmw.550.0.0.1331820 VMware VMwareCertified -

ata-pata-atiixp 0.4.6-4vmw.550.0.0.1331820 VMware VMwareCertified -

ata-pata-cmd64x 0.2.5-3vmw.550.0.0.1331820 VMware VMwareCertified -

ata-pata-hpt3x2n 0.3.4-3vmw.550.0.0.1331820 VMware VMwareCertified -

ata-pata-pdc2027x 1.0-3vmw.550.0.0.1331820 VMware VMwareCertified -

ata-pata-serverworks 0.4.3-3vmw.550.0.0.1331820 VMware VMwareCertified -

ata-pata-sil680 0.4.8-3vmw.550.0.0.1331820 VMware VMwareCertified -

ata-pata-via 0.3.3-2vmw.550.0.0.1331820 VMware VMwareCertified -

block-cciss 3.6.14-10vmw.550.0.0.1331820 VMware VMwareCertified -

ehci-ehci-hcd 1.0-3vmw.550.0.0.1331820 VMware VMwareCertified -

elxnet 10.0.100.0v-1vmw.550.0.0.1331820 VMware VMwareCertified -

epsec-mux 5.1.0-01814505 VMware VMwareCertified -

esx-base 5.5.0-2.62.2702864 VMware VMwareCertified -

esx-dvfilter-generic-fastpath 5.5.0-0.0.1331820 VMware VMwareCertified -

esx-tboot 5.5.0-2.33.2068190 VMware VMwareCertified -

esx-xlibs 5.5.0-0.0.1331820 VMware VMwareCertified -

esx-xserver 5.5.0-0.0.1331820 VMware VMwareCertified -

ima-qla4xxx 2.01.31-1vmw.550.0.0.1331820 VMware VMwareCertified -

ipmi-ipmi-devintf 39.1-4vmw.550.0.0.1331820 VMware VMwareCertified -

ipmi-ipmi-msghandler 39.1-4vmw.550.0.0.1331820 VMware VMwareCertified -

ipmi-ipmi-si-drv 39.1-4vmw.550.0.0.1331820 VMware VMwareCertified -

lpfc 10.0.100.1-1vmw.550.0.0.1331820 VMware VMwareCertified -

lsi-mr3 0.255.03.01-2vmw.550.1.16.1746018 VMware VMwareCertified -

lsi-msgpt3 00.255.03.03-1vmw.550.1.15.1623387 VMware VMwareCertified -

misc-cnic-register 1.72.1.v50.1i-1vmw.550.0.0.1331820 VMware VMwareCertified -

misc-drivers 5.5.0-2.62.2702864 VMware VMwareCertified -

mtip32xx-native 3.3.4-1vmw.550.1.15.1623387 VMware VMwareCertified -

net-be2net 4.6.100.0v-1vmw.550.0.0.1331820 VMware VMwareCertified -

net-bnx2 2.2.3d.v55.2-1vmw.550.0.0.1331820 VMware VMwareCertified -

net-bnx2x 1.72.56.v55.2-1vmw.550.0.0.1331820 VMware VMwareCertified -

net-cnic 1.72.52.v55.1-1vmw.550.0.0.1331820 VMware VMwareCertified -

net-e1000 8.0.3.1-3vmw.550.0.0.1331820 VMware VMwareCertified -

net-e1000e 1.1.2-4vmw.550.1.15.1623387 VMware VMwareCertified -

net-forcedeth 0.61-2vmw.550.0.0.1331820 VMware VMwareCertified -

net-igb 5.0.5.1.1-1vmw.550.2.54.2403361 VMware VMwareCertified -

net-ixgbe 3.7.13.7.14iov-12vmw.550.2.62.2702864 VMware VMwareCertified -

net-mlx4-core 1.9.7.0-1vmw.550.0.0.1331820 VMware VMwareCertified -

net-mlx4-en 1.9.7.0-1vmw.550.0.0.1331820 VMware VMwareCertified -

net-nx-nic 5.0.621-1vmw.550.0.0.1331820 VMware VMwareCertified -

net-tg3 3.123c.v55.5-1vmw.550.2.33.2068190 VMware VMwareCertified -

net-vmxnet3 1.1.3.0-3vmw.550.2.39.2143827 VMware VMwareCertified -

ohci-usb-ohci 1.0-3vmw.550.0.0.1331820 VMware VMwareCertified -

qlnativefc 1.0.12.0-1vmw.550.0.0.1331820 VMware VMwareCertified -

rste 2.0.2.0088-4vmw.550.1.15.1623387 VMware VMwareCertified -

sata-ahci 3.0-21vmw.550.2.54.2403361 VMware VMwareCertified -

sata-ata-piix 2.12-10vmw.550.2.33.2068190 VMware VMwareCertified -

sata-sata-nv 3.5-4vmw.550.0.0.1331820 VMware VMwareCertified -

sata-sata-promise 2.12-3vmw.550.0.0.1331820 VMware VMwareCertified -

sata-sata-sil24 1.1-1vmw.550.0.0.1331820 VMware VMwareCertified -

sata-sata-sil 2.3-4vmw.550.0.0.1331820 VMware VMwareCertified -

sata-sata-svw 2.3-3vmw.550.0.0.1331820 VMware VMwareCertified -

scsi-aacraid 1.1.5.1-9vmw.550.0.0.1331820 VMware VMwareCertified -

scsi-adp94xx 1.0.8.12-6vmw.550.0.0.1331820 VMware VMwareCertified -

scsi-aic79xx 3.1-5vmw.550.0.0.1331820 VMware VMwareCertified -

scsi-bnx2fc 1.72.53.v55.1-1vmw.550.0.0.1331820 VMware VMwareCertified -

scsi-bnx2i 2.72.11.v55.4-1vmw.550.0.0.1331820 VMware VMwareCertified -

scsi-hpsa 5.5.0-44vmw.550.0.0.1331820 VMware VMwareCertified -

scsi-ips 7.12.05-4vmw.550.0.0.1331820 VMware VMwareCertified -

scsi-lpfc820 8.2.3.1-129vmw.550.0.0.1331820 VMware VMwareCertified -

scsi-megaraid-mbox 2.20.5.1-6vmw.550.0.0.1331820 VMware VMwareCertified -

scsi-megaraid-sas 5.34-9vmw.550.2.33.2068190 VMware VMwareCertified -

scsi-megaraid2 2.00.4-9vmw.550.0.0.1331820 VMware VMwareCertified -

scsi-mpt2sas 14.00.00.00-3vmw.550.1.15.1623387 VMware VMwareCertified -

scsi-mptsas 4.23.01.00-9vmw.550.0.0.1331820 VMware VMwareCertified -

scsi-mptspi 4.23.01.00-9vmw.550.0.0.1331820 VMware VMwareCertified -

scsi-qla2xxx 902.k1.1-9vmw.550.0.0.1331820 VMware VMwareCertified -

scsi-qla4xxx 5.01.03.2-6vmw.550.0.0.1331820 VMware VMwareCertified -

uhci-usb-uhci 1.0-3vmw.550.0.0.1331820 VMware VMwareCertified -

vcloud-agent 5.5.0-1280396 VMware VMwareCertified -

vmware-fdm 5.5.0-2001466 VMware VMwareCertified -

vshield 5.5.3-2172759 VMware VMwareCertified -

xhci-xhci 1.0-2vmw.550.2.39.2143827 VMware VMwareCertified -

tools-light 5.5.0-2.62.2702864 VMware VMwareCertified -

scsi-fnic 1.6.0.5-1OEM.500.0.0.472560 cisco VMwareCertified -

Any help is appreciated. Thanks.

Sreec · ‎05-20-2015

Hi,

When you say network isolation doesn't work,can you be bit more precise on that point?

1.Is the network pools not getting created/failed?

2.Virtual Machines do not have network connectivity even if they are placed in the same host?

3.Do we have any working network pools(VXLAN/VLAN/PG backed) in this set-up?

Cheers,
Sree | VCIX-5X| VCAP-5X| VExpert 7x|Cisco Certified Specialist
Please KUDO helpful posts and mark the thread as solved if answered

Jackal830 · ‎05-21-2015

To clarify, I'm talking about an Isolated Organization vDC Network.

For example, in Org VDC Networks, if I were to add one, I'd select "Create an isolated network within this virtual datacenter"

The network shows OK status, the hosts report "VCD Network Isolation Capable" as OK, but it won't work until I unprepare and re-prepare the hosts running VMs that need to be on that isolated network.

The odd thing is, the 4 hosts that do work straight from boot, continue to work after reboots. They are all using the same base image profile and the same auto-deploy image.

Does this answer your question?

Sreec · ‎05-21-2015

So you are using VCDNI network pool and whenever we are creating Isolated org-vdc network you have issue? Please correct me if i'm wrong?

What type of issue you have with Isolated org-vdc networks? VM's cannot reach other ? Like i asked in my first reply,does it work if VM's are running on same host?

How about a direct org-vdc/vapp/routed network backed by VCDNI network pool.Do they work?

Cheers,
Sree | VCIX-5X| VCAP-5X| VExpert 7x|Cisco Certified Specialist
Please KUDO helpful posts and mark the thread as solved if answered

Jackal830 · ‎05-21-2015

Even if the VMs are on the same host, they cannot communicate with each other over the private network (pings fail, etc) until I unprepare and prepare the host (and move the VMs back to it)

The issue happens to any VCDNI network (already created or newly created) No traffic works until the vcloud agent is uninstalled and reinstalled (which shouldn't be required, since the VIB is included in my autodeploy image). There are 4 hosts that do not require the unprepare and prepare, they just work upon startup.

Type "Direct" works just fine. A routed network connecting to an edge gateway does not work and requires all VMs to be on a host that has been unprepared and then prepared (or on one of the 4 hosts that work correctly upon boot-up).

Jackal830 · ‎05-21-2015

I have been using one host for testing (one that doesn't work upon startup). I've rebooted it several times today. One reboot, it actually worked right away, but the others it didn't. This is very strange.

Edit: Just to be sure that it did work at least once right after a reboot, I did several more reboots and got it to work another time (out of many).

Could it be something weird with autodeploy and enabling host spanning? Could it be some sort of weird timing issue?

Sreec · ‎05-21-2015

You said,even VM's on same host doesn't ping each other.Are we sure that they were on same port-group and host when you tested? Scenario will be same even if its a routed org network,VSE and respective VM'S might be residing on different host during your test.

Worth enough to re-check if Vlan-id and MTU is properly set in DVS/Physical switch and VCDNI network pool properties in VCD as well.Adding to that it can be issue with fence module ,try running /opt/vmware/vslad/fence-util moduleinfo which would give you a better output(To get per host vCDNI statistics use)

Cheers,
Sree | VCIX-5X| VCAP-5X| VExpert 7x|Cisco Certified Specialist
Please KUDO helpful posts and mark the thread as solved if answered

Jackal830 · ‎05-21-2015

Right now, on my test host (that is not working), I have two VMs, one named "Test 1" and the other "Test 2" They only have an isolated network on them (named "Test Isolated Network") that is in my VDC, and pings do not work between them. If I move them to any other host (that have been unprepared and prepared), pings work. I can move both to the same host, or different hosts.

Here is the fence info on this host, I'm not really sure how to interpret this:

~ # esxcli vcloud fence getfenceinfo

Module Parameters:

Host Key: 0x104b0

Configured LAN MTUs:

+------------------------------------------------------------------------------------------+

| LAN ID | 1 2 3 4 5 6 7 8 9 10 - - - - - - |

| MTU | 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 - - - - - - |

+------------------------------------------------------------------------------------------+

Active Ports:

+-----------------------------------------+

| ID | OPI | LanID | MTU |

+-----------------------------------------+

| 410b1d4ce7d0 | 01,0001a2 | 4 | 1500 |

| 410b1d4cfde0 | 01,0001a2 | 4 | 1500 |

+-----------------------------------------+

Switch State:

+-----------------------------------------------------+

+-----------------------------------------------------+

+ ............................... Port:0x410b1d4ce7d0 +

+-----------------------------------------------------+

+ ............................... Port:0x410b1d4cfde0 +

| 00:50:56:01:06:16 | 00:13:F5:01:04:B4 | 1 | 1 | 1 |

+-----------------------------------------------------+

Port Statistics Summary:

+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| 410b1d4ce7d0 | 01,0001a2 | 3991 | 2 | 3976 | 0 | 0 | 0 | 13 | 0 | 0 | 0 | 0 | 0 | 0 | 679 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |

+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| 410b1d4cfde0 | 01,0001a2 | 4478 | 2 | 3860 | 0 | 0 | 0 | 616 | 0 | 0 | 0 | 0 | 0 | 0 | 626 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |

+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

And here is the output of the command a little while later on the same host (while my test pings are going)

~ # esxcli vcloud fence getfenceinfo

Module Parameters:

Host Key: 0x104b0

Configured LAN MTUs:

+------------------------------------------------------------------------------------------+

| LAN ID | 1 2 3 4 5 6 7 8 9 10 - - - - - - |

| MTU | 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 - - - - - - |

+------------------------------------------------------------------------------------------+

Active Ports:

+-----------------------------------------+

| ID | OPI | LanID | MTU |

+-----------------------------------------+

| 410b1d4ce7d0 | 01,0001a2 | 4 | 1500 |

| 410b1d4cfde0 | 01,0001a2 | 4 | 1500 |

+-----------------------------------------+

Switch State:

+-----------------------------------------------------+

+-----------------------------------------------------+

+ ............................... Port:0x410b1d4ce7d0 +

+-----------------------------------------------------+

+ ............................... Port:0x410b1d4cfde0 +

| 00:50:56:01:06:16 | 00:13:F5:01:04:B4 | 1 | 1 | 1 |

+-----------------------------------------------------+

Port Statistics Summary:

+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| 410b1d4ce7d0 | 01,0001a2 | 4696 | 2 | 4681 | 0 | 0 | 0 | 13 | 0 | 0 | 0 | 0 | 0 | 0 | 796 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |

+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| 410b1d4cfde0 | 01,0001a2 | 5300 | 2 | 4565 | 0 | 0 | 0 | 733 | 0 | 0 | 0 | 0 | 0 | 0 | 743 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |

+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Looking at the VDS, the Maximum MTU is set to 1500. Looking at dvs.VCDVSTest Isolated Network-ca9a45c9-1dec-4f95-8b96-68d9c05b2a5d, I see it's VLAN is set to 2535 (which matches the network pool settings) and number of ports is 16

Jackal830 · ‎05-21-2015

Here is another output, this time after rebooting the host, confirming network connectivity didn't work, unpreparing and preparing the host. The same two VMs are present

~ # esxcli vcloud fence getfenceinfo

Module Parameters:

Host Key: 0x104b0

Configured LAN MTUs:

+------------------------------------------------------------------------------------------+

| LAN ID | 1 2 3 4 5 6 7 8 9 10 - - - - - - |

| MTU | 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 - - - - - - |

+------------------------------------------------------------------------------------------+

Active Ports:

+-----------------------------------------+

| ID | OPI | LanID | MTU |

+-----------------------------------------+

| 500000d | 00,fffffe | 4 | 1500 |

| 410b1ca8a7d0 | 01,0001a2 | 4 | 1500 |

| 410b1ca8bde0 | 01,0001a2 | 4 | 1500 |

+-----------------------------------------+

Switch State:

+-----------------------------------------------------+

+-----------------------------------------------------+

+ ............................... Port:0x500000d +

+-----------------------------------------------------+

+ ............................... Port:0x410b1ca8a7d0 +

| 00:50:56:01:06:16 | 00:13:F5:01:04:B4 | 1 | 1 | 1 |

+-----------------------------------------------------+

+ ............................... Port:0x410b1ca8bde0 +

| 00:50:56:01:06:17 | 00:13:F5:01:04:B4 | 1 | 0 | 1 |

+-----------------------------------------------------+

Port Statistics Summary:

+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| 500000d | 00,fffffe | 255 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 255 | 0 |

+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| 410b1ca8a7d0 | 01,0001a2 | 720 | 2 | 578 | 0 | 0 | 0 | 140 | 0 | 0 | 0 | 0 | 0 | 0 | 140 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |

+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| 410b1ca8bde0 | 01,0001a2 | 713 | 2 | 576 | 0 | 0 | 0 | 135 | 0 | 0 | 0 | 0 | 0 | 0 | 139 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |

+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

I notice there is another Port ID listed in this output. Is this my problem? A required port ID is not being created upon bootup?

Sreec · ‎05-21-2015

VCDNI is a MAC-MAC encapsulation technology.For the same reason MTU should be increased MTU Increase (24Bytes, 1500 –> 1524).Increase the MTU size of network devices in the transport VLAN to at least 1600 to accommodate the additional information needed for VCD-NI.

Please do change the MTU settings on

1.VCDNI network Pool properties in vCloud Director--MTU should be 1600

2.Set the same MTU(1600) in the DVS which is backing the VCDNI network Pool

3.Set the same MTU(1600) in upstream or downstream switches that this packet can traverse.

Note:Your external networks and network pools can't share the same VLAN.

Cheers,
Sree | VCIX-5X| VCAP-5X| VExpert 7x|Cisco Certified Specialist
Please KUDO helpful posts and mark the thread as solved if answered

Jackal830 · ‎05-22-2015

While I understand what you are saying here, I don't think this has anything to do with the issue I am running into.

I am able to send 1500 byte sized pings across an isolated network, so there must be some sort of segmentation (edit: fragmentation) that occurs if the MTU is below 1524.

Changing the MTU settings on all our upstream equipment is not going to be a viable option for us.

Now, if there is a reason why this MTU setting would cause the issue I'm running into, I'm all ears. Before we switched to Auto-Deploy, we had no reach-ability issues on isolated networks, and we have some hosts that boot up on Auto-Deploy with it working as well. If it was a MTU issue, wouldn't we see the same issue on each host, every time? Wouldn't I also be able to send a 64 byte ping without issue (something I cannot do before I unprepare and then prepare the host)

We have another vCloud cluster that is not using auto-deploy, it also has a MTU of 1500 and it has no (perceived) issues with isolated networks.

Edit: If I send a don't fragment flag to ping and send a 1500 byte packet, it does run into an error saying fragmentation is needed. I don't doubt we aren't following best practice, what I do doubt is that this setting is causing the reachability issues we are seeing.

Sreec · ‎05-22-2015

What i pin-pointed was one of the core prerequites for VCDNI network pool.Since you already have Support Request created and if it is really a host specific issue i'm pretty sure there will be some valid messages in the host logs which they will share with you.However changing the MTU is mandatory and same applies for VXLAN network as well.Please refer below documentation for detailed referrence.

http://download3.vmware.com/vcat/vcat31_documentation_center/index.html#page/Architecting%2520a%2520...

VMware vCloud Director 5.5 Documentation Center

http://download3.vmware.com/vcat/vcat31_documentation_center/index.html#page/Implementation%2520Exam...

http://www.yellow-bricks.com/2010/09/09/vcd-networking-part-2-network-pools/

Cheers,
Sree | VCIX-5X| VCAP-5X| VExpert 7x|Cisco Certified Specialist
Please KUDO helpful posts and mark the thread as solved if answered

Jackal830 · ‎05-22-2015

I do agree that we are not following best practice here, but I don't think it is "mandatory" From these links, I see the following:

Increase the MTU size of network devices in the transport VLAN to at least 1600 to accommodate the additional information needed for VCD-NI. The information includes all physical switches and vSphere Distributed Switches. Failure to increase the MTU size causes packet fragmentation, negatively affecting network throughput performance of vCloud workloads.

When you configure the virtual machine guest operating system and the underlying physical infrastructure with the standard MTU (1500 bytes), the VMware network isolation protocol fragments frames. To avoid frame fragmentation, increase the MTU to at least 1600 bytes for the network pool and the underlying physical network. You can increase the network pool MTU up to, but not greater than, the MTU of the physical network.

It is recommended to increase the MTU size on the physical devices backing the VCDNI network pool to 1524 bytes.

It is also recommended to increase the MTU size on the network pool itself to 1524 bytes.

So, with a setting of 1500, we get packet fragmentation, which is not optimal, but connectivity should exist. A 64-byte ping should work and not have any fragmentation. I have confirmed this in my fragmentation tests. Fragmentation, in our environment, starts at 1480 bytes (and it still works, just fragmented).

Sreec · ‎05-22-2015

That is correct,connectivity should certainly be there.If you are OK with packet fragmentation in a production environment we don't need to make that change.If you dont mind can you provide me with SR number,i would like to have a look at the logs.Apart from that there is nothing much we can pin-point as per our conversation so far.Please expect a delay in my response,i primarily check and respond during ANZ/IST timings

Cheers,
Sree | VCIX-5X| VCAP-5X| VExpert 7x|Cisco Certified Specialist
Please KUDO helpful posts and mark the thread as solved if answered

Jackal830 · ‎05-26-2015

Sure deal, it's 15670628905.

EcoBassam · ‎11-04-2015

Hello,

Any news on this case, I may be running into sumilar issue, I would appreciate if you can share any useful details ?

Regards,

All

Auto Deploy and VCD Network Isolation - Not Working