Highlighted
Contributor
Contributor

Auto Deploy and VCD Network Isolation - Not Working

Hello,

I have opened a support case about this issue, but the case is moving a lot slower than I would like to see. I thought I'd post here to see if anyone has ran into anything similar.

We recently switched our 24-host vCloud cluster (more info about this cluster below) over to Auto-Deploy (was using boot from USB). After the switch, we had a few complaints from customers that use isolated networks. It turns out that only 4 of our 24 hosts were working properly with isolated networks. All 24-hosts are using the same auto-deploy image.

When the hosts boot up, the vCloud Director web interface shows the following with green check marks: "Status", "Enabled", "Ready", "Available", and "VCD Network Isolation Capable".

To get my other 20 hosts working, I disabled all 20 hosts in the vCloud interface, unprepared them, and prepared them. After this, network isolation worked for those 20 hosts. When I reboot one of the hosts, the host still looks like it's good in the interface, but network isolation doesn't work until I disable, unprepare, and prepare.

Here is some more information about our environment:

vCenter build 2001466

ESXi Host Build 2702864

VCD Build 5.5.2.2000523

vShield Build 5.5.3

Here is the Deploy Rule that the hosts are using:

Name        : UCS-2.2.1-pcloud.5.5.aln

PatternList : {oemstring=$SPT:ESX_PublicCloud_autod.2.2.x}

ItemList    : {ESXi-5.5.0-autod-2.2.1.vshield}

I have confirmed that this is indeed the profile being used by looking at the Summary tab for the hosts and seeing the listed Image Profile.

And here is what's in that item:

PowerCLI D:\ImageBuild> Get-EsxSoftwarePackage

Name                     Version                        Vendor     Creation Date

----                     -------                        ------     -------------

misc-cnic-register       1.72.1.v50.1i-1vmw.550.0.0.... VMware     9/19/2013 6:0...

scsi-lpfc820             8.2.3.1-129vmw.550.0.0.1331820 VMware     9/19/2013 6:0...

scsi-megaraid-mbox       2.20.5.1-6vmw.550.0.0.1331820  VMware     9/19/2013 6:0...

esx-xlibs                5.5.0-0.0.1331820              VMware     9/19/2013 6:0...

lpfc                     10.0.100.1-1vmw.550.0.0.133... VMware     9/19/2013 6:0...

mtip32xx-native          3.3.4-1vmw.550.1.15.1623387    VMware     2/22/2014 1:1...

net-nx-nic               5.0.621-1vmw.550.0.0.1331820   VMware     9/19/2013 6:0...

block-cciss              3.6.14-10vmw.550.0.0.1331820   VMware     9/19/2013 6:0...

scsi-fnic                1.6.0.5-1OEM.500.0.0.472560    cisco      9/30/2013 11:...

net-enic                 2.1.2.42-1OEM.500.0.0.472560   Cisco      9/5/2013 8:30...

sata-sata-sil24          1.1-1vmw.550.0.0.1331820       VMware     9/19/2013 6:0...

rste                     2.0.2.0088-4vmw.550.1.15.16... VMware     2/22/2014 1:1...

elxnet                   10.0.100.0v-1vmw.550.0.0.13... VMware     9/19/2013 6:0...

scsi-aacraid             1.1.5.1-9vmw.550.0.0.1331820   VMware     9/19/2013 6:0...

ata-pata-cmd64x          0.2.5-3vmw.550.0.0.1331820     VMware     9/19/2013 6:0...

net-cnic                 1.72.52.v55.1-1vmw.550.0.0.... VMware     9/19/2013 6:0...

scsi-adp94xx             1.0.8.12-6vmw.550.0.0.1331820  VMware     9/19/2013 6:0...

net-be2net               4.6.100.0v-1vmw.550.0.0.133... VMware     9/19/2013 6:0...

net-ixgbe                3.7.13.7.14iov-12vmw.550.2.... VMware     4/29/2015 6:4...

net-igb                  5.0.5.1.1-1vmw.550.2.54.240... VMware     1/1/2015 8:00...

epsec-mux                5.1.0-01814505                 VMware     5/13/2014 4:3...

esx-base                 5.5.0-2.62.2702864             VMware     4/29/2015 6:4...

ata-pata-sil680          0.4.8-3vmw.550.0.0.1331820     VMware     9/19/2013 6:0...

ipmi-ipmi-msghandler     39.1-4vmw.550.0.0.1331820      VMware     9/19/2013 6:0...

ata-pata-hpt3x2n         0.3.4-3vmw.550.0.0.1331820     VMware     9/19/2013 6:0...

scsi-megaraid-sas        5.34-9vmw.550.2.33.2068190     VMware     8/23/2014 1:5...

scsi-mptsas              4.23.01.00-9vmw.550.0.0.133... VMware     9/19/2013 6:0...

net-bnx2                 2.2.3d.v55.2-1vmw.550.0.0.1... VMware     9/19/2013 6:0...

ata-pata-via             0.3.3-2vmw.550.0.0.1331820     VMware     9/19/2013 6:0...

ima-qla4xxx              2.01.31-1vmw.550.0.0.1331820   VMware     9/19/2013 6:0...

ata-pata-amd             0.3.10-3vmw.550.0.0.1331820    VMware     9/19/2013 6:0...

ata-pata-serverworks     0.4.3-3vmw.550.0.0.1331820     VMware     9/19/2013 6:0...

sata-sata-promise        2.12-3vmw.550.0.0.1331820      VMware     9/19/2013 6:0...

sata-sata-nv             3.5-4vmw.550.0.0.1331820       VMware     9/19/2013 6:0...

ipmi-ipmi-devintf        39.1-4vmw.550.0.0.1331820      VMware     9/19/2013 6:0...

scsi-ips                 7.12.05-4vmw.550.0.0.1331820   VMware     9/19/2013 6:0...

sata-sata-svw            2.3-3vmw.550.0.0.1331820       VMware     9/19/2013 6:0...

scsi-mptspi              4.23.01.00-9vmw.550.0.0.133... VMware     9/19/2013 6:0...

net-e1000e               1.1.2-4vmw.550.1.15.1623387    VMware     2/22/2014 1:1...

esx-xserver              5.5.0-0.0.1331820              VMware     9/19/2013 6:0...

net-tg3                  3.123c.v55.5-1vmw.550.2.33.... VMware     8/23/2014 1:5...

net-forcedeth            0.61-2vmw.550.0.0.1331820      VMware     9/19/2013 6:0...

scsi-aic79xx             3.1-5vmw.550.0.0.1331820       VMware     9/19/2013 6:0...

sata-ata-piix            2.12-10vmw.550.2.33.2068190    VMware     8/23/2014 1:5...

scsi-bnx2i               2.72.11.v55.4-1vmw.550.0.0.... VMware     9/19/2013 6:0...

ohci-usb-ohci            1.0-3vmw.550.0.0.1331820       VMware     9/19/2013 6:0...

scsi-qla4xxx             5.01.03.2-6vmw.550.0.0.1331820 VMware     9/19/2013 6:0...

scsi-qla2xxx             902.k1.1-9vmw.550.0.0.1331820  VMware     9/19/2013 6:0...

net-mlx4-core            1.9.7.0-1vmw.550.0.0.1331820   VMware     9/19/2013 6:0...

ata-pata-atiixp          0.4.6-4vmw.550.0.0.1331820     VMware     9/19/2013 6:0...

misc-drivers             5.5.0-2.62.2702864             VMware     4/29/2015 6:4...

esx-dvfilter-generic-... 5.5.0-0.0.1331820              VMware     9/19/2013 6:0...

ata-pata-pdc2027x        1.0-3vmw.550.0.0.1331820       VMware     9/19/2013 6:0...

qlnativefc               1.0.12.0-1vmw.550.0.0.1331820  VMware     9/19/2013 6:0...

lsi-mr3                  0.255.03.01-2vmw.550.1.16.1... VMware     4/15/2014 9:0...

vshield                  5.5.3-2172759                  VMware     9/30/2014 2:3...

net-vmxnet3              1.1.3.0-3vmw.550.2.39.2143827  VMware     9/18/2014 11:...

scsi-hpsa                5.5.0-44vmw.550.0.0.1331820    VMware     9/19/2013 6:0...

tools-light              5.5.0-2.62.2702864             VMware     4/29/2015 6:4...

scsi-mpt2sas             14.00.00.00-3vmw.550.1.15.1... VMware     2/22/2014 1:1...

scsi-bnx2fc              1.72.53.v55.1-1vmw.550.0.0.... VMware     9/19/2013 6:0...

lsi-msgpt3               00.255.03.03-1vmw.550.1.15.... VMware     2/22/2014 1:1...

net-e1000                8.0.3.1-3vmw.550.0.0.1331820   VMware     9/19/2013 6:0...

xhci-xhci                1.0-2vmw.550.2.39.2143827      VMware     9/18/2014 11:...

ipmi-ipmi-si-drv         39.1-4vmw.550.0.0.1331820      VMware     9/19/2013 6:0...

sata-ahci                3.0-21vmw.550.2.54.2403361     VMware     1/1/2015 8:00...

net-bnx2x                1.72.56.v55.2-1vmw.550.0.0.... VMware     9/19/2013 6:0...

scsi-megaraid2           2.00.4-9vmw.550.0.0.1331820    VMware     9/19/2013 6:0...

ehci-ehci-hcd            1.0-3vmw.550.0.0.1331820       VMware     9/19/2013 6:0...

esx-tboot                5.5.0-2.33.2068190             VMware     8/23/2014 1:5...

uhci-usb-uhci            1.0-3vmw.550.0.0.1331820       VMware     9/19/2013 6:0...

sata-sata-sil            2.3-4vmw.550.0.0.1331820       VMware     9/19/2013 6:0...

vcloud-agent             5.5.0-1280396                  VMware     8/17/2013 4:0...

net-mlx4-en              1.9.7.0-1vmw.550.0.0.1331820   VMware     9/19/2013 6:0...

And here is the output of testing the deploy rule compliance:

PowerCLI D:\ImageBuild> Get-Cluster 0000000-ESXVCLOUDCL1.ALN | Get-VMHost |Test-

DeployRuleSetCompliance

VMHost               ItemList

------               --------

esx142269.vm.seo.... {}

esx140622.vm.seo.... {}

esx139784.vm.seo.... {}

esx140617.vm.seo.... {}

esx138793.vm.seo.... {}

esx135523.vm.seo.... {}

esx138945.vm.seo.... {}

esx138794.vm.seo.... {}

esx139783.vm.seo.... {}

esx140309.vm.seo.... {}

esx140310.vm.seo.... {}

esx140311.vm.seo.... {}

esx140313.vm.seo.... {}

esx140339.vm.seo.... {}

esx140614.vm.seo.... {}

esx140615.vm.seo.... {}

esx140616.vm.seo.... {}

esx140618.vm.seo.... {}

esx140619.vm.seo.... {}

esx140621.vm.seo.... {}

esx141947.vm.seo.... {}

esx141945.vm.seo.... {}

esx142271.vm.seo.... {}

esx142270.vm.seo.... {}

Here is another vib list from a host:

~ # esxcli software vib list

Name                           Version                                Vendor  Acceptance Level  Install Date

-----------------------------  -------------------------------------  ------  ----------------  ------------

net-enic                       2.1.2.42-1OEM.500.0.0.472560           Cisco   VMwareCertified   -

ata-pata-amd                   0.3.10-3vmw.550.0.0.1331820            VMware  VMwareCertified   -

ata-pata-atiixp                0.4.6-4vmw.550.0.0.1331820             VMware  VMwareCertified   -

ata-pata-cmd64x                0.2.5-3vmw.550.0.0.1331820             VMware  VMwareCertified   -

ata-pata-hpt3x2n               0.3.4-3vmw.550.0.0.1331820             VMware  VMwareCertified   -

ata-pata-pdc2027x              1.0-3vmw.550.0.0.1331820               VMware  VMwareCertified   -

ata-pata-serverworks           0.4.3-3vmw.550.0.0.1331820             VMware  VMwareCertified   -

ata-pata-sil680                0.4.8-3vmw.550.0.0.1331820             VMware  VMwareCertified   -

ata-pata-via                   0.3.3-2vmw.550.0.0.1331820             VMware  VMwareCertified   -

block-cciss                    3.6.14-10vmw.550.0.0.1331820           VMware  VMwareCertified   -

ehci-ehci-hcd                  1.0-3vmw.550.0.0.1331820               VMware  VMwareCertified   -

elxnet                         10.0.100.0v-1vmw.550.0.0.1331820       VMware  VMwareCertified   -

epsec-mux                      5.1.0-01814505                         VMware  VMwareCertified   -

esx-base                       5.5.0-2.62.2702864                     VMware  VMwareCertified   -

esx-dvfilter-generic-fastpath  5.5.0-0.0.1331820                      VMware  VMwareCertified   -

esx-tboot                      5.5.0-2.33.2068190                     VMware  VMwareCertified   -

esx-xlibs                      5.5.0-0.0.1331820                      VMware  VMwareCertified   -

esx-xserver                    5.5.0-0.0.1331820                      VMware  VMwareCertified   -

ima-qla4xxx                    2.01.31-1vmw.550.0.0.1331820           VMware  VMwareCertified   -

ipmi-ipmi-devintf              39.1-4vmw.550.0.0.1331820              VMware  VMwareCertified   -

ipmi-ipmi-msghandler           39.1-4vmw.550.0.0.1331820              VMware  VMwareCertified   -

ipmi-ipmi-si-drv               39.1-4vmw.550.0.0.1331820              VMware  VMwareCertified   -

lpfc                           10.0.100.1-1vmw.550.0.0.1331820        VMware  VMwareCertified   -

lsi-mr3                        0.255.03.01-2vmw.550.1.16.1746018      VMware  VMwareCertified   -

lsi-msgpt3                     00.255.03.03-1vmw.550.1.15.1623387     VMware  VMwareCertified   -

misc-cnic-register             1.72.1.v50.1i-1vmw.550.0.0.1331820     VMware  VMwareCertified   -

misc-drivers                   5.5.0-2.62.2702864                     VMware  VMwareCertified   -

mtip32xx-native                3.3.4-1vmw.550.1.15.1623387            VMware  VMwareCertified   -

net-be2net                     4.6.100.0v-1vmw.550.0.0.1331820        VMware  VMwareCertified   -

net-bnx2                       2.2.3d.v55.2-1vmw.550.0.0.1331820      VMware  VMwareCertified   -

net-bnx2x                      1.72.56.v55.2-1vmw.550.0.0.1331820     VMware  VMwareCertified   -

net-cnic                       1.72.52.v55.1-1vmw.550.0.0.1331820     VMware  VMwareCertified   -

net-e1000                      8.0.3.1-3vmw.550.0.0.1331820           VMware  VMwareCertified   -

net-e1000e                     1.1.2-4vmw.550.1.15.1623387            VMware  VMwareCertified   -

net-forcedeth                  0.61-2vmw.550.0.0.1331820              VMware  VMwareCertified   -

net-igb                        5.0.5.1.1-1vmw.550.2.54.2403361        VMware  VMwareCertified   -

net-ixgbe                      3.7.13.7.14iov-12vmw.550.2.62.2702864  VMware  VMwareCertified   -

net-mlx4-core                  1.9.7.0-1vmw.550.0.0.1331820           VMware  VMwareCertified   -

net-mlx4-en                    1.9.7.0-1vmw.550.0.0.1331820           VMware  VMwareCertified   -

net-nx-nic                     5.0.621-1vmw.550.0.0.1331820           VMware  VMwareCertified   -

net-tg3                        3.123c.v55.5-1vmw.550.2.33.2068190     VMware  VMwareCertified   -

net-vmxnet3                    1.1.3.0-3vmw.550.2.39.2143827          VMware  VMwareCertified   -

ohci-usb-ohci                  1.0-3vmw.550.0.0.1331820               VMware  VMwareCertified   -

qlnativefc                     1.0.12.0-1vmw.550.0.0.1331820          VMware  VMwareCertified   -

rste                           2.0.2.0088-4vmw.550.1.15.1623387       VMware  VMwareCertified   -

sata-ahci                      3.0-21vmw.550.2.54.2403361             VMware  VMwareCertified   -

sata-ata-piix                  2.12-10vmw.550.2.33.2068190            VMware  VMwareCertified   -

sata-sata-nv                   3.5-4vmw.550.0.0.1331820               VMware  VMwareCertified   -

sata-sata-promise              2.12-3vmw.550.0.0.1331820              VMware  VMwareCertified   -

sata-sata-sil24                1.1-1vmw.550.0.0.1331820               VMware  VMwareCertified   -

sata-sata-sil                  2.3-4vmw.550.0.0.1331820               VMware  VMwareCertified   -

sata-sata-svw                  2.3-3vmw.550.0.0.1331820               VMware  VMwareCertified   -

scsi-aacraid                   1.1.5.1-9vmw.550.0.0.1331820           VMware  VMwareCertified   -

scsi-adp94xx                   1.0.8.12-6vmw.550.0.0.1331820          VMware  VMwareCertified   -

scsi-aic79xx                   3.1-5vmw.550.0.0.1331820               VMware  VMwareCertified   -

scsi-bnx2fc                    1.72.53.v55.1-1vmw.550.0.0.1331820     VMware  VMwareCertified   -

scsi-bnx2i                     2.72.11.v55.4-1vmw.550.0.0.1331820     VMware  VMwareCertified   -

scsi-hpsa                      5.5.0-44vmw.550.0.0.1331820            VMware  VMwareCertified   -

scsi-ips                       7.12.05-4vmw.550.0.0.1331820           VMware  VMwareCertified   -

scsi-lpfc820                   8.2.3.1-129vmw.550.0.0.1331820         VMware  VMwareCertified   -

scsi-megaraid-mbox             2.20.5.1-6vmw.550.0.0.1331820          VMware  VMwareCertified   -

scsi-megaraid-sas              5.34-9vmw.550.2.33.2068190             VMware  VMwareCertified   -

scsi-megaraid2                 2.00.4-9vmw.550.0.0.1331820            VMware  VMwareCertified   -

scsi-mpt2sas                   14.00.00.00-3vmw.550.1.15.1623387      VMware  VMwareCertified   -

scsi-mptsas                    4.23.01.00-9vmw.550.0.0.1331820        VMware  VMwareCertified   -

scsi-mptspi                    4.23.01.00-9vmw.550.0.0.1331820        VMware  VMwareCertified   -

scsi-qla2xxx                   902.k1.1-9vmw.550.0.0.1331820          VMware  VMwareCertified   -

scsi-qla4xxx                   5.01.03.2-6vmw.550.0.0.1331820         VMware  VMwareCertified   -

uhci-usb-uhci                  1.0-3vmw.550.0.0.1331820               VMware  VMwareCertified   -

vcloud-agent                   5.5.0-1280396                          VMware  VMwareCertified   -

vmware-fdm                     5.5.0-2001466                          VMware  VMwareCertified   -

vshield                        5.5.3-2172759                          VMware  VMwareCertified   -

xhci-xhci                      1.0-2vmw.550.2.39.2143827              VMware  VMwareCertified   -

tools-light                    5.5.0-2.62.2702864                     VMware  VMwareCertified   -

scsi-fnic                      1.6.0.5-1OEM.500.0.0.472560            cisco   VMwareCertified   -

Any help is appreciated. Thanks.

0 Kudos
15 Replies
Highlighted
VMware Employee
VMware Employee

Hi,

      When you say network isolation doesn't work,can you be bit more precise on that point?

1.Is the network pools not getting created/failed?

2.Virtual Machines do not have network connectivity even if they are placed in the same host?

3.Do we have any working network pools(VXLAN/VLAN/PG backed) in this set-up?

Cheers,
Sree | CKA|VCAP-NSX-T| VCIX-3X| VCAP-3X| VExpert 4x
0 Kudos
Highlighted
Contributor
Contributor

To clarify, I'm talking about an Isolated Organization vDC Network.

For example, in Org VDC Networks, if I were to add one, I'd select "Create an isolated network within this virtual datacenter"

The network shows OK status, the hosts report "VCD Network Isolation Capable" as OK, but it won't work until I unprepare and re-prepare the hosts running VMs that need to be on that isolated network.

The odd thing is, the 4 hosts that do work straight from boot, continue to work after reboots. They are all using the same base image profile and the same auto-deploy image.

Does this answer your question?

0 Kudos
Highlighted
VMware Employee
VMware Employee

So you are using VCDNI network pool and whenever we are creating Isolated org-vdc network you have issue? Please correct me if i'm wrong?

What type of issue you have with Isolated org-vdc networks? VM's cannot reach other ? Like i asked in my first reply,does it work if VM's are running on same host?

How about a direct org-vdc/vapp/routed network backed by VCDNI network pool.Do they work?

Cheers,
Sree | CKA|VCAP-NSX-T| VCIX-3X| VCAP-3X| VExpert 4x
0 Kudos
Highlighted
Contributor
Contributor

Even if the VMs are on the same host, they cannot communicate with each other over the private network (pings fail, etc) until I unprepare and prepare the host (and move the VMs back to it)

The issue happens to any VCDNI network (already created or newly created) No traffic works until the vcloud agent is uninstalled and reinstalled (which shouldn't be required, since the VIB is included in my autodeploy image). There are 4 hosts that do not require the unprepare and prepare, they just work upon startup.

Type "Direct" works just fine. A routed network connecting to an edge gateway does not work and requires all VMs to be on a host that has been unprepared and then prepared (or on one of the 4 hosts that work correctly upon boot-up).

0 Kudos
Highlighted
Contributor
Contributor

I have been using one host for testing (one that doesn't work upon startup). I've rebooted it several times today. One reboot, it actually worked right away, but the others it didn't. This is very strange.

Edit: Just to be sure that it did work at least once right after a reboot, I did several more reboots and got it to work another time (out of many).

Could it be something weird with autodeploy and enabling host spanning? Could it be some sort of weird timing issue?

0 Kudos
Highlighted
VMware Employee
VMware Employee

You said,even VM's on same host doesn't ping each other.Are we sure that they were on same port-group and host when you tested? Scenario will be same even if its a routed org network,VSE and respective VM'S might be residing on different host during your test.

Worth enough to re-check if  Vlan-id and MTU is properly set in DVS/Physical switch and VCDNI network pool properties in VCD as well.Adding to that it can be issue with fence module ,try running /opt/vmware/vslad/fence-util moduleinfo which would give you a better output(To get per host vCDNI statistics use)

Cheers,
Sree | CKA|VCAP-NSX-T| VCIX-3X| VCAP-3X| VExpert 4x
0 Kudos
Highlighted
Contributor
Contributor

Right now, on my test host (that is not working), I have two VMs, one named "Test 1" and the other "Test 2" They only have an isolated network on them (named "Test Isolated Network") that is in my VDC, and pings do not work between them. If I move them to any other host (that have been unprepared and prepared), pings work. I can move both to the same host, or different hosts.

Here is the fence info on this host, I'm not really sure how to interpret this:

~ # esxcli vcloud fence getfenceinfo

   Module Parameters:

        Host Key: 0x104b0

        Configured LAN MTUs:

        +------------------------------------------------------------------------------------------+

        | LAN ID |    1    2    3    4    5    6    7    8    9   10    -    -    -    -    -    - |

        |    MTU | 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500    -    -    -    -    -    - |

        +------------------------------------------------------------------------------------------+

   Active Ports:

        +-----------------------------------------+

        |     ID       |    OPI    | LanID |  MTU |

        +-----------------------------------------+

        | 410b1d4ce7d0 | 01,0001a2 |     4 | 1500 |

        | 410b1d4cfde0 | 01,0001a2 |     4 | 1500 |

        +-----------------------------------------+

   Switch State:

        +-----------------------------------------------------+

        |   Inner MAC    |   Outer MAC   | used | age | seen  |

        +-----------------------------------------------------+

        + ............................... Port:0x410b1d4ce7d0 +

        +-----------------------------------------------------+

        + ............................... Port:0x410b1d4cfde0 +

        | 00:50:56:01:06:16 | 00:13:F5:01:04:B4 | 1  | 1  | 1 |

        +-----------------------------------------------------+

   Port Statistics Summary:

        +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

        |       Identity           |               To VM dropped                         | To VM passed    |     From VM dropped                        |                 From VM passed                      |  Reflect        |

        |   Port ID    | Fence ID  | total  | misunf | misfen | stored | frag   | other  | fenced | join   | unfen  | csum   | frag   | GVT    | other  | fenced | tso    | frag   | csum   | GVT    | ufport | pass   |  error |

        +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

        | 410b1d4ce7d0 | 01,0001a2 |   3991 |      2 |   3976 |      0 |      0 |      0 |     13 |      0 |      0 |      0 |      0 |      0 |      0 |    679 |      0 |      0 |      0 |      0 |      0 |      0 |      0 |

        +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

        | 410b1d4cfde0 | 01,0001a2 |   4478 |      2 |   3860 |      0 |      0 |      0 |    616 |      0 |      0 |      0 |      0 |      0 |      0 |    626 |      0 |      0 |      0 |      0 |      0 |      0 |      0 |

        +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

And here is the output of the command a little while later on the same host (while my test pings are going)

~ # esxcli vcloud fence getfenceinfo

   Module Parameters:

        Host Key: 0x104b0

        Configured LAN MTUs:

        +------------------------------------------------------------------------------------------+

        | LAN ID |    1    2    3    4    5    6    7    8    9   10    -    -    -    -    -    - |

        |    MTU | 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500    -    -    -    -    -    - |

        +------------------------------------------------------------------------------------------+

   Active Ports:

        +-----------------------------------------+

        |     ID       |    OPI    | LanID |  MTU |

        +-----------------------------------------+

        | 410b1d4ce7d0 | 01,0001a2 |     4 | 1500 |

        | 410b1d4cfde0 | 01,0001a2 |     4 | 1500 |

        +-----------------------------------------+

   Switch State:

        +-----------------------------------------------------+

        |   Inner MAC    |   Outer MAC   | used | age | seen  |

        +-----------------------------------------------------+

        + ............................... Port:0x410b1d4ce7d0 +

        +-----------------------------------------------------+

        + ............................... Port:0x410b1d4cfde0 +

        | 00:50:56:01:06:16 | 00:13:F5:01:04:B4 | 1  | 1  | 1 |

        +-----------------------------------------------------+

   Port Statistics Summary:

        +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

        |       Identity           |               To VM dropped                         | To VM passed    |     From VM dropped                        |                 From VM passed                      |  Reflect        |

        |   Port ID    | Fence ID  | total  | misunf | misfen | stored | frag   | other  | fenced | join   | unfen  | csum   | frag   | GVT    | other  | fenced | tso    | frag   | csum   | GVT    | ufport | pass   |  error |

        +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

        | 410b1d4ce7d0 | 01,0001a2 |   4696 |      2 |   4681 |      0 |      0 |      0 |     13 |      0 |      0 |      0 |      0 |      0 |      0 |    796 |      0 |      0 |      0 |      0 |      0 |      0 |      0 |

        +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

        | 410b1d4cfde0 | 01,0001a2 |   5300 |      2 |   4565 |      0 |      0 |      0 |    733 |      0 |      0 |      0 |      0 |      0 |      0 |    743 |      0 |      0 |      0 |      0 |      0 |      0 |      0 |

        +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Looking at the VDS, the Maximum MTU is set to 1500. Looking at dvs.VCDVSTest Isolated Network-ca9a45c9-1dec-4f95-8b96-68d9c05b2a5d, I see it's VLAN is set to 2535 (which matches the network pool settings) and number of ports is 16

0 Kudos
Highlighted
Contributor
Contributor

Here is another output, this time after rebooting the host, confirming network connectivity didn't work, unpreparing and preparing the host. The same two VMs are present

~ # esxcli vcloud fence getfenceinfo

   Module Parameters:

        Host Key: 0x104b0

        Configured LAN MTUs:

        +------------------------------------------------------------------------------------------+

        | LAN ID |    1    2    3    4    5    6    7    8    9   10    -    -    -    -    -    - |

        |    MTU | 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500    -    -    -    -    -    - |

        +------------------------------------------------------------------------------------------+

   Active Ports:

        +-----------------------------------------+

        |     ID       |    OPI    | LanID |  MTU |

        +-----------------------------------------+

        |      500000d | 00,fffffe |     4 | 1500 |

        | 410b1ca8a7d0 | 01,0001a2 |     4 | 1500 |

        | 410b1ca8bde0 | 01,0001a2 |     4 | 1500 |

        +-----------------------------------------+

   Switch State:

        +-----------------------------------------------------+

        |   Inner MAC    |   Outer MAC   | used | age | seen  |

        +-----------------------------------------------------+

        + ............................... Port:0x500000d      +

        +-----------------------------------------------------+

        + ............................... Port:0x410b1ca8a7d0 +

        | 00:50:56:01:06:16 | 00:13:F5:01:04:B4 | 1  | 1  | 1 |

        +-----------------------------------------------------+

        + ............................... Port:0x410b1ca8bde0 +

        | 00:50:56:01:06:17 | 00:13:F5:01:04:B4 | 1  | 0  | 1 |

        +-----------------------------------------------------+

   Port Statistics Summary:

        +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

        |       Identity           |               To VM dropped                         | To VM passed    |     From VM dropped                        |                 From VM passed                      |  Reflect        |

        |   Port ID    | Fence ID  | total  | misunf | misfen | stored | frag   | other  | fenced | join   | unfen  | csum   | frag   | GVT    | other  | fenced | tso    | frag   | csum   | GVT    | ufport | pass   |  error |

        +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

        |      500000d | 00,fffffe |    255 |      0 |      0 |      0 |      0 |      0 |      0 |      0 |      0 |      0 |      0 |      0 |      0 |      0 |      0 |      0 |      0 |      0 |      0 |    255 |      0 |

        +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

        | 410b1ca8a7d0 | 01,0001a2 |    720 |      2 |    578 |      0 |      0 |      0 |    140 |      0 |      0 |      0 |      0 |      0 |      0 |    140 |      0 |      0 |      0 |      0 |      0 |      0 |      0 |

        +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

        | 410b1ca8bde0 | 01,0001a2 |    713 |      2 |    576 |      0 |      0 |      0 |    135 |      0 |      0 |      0 |      0 |      0 |      0 |    139 |      0 |      0 |      0 |      0 |      0 |      0 |      0 |

        +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

I notice there is another Port ID listed in this output. Is this my problem? A required port ID is not being created upon bootup?

0 Kudos
Highlighted
VMware Employee
VMware Employee

VCDNI is a MAC-MAC encapsulation technology.For the same reason MTU should be increased MTU Increase (24Bytes, 1500 –> 1524).Increase the MTU size of network devices in the transport VLAN to at least 1600 to accommodate the additional information needed for VCD-NI.


Please do change the MTU settings on


1.VCDNI network Pool properties in vCloud Director--MTU should be 1600

2.Set the same MTU(1600) in the DVS which is backing the VCDNI network Pool

3.Set the same MTU(1600) in upstream or downstream switches that this packet can traverse.


Note:Your external networks and network pools can't share the same VLAN.

Cheers,
Sree | CKA|VCAP-NSX-T| VCIX-3X| VCAP-3X| VExpert 4x
0 Kudos
Highlighted
Contributor
Contributor

While I understand what you are saying here, I don't think this has anything to do with the issue I am running into.

I am able to send 1500 byte sized pings across an isolated network, so there must be some sort of segmentation (edit: fragmentation) that occurs if the MTU is below 1524.

Changing the MTU settings on all our upstream equipment is not going to be a viable option for us.

Now, if there is a reason why this MTU setting would cause the issue I'm running into, I'm all ears. Before we switched to Auto-Deploy, we had no reach-ability issues on isolated networks, and we have some hosts that boot up on Auto-Deploy with it working as well. If it was a MTU issue, wouldn't we see the same issue on each host, every time? Wouldn't I also be able to send a 64 byte ping without issue (something I cannot do before I unprepare and then prepare the host)

We have another vCloud cluster that is not using auto-deploy, it also has a MTU of 1500 and it has no (perceived) issues with isolated networks.

Edit: If I send a don't fragment flag to ping and send a 1500 byte packet, it does run into an error saying fragmentation is needed. I don't doubt we aren't following best practice, what I do doubt is that this setting is causing the reachability issues we are seeing.

0 Kudos
Highlighted
VMware Employee
VMware Employee

What i pin-pointed was one of the core prerequites for VCDNI network pool.Since you already have Support Request created and if it is really a host specific issue i'm pretty sure there will be some valid messages in the host logs which they will share with you.However changing the MTU is mandatory and same applies for VXLAN network as well.Please refer below documentation for detailed referrence.

http://download3.vmware.com/vcat/vcat31_documentation_center/index.html#page/Architecting%2520a%2520...

VMware vCloud Director 5.5 Documentation Center

http://download3.vmware.com/vcat/vcat31_documentation_center/index.html#page/Implementation%2520Exam...

http://www.yellow-bricks.com/2010/09/09/vcd-networking-part-2-network-pools/

Cheers,
Sree | CKA|VCAP-NSX-T| VCIX-3X| VCAP-3X| VExpert 4x
0 Kudos
Highlighted
Contributor
Contributor

I do agree that we are not following best practice here, but I don't think it is "mandatory" From these links, I see the following:

*Increase the MTU size of network devices in the transport VLAN to at least 1600 to accommodate the additional information needed for VCD-NI. The information includes all physical switches and vSphere Distributed Switches. Failure to increase the MTU size causes packet fragmentation, negatively affecting network throughput performance of vCloud workloads.

When you configure the virtual machine guest operating system and the underlying physical infrastructure with the standard MTU (1500 bytes), the VMware network isolation protocol fragments frames. To avoid frame fragmentation, increase the MTU to at least 1600 bytes for the network pool and the underlying physical network. You can increase the network pool MTU up to, but not greater than, the MTU of the physical network.

*It is recommended to increase the MTU size on the physical devices backing the VCDNI network pool to 1524 bytes.

*It is also recommended to increase the MTU size on the network pool itself to 1524 bytes.

So, with a setting of 1500, we get packet fragmentation, which is not optimal, but connectivity should exist. A 64-byte ping should work and not have any fragmentation. I have confirmed this in my fragmentation tests. Fragmentation, in our environment, starts at 1480 bytes (and it still works, just fragmented).

0 Kudos
Highlighted
VMware Employee
VMware Employee

That is correct,connectivity should certainly be there.If you are OK with packet fragmentation in a production environment we don't need to make that change.If you dont mind can you provide me with SR number,i would like to have a look at the logs.Apart from that there is nothing much we can pin-point as per our conversation so far.Please expect a delay in my response,i primarily check and respond during ANZ/IST timings Smiley Happy

Cheers,
Sree | CKA|VCAP-NSX-T| VCIX-3X| VCAP-3X| VExpert 4x
0 Kudos
Highlighted
Contributor
Contributor

Sure deal, it's 15670628905.

0 Kudos
Highlighted
Enthusiast
Enthusiast

Hello,

Any news on this case, I may be running into sumilar issue, I would appreciate if you can share any useful details ?

Regards,

0 Kudos