1 2 Previous Next 17 Replies Latest reply on Jul 21, 2020 3:53 AM by mauricioamorim

    Even after enabling BGP on UI on Tire0-GW, still on the CLI, it shows as BGP as "Administratively shut down".

    dtrajan Novice

      Even after enabling BGP on UI on Tire0 Gateway, still on the CLI, it shows as BGP as "Administratively shut down".

      I am able to ping neighbor address from nsxt-edge1(tier0_sr). Only BGP shows as "Administratively shut down"

      Can anyone pls let me know, how to fix this?

      Tire-1 gateway is working fine.

       

      nsxt-edge01(tier0_sr)> get bgp neighbor

      BGP neighbor is 192.168.100.1, remote AS 65100, local AS 65000, external link

      Administratively shut down

        BGP version 4, remote router ID 0.0.0.0, local router ID 192.168.100.2

        BGP state = Idle

        Last read 19:53:42, Last write never

        Hold time is 180, keepalive interval is 60 seconds

        Configured hold time is 180, keepalive interval is 60 seconds

        Graceful restart information:

          Local GR Mode  : Helper*

          Remote GR Mode : NotApplicable

          R bit          : False

          Timers :

           Configured Restart Time(sec)  : 180

           Received Restart Time(sec)    : 0

        Message statistics:

          Inq depth is 0

          Outq depth is 0

                               Sent       Rcvd

          Opens:                  0          0

          Notifications:          0          0

          Updates:                0          0

          Keepalives:             0          0

          Route Refresh:          0          0

          Capability:             0          0

          Total:                  0          0

        Minimum time between advertisement runs is 0 seconds

        Update source is 192.168.100.2

       

      For address family: IPv4 Unicast

        Not part of any update group

        Community attribute sent to this neighbor(all)

        0 accepted prefixes

       

        Connections established 0; dropped 0

        Last reset never

      BGP Connect Retry Timer in Seconds: 10

      Read thread: off  Write thread: off

       

       

       

      nsxt-edge01(tier0_sr)> ping 192.168.100.1 source 192.168.100.2 repeat 6

      PING 192.168.100.1 (192.168.100.1) from 192.168.100.2: 56 data bytes

      64 bytes from 192.168.100.1: icmp_seq=0 ttl=64 time=2.020 ms

      64 bytes from 192.168.100.1: icmp_seq=1 ttl=64 time=2.382 ms

      64 bytes from 192.168.100.1: icmp_seq=2 ttl=64 time=2.396 ms

      64 bytes from 192.168.100.1: icmp_seq=3 ttl=64 time=1.961 ms

      64 bytes from 192.168.100.1: icmp_seq=4 ttl=64 time=1.687 ms

      64 bytes from 192.168.100.1: icmp_seq=5 ttl=64 time=1.891 ms

       

      --- 192.168.100.1 ping statistics ---

      6 packets transmitted, 6 packets received, 0.0% packet loss

      round-trip min/avg/max/stddev = 1.687/2.056/2.396/0.257 ms

       

       

        • 1. Re: Even after enabling BGP on UI on Tire0-GW, still on the CLI, it shows as BGP as "Administratively shut down".
          Sreec Master
          VMware EmployeesvExpertCommunity Warriors

          Can you perform a BGP debug on both the routers ?

          Also please provide get interfaces output from tier-0

          • 2. Re: Even after enabling BGP on UI on Tire0-GW, still on the CLI, it shows as BGP as "Administratively shut down".
            dtrajan Novice

            Thanks for the reply. Pls find the details of get interfaces....

             

            On the other end (router), i did the debug, actually, it is establishing TCP connection and the other end, sends the BGP-OPEN message, then NSX sends TCP reset message...

             

            Other end (router) is trying to establishing the TCP and sending BGP OPEN message, but NSX-T send TCP reset message, and not sending any BGP OPEN message...

            its is like there is not BGP configured on NSX-T tire-0 gateway... On tire-0 it shows as "Administratively shutdown" , even with BGP enabled on UI...

             

             

             

             

            On NSX-T, Snippet of the uplink interface, where BGP connection trying to establish:

            =================================================================

                Interface     : 5093c7c7-b3ad-400e-9782-583a4047eae2

                Ifuid         : 288

                Name          : uplink01

                Fwd-mode      : IPV4_ONLY

                Internal name : uplink-288

                Mode          : lif

                Port-type     : uplink

                IP/Mask       : 192.168.100.2/24

                MAC           : 00:50:56:b6:c2:38

                VLAN          : None

                Access-VLAN   : None

                LS port       : de6659c2-b0af-4862-ba71-d5e1d02af763

                Urpf-mode     : STRICT_MODE

                DAD-mode      : LOOSE

                RA-mode       : SLAAC_DNS_TRHOUGH_RA(M=0, O=0)

                Admin         : up

                Op_state      : up

                MTU           : 9000

             

             

             

            Complete Output on vrf:

            -----------------------------

            nsxt-edge01(tier0_sr)> get interfaces

            Logical Router

            UUID                                   VRF    LR-ID  Name                              Type                      

            f2dbfc36-b3df-4b9f-8c50-95107e4fea02   5      2052   DR-Tire0-GW                       DISTRIBUTED_ROUTER_TIER0  

            Interfaces (IPv6 DAD Status A-Assigned, D-Duplicate, T-Tentative)

                Interface     : 1eb95b92-3457-442a-861d-1919ba8b8d77

                Ifuid         : 301

                Name          : Tire0-GW-Tire1-GW-t0_lrp

                Fwd-mode      : IPV4_ONLY

                Internal name : downlink-301

                Mode          : lif

                Port-type     : downlink

                IP/Mask       : 100.64.112.0/31;fc7c:29ed:e1ac:d000::1/64(NA);fe80::50:56ff:fe56:4452/64(NA)

                MAC           : 02:50:56:56:44:52

                VNI           : 71689

                Access-VLAN   : None

                LS port       : d895832e-c514-48bf-820b-d3bb875c1cc6

                Urpf-mode     : PORT_CHECK

                DAD-mode      : LOOSE

                RA-mode       : SLAAC_DNS_TRHOUGH_RA(M=0, O=0)

                Admin         : up

                Op_state      : up

                MTU           : 1500

             

                Interface     : 41a82333-39b2-45e1-96aa-9c749805ca88

                Ifuid         : 296

                Name          : bp-dr-port

                Fwd-mode      : IPV4_ONLY

                Mode          : lif

                Port-type     : backplane

                IP/Mask       : 169.254.0.1/25;fe80::50:56ff:fe56:4452/64(NA)

                MAC           : 02:50:56:56:44:52

                VNI           : 71690

                Access-VLAN   : None

                LS port       : 8c064dea-e15b-43a5-b789-bd437eb0c7b9

                Urpf-mode     : PORT_CHECK

                DAD-mode      : LOOSE

                RA-mode       : RA_INVALID

                Admin         : up

                Op_state      : up

                MTU           : 1500

             

                Interface     : 1c535e28-1211-5c61-af7c-7f01c834d3d4

                Ifuid         : 293

                Mode          : cpu

                Port-type     : cpu

             

                Interface     : 57306016-9058-5171-82fc-0559d6e6e108

                Ifuid         : 294

                Mode          : blackhole

                Port-type     : blackhole

             

            Logical Router

            UUID                                   VRF    LR-ID  Name                              Type                      

            e8aae4b9-d9bf-4966-9721-4ba970bdc53f   3      2053   SR-Tire0-GW                       SERVICE_ROUTER_TIER0      

            Interfaces (IPv6 DAD Status A-Assigned, D-Duplicate, T-Tentative)

                Interface     : 56d780ea-31b0-501a-9030-310b4b5fd645

                Ifuid         : 281

                Mode          : cpu

                Port-type     : cpu

             

                Interface     : 610111d2-abb6-57c1-a8ef-31418ac71cc4

                Ifuid         : 282

                Mode          : blackhole

                Port-type     : blackhole

             

                Interface     : 24420827-5a60-424d-ae7b-4240ad08849f

                Ifuid         : 286

                Name          : sr0-internal-routing-port

                Fwd-mode      : IPV4_ONLY

                Internal name : inter-sr-286

                Mode          : lif

                Port-type     : internal-routing

                IP/Mask       : 169.254.0.130/25;fe80::50:56ff:fe56:5201/64(NA)

                MAC           : 02:50:56:56:52:00

                VNI           : 71687

                Access-VLAN   : None

                LS port       : 42fd04d2-90a3-4194-a595-88f0432db89a

                Urpf-mode     : PORT_CHECK

                DAD-mode      : LOOSE

                RA-mode       : RA_INVALID

                Admin         : up

                Op_state      : up

                MTU           : 1500

             

                Interface     : a3f81c59-d705-4524-b0a6-78890102abfd

                Ifuid         : 290

                Name          : bp-sr0-port

                Fwd-mode      : IPV4_ONLY

                Internal name : downlink-290

                Mode          : lif

                Port-type     : backplane

                IP/Mask       :

                MAC           : 02:50:56:56:53:00

                VNI           : 71690

                Access-VLAN   : None

                LS port       : a73cbb1c-986d-4258-86b3-c04eff9afac0

                Urpf-mode     : NONE

                DAD-mode      : LOOSE

                RA-mode       : RA_INVALID

                Admin         : up

                Op_state      : down

                MTU           : 1500

             

                Interface     : 5093c7c7-b3ad-400e-9782-583a4047eae2

                Ifuid         : 288

                Name          : uplink01

                Fwd-mode      : IPV4_ONLY

                Internal name : uplink-288

                Mode          : lif

                Port-type     : uplink

                IP/Mask       : 192.168.100.2/24

                MAC           : 00:50:56:b6:c2:38

                VLAN          : None

                Access-VLAN   : None

                LS port       : de6659c2-b0af-4862-ba71-d5e1d02af763

                Urpf-mode     : STRICT_MODE

                DAD-mode      : LOOSE

                RA-mode       : SLAAC_DNS_TRHOUGH_RA(M=0, O=0)

                Admin         : up

                Op_state      : up

                MTU           : 9000

             

                Interface     : a812a72c-27a6-4152-9904-f90a0f5d2272

                Ifuid         : 291

                Mode          : loopback

                Port-type     : loopback

                IP/Mask       : 127.0.0.1/8;::1/128(NA)

             

             

             

             

            Output on tire-0 box: (removed eth0 IP details from the output, rest of them, are all correct)

            --------------------------------------------------------------------------------------------------------------

             

            nsxt-edge01> get interfaces

            Interface: bond0

              Address: unknown

              MAC address: be:7c:bb:9a:76:d4

              MTU: 1500

              Broadcast address: None

              KNI: False

              Bond mode: ROUND_ROBIN

              Bond slaves:

              Link status: down

              Admin status: down

              RX packets: 0

              RX bytes: 0

              RX errors: 0

              RX dropped: 0

              TX packets: 0

              TX bytes: 0

              TX errors: 0

              TX dropped: 0

              TX collisions: 0

             

            Interface: eth0

              Address: [REMOVED FROM OUTPUT]

              MAC address: 00:50:56:b6:0f:ca

              MTU: 1500

              Default gateway: [REMOVED FROM OUTPUT]

              Broadcast address: [REMOVED FROM OUTPUT]

              KNI: False

              Link status: up

              Admin status: up

              RX packets: 2498917

              RX bytes: 176892161

              RX errors: 0

              RX dropped: 0

              TX packets: 80014

              TX bytes: 18575173

              TX errors: 0

              TX dropped: 0

              TX collisions: 0

             

            Interface: fp-eth0

              ID: 0

              Link status: up

              MAC address: 00:50:56:b6:dc:b5

              MTU: 1600

              PCI: 0000:0b:00:00

              Offload Capabilities: TX_VLAN_INSERT TX_UDP_CKSUM TX_TCP_CKSUM TX_TCP_TSO RX_VLAN_STRIP RX_IPV4_CKSUM RX_UDP_CKSUM RX_TCP_CKSUM RX_TCP_LRO

              Polling Status: active

              Driver: net_vmxnet3

              Rx queue: 2

              Tx queue: 2

              Socket: 0

              RX packets: 9555

              RX bytes: 859890

              RX errors: 0

              RX badcrc: unknown

              RX badlen: unknown

              RX misses: 0

              RX nombufs: 0

              RX pause xoff: unknown

              RX pause xon: unknown

              TX packets: 139993

              TX bytes: 5879706

              TX errors: 0

              TX pause xoff: unknown

              TX pause xon: unknown

             

            Interface: fp-eth1

              ID: 1

              Link status: up

              MAC address: 00:50:56:b6:c2:38

              MTU: 1600

              PCI: 0000:13:00:00

              Offload Capabilities: TX_VLAN_INSERT TX_UDP_CKSUM TX_TCP_CKSUM TX_TCP_TSO RX_VLAN_STRIP RX_IPV4_CKSUM RX_UDP_CKSUM RX_TCP_CKSUM RX_TCP_LRO

              Polling Status: active

              Driver: net_vmxnet3

              Rx queue: 2

              Tx queue: 2

              Socket: 0

              RX packets: 2820

              RX bytes: 247591

              RX errors: 0

              RX badcrc: unknown

              RX badlen: unknown

              RX misses: 0

              RX nombufs: 0

              RX pause xoff: unknown

              RX pause xon: unknown

              TX packets: 8192

              TX bytes: 539708

              TX errors: 0

              TX pause xoff: unknown

              TX pause xon: unknown

             

            Interface: fp-eth2

              ID: 2

              Link status: up

              MAC address: 00:50:56:b6:2c:cc

              MTU: 1500

              PCI: 0000:1b:00:00

              Offload Capabilities: TX_VLAN_INSERT TX_UDP_CKSUM TX_TCP_CKSUM TX_TCP_TSO RX_VLAN_STRIP RX_IPV4_CKSUM RX_UDP_CKSUM RX_TCP_CKSUM RX_TCP_LRO

              Polling Status: active

              Driver: net_vmxnet3

              Rx queue: 2

              Tx queue: 2

              Socket: 0

              RX packets: 3059847

              RX bytes: 205469196

              RX errors: 0

              RX badcrc: unknown

              RX badlen: unknown

              RX misses: 0

              RX nombufs: 0

              RX pause xoff: unknown

              RX pause xon: unknown

              TX packets: 0

              TX bytes: 0

              TX errors: 0

              TX pause xoff: unknown

              TX pause xon: unknown

            • 3. Re: Even after enabling BGP on UI on Tire0-GW, still on the CLI, it shows as BGP as "Administratively shut down".
              Sreec Master
              Community WarriorsvExpertVMware Employees

              Interface config looks fine. Well i believe its AS number issue . You should cross check if BPG peering config AS is correct on both the sides

               

              As per your config remote AS is 65100 and local AS is 65000

              • 4. Re: Even after enabling BGP on UI on Tire0-GW, still on the CLI, it shows as BGP as "Administratively shut down".
                dtrajan Novice

                I cross-checked, configs on router and its correct....

                 

                Router side:

                ------------------

                local-as: 65100

                peer-as: 65000

                 

                tier-0 (NSX-T):

                --------------------

                local-as: 65000

                peer-as: 65100

                 

                I am puzzled, why on tier-0 it shows BGP peer as "Administratively shutdown", even with UI shows as BGP enabled...

                Even the ping between two peers are successful on both the side...

                 

                Any idea, under what all the scenarios, on nsx-t (tire-0 GW), it shows BGP peer as "Administratively shutdown"?

                • 5. Re: Even after enabling BGP on UI on Tire0-GW, still on the CLI, it shows as BGP as "Administratively shut down".
                  Sreec Master
                  vExpertCommunity WarriorsVMware Employees

                  If possible please share the screenshots of Interface and BGP config from U.I as well.

                  • 6. Re: Even after enabling BGP on UI on Tire0-GW, still on the CLI, it shows as BGP as "Administratively shut down".
                    dtrajan Novice

                    Thanks for looking into this issue... Please find the details below:

                    Note: I tried with BFD enabled and disabled, in both case, BGP neighbor shows as "Administratively shut down"

                     

                     

                     

                     

                     

                     

                     

                     

                     

                     

                     

                     

                     

                     

                     

                    With different VIew:

                    ------------------------------

                     

                     

                     

                     

                     

                     

                    • 7. Re: Even after enabling BGP on UI on Tire0-GW, still on the CLI, it shows as BGP as "Administratively shut down".
                      mauricioamorim Expert
                      VMware Employees

                      Do you have a single Edge node in the Edge cluster where this T0 is on?

                      • 8. Re: Even after enabling BGP on UI on Tire0-GW, still on the CLI, it shows as BGP as "Administratively shut down".
                        dtrajan Novice

                        Yes, currently I have single Edge Node in the Edge cluster. Will having two edge-node in the same cluster will solve the BGP - "Administratively shut down" issue?

                        Thanks.

                        • 9. Re: Even after enabling BGP on UI on Tire0-GW, still on the CLI, it shows as BGP as "Administratively shut down".
                          mauricioamorim Expert
                          VMware Employees

                          It should work even with a single edge node in the cluster.

                           

                          There is an option to administratively disable a BGP neighbor and it seems that it was switched on. This is the only way I found to reproduce what you have. This switch is only available in the Advanced Networking and Security (NSX 2.5 and below) or on Manager mode on NSX-T 3.0+.

                           

                          The steps below were tested on NSX-T 3.0:

                           

                          If you do not have the Policy/Manager button in the upper right corner go to System > User Interface Settings (Last option) and Toggle Visibility so that it appears.

                           

                          Then go to Networking (click and make sure the Manager option in the upper right corner is selected)

                          Click on Tier-0 Logical Routers > Click on the T0

                          Under Routing > BGP > select the neighbor in the lower part of the screen and click on EDIT

                          There will be an option to change de Admin Status > Change to enable

                           

                          Hope this helps

                          • 10. Re: Even after enabling BGP on UI on Tire0-GW, still on the CLI, it shows as BGP as "Administratively shut down".
                            dtrajan Novice

                            Thanks for looking into this issue... yeah, i have verified the same on UI and the BGP neighbor admin-status is "Enabled:... Still no luck... on the CLI it shows BGP "Administratively shutdown"...

                            Please find the details below:

                             

                            Does this related to any physical NIC supportablility? I tried with both 10G and 1G physical NIC too...

                            But, i am able to successfully ping between the neighbor router and Tire-0 interface.

                             

                             

                             

                             

                             

                             

                             

                             

                             

                             

                             

                             

                             

                             

                             

                             

                             

                            With different VIew:

                            ------------------------------

                             

                             

                             

                             

                             

                             

                             

                            • 12. Re: Even after enabling BGP on UI on Tire0-GW, still on the CLI, it shows as BGP as "Administratively shut down".
                              dtrajan Novice

                              Sorry for the late reply. I am seeing the below behavior:

                              Please let me know, if there is anything I am missing during my config? I am not sure, why my BGP is "Administratively shut down", as soon as I attach VLANs (either to Tire-0 or Tire-1 or VRF)

                               

                              Note: I have VMs attached to VLAN segments.

                               

                               

                              Case-1) If I do not attach any VLAN segments to either Tire-0 or Tire-1 or VRF, then BGP session is up between Tire-0 and external-router.

                               

                              Case-2) If I attach VLAN segment to any node (either Tire-0 or Tire-1 or VRF), then the BGP on Tire-0 goes to "Administratively shut down"

                               

                              Note: I have NOT configured BFD...

                               

                               

                              Reg Alarms, I see the below in the NSX-Manager:

                              ==========================================

                              1)

                              Routing Routing Down nsxt-edge1

                              Open

                               

                              Description

                              All BGP/BFD sessions are down.

                              Recommended Action

                              Invoke the NSX CLI command `get logical-routers` to get the tier0 service router and switch to this vrf, then invoke the following NSX CLI commands. 1. `ping <BFD peer IP address>` to verifyconnectivity. 2. `get bfd-config` and `get bfd-sessions` to check if BFD is running well. 3. `get bgp neighbor summary` to check if BGP is running well. Also check /var/log/syslog to see if there are any errors related to BGP connectivity.

                               

                               

                               

                              2)

                              Infrastructure Communication  Edge Tunnels Down  nsxt-edge1  nsxt-edge1

                              Open

                               

                              Description

                              The overall tunnel status of Edge node 01054703-43cc-4348-93b8-be2c9d38aded is down.

                              Recommended Action

                              Invoke the NSX CLI command `get tunnel-ports` to get all tunnel ports, then check each tunnel's stats by invoking NSX CLI command `get tunnel-port <UUID> stats` to check if there are any drops. Also check /var/log/syslog if there are tunnel related errors.

                               

                               

                              3)

                              Routing   BGP Down   nsxt-edge1

                               

                              Description

                              In Router 66e152c1-606c-49e8-a89a-6c25e46fea9a, BGP neighbor ip_address:192.168.100.1 is down, reason: Network or config error.

                              Recommended Action

                              1. Invoke the NSX CLI command `get logical-routers`. 2. Switch to service-router ed9bf441-e57f-4ce7-a69a-26cec15fa5cf. 3. Invoke the NSX CLI command `get bgp neighbor summary` to check the BGP neighbor status. 4. Check /var/log/syslog to see if there are any errors related to BGP connectivity.

                              • 13. Re: Even after enabling BGP on UI on Tire0-GW, still on the CLI, it shows as BGP as "Administratively shut down".
                                mauricioamorim Expert
                                VMware Employees

                                Are you attaching VMs to the same VLAN as the uplinks of the Tier0? Please send us some screenshots of your Edge node config.

                                • 14. Re: Even after enabling BGP on UI on Tire0-GW, still on the CLI, it shows as BGP as "Administratively shut down".
                                  dtrajan Novice

                                  When I create VM on vcenter, I use the segment (overlay-segment). Pls see the below screen-shot for "app90" name...

                                  As soon as I create VM with "app90" as network adaptor (and attach to VRF), then BGP goes to "Administratively shut down".

                                  I am attaching the VM to the segment (overlay-segment).

                                   

                                   

                                   

                                  vlan-segment is attached to the Tier-0 interface (and NOT creating any VM on this vlan-sgment). This is just for uplink (connection to router for BGP session)...

                                   

                                  Please find the config below and let me know, if you need more details:

                                  ==========================================================

                                   

                                  overlay-segment:

                                  ---------------------

                                   

                                   

                                   

                                   

                                  vlan-segment:

                                  ------------------

                                   

                                   

                                  Tire-1:

                                  ------------

                                   

                                   

                                   

                                   

                                   

                                   

                                  Tire-0:

                                  ------------

                                   

                                   

                                   

                                   

                                  VRF:

                                  -----------

                                   

                                  1 2 Previous Next