VMware Networking Community
dtrajan
Contributor
Contributor

Even after enabling BGP on UI on Tire0-GW, still on the CLI, it shows as BGP as "Administratively shut down".

Even after enabling BGP on UI on Tire0 Gateway, still on the CLI, it shows as BGP as "Administratively shut down".

I am able to ping neighbor address from nsxt-edge1(tier0_sr). Only BGP shows as "Administratively shut down"

Can anyone pls let me know, how to fix this?

Tire-1 gateway is working fine.

nsxt-edge01(tier0_sr)> get bgp neighbor

BGP neighbor is 192.168.100.1, remote AS 65100, local AS 65000, external link

Administratively shut down

  BGP version 4, remote router ID 0.0.0.0, local router ID 192.168.100.2

  BGP state = Idle

  Last read 19:53:42, Last write never

  Hold time is 180, keepalive interval is 60 seconds

  Configured hold time is 180, keepalive interval is 60 seconds

  Graceful restart information:

    Local GR Mode  : Helper*

    Remote GR Mode : NotApplicable

    R bit          : False

    Timers :

     Configured Restart Time(sec)  : 180

     Received Restart Time(sec)    : 0

  Message statistics:

    Inq depth is 0

    Outq depth is 0

                         Sent       Rcvd

    Opens:                  0          0

    Notifications:          0          0

    Updates:                0          0

    Keepalives:             0          0

    Route Refresh:          0          0

    Capability:             0          0

    Total:                  0          0

  Minimum time between advertisement runs is 0 seconds

  Update source is 192.168.100.2

For address family: IPv4 Unicast

  Not part of any update group

  Community attribute sent to this neighbor(all)

  0 accepted prefixes

  Connections established 0; dropped 0

  Last reset never

BGP Connect Retry Timer in Seconds: 10

Read thread: off  Write thread: off

nsxt-edge01(tier0_sr)> ping 192.168.100.1 source 192.168.100.2 repeat 6

PING 192.168.100.1 (192.168.100.1) from 192.168.100.2: 56 data bytes

64 bytes from 192.168.100.1: icmp_seq=0 ttl=64 time=2.020 ms

64 bytes from 192.168.100.1: icmp_seq=1 ttl=64 time=2.382 ms

64 bytes from 192.168.100.1: icmp_seq=2 ttl=64 time=2.396 ms

64 bytes from 192.168.100.1: icmp_seq=3 ttl=64 time=1.961 ms

64 bytes from 192.168.100.1: icmp_seq=4 ttl=64 time=1.687 ms

64 bytes from 192.168.100.1: icmp_seq=5 ttl=64 time=1.891 ms

--- 192.168.100.1 ping statistics ---

6 packets transmitted, 6 packets received, 0.0% packet loss

round-trip min/avg/max/stddev = 1.687/2.056/2.396/0.257 ms

pastedImage_0.png

Tags (1)
Reply
0 Kudos
23 Replies
Sreec
VMware Employee
VMware Employee

Can you perform a BGP debug on both the routers ?

Also please provide get interfaces output from tier-0

Cheers,
Sree | VCIX-5X| VCAP-5X| VExpert 6x|Cisco Certified Specialist
Please KUDO helpful posts and mark the thread as solved if answered
Reply
0 Kudos
dtrajan
Contributor
Contributor

Thanks for the reply. Pls find the details of get interfaces....

On the other end (router), i did the debug, actually, it is establishing TCP connection and the other end, sends the BGP-OPEN message, then NSX sends TCP reset message...

Other end (router) is trying to establishing the TCP and sending BGP OPEN message, but NSX-T send TCP reset message, and not sending any BGP OPEN message...

its is like there is not BGP configured on NSX-T tire-0 gateway... On tire-0 it shows as "Administratively shutdown" , even with BGP enabled on UI...

On NSX-T, Snippet of the uplink interface, where BGP connection trying to establish:

=================================================================

    Interface     : 5093c7c7-b3ad-400e-9782-583a4047eae2

    Ifuid         : 288

    Name          : uplink01

    Fwd-mode      : IPV4_ONLY

    Internal name : uplink-288

    Mode          : lif

    Port-type     : uplink

    IP/Mask       : 192.168.100.2/24

    MAC           : 00:50:56:b6:c2:38

    VLAN          : None

    Access-VLAN   : None

    LS port       : de6659c2-b0af-4862-ba71-d5e1d02af763

    Urpf-mode     : STRICT_MODE

    DAD-mode      : LOOSE

    RA-mode       : SLAAC_DNS_TRHOUGH_RA(M=0, O=0)

    Admin         : up

    Op_state      : up

    MTU           : 9000

Complete Output on vrf:

-----------------------------

nsxt-edge01(tier0_sr)> get interfaces

Logical Router

UUID                                   VRF    LR-ID  Name                              Type                      

f2dbfc36-b3df-4b9f-8c50-95107e4fea02   5      2052   DR-Tire0-GW                       DISTRIBUTED_ROUTER_TIER0  

Interfaces (IPv6 DAD Status A-Assigned, D-Duplicate, T-Tentative)

    Interface     : 1eb95b92-3457-442a-861d-1919ba8b8d77

    Ifuid         : 301

    Name          : Tire0-GW-Tire1-GW-t0_lrp

    Fwd-mode      : IPV4_ONLY

    Internal name : downlink-301

    Mode          : lif

    Port-type     : downlink

    IP/Mask       : 100.64.112.0/31;fc7c:29ed:e1ac:d000::1/64(NA);fe80::50:56ff:fe56:4452/64(NA)

    MAC           : 02:50:56:56:44:52

    VNI           : 71689

    Access-VLAN   : None

    LS port       : d895832e-c514-48bf-820b-d3bb875c1cc6

    Urpf-mode     : PORT_CHECK

    DAD-mode      : LOOSE

    RA-mode       : SLAAC_DNS_TRHOUGH_RA(M=0, O=0)

    Admin         : up

    Op_state      : up

    MTU           : 1500

    Interface     : 41a82333-39b2-45e1-96aa-9c749805ca88

    Ifuid         : 296

    Name          : bp-dr-port

    Fwd-mode      : IPV4_ONLY

    Mode          : lif

    Port-type     : backplane

    IP/Mask       : 169.254.0.1/25;fe80::50:56ff:fe56:4452/64(NA)

    MAC           : 02:50:56:56:44:52

    VNI           : 71690

    Access-VLAN   : None

    LS port       : 8c064dea-e15b-43a5-b789-bd437eb0c7b9

    Urpf-mode     : PORT_CHECK

    DAD-mode      : LOOSE

    RA-mode       : RA_INVALID

    Admin         : up

    Op_state      : up

    MTU           : 1500

    Interface     : 1c535e28-1211-5c61-af7c-7f01c834d3d4

    Ifuid         : 293

    Mode          : cpu

    Port-type     : cpu

    Interface     : 57306016-9058-5171-82fc-0559d6e6e108

    Ifuid         : 294

    Mode          : blackhole

    Port-type     : blackhole

Logical Router

UUID                                   VRF    LR-ID  Name                              Type                      

e8aae4b9-d9bf-4966-9721-4ba970bdc53f   3      2053   SR-Tire0-GW                       SERVICE_ROUTER_TIER0      

Interfaces (IPv6 DAD Status A-Assigned, D-Duplicate, T-Tentative)

    Interface     : 56d780ea-31b0-501a-9030-310b4b5fd645

    Ifuid         : 281

    Mode          : cpu

    Port-type     : cpu

    Interface     : 610111d2-abb6-57c1-a8ef-31418ac71cc4

    Ifuid         : 282

    Mode          : blackhole

    Port-type     : blackhole

    Interface     : 24420827-5a60-424d-ae7b-4240ad08849f

    Ifuid         : 286

    Name          : sr0-internal-routing-port

    Fwd-mode      : IPV4_ONLY

    Internal name : inter-sr-286

    Mode          : lif

    Port-type     : internal-routing

    IP/Mask       : 169.254.0.130/25;fe80::50:56ff:fe56:5201/64(NA)

    MAC           : 02:50:56:56:52:00

    VNI           : 71687

    Access-VLAN   : None

    LS port       : 42fd04d2-90a3-4194-a595-88f0432db89a

    Urpf-mode     : PORT_CHECK

    DAD-mode      : LOOSE

    RA-mode       : RA_INVALID

    Admin         : up

    Op_state      : up

    MTU           : 1500

    Interface     : a3f81c59-d705-4524-b0a6-78890102abfd

    Ifuid         : 290

    Name          : bp-sr0-port

    Fwd-mode      : IPV4_ONLY

    Internal name : downlink-290

    Mode          : lif

    Port-type     : backplane

    IP/Mask       :

    MAC           : 02:50:56:56:53:00

    VNI           : 71690

    Access-VLAN   : None

    LS port       : a73cbb1c-986d-4258-86b3-c04eff9afac0

    Urpf-mode     : NONE

    DAD-mode      : LOOSE

    RA-mode       : RA_INVALID

    Admin         : up

    Op_state      : down

    MTU           : 1500

    Interface     : 5093c7c7-b3ad-400e-9782-583a4047eae2

    Ifuid         : 288

    Name          : uplink01

    Fwd-mode      : IPV4_ONLY

    Internal name : uplink-288

    Mode          : lif

    Port-type     : uplink

    IP/Mask       : 192.168.100.2/24

    MAC           : 00:50:56:b6:c2:38

    VLAN          : None

    Access-VLAN   : None

    LS port       : de6659c2-b0af-4862-ba71-d5e1d02af763

    Urpf-mode     : STRICT_MODE

    DAD-mode      : LOOSE

    RA-mode       : SLAAC_DNS_TRHOUGH_RA(M=0, O=0)

    Admin         : up

    Op_state      : up

    MTU           : 9000

    Interface     : a812a72c-27a6-4152-9904-f90a0f5d2272

    Ifuid         : 291

    Mode          : loopback

    Port-type     : loopback

    IP/Mask       : 127.0.0.1/8;::1/128(NA)

Output on tire-0 box: (removed eth0 IP details from the output, rest of them, are all correct)

--------------------------------------------------------------------------------------------------------------

nsxt-edge01> get interfaces

Interface: bond0

  Address: unknown

  MAC address: be:7c:bb:9a:76:d4

  MTU: 1500

  Broadcast address: None

  KNI: False

  Bond mode: ROUND_ROBIN

  Bond slaves:

  Link status: down

  Admin status: down

  RX packets: 0

  RX bytes: 0

  RX errors: 0

  RX dropped: 0

  TX packets: 0

  TX bytes: 0

  TX errors: 0

  TX dropped: 0

  TX collisions: 0

Interface: eth0

  Address: [REMOVED FROM OUTPUT]

  MAC address: 00:50:56:b6:0f:ca

  MTU: 1500

  Default gateway: [REMOVED FROM OUTPUT]

  Broadcast address: [REMOVED FROM OUTPUT]

  KNI: False

  Link status: up

  Admin status: up

  RX packets: 2498917

  RX bytes: 176892161

  RX errors: 0

  RX dropped: 0

  TX packets: 80014

  TX bytes: 18575173

  TX errors: 0

  TX dropped: 0

  TX collisions: 0

Interface: fp-eth0

  ID: 0

  Link status: up

  MAC address: 00:50:56:b6:dc:b5

  MTU: 1600

  PCI: 0000:0b:00:00

  Offload Capabilities: TX_VLAN_INSERT TX_UDP_CKSUM TX_TCP_CKSUM TX_TCP_TSO RX_VLAN_STRIP RX_IPV4_CKSUM RX_UDP_CKSUM RX_TCP_CKSUM RX_TCP_LRO

  Polling Status: active

  Driver: net_vmxnet3

  Rx queue: 2

  Tx queue: 2

  Socket: 0

  RX packets: 9555

  RX bytes: 859890

  RX errors: 0

  RX badcrc: unknown

  RX badlen: unknown

  RX misses: 0

  RX nombufs: 0

  RX pause xoff: unknown

  RX pause xon: unknown

  TX packets: 139993

  TX bytes: 5879706

  TX errors: 0

  TX pause xoff: unknown

  TX pause xon: unknown

Interface: fp-eth1

  ID: 1

  Link status: up

  MAC address: 00:50:56:b6:c2:38

  MTU: 1600

  PCI: 0000:13:00:00

  Offload Capabilities: TX_VLAN_INSERT TX_UDP_CKSUM TX_TCP_CKSUM TX_TCP_TSO RX_VLAN_STRIP RX_IPV4_CKSUM RX_UDP_CKSUM RX_TCP_CKSUM RX_TCP_LRO

  Polling Status: active

  Driver: net_vmxnet3

  Rx queue: 2

  Tx queue: 2

  Socket: 0

  RX packets: 2820

  RX bytes: 247591

  RX errors: 0

  RX badcrc: unknown

  RX badlen: unknown

  RX misses: 0

  RX nombufs: 0

  RX pause xoff: unknown

  RX pause xon: unknown

  TX packets: 8192

  TX bytes: 539708

  TX errors: 0

  TX pause xoff: unknown

  TX pause xon: unknown

Interface: fp-eth2

  ID: 2

  Link status: up

  MAC address: 00:50:56:b6:2c:cc

  MTU: 1500

  PCI: 0000:1b:00:00

  Offload Capabilities: TX_VLAN_INSERT TX_UDP_CKSUM TX_TCP_CKSUM TX_TCP_TSO RX_VLAN_STRIP RX_IPV4_CKSUM RX_UDP_CKSUM RX_TCP_CKSUM RX_TCP_LRO

  Polling Status: active

  Driver: net_vmxnet3

  Rx queue: 2

  Tx queue: 2

  Socket: 0

  RX packets: 3059847

  RX bytes: 205469196

  RX errors: 0

  RX badcrc: unknown

  RX badlen: unknown

  RX misses: 0

  RX nombufs: 0

  RX pause xoff: unknown

  RX pause xon: unknown

  TX packets: 0

  TX bytes: 0

  TX errors: 0

  TX pause xoff: unknown

  TX pause xon: unknown

Reply
0 Kudos
Sreec
VMware Employee
VMware Employee

Interface config looks fine. Well i believe its AS number issue . You should cross check if BPG peering config AS is correct on both the sides

As per your config remote AS is 65100 and local AS is 65000

Cheers,
Sree | VCIX-5X| VCAP-5X| VExpert 6x|Cisco Certified Specialist
Please KUDO helpful posts and mark the thread as solved if answered
Reply
0 Kudos
dtrajan
Contributor
Contributor

I cross-checked, configs on router and its correct....

Router side:

------------------

local-as: 65100

peer-as: 65000

tier-0 (NSX-T):

--------------------

local-as: 65000

peer-as: 65100

I am puzzled, why on tier-0 it shows BGP peer as "Administratively shutdown", even with UI shows as BGP enabled...

Even the ping between two peers are successful on both the side...

Any idea, under what all the scenarios, on nsx-t (tire-0 GW), it shows BGP peer as "Administratively shutdown"?

Reply
0 Kudos
Sreec
VMware Employee
VMware Employee

If possible please share the screenshots of Interface and BGP config from U.I as well.

Cheers,
Sree | VCIX-5X| VCAP-5X| VExpert 6x|Cisco Certified Specialist
Please KUDO helpful posts and mark the thread as solved if answered
Reply
0 Kudos
dtrajan
Contributor
Contributor

Thanks for looking into this issue... Please find the details below:

Note: I tried with BFD enabled and disabled, in both case, BGP neighbor shows as "Administratively shut down"

pastedImage_0.png

pastedImage_1.png

pastedImage_2.png

With different VIew:

------------------------------

pastedImage_3.png

Reply
0 Kudos
mauricioamorim
VMware Employee
VMware Employee

Do you have a single Edge node in the Edge cluster where this T0 is on?

Reply
0 Kudos
dtrajan
Contributor
Contributor

Yes, currently I have single Edge Node in the Edge cluster. Will having two edge-node in the same cluster will solve the BGP - "Administratively shut down" issue?

Thanks.

Reply
0 Kudos
mauricioamorim
VMware Employee
VMware Employee

It should work even with a single edge node in the cluster.

There is an option to administratively disable a BGP neighbor and it seems that it was switched on. This is the only way I found to reproduce what you have. This switch is only available in the Advanced Networking and Security (NSX 2.5 and below) or on Manager mode on NSX-T 3.0+.

The steps below were tested on NSX-T 3.0:

If you do not have the Policy/Manager button in the upper right corner go to System > User Interface Settings (Last option) and Toggle Visibility so that it appears.

Then go to Networking (click and make sure the Manager option in the upper right corner is selected)

Click on Tier-0 Logical Routers > Click on the T0

Under Routing > BGP > select the neighbor in the lower part of the screen and click on EDIT

There will be an option to change de Admin Status > Change to enable

Hope this helps

Reply
0 Kudos
dtrajan
Contributor
Contributor

Thanks for looking into this issue... yeah, i have verified the same on UI and the BGP neighbor admin-status is "Enabled:... Still no luck... on the CLI it shows BGP "Administratively shutdown"...

Please find the details below:

Does this related to any physical NIC supportablility? I tried with both 10G and 1G physical NIC too...

But, i am able to successfully ping between the neighbor router and Tire-0 interface.

495654_495654.pngpastedImage_0.png

495757_495757.pngpastedImage_1.png

495758_495758.pngpastedImage_2.png

With different VIew:

------------------------------

495759_495759.pngpastedImage_3.png

 

Reply
0 Kudos
mauricioamorim
VMware Employee
VMware Employee

Does your edge node have any alarms?

Reply
0 Kudos
dtrajan
Contributor
Contributor

Sorry for the late reply. I am seeing the below behavior:

Please let me know, if there is anything I am missing during my config? I am not sure, why my BGP is "Administratively shut down", as soon as I attach VLANs (either to Tire-0 or Tire-1 or VRF)

Note: I have VMs attached to VLAN segments.

Case-1) If I do not attach any VLAN segments to either Tire-0 or Tire-1 or VRF, then BGP session is up between Tire-0 and external-router.

Case-2) If I attach VLAN segment to any node (either Tire-0 or Tire-1 or VRF), then the BGP on Tire-0 goes to "Administratively shut down"

Note: I have NOT configured BFD...

Reg Alarms, I see the below in the NSX-Manager:

==========================================

1)

Routing Routing Down nsxt-edge1

Open

Description

All BGP/BFD sessions are down.

Recommended Action

Invoke the NSX CLI command `get logical-routers` to get the tier0 service router and switch to this vrf, then invoke the following NSX CLI commands. 1. `ping <BFD peer IP address>` to verifyconnectivity. 2. `get bfd-config` and `get bfd-sessions` to check if BFD is running well. 3. `get bgp neighbor summary` to check if BGP is running well. Also check /var/log/syslog to see if there are any errors related to BGP connectivity.

2)

Infrastructure Communication  Edge Tunnels Down  nsxt-edge1  nsxt-edge1

Open

Description

The overall tunnel status of Edge node 01054703-43cc-4348-93b8-be2c9d38aded is down.

Recommended Action

Invoke the NSX CLI command `get tunnel-ports` to get all tunnel ports, then check each tunnel's stats by invoking NSX CLI command `get tunnel-port <UUID> stats` to check if there are any drops. Also check /var/log/syslog if there are tunnel related errors.

3)

Routing   BGP Down   nsxt-edge1

Description

In Router 66e152c1-606c-49e8-a89a-6c25e46fea9a, BGP neighbor ip_address:192.168.100.1 is down, reason: Network or config error.

Recommended Action

1. Invoke the NSX CLI command `get logical-routers`. 2. Switch to service-router ed9bf441-e57f-4ce7-a69a-26cec15fa5cf. 3. Invoke the NSX CLI command `get bgp neighbor summary` to check the BGP neighbor status. 4. Check /var/log/syslog to see if there are any errors related to BGP connectivity.

Reply
0 Kudos
mauricioamorim
VMware Employee
VMware Employee

Are you attaching VMs to the same VLAN as the uplinks of the Tier0? Please send us some screenshots of your Edge node config.

Reply
0 Kudos
dtrajan
Contributor
Contributor

When I create VM on vcenter, I use the segment (overlay-segment). Pls see the below screen-shot for "app90" name...

As soon as I create VM with "app90" as network adaptor (and attach to VRF), then BGP goes to "Administratively shut down".

I am attaching the VM to the segment (overlay-segment).

vlan-segment is attached to the Tier-0 interface (and NOT creating any VM on this vlan-sgment). This is just for uplink (connection to router for BGP session)...

Please find the config below and let me know, if you need more details:

==========================================================

overlay-segment:

---------------------

vlan-segment:

------------------

Tire-1:

------------

Tire-0:

------------

VRF:

-----------

Reply
0 Kudos
dtrajan
Contributor
Contributor

overlay-segment:

---------------------

pastedImage_0.png

Reply
0 Kudos
dtrajan
Contributor
Contributor

vlan-segment:

------------------

pastedImage_0.png

Reply
0 Kudos
mauricioamorim
VMware Employee
VMware Employee

There are some problems with the screenshots that are not showing.

Please don't forget to send the screenshots of the Edge node configuration and status (System > Fabric > Nodes > Edge Transport Nodes).

Reply
0 Kudos
tyrebyter
Contributor
Contributor

What was the resolution to this problem? We have the same issue in one environment (of 5 built).

Reply
0 Kudos
suhailsaeed
Contributor
Contributor

For us the issue got resolved by making sure all the Tunnels for Edges are in UP state in NSX-T Manager

Reply
0 Kudos